Infrastructure
Open-source LLMs on your own servers
Hardware capability
Our in-house AI server
- GPUs
- 2 × NVIDIA GeForce RTX 5060 Ti 16 GB
- Total VRAM
- 32 GB
- CPU
- AMD Ryzen 5 5600
- System RAM
- 32 GB
- Model serving
- vLLM and Ollama
- Gateway
- LiteLLM (OpenAI-compatible API, keys, quotas, logs)
- Models
- Open-source models only
32 GB of VRAM across two GPUs is enough for quantised open-source models in roughly the 7B–32B parameter range. Client deployments are sized to the client's models and load, and are usually larger.
Build video of the Surge in-house AI server.
What we install for clients
The private AI stack
| Layer | What it does | Typical tools |
|---|---|---|
| Applications | Agents, RAG assistants and integrations we build for you | Your systems via APIs; human approval steps where needed |
| Gateway | One OpenAI-compatible endpoint for every model, with keys per team, quotas, logging and routing to an optional backup | LiteLLM |
| Retrieval | Document ingestion, embeddings and vector search over your content | pgvector or Qdrant; open-source embedding models |
| Model serving | Open-source LLMs served with batching and quantisation to fit your GPUs | vLLM for throughput; Ollama for small footprints |
| Observability | Latency, throughput, GPU use, cost per use case, and traces of agent runs | Prometheus and Grafana; LLM tracing |
| Hardware | GPU servers on your premises or in your data centre | Sized in the assessment to your models, users and response-time targets |
How it goes
From sizing to running
- 01
Size
We estimate the models you need, concurrent users and response times, and size GPUs, memory and storage. You buy the hardware, or use servers you already own.
- 02
Install
We set up the OS, drivers, model serving, gateway, vector store and monitoring, and harden access.
- 03
Evaluate
We test candidate open-source models on your own tasks and pick the smallest one that meets the bar.
- 04
Connect
Your applications call one gateway endpoint. Teams get their own keys and quotas.
- 05
Run
We monitor, patch, tune on your data and upgrade models as better open-source releases appear.
Model choice
Open-source in-house, backup only where you approve
Default · in-house
Open-source models on your infrastructure
- Runs on your own GPU servers, on-prem or in your data centre.
- Your data, prompts, logs and tuned model weights stay with you.
- No per-token bill for the work that runs in-house.
- Swap to newer open-source models without changing your applications.
Optional · backup
Foundation-model subscription
- Claude, GPT or Gemini, through your own subscription.
- Used only for the specific needs you approve in the assessment.
- Routed through the same gateway, so every call is logged and capped.
- Everything else stays on your infrastructure.
You choose. On-prem means open-source models. A foundation-model backup is a decision you make use case by use case, not a default we build in.
Ownership
What stays yours
- The hardware, on your premises or in your data centre
- Your documents, prompts, conversation logs and embeddings
- Model weights you fine-tune on your data
- Gateway configuration, keys and usage records
- Runbooks and documentation, so another team can operate it
Plan your private AI server
Tell us your use cases and users. We will come back with a sizing, a model shortlist and a budget range.