logoGPU/docs
Dashboard← Back to site

Inference Backends

Native nodes use Ollama. Browser nodes use WebLLM / WebGPU. The orchestrator routes tasks to a backend that has the requested model cached.

Native backend — Ollama

Ollama is the default native runtime. It wraps llama.cpp with automatic Metal (Apple Silicon) and CUDA (NVIDIA) acceleration. Our installer pulls Ollama and starts its daemon for you.

  • Model format: GGUF, 4-bit quantised by default
  • API: Ollama's local /api/chat with stream support
  • Acceleration: Metal on Mac, CUDA on Linux NVIDIA, CPU fallback elsewhere
  • Catalogue: llama3.2:1b/3b, gemma2:2b, qwen2.5:1.5b/3b, phi3.5:3.8b, mistral:7b

Inspect what your node has cached

bash
curl http://localhost:11434/api/tags

The provider script reports this list to the orchestrator at registration, so /v1/models can show accurate nodes_available counts per model.

Browser backend — WebLLM

Browser nodes use @mlc-ai/web-llm to run quantised models directly in the tab via WebGPU. Models are downloaded once and cached in IndexedDB; subsequent sessions take seconds.

  • Runtime: WebGPU through MLC's WASM compute kernels
  • Engine lives in a Web Worker to survive background-tab throttling
  • A silent AudioContext keeps Chrome from idling the tab
  • Six model sizes available — see Browser Nodes
Backend choice is automatic
The orchestrator picks a provider that already has the requested model cached. If you call model: "mistral-7b" and no provider has it, the request auto-routes to the closest model in the same tier/family. The response includes a routed_from field so the caller knows.

What we do NOT support yet

  • oMLX (Apple MLX): on roadmap. Native MLX is faster than llama.cpp on M-series for some models, especially Qwen-class reasoning.
  • Split-model inference: on roadmap. Two providers cooperate to serve one model larger than either can hold alone.
  • ONNX Runtime Web: not planned. WebLLM covers our browser cases.
Next →
Browser Nodes