Architecture
Everything behind the chat box.
Seven layers, from bare metal to this public page. Model names and open-source tools below are real; any network address, hostname, or account detail is intentionally generalized.
How a live demo request actually travels
Visitor
Browses this Vercel-hosted site
Next.js API route
Serverless function, server-side only
Tailscale Funnel
One authenticated public doorway
Home server
Local model runtime + knowledge banks
If the home server is offline or the bridge times out, the site falls back to a pre-written example response instead of showing an error.
A single high-memory workstation, not a cloud cluster.
Apple Silicon workstation
Ultra-class chip, 24+ CPU cores, 60+ GPU cores
Unified memory
96 GB shared CPU/GPU pool — the key unlock for large local models
Dedicated storage volume
2 TB+ SSD reserved entirely for models, knowledge banks, logs
Always-on networking
Wired Ethernet + static local address for 24/7 availability
The runtime that turns downloaded weights into a live API.
Ollama
OpenAI-compatible API backbone at /v1 — every tool below speaks to this
LM Studio
Optional GUI for trialing new model releases on Apple's MLX engine
llama.cpp / whisper.cpp
Underlying inference engines for text and speech
Open-weight models chosen to fit a 96 GB unified-memory budget.
gpt-oss-120b (MoE)
~64 GB class — frontier-level reasoning, fully offline
Qwen3.6-35B-A3B (MoE)
Fast daily-driver: coding, long context, vision
Qwen3-VL-30B-A3B
Image, screenshot, and video-frame understanding
Whisper large-v3-turbo
Speech-to-text for meetings and recordings
nomic-embed-text
Embedding model powering retrieval search
How humans (and other apps) actually reach the models.
Open WebUI
Multi-user chat UI, logins, per-chat model switching, tool calling
Tailscale mesh VPN
Private remote access from anywhere, zero exposed ports
Tailscale Funnel
The one deliberate, authenticated doorway this public site uses
Three tiers, added only as each one proves insufficient.
Tier 1 — Vector RAG
Domain knowledge banks indexed for retrieval-augmented answers
Tier 2 — LLM Wiki
Obsidian vault of distilled, human-corrected concept pages
Tier 3 — Graph Knowledge
LightRAG for relationship/"what's affected if" questions
Turning a chat server into a platform for real project work.
VS Code + Continue/Cline
Local coding assistant against the workhorse model
Python + FastAPI + ChromaDB
Custom retrieval API for programmatic project use
Per-project AI registry
One config per project: model, endpoint, prompt, knowledge bank
The public layer you're looking at right now.
Next.js 14 + TypeScript
App Router, React Server Components
Tailwind CSS
Design system, dark mode default
Neon Postgres
Serverless database backing the setup checklist
Vercel
Hosting, edge network, environment-managed deploys
Tailscale Funnel bridge
Lets this site call the real home server's API, safely
A note on safety
The home model server is never opened to the public internet as a whole — only a single narrow endpoint is bridged out, behind a shared secret, for this one demo feature. Everything else (the admin chat UI, the raw model API, project files) stays private to the home network and its mesh VPN.