Tabby Tavern UNIFIED

Taproot + Basecamp + Tabby Tavern on one Compose project and one Docker network.

Default GPU engine is Qwen3.5-9B (llama.cpp CUDA). TabbyAPI EXL3 and Ollama are real, and they sit behind --profile alternate-inference so a 12 GB card does not load three engines at once.

Layer	Software
Primary inference	Qwen3.5-9B · llama.cpp CUDA · `http://qwen:8080/v1`
Alternate inference	TabbyAPI EXL3 + Ollama (profile)
Canonical chat	Open WebUI
Character chat	SillyTavern + shipped character cards
Private search	SearXNG (JSON)
MCP	MCPO + FastMCP · optional dockroot diagnostics
Coding pack (ex-Basecamp)	code-server, Continue, Qdrant, Chroma, n8n, Lobe, AnythingLLM, LibreChat, Mongo, Meilisearch, pgvector

This is a private-lab / portfolio stack — production-shaped, not a multi-tenant SaaS product and not a hosted inference endpoint.

Lab baseline hardware: NVIDIA GeForce RTX 4070 (Linux + Docker Compose + NVIDIA Container Toolkit, WSL2 Ubuntu).

Release v3.0.0 UNIFIED (Aug 2026) matches the live lab after Hermes reconciled Taproot and Basecamp into this tree: 15 default services on tabby-tavern_ai-network, Qwen healthy, MCPO /docs 200, character cards in cards/.

Surface	URL
This HF card	https://huggingface.co/jpanasuk/tabby-tavern-stack
GitHub source of truth	https://github.com/jpanasuk-netizen/tabby-tavern-stack
Sell sheet (Space)	https://huggingface.co/spaces/jpanasuk/tabby-tavern-sell-sheet
Telemetry / benchmarks	https://github.com/jpanasuk-netizen/local_grid_suite
Multi-agent prototype	https://github.com/jpanasuk-netizen/multi-agent-dungeon-crawler
Author	https://huggingface.co/jpanasuk · @jpanasuk-netizen

Weights are not included. You download EXL3 and/or GGUF models yourself. Secrets are not included. Copy examples and generate your own keys. Character cards are included. Import the PNGs under cards/ into SillyTavern.

Service ports (compose defaults)

Default path is Qwen → Open WebUI. Inside Docker use the service name, not localhost.

Service	Host	In-network	Role
`qwen`	1234	`http://qwen:8080/v1`	Primary llama.cpp CUDA · Qwen3.5-9B
`open-webui`	3000	8080	Canonical chat UI
`sillytavern`	8000	8000	Character frontend + cards
`searxng`	8080	8080	Private JSON search
`mcpo`	8001	8000	MCP → OpenAPI
`code-server`	8443	8080	Browser VS Code + Continue
`librechat`	3080	3080	Extra chat UI
`lobe-chat`	3210	3210	Extra chat UI
`anythingllm`	3002	3001	RAG workspace
`n8n`	5678	5678	Workflows
`qdrant`	6333/6334	6333	Vector DB
`chroma`	8005	8000	Embeddings DB
`meilisearch`	7700	7700	Full-text
`postgres`	5432	5432	pgvector / Hermes memory
`mongo`	27017	27017	LibreChat
`firefox`	3010	3000	In-stack browser with Tabby favorites
`tabbyapi`	5000	5000	EXL3 — `--profile alternate-inference`
`ollama`	11435	11434	GGUF — `--profile alternate-inference`

Host bookmark bar (Chrome / Firefox)

These are the host homes for every UNIFIED add-on with a web UI. Import bookmarks.html into Chrome (Bookmarks → Import bookmarks and settings → Bookmarks HTML file) or Firefox. The in-stack browser is already seeded: http://127.0.0.1:3010

Bookmark	Host URL
Open WebUI	http://127.0.0.1:3000
Qwen llama.cpp	http://127.0.0.1:1234
SillyTavern	http://127.0.0.1:8000
MCPO docs	http://127.0.0.1:8001/docs
SearXNG	http://127.0.0.1:8080
code-server	http://127.0.0.1:8443
LibreChat	http://127.0.0.1:3080
AnythingLLM	http://127.0.0.1:3002
Lobe Chat	http://127.0.0.1:3210
n8n	http://127.0.0.1:5678
Qdrant	http://127.0.0.1:6333/dashboard
Chroma	http://127.0.0.1:8005
Meilisearch	http://127.0.0.1:7700
Tavern Firefox	http://127.0.0.1:3010

Mongo (:27017) and Postgres (:5432) have no HTML home. TabbyAPI (:5000) and Ollama (:11435) stay behind --profile alternate-inference.

Open WebUI on the unified stack:

ENABLE_OLLAMA_API=false
ENABLE_OPENAI_API=true
OPENAI_API_BASE_URL=http://qwen:8080/v1
ENABLE_RAG_WEB_SEARCH=true
RAG_WEB_SEARCH_ENGINE=searxng
SEARXNG_QUERY_URL=http://searxng:8080/search?q=<query>

All services share the bridge network ai-network.

System requirements

Host

OS: Linux native, or Windows WSL2 with Docker Desktop / Docker Engine
GPU: NVIDIA GPU with recent drivers (lab used RTX 4070)
Docker: Docker Engine + Compose plugin (docker compose version)
NVIDIA Container Toolkit (required for GPU passthrough into containers)
Disk: room for Docker images plus model weights (EXL3 8B class is often tens of GB; GGUF varies by quant)
RAM / VRAM: 8B-class EXL3 fits a 12 GB class card with headroom when KV/cache is tuned; larger models need more VRAM

One-time NVIDIA Container Toolkit check

nvidia-smi

docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

If nvidia-smi works on the host but fails in Docker, fix the toolkit / Docker daemon GPU runtime before bringing the stack up.

Optional host tools

pipx install "huggingface_hub[cli]"
# or: python3 -m venv .venv && . .venv/bin/activate && pip install -U "huggingface_hub[cli]"

Cold start (stranger path)

Prefer cloning GitHub for day-to-day work. This HF repo is the public model card + sanitized mirror of the same layout.

0) Clone

git clone https://github.com/jpanasuk-netizen/tabby-tavern-stack.git
cd tabby-tavern-stack

Alternate (HF mirror — same public layout, may lag GitHub):

git lfs install   # only if you later pull large assets; weights still not shipped
git clone https://huggingface.co/jpanasuk/tabby-tavern-stack
cd tabby-tavern-stack

1) Build the TabbyAPI image used by compose

Compose expects this exact local tag:

docker build -f Dockerfile.tabby -t local/tabbyapi:exl3-fixed .

Dockerfile.tabby is the WSL2-tested image: ghcr.io/theroyallab/tabbyapi:latest plus build deps and a libcuda.so symlink so ExLlamaV3/Triton can link inside WSL2.

2A) Download an EXL3 model for TabbyAPI

Weights go under tabby_models/ (gitignored). Compose mounts:

./tabby_models  →  /app/models   (inside tabbyapi)

TabbyAPI model_name is a directory name under that mount.

mkdir -p tabby_models

huggingface-cli download turboderp/Llama-3.1-8B-Instruct-exl3 \
  --revision 6.0bpw \
  --local-dir tabby_models/Llama-3.1-8B-Instruct-exl3-6.0bpw

ls -la tabby_models/Llama-3.1-8B-Instruct-exl3-6.0bpw | head

EXL2 will not load. Current TabbyAPI dropped the ExLlamaV2 backend. Use EXL3 quants only.

2B) Configure TabbyAPI (required before first launch)

cp -n tabby_config/config.example.yml tabby_config/config.yml

Generate real keys and paste them in:

python3 -c "import secrets; print('admin_key:', secrets.token_hex(32)); print('api_key:  ', secrets.token_hex(32))"

Set model_name to the folder you downloaded. For a 12 GB card the lab uses:

model:
  model_dir: models
  model_name: Llama-3.1-8B-Instruct-exl3-6.0bpw
  max_seq_len: 8192
  cache_size: 8192
  cache_8bit: true

Compose always mounts:

./tabby_config/config.yml  →  /app/config.yml

Put the same api_key in:

open-webui OPENAI_API_KEY
mcpo/config.json TABBYAPI_KEY
SillyTavern API settings

2C) Ollama / GGUF path (secondary backend)

After the stack is up:

docker exec -it tabby-tavern-ollama-1 ollama pull llama3.1:8b
docker exec -it tabby-tavern-ollama-1 ollama list
docker exec -it tabby-tavern-ollama-1 ollama run llama3.1:8b "Say hello in one sentence."

You can run TabbyAPI (EXL3) and Ollama (GGUF) together; on a 12 GB card load one heavy model at a time unless you know your headroom. Lab measurement: ~8 GB VRAM for Llama-3.1-8B EXL3 6.0bpw + 8-bit cache, ~4 GB left for a small GGUF.

3) Frontend / search secrets (placeholders only)

SillyTavern (sillytavern_config/config.yaml):

Default listen port 8000
browserLaunch.enabled: false (no browser inside Docker)
basicAuthMode: true — ST refuses to start on 0.0.0.0 without auth
Replace:

basicAuthUser:
  username: "YOUR_ST_USERNAME_HERE"
  password: "YOUR_ST_PASSWORD_HERE"

SearXNG (searxng_config/settings.yml):

use_default_settings: true
server:
  secret_key: "YOUR_SEARXNG_SECRET_KEY_HERE"
  image_proxy: true
search:
  formats:
    - html
    - json

JSON format is required for MCPO searxng_search and Open WebUI RAG.

4) Character cards (SillyTavern)

This release ships Tavern character cards under cards/:

File	What it is
`cards/default_Seraphina.png`	SillyTavern PNG character card (embedded spec)
`cards/Seraphina/*.png`	Expression sprites used by the card
`cards/README.md`	Import notes

Use the cards:

Start SillyTavern → http://localhost:8000
Characters → Import → pick cards/default_Seraphina.png
Copy cards/Seraphina/ into SillyTavern's character expressions folder if the importer does not pull sprites automatically (sillytavern_data/default-user/characters/Seraphina/)

Do not commit chats, secrets.json, or live API keys from a running data directory.

5) Launch

docker compose up -d
docker compose ps

Or:

chmod +x start-stack.sh load-model.sh
./start-stack.sh

6) Health checks (every service)

docker compose ps
docker compose logs --tail=80 tabbyapi
docker compose logs --tail=40 ollama
docker compose logs --tail=40 sillytavern
docker compose logs --tail=40 open-webui
docker compose logs --tail=40 searxng
docker compose logs --tail=40 mcpo

curl -sS -o /dev/null -w "sillytavern  %{http_code}\n" http://127.0.0.1:8000/ || true
curl -sS -o /dev/null -w "open-webui   %{http_code}\n" http://127.0.0.1:3000/ || true
curl -sS -o /dev/null -w "tabbyapi     %{http_code}\n" http://127.0.0.1:5000/ || true
curl -sS -o /dev/null -w "ollama       %{http_code}\n" http://127.0.0.1:11435/ || true
curl -sS -o /dev/null -w "searxng      %{http_code}\n" http://127.0.0.1:8080/ || true
curl -sS -o /dev/null -w "mcpo         %{http_code}\n" http://127.0.0.1:8001/docs || true

docker exec -it tabby-tavern-tabbyapi-1 nvidia-smi || true

Browser targets:

SillyTavern → http://localhost:8000
Open WebUI → http://localhost:3000
TabbyAPI → http://localhost:5000
Ollama (host) → http://localhost:11435
SearXNG → http://localhost:8080
MCPO Swagger → http://localhost:8001/docs
Firefox (in-stack) → http://localhost:3010
AnythingLLM → http://localhost:3002
LibreChat → http://localhost:3080
Lobe Chat → http://localhost:3210
n8n → http://localhost:5678
Qdrant dashboard → http://localhost:6333/dashboard
Chroma → http://localhost:8005
Meilisearch → http://localhost:7700
code-server → http://localhost:8443

7) Point SillyTavern at TabbyAPI

In SillyTavern API settings:

API type: OpenAI-compatible / TabbyAPI
Endpoint: http://tabbyapi:5000/v1 from another container, or http://127.0.0.1:5000/v1 from the host
API key: the api_key in tabby_config/config.yml

If ST cannot reach TabbyAPI, check:

Both containers on ai-network
Keys match
TabbyAPI finished loading the EXL3 model (docker compose logs -f tabbyapi)

8) Switch / reload TabbyAPI model

# Edit model_name in tabby_config/config.yml to another folder under tabby_models/
docker compose restart tabbyapi
docker compose logs -f tabbyapi

MCPO (MCP → OpenAPI)

MCPO (ghcr.io/open-webui/mcpo:main) launches MCP servers as subprocesses and exposes their tools as OpenAPI HTTP endpoints.

mcp-servers/server.py is a FastMCP server (not raw JSON-RPC) with 7 tools:

Tool	Description
`list_tabbyapi_models`	List loaded EXL3 models in TabbyAPI
`tabbyapi_chat`	Chat with the TabbyAPI EXL3 model
`list_ollama_models`	List GGUF models in Ollama
`ollama_pull_model`	Pull a model into Ollama
`ollama_chat`	Chat with an Ollama model
`get_stack_status`	Health check all services
`searxng_search`	Web search via SearXNG (JSON)

Config notes:

mcpo/config.json must call /app/.venv/bin/python3 (MCPO image venv has mcp)
Env: TABBYAPI_URL, OLLAMA_URL, TABBYAPI_KEY (placeholder in public tree)
Tools are served under /host-master/ (server name in config)
Empty "mcpServers": {} crashes MCPO — keep at least one entry
Set --api-key in compose (public tree uses REPLACE_WITH_YOUR_MCPO_API_KEY)

A second server, mcp-servers/tavern_mcp.py, is the lab connectivity toolkit (status, self_check, wire, models, chat). Wire it only if you also ship discover.py next to it.

GPU / compose tuning (what the lab actually ships)

Setting	Value	Why
`deploy.resources.reservations.devices`	NVIDIA GPU `count: all`	Passthrough
`shm_size`	16g	Avoid shared-memory OOM during load/attn
`CUDA_VISIBLE_DEVICES`	`0`	Pin first GPU
`CUDA_DEVICE_ORDER`	`PCI_BUS_ID`	Stable device ordering
`PYTORCH_CUDA_ALLOC_CONF`	`max_split_size_mb:128`	Allocator fragmentation control
`EXLLAMA_GPU_LAYERS`	`999`	Prefer full GPU offload
`EXLLAMA_KV_CACHE`	`q8_0`	KV cache quant tradeoff
`EXLLAMA_FLASH_ATTENTION`	`1`	Flash-attn path when available
`OLLAMA_FLASH_ATTENTION`	`1`	Ollama flash-attn
`OLLAMA_KV_CACHE_TYPE`	`q8_0`	Ollama KV quant
TabbyAPI `cache_8bit`	`true`	Halves KV VRAM on 12 GB cards

If you OOM: lower max_seq_len / cache_size, use a smaller bpw EXL3, or stop Ollama models while TabbyAPI holds a large model.

Repository structure

tabby-tavern-stack/
├── docker-compose.yml              # lab orchestration (6 services)
├── docker-compose.starter.yml      # optional coding-tools overlay
├── Dockerfile / Dockerfile.tabby   # TabbyAPI image (WSL2 libcuda fix)
├── start-stack.sh
├── load-model.sh
├── cards/                          # SillyTavern character cards (PNG)
├── mcpo/config.json                # MCPO server map (placeholders)
├── mcp-servers/server.py           # FastMCP stack tools
├── tabby_config/config.example.yml
├── sillytavern_config/config.yaml
├── searxng_config/settings.yml
├── tabby_models/                   # EXL3 weights (gitignored contents)
├── SECURITY.md
├── DEVLOG.md
├── LICENSE
└── docs/                           # sell-sheet extras

Compose service names (authoritative):

tabbyapi · sillytavern · ollama · open-webui · searxng · mcpo

Environment & secrets guidance

Secret / file	Where	Rule
TabbyAPI `admin_key` / `api_key`	`tabby_config/config.yml`	Generate yourself; never commit live values
Open WebUI `OPENAI_API_KEY`	compose env	Same value as TabbyAPI `api_key`
MCPO `--api-key` / `TABBYAPI_KEY`	compose + `mcpo/config.json`	Placeholders in public tree
SillyTavern basic auth	`sillytavern_config/config.yaml`	Replace `YOUR_ST_*` placeholders
SearXNG `secret_key`	`searxng_config/settings.yml`	Replace placeholder
Open WebUI DB	`openwebui_data/`	gitignored
Ollama keys/models	host `~/.ollama` or `ollama_data/`	gitignored

Private-lab defaults are intentional. This is not hardened multi-tenant hosting.

Before any LAN/WAN exposure:

Replace every placeholder credential
Prefer binding host ports to 127.0.0.1
Put a reverse proxy + TLS in front if you leave the machine
Read SECURITY.md

Older public revisions of this mirror contained lab convenience keys (including a TabbyAPI key in mcpo/config.json). Treat any key you ever saw in a public file as burned and rotate it.

Common failure modes

Symptom	Likely cause	Fix
`local/tabbyapi:exl3-fixed` start fails	Image never built	`docker build -f Dockerfile.tabby -t local/tabbyapi:exl3-fixed .`
TabbyAPI: `exl2` backend no longer supported	Pointed at an EXL2 directory	Download an EXL3 revision instead
`/usr/bin/ld: cannot find -lcuda`	WSL2 CUDA libs not on linker path	Use the patched `Dockerfile.tabby` in this repo
TabbyAPI cannot find model	`model_name` ≠ folder under `tabby_models/`	Align names; confirm mount
CUDA / GPU errors in container	Toolkit missing	Fix NVIDIA Container Toolkit first
OOM during load	Model + KV + dual backends too large	Smaller quant; `cache_8bit`; unload Ollama
ST cannot talk to TabbyAPI	Auth/network mismatch	Same network; matching API key; wait for load
Open WebUI shows no models	Ollama empty or URL wrong	`ollama pull`; confirm `OLLAMA_BASE_URL`
Host port 11434 in use	Host Ollama already running	Keep compose map `11435:11434`
SearXNG 500s / no JSON	Placeholder secret or HTML-only formats	Set secret; add `json` to `search.formats`
MCPO crash: no mcpServers	Empty config	Keep at least one server entry
SillyTavern refuses to start	Listen 0.0.0.0 with no auth	Keep `basicAuthMode: true`
NVIDIA apt `Malformed entry`	Wrong toolkit repo line	See WSL2 notes below

Measured lab results (defensible only)

Numbers from checked-in sample telemetry in local_grid_suite (benchmarks/sample_hardware_runs.json). Single-box lab runs — not a product SLA.

Stage	Model	Decode tok/s
Baseline	`qwen3:8b`	1.39
GPU-routed	`qwen-gpu:latest`	29.7 – 39.3
Stabilized	`qwen-gpu:latest`	37.47

→ ~27× decode uplift on that run series after GPU routing / tuning.

Warm stream suite on qwen3:8b: ~76 tok/s (400-token runs).

WSL2 EXL3 chat (Llama-3.1-8B-Instruct 6.0bpw, RTX 4070): ~53 tok/s processing in the verified 24/24 check pass.

Re-measure on your hardware. Do not advertise these as guaranteed throughput.

Engineering notes

See DEVLOG.md for the build log:

Compose consolidation (TabbyAPI + ST + Open WebUI + Ollama + SearXNG + MCPO)
EXL3 adoption; EXL2 dropped upstream
GPU env tuning (shm_size, flash-attn / KV cache, cache_8bit)
Container-to-TabbyAPI auth / whitelist fixes
WSL2 NVIDIA toolkit + libcuda.so linker fix
MCPO FastMCP server (7 tools verified)
Character cards shipped in cards/

Optional coding starter overlay

docker-compose.starter.yml is an optional overlay (browser IDE, vector DBs, n8n, extra chat UIs). It is not required for the core six-service lab. It expects the core stack network ai-network to already exist. Change every placeholder password before up.

docker compose up -d
docker compose -f docker-compose.starter.yml up -d

WSL2 field notes — fresh-install lessons (Aug 2026)

A from-scratch rebuild on a clean WSL2 Ubuntu environment (same RTX 4070). Additive to the cold-start guide.

EXL2 vs EXL3 — TabbyAPI dropped EXL2 support

ValueError: Models quantized with 'exl2' require the exllamav2 backend, which is no longer supported. Please use an exl3 or unquantized model.

Use EXL3 quants only (turboderp/Llama-3.1-8B-Instruct-exl3 branches 2.0–8.0 bpw).

NVIDIA Container Toolkit repo URL

The deb .../noble main line is wrong for this repo (apt: Malformed entry (Component)). NVIDIA uses a flat structure:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
  | sudo gpg --dearmor --yes -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

echo 'deb [arch=amd64 signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://nvidia.github.io/libnvidia-container/stable/deb/amd64 /' \
  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

libcuda.so not found inside TabbyAPI on WSL2

Even with torch.cuda.is_available() == True, model load can fail with cannot find -lcuda. The patched Dockerfile in this repo creates:

RUN ln -sf /usr/local/cuda-12.8/compat/libcuda.so /usr/lib/x86_64-linux-gnu/libcuda.so && ldconfig

Triton “not supported, roll back to CPU” warnings are cosmetic — exllamav3 uses its own CUDA kernels.

Port conflict when host Ollama is already running

Compose maps 11435:11434. Open WebUI still uses http://ollama:11434 internally.

Verified working state

After the v2.0.0 tree: 24/24 checks passing on the lab box — six containers, EXL3 chat, Ollama llama3.1:8b, Open WebUI dual backend, SillyTavern with cards, SearXNG JSON, MCPO 7 tools, ~8 GB / 12 GB VRAM.

Related spine

Project	Link
Stack (this card)	https://huggingface.co/jpanasuk/tabby-tavern-stack
Stack (GitHub)	https://github.com/jpanasuk-netizen/tabby-tavern-stack
Sell sheet Space	https://huggingface.co/spaces/jpanasuk/tabby-tavern-sell-sheet
Local Grid Suite	https://github.com/jpanasuk-netizen/local_grid_suite
Multi-agent prototype	https://github.com/jpanasuk-netizen/multi-agent-dungeon-crawler
Collection	https://huggingface.co/collections/jpanasuk/independent-ai-lab-spine-6a727803ed9c6d16164f5130

Author

Jeremy Panasuk — enterprise data / platform background; Aug 2024–present independent local-AI systems year (private Docker LLM lab, decode telemetry, multi-agent prototypes).

GitHub: @jpanasuk-netizen
Hugging Face: jpanasuk
LinkedIn: jeremy-p-34203322

License

MIT

Downloads last month: -; Downloads are not tracked for this model. How to track

Spaces using jpanasuk/tabby-tavern-stack 8

Collection including jpanasuk/tabby-tavern-stack

Independent AI Lab Spine

Collection

Stack + telemetry + multi-agent prototype companions for the independent engineering year. • 3 items • Updated 5 days ago