# Runbook

## Daily checks

```bash
docker compose ps
docker compose logs --since 1h --tail 100
./verify.sh
```

## Common problems

**Model pull is slow or fails.**
Model weights are gigabytes. Confirm disk space (`df -h`) and that the
container can reach the registry. For a fully offline host, pull on a
connected machine and copy the volume.

**Out of memory during inference.**
A 7B model in fp16 needs roughly 14 GB. Use a quantised build (Q4 needs about
4-5 GB) or a smaller model. Ollama picks a quantisation automatically; for
llama.cpp pass an explicit `-ngl` and quantised GGUF.

**Port already in use.**
Change the host side of the mapping in `docker-compose.yml`, e.g.
`"11435:11434"`. Update `.env` and any client config to match.

**GPU not detected.**
Install the NVIDIA container toolkit and confirm `docker run --rm --gpus all
nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi` works before blaming the stack.

## Backup and restore

State lives in named volumes:

```bash
docker run --rm -v atlasstack_qdrant_data:/data -v "$PWD":/backup \
  alpine tar czf /backup/qdrant-$(date +%F).tgz -C /data .
```

Restore is the reverse. Stop the service first so the volume is not mid-write.

## Upgrading

Tags default to `latest`. Pin them in `.env` for reproducible deploys:

```
OLLAMA_TAG=0.3.12
```

Then `docker compose pull && docker compose up -d`. Read the upstream release
notes before a major bump - vector store storage formats occasionally change.

## Services in this tier

- `ollama`
- `tgi`
- `vllm`
- `localai`
- `qdrant`
- `chroma`
- `milvus`