Component guide
tgi
Production model serving from Hugging Face.
Atlas
in AtlasStack-Atlas-Pro-+-Support-200
text-generation-inference
Licence: Apache-2.0 | Language: Python | Source: https://github.com/huggingface/text-generation-inference
Role in the stack
Production Hugging Face model serving
Why it's here
Continuous batching, token streaming and a production HTTP API for open models.
Using it
See the project README in the fetched source:
./fetch-sources.sh ./sources
tar xzf ./sources/text-generation-inference.tar.gz -C ./sourcesRunning it
Started by docker compose up -d on port 8080.
This component ships inside the Atlas
stack. See pricing →