Local LLM demo box
Model: SmolLM2-1.7B-Instruct (GGUF Q4_K_M)
A large language model is the thing behind a chat assistant: you give it text, it continues it. Every one you have used through a website answers somewhere else, on someone else's hardware, behind an account and an API key.
This demo takes SmolLM2-1.7B-Instruct, quantised to 4 bits in GGUF form, and ships it as a signed, self-contained box that answers on your own machine.
$ scrollcase run .scrollcase/.../*.release.json \
-- "What is the capital of Italy?"
Rome.You define what the box should contain, Scrollcase handles it:
No Python environment to prepare, no model to download at run time, no container. The model and everything it needs are inside the box, which runs it in its own environment.
There is no network call in that command, no key, and no account — and unlike a hosted assistant, nothing about the question leaves the machine.
Try the demo
Download a signed box and run it in a minute, or package one yourself in a Codespace. Both end with the same box; only one of them asks you to build it.
Build it yourself
The demo repository is almost empty on purpose: you package the model yourself, and its README is the walkthrough. Install the CLI, initialise the workspace, create the scroll, declare the model's pinned file, lock, commit, sign and build. Longer than the sentiment demo: most of the wait is the 1.06 GB model, fetched once to pin its hash and once to build.
Open in GitHub Codespaces
Two modes, one box
Give it words and it answers once. Give it nothing and it opens a chat:
$ scrollcase run .scrollcase/.../*.release.json
loading smollm2-1.7b-instruct-q4_k_m.gguf …
ready in 1.4s · 2 threads · 2048-token context
/exit or Ctrl-D to quit, Ctrl-C to cancel an answer
> what is a hash function?
generating …
A hash function maps data of any size to a fixed-size value. …
> give me an example of oneShape of a session, not a recording.
The second question has no subject in it, and it still works: the box keeps the conversation and sends it back each turn, on a single load of the weights. That is the difference between a chat and a loop that calls a one-shot twice — and on CPU, where loading a gigabyte is a visible cost, it is also the difference between the box feeling like a program and feeling like a conversation.
Nothing is rebuilt to get it. Same release file, same signature, same entrypoint: the mode is decided by whether there are arguments, because execution.defaultArgs is [] and a bare run therefore reaches the box with an empty argument list. A --chat flag would have had to be declared in the scroll and signed into the release.
What the demo shows
A signed release, verified before execution. The release document commits to the archive by size and SHA-256, and the archive's filename is its content hash. verify checks the signature against a public key you obtain independently of the download. The mechanics are the same as any other box; what changes here is what the guarantee covers, which is a whole language model rather than a configuration file.
The whole model is one file. A GGUF is a single container holding the weights, the tokenizer and the chat template, so the scroll declares exactly one asset where the sentiment demo needs three — and there is no tokenizer that can drift out of step with the weights it belongs to. It is pinned to an immutable upstream commit, with the size and SHA-256 that add asset recorded when it fetched the file. With weights: embed it is packed into the archive, so the box installs and runs air-gapped.
Offline because there is no downloader, not because a variable says so. The sentiment demo declares HF_HUB_OFFLINE=1 and two siblings as defence in depth, and it needs them: a Hugging Face client really is present in its environment, pulled in transitively. This stack has none. entrypoint.py imports llama_cpp and nothing else, and what keeps the box offline is that there is no code in it that could phone home. Copying those variables across would have looked reassuring and guaranteed nothing, so they are deliberately absent.
The one environment variable that does earn its place. PYTHONDONTWRITEBYTECODE=1, because a .pyc carries a timestamp. Without it the self-test's own import entrypoint writes one into the payload before the payload is hashed, and a box that was extracted, run, and verified again fails the second verification — twice defeated by a cache file nobody asked for.
A self-test that has to actually generate. It loads the gigabyte with the box's own interpreter and asserts that the answer to What is the capital of Italy? contains rome. Greedy decoding (temperature=0.0) is what makes that reproducible enough to assert on content rather than merely on the model having emitted something — and it asserts a substring, not a sentence, so a llama.cpp point release that rewords the answer does not fail a build for a reason nobody cares about.
One declared dependency is not one dependency shipped. The scroll declares llama-cpp-python and nothing else. What arrives with it is the compiled llama.cpp — and also fastapi, uvicorn, pydantic-settings, numpy and diskcache, because the upstream package ships an OpenAI-compatible server this box never starts. Every one of them appears by name in the licence inventory that audit derives from the lock file, which is the point of having one.
What you write
The walkthrough packages a single target, so unlike the sentiment demo there is no split scroll — just one directory:
scrolls/llm-demo/
linux-x86_64-cpu/
scroll.json # identity, the asset, the notices, the environment, the self-test
pixi.toml # python 3.11 + llama-cpp-python
self_test.py # the one file you open in an editorscrollcase new scroll asks eight questions and writes the rest. The model and runtime identity, the box version, the pixi version and the interpreter path are all defaults worth taking; what it cannot guess is the target, what the box is called, which revision of the model is inside, where you will publish it, and what runs when someone starts it.
No hash is typed by hand anywhere. scrollcase add asset fetches the GGUF once and records the size and SHA-256 it found; the notices and the entrypoint are pinned the same way, and scrollcase refresh moves those digests after a reviewed change.
The scroll declares weights: embed, a 4 GB RAM floor and execution as a python-script. That floor is arithmetic rather than a guess: the quantised weights occupy about 1.0 GB and the attention cache adds 384 MiB at the 2048-token context the entrypoint asks for, which lands around 1.5–1.8 GB resident. It is a fact a consumer can check before unpacking a gigabyte.
The packaged version of the same box, the one CI builds for all three operating systems, is examples/llm-demo/ in the Scrollcase repository. There the three targets do share a split scroll: one base carrying the identity, the asset, the environment and the self-test, and three target files of nine lines each — twelve on macOS, which switches the packaged Metal backend off with GGML_METAL_DEVICES=0 so that a box named cpu is one (why that is not automatic). Its entrypoint.py is byte for byte the one the walkthrough ships, and a test asserts the declared hashes still match, so the two copies cannot drift apart quietly.
Measured
The box has been built, self-tested, verified and run on all three CPU targets — Linux, macOS and Windows — by the workflow that publishes it. Each one loads its own gigabyte with its own interpreter and has to answer What is the capital of Italy? with Rome before it is allowed to be signed.
On an M1 MacBook Air the published archive is 1.16 GB, unpacks to 1.3 GB, loads in two to four seconds and generates about 13 tokens per second on eight threads.
What to expect
Running one:
- Generation is CPU-bound. On the 2 vCPU a default Codespace gives you, expect single-digit tokens per second, so a long answer takes tens of seconds. Output is capped at 160 tokens, and a
generating …line on stderr keeps the wait from looking like a hang. - The context is 2048 tokens, shared between the conversation and the answer. In chat mode the oldest exchanges are dropped when it fills, and the box says so on stderr rather than failing.
Building one:
- About 2.1 GB is downloaded — the 1.06 GB GGUF once when
add assetrecords its hash, and again when the build fetches it. Fast inside a Codespace, but worth knowing before you start. - 5–6 GB of disk goes to environment, payload, archive and downloads, against a Codespace's 32 GB. It fits; a second build in the same session does not leave much room.
Scope and limitations
This is a demonstration of packaging, not an assistant product, and the model is small: 1.7 billion parameters. It states false things fluently and gives no signal that it is doing so, it knows nothing of events after its training data, and it cannot reliably do arithmetic or cite sources.
The 4-bit quantisation is a lossy transformation of the original bfloat16 checkpoint — it is what turns 3.4 GB of parameters into a 1.06 GB file that loads on a laptop, and it shifts outputs unevenly. A prompt the original answers correctly is not guaranteed to be answered correctly here, so every limitation the model card documents applies at least as strongly to this box.
It was trained primarily on English and its outputs reflect the biases of that data. Do not use it for factual lookup, for decisions about people, or for anything you would not check yourself.
Both the original checkpoint and the GGUF conversion are published by the same upstream party under Apache-2.0, and the box ships the full licence text next to the model notice.