Lightning.
Treat a model's context as a checkpoint you continue from, not a transcript you replay.
- Status:
- research (Early scaffold and design essay)
- Built with:
- TypeScript, llama.cpp
- Licence:
- Apache-2.0
What it is
Lightning is a sketch of a different shape for LLM runtimes. Almost every chat system re-sends the conversation history each turn and hopes the server's prefix cache still matches. Lightning's loop is request plus checkpoint in, response plus next checkpoint out.
It is an idea with a runnable first slice, not a finished runtime: a small, typed, tested TypeScript scaffold with no runtime dependencies.
The thesis
Continuity lives in a tree of mostly append-only branches, each with a parent, a responsibility and a lineage. Branches are stratified by how often they are written, so fast writes stay away from slow state and caches stay valid. Compaction creates a successor branch instead of rewriting the old one.
Being able to answer prompts is not enough: a model has to hold up under checkpointed, branch-aware continuation, and that should be evaluated, not assumed.
What works today
- An in-memory branch store with create, fork, freeze, thaw, checkpoint and restore, and a thermal planner that ranks branches hot, warm or cold.
- A llama.cpp adapter with native KV-cache checkpoints when llama-server runs with a slot save path. Otherwise it reports those operations as emulated and says why.
- Fan-out: one request answered from several saved branch tips in parallel, one slot each.
- Managed slot files: tiered, deleted when nothing holds them, and kept under a disk limit.
- A router and a compaction v0, measured on a small synthetic set.
- A local HTTP API on 127.0.0.1, including an OpenAI-compatible chat completions endpoint, and a branch inspector page.
- One measured setup (a 4-core CPU host, a 0.5B model): answering one question from 4 restored branch tips took about 3.4 s, against about 57 s for 4 cold prefills.
Limitations and scope
- An early scaffold and design essay. The architecture is the point; the code is the first slice that makes it runnable.
- The branch store and checkpoint metadata live in memory. There is no persistence, no background maintainer and no streaming yet.
- Measurements come from one small CPU host and one small model. The router and compaction numbers come from a small synthetic set and are not held out.
- Compaction is extractive. At low budgets it dropped the planted facts and the answers failed.
- Fan-out width is capped by llama-server's slots; extra requests wait.
Source
The code is public. Everything on this page comes from Lightning's own README; the README is the place to check what has changed since.