What does O(1) hot state and O(log N) probes mean?

Hot state is what a decoding step must keep live: for HELIX that is a window-sized ring buffer, one delta-rule matrix and two convolution states per head, all fixed-size regardless of context length. The key/value history and landmark tree grow with N but are append-only, never rewritten, and can sit in cold memory. A step reads only O(index_topk x block_size + index_branching x log N) of that. It is explicitly not O(1) total memory, which is impossible for anything that can quote an exact name from a million tokens back.

How does the landmark index stay causal?

Two rules. A node is eligible for query block J only if the entire span it summarises ends at or before J, so a summary is never consulted by a query it partly describes — which also means a landmark is computed once and reused by every later query. And routing for block J comes from the hidden state at the end of block J-1, which is strictly past for every token in block J, so one gather serves the whole block. Anything routing could not anticipate within that block is covered exactly by the local strand.

How much faster is HELIX than full attention?

On the published measurements, HELIX forms a constant 8,192 attention pairs per token at every context length, fitting N^1.00, while full attention fits N^2.00. At 32,768 tokens that is 32 times fewer pairs and 1.6 times faster wall-clock — and that comparison runs unfused reference PyTorch against a fused SDPA kernel, so the constant factor is working against HELIX, not for it. At 4,096 tokens full attention is still the faster of the two.

Does the index strand actually matter?

That is the one result the recall experiment establishes. On multi-query associative recall with bindings placed well outside the local window, HELIX scores 30.8 percent against 6.0 percent for the same model with the index strand removed — a gap of roughly 21 standard errors, with chance at 6.2 percent. Without the index the model sits at chance, because its window cannot see the bindings and its recurrent state cannot hold them. Against full attention at 32.0 percent, HELIX is a tie rather than a win: 1.2 points with a 1.7-point standard error.

How do I install and try HELIX?

Run pip install helix-lm. The package ships a command-line tool: helix info explains what it is and what it costs, helix demo builds a model and verifies the decode state, helix bench measures cost against context length, and helix recall trains with and without the index strand and compares them. It requires Python 3.10 or newer and is licensed Apache-2.0.

HELIX — Linear-Scaling Sequence Architecture With Exact Long-Range Recall

Q: What is HELIX?

HELIX — Hierarchical Episodic Linear IndeX — is a sequence architecture that scales linearly in context length while keeping the exact long-range recall that attention is good at. Every block braids three mixers over the same residual stream: a local strand reading a fixed window exactly, a recurrent strand compressing the whole past into a matrix-valued delta-rule state, and an index strand retrieving a handful of memory blocks from anywhere in the past. It is published as the helix-lm package under Apache-2.0 and was built by Nathan at CorX Labs in Jamaica.

Q: Why can't a linear-time model just compress the past?

Because of a counting argument, not an engineering gap. A fixed state of S bits cannot distinguish more than 2^S different histories, so a model that squeezes everything into a fixed-size state eventually cannot quote back a name it saw once. HELIX does not try to compress its way around this. It separates memory capacity from memory bandwidth: storage grows linearly and stays append-only, while what any single token touches stays bounded.

Q: Has HELIX been trained on real text?

No. There are no trained checkpoints and nobody has trained HELIX on real language data. Everything published about it concerns complexity, causality and recall mechanics — properties provable on an untrained model or measurable on a synthetic probe — not perplexity or language quality. It is an architecture proposal with a working, tested reference implementation, not a better model.

HELIX is a sequence architecture built from scratch by CorX Labs in Jamaica, published as the helix-lm package. A transformer compares every token to every other token: ten thousand tokens is fifty million comparisons, and doubling the input quadruples the work. Linear-time alternatives fix that by squeezing the past into a fixed-size state — and then they cannot quote back a name they saw once, because a state of S bits cannot tell apart more than 2^S histories. That is a counting argument, not an engineering gap.

HELIX does not try to compress its way around it. It separates memory capacity from memory bandwidth: storage grows linearly, stays append-only and can live in cold memory, while what any one token touches stays bounded. The claim is O(1) hot state and O(log N) probes — not O(1) total memory, which is impossible for anything that can quote an exact name from a million tokens back.

There are no trained checkpoints. Nobody has trained HELIX on real text. Everything measured here is about complexity, causality and recall mechanics — properties provable on an untrained model or measurable on a synthetic probe — not perplexity or language quality. Treat this as an architecture proposal with a working, tested reference implementation, not a better model. The full caveats are below.

Package: helix-lm 0.1.0
Mixers per block: 3 (local, recurrent, index)
Attention pairs: N^1.00 measured
Hot decode state: O(1)
Probes per step: O(log N)
Storage: O(N), append-only
Runtime: PyTorch
Licence: Apache-2.0
Status: Untrained reference implementation

1. One HELIX block

Every block braids three mixers over the same residual stream. They differ in what they read, what they cost to train, and — the part that matters at inference — what they have to hold on to while decoding. Solid lines carry representations forward; dashed lines are reads against stored memory.

Representation forward Read against stored memory Capacity grows; bandwidth does not

One HELIX block. The same residual stream feeds all three mixers, and their outputs are braided back into it. The middle column is what each strand costs at training time; the right column is the part that decides whether the thing is usable at inference — what has to stay live per step.

The three HELIX strands: what each reads, its training cost, and its decode-time state
Strand	Reads	Training cost	Held at decode
L — local	The last `local_blocks × block_size` tokens, exactly, with a different window per head	O(N · span)	A window-sized ring buffer
R — recurrent	Everything, compressed into a matrix-valued delta-rule state	O(N · d_k · d_v)	One matrix plus two convolution states per head
I — index	`index_topk` blocks chosen anywhere in the past, by a beam descent over a landmark tree	O(N · topk · block_size) + O(N log N) routing	The key/value history and a landmark tree, of which a step reads O(topk · block_size + branching · log N)

2. How the index works

The past is cut into blocks of block_size tokens. Each closed block gets a landmark — a learned pooled summary that mixes a plain mean with an attention pool, because a mean alone erases the rare token a later query will be hunting for. Landmarks are pooled again, index_branching at a time, into a tree, using one weight-shared pooler at every internal level.

Pooling & tree structure Routing & retrieval Eligible spans only — causal by construction

The landmark tree. Blocks are pooled into landmarks, landmarks into internal nodes, and a query block descends from the root to a handful of blocks it then reads in full. A node is only eligible if the entire span it summarises closed before the query block, so a summary is never consulted by a query it partly describes.

A query block retrieves in four moves:

Start at the root, using routing taken from the end of the previous block.
Keep index_beam_width nodes per level, scoring the beam's children plus that level's frontier.
Descend to the landmarks, accumulating selection scores along the path.
Read index_topk whole blocks exactly — the tokens themselves, not their summaries.

Two rules keep it causal and parallel:

A node is eligible for query block J only if the whole span it summarises ends at or before J. A summary is never consulted by a query it partly describes, so a landmark is computed once and reused by every query.
Routing for block J comes from the hidden state at the end of block J − 1 — strictly past for every token of block J, so one gather serves block_size queries. Anything routing could not anticipate inside that block is covered exactly by strand L.

The descent keeps index_beam_width nodes per level. Candidates at each level are the beam's children plus that level's frontier — eligible nodes whose parent is not eligible. At most branching − 1 per level, and the frontiers across all levels tile [0, J) exactly, so nothing reachable is cut off by a beam that descended elsewhere. Selection scores are fed back as additive attention logits, accumulated along the descent path, which is what lets gradient reach every level of the tree rather than just the leaves.

3. What has been measured

On small models, 4 CPU cores. Reproduce with helix bench and helix recall.

Cost against context length

Attention pairs are exact — the query/key products the forward pass actually forms, counted by instrumenting the attention call, not estimated from the formula.

Attention pairs per token and wall-clock time for HELIX against full attention, by context length
Tokens	HELIX pairs/token	Full attention pairs/token	HELIX	Full attention
4 096	8 192	32 776	0.4 s	0.2 s
8 192	8 192	65 544	0.9 s	0.6 s
16 384	8 192	131 080	1.7 s	1.9 s
32 768	8 192	262 152	4.3 s	6.7 s

HELIX fits N^1.00 in attention pairs; full attention fits N^2.00. By 32k tokens HELIX forms 32× fewer pairs and runs 1.6× faster — as unfused reference PyTorch against a fused SDPA kernel. At 4k tokens full attention is still the faster of the two; the crossover is somewhere between 8k and 16k on this hardware.

Recall from far back

Multi-query associative recall: bindings written near the start, queried roughly 200 tokens later, well outside the local window.

Multi-query associative recall accuracy with 95 percent confidence intervals
Model	Recall	95% CI
HELIX (L+R+I)	30.8%	[28.5%, 33.1%]
HELIX no index (L+R)	6.0%	[4.8%, 7.2%]
Full attention	32.0%	[29.7%, 34.3%]
Chance	6.2%	—

Read the two gaps separately. HELIX against its own ablation is the result: strip the index strand and the same model — same width, same depth — sits at chance, because its window cannot see the bindings and its recurrent state cannot hold them. That gap is about 21 standard errors.

HELIX against full attention is a tie, not a win or a loss: 1.2 points with a 1.7-point standard error, z = 0.72. Anyone reporting that as a victory over attention would be reading noise.

4. Correctness

pytest covers the properties you can check without training anything:

Strictly causal — editing token t leaves every earlier logit bitwise unchanged.
Decode == prefill — token-by-token generation from the fixed-size state reproduces the one-shot forward, as does unaligned chunked prefill.
Right padding is exactly invariant.
Constant work per token — each query reads the same number of keys at any context length.
The cache is what the architecture claims — fixed-size recurrent and convolution states everywhere, a window-capped key/value cache on non-indexing layers.
Tiling is bitwise inert — attention_tile_blocks caps peak activation memory and changes nothing about the result.
Every parameter gets gradient, landmark poolers included.

5. What this is not

There are no trained checkpoints. Nobody has trained HELIX on real text. Everything above is about complexity, causality and recall mechanics — properties provable on an untrained model or measurable on a synthetic probe — not about perplexity or language quality. The inductive-bias arguments (multi-scale windows, short convolutions, surprise-gated writes) are untested hypotheses.

Treat this as an architecture proposal with a working, tested reference implementation, not a better model. Finding out whether it is actually better needs real training runs on real hardware.

Known design limits, stated plainly

One retrieval serves a whole query block, so index_topk must cover the diversity a block_size span asks for.
Routing is one block stale — the index cannot react to the token currently being generated.
Landmarks are pooled from keys that already carry their rotary phase, so routing is not purely content-addressed.
Left padding shifts the block grid, so it is not bit-identical to an unpadded run. Right padding is.
No fused kernel for the gathered attention yet; the scaling is right, the constant factor is not.

6. Using it

Install from PyPI:

pip install helix-lm

Build a model and run it:

import torch
from helix_lm import HelixConfig, HelixForCausalLM

model = HelixForCausalLM(HelixConfig(vocab_size=32000, hidden_size=1024, num_hidden_layers=12))
logits = model(torch.randint(0, 32000, (1, 8192))).logits
text = model.generate(torch.randint(0, 32000, (1, 64)), max_new_tokens=32, temperature=0.8)

The package ships a command-line tool, which is the fastest way to see the claims hold up on your own machine rather than taking them on trust:

helix info      # what it is and what it costs
helix demo      # build a model, generate, verify the decode state
helix bench     # cost against context length
helix recall    # train with and without the index strand, and compare

Configuration

Every knob is on HelixConfig, documented in its docstring. The ones that matter most:

HelixConfig(
    block_size=64,          # memory-block granularity
    local_blocks=4,         # strand L sees 4 previous blocks plus its own
    num_window_scales=4,    # head groups get geometrically different windows
    index_topk=8,           # blocks strand I retrieves per query block
    index_layer_stride=3,   # ...on every third layer; the rest keep an O(1) KV cache
    index_branching=8,      # landmark tree fan-out
    attention_tile_blocks=64,  # caps peak activation memory; no effect on the result
)

Licence. Apache-2.0. Portions of the delta-rule and rotary helpers derive from HuggingFace Transformers (Apache-2.0); see NOTICE.

7. Questions

What is HELIX, in one paragraph?

Hierarchical Episodic Linear IndeX — a sequence architecture that scales linearly in context length while keeping the exact long-range recall that attention is good at. Every block braids three mixers over the same residual stream: a local strand reading a fixed window exactly, a recurrent strand compressing the whole past into a matrix-valued delta-rule state, and an index strand retrieving a handful of memory blocks from anywhere in the past. Published as helix-lm under Apache-2.0.

Why can’t a linear-time model just compress the past?

A counting argument stops it. A fixed state of S bits cannot tell apart more than 2^S different histories, so a model that squeezes everything into a fixed-size state eventually cannot quote back a name it saw once. That is not an engineering gap that a better compressor closes. HELIX separates capacity from bandwidth instead: storage grows linearly and stays append-only, while what any single token touches stays bounded.

What exactly is O(1), and what isn’t?

The hot state is O(1): a window-sized ring buffer, one delta-rule matrix and two convolution states per head, all fixed-size whatever the context length. The key/value history and landmark tree are O(N) — but append-only, never rewritten, and able to live in cold memory. A step reads only O(index_topk × block_size + branching × log N) of it. Total memory is explicitly not O(1), which would be impossible for anything that can quote an exact name from a million tokens back.

How does the index stay causal if summaries are shared?

A node is eligible for query block J only if the entire span it summarises ends at or before J — so a summary is never consulted by a query it partly describes, which is also what lets a landmark be computed once and reused by every later query. Routing for block J comes from the hidden state at the end of block J − 1, strictly past for every token in the block, so a single gather serves all block_size queries. Whatever routing could not anticipate inside that block is covered exactly by the local strand.

Has it been trained on real text?

No. There are no trained checkpoints and no language-quality numbers of any kind. The published results cover complexity, causality and recall mechanics only. Whether HELIX is actually better than the alternatives is an open question that needs real training runs on real hardware to answer.

Is it faster than attention?

At long context, on the measurements published: 32× fewer attention pairs and 1.6× faster wall-clock at 32k tokens, with HELIX running unfused reference PyTorch against a fused SDPA kernel. At 4k tokens attention is still faster. The scaling is the claim; the constant factor is not, and a fused kernel for the gathered attention does not exist yet.

Does the index strand actually earn its place?

That is the one thing the recall experiment does establish. With the index: 30.8%. Same model with the index strand stripped out: 6.0%, against a chance floor of 6.2% — indistinguishable from guessing. The gap is roughly 21 standard errors. Against full attention at 32.0% it is a tie, not a win.

Where does it fall short?

One retrieval serves a whole query block, so index_topk has to cover everything a block_size span might ask for. Routing is one block stale, so the index cannot react to the token being generated right now. Landmarks are pooled from keys that already carry their rotary phase, so routing is not purely content-addressed. Left padding shifts the block grid and so is not bit-identical to an unpadded run. And there is no fused kernel yet.

Try it yourself

The claims are reproducible in one command.

helix bench counts the attention pairs the forward pass actually forms, and helix recall trains the ablation alongside the full model. Neither takes special hardware — the published numbers came off 4 CPU cores.

View on PyPI Back to all models