Needle 2 Is a 45M-Parameter Model That Can Only Call Tools

A 14MB binary that runs a full session in 28MB of RAM and hits 500 tokens per second on a Raspberry Pi 5. It cannot hold a conversation. That is the design, and the benchmark table has one number worth arguing about.

Needle 2 has 45 million parameters. Not 45 billion. The entire model ships as a single 14MB binary that runs a full session in roughly 28MB of RAM, which puts it inside an ESP32-S3 microcontroller, and it decodes at over 500 tokens per second on a Raspberry Pi 5.

Ask it a question and it will not answer. It has exactly two possible outputs: a structured function call, or an empty list meaning no declared tool can serve the request. There is no free-text fallback. The README states it as a contract: "Needle solves every problem as a function call."

Most of the on-device AI conversation for the past two years has been about how much world knowledge you can compress onto a phone. Cactus Compute's bet points the other way, and the argument is sharper than the compression numbers.

The bet: turning on a light needs no world knowledge

Here is the reframe, in Cactus's words from the Needle 2 launch page: "Smartwatches, home assistants and robots already expose their abilities as functions with typed parameters, so the only hard part is mapping a messy sentence onto them: which function, and which arguments. Framed that way, the problem needs no world knowledge and no open-ended prose. That is why 45M parameters suffice whereas chat requires billions."

I think that is correct, and I think most local-model work has been solving the wrong problem for device control. If your smart speaker needs to know who wrote Middlemarch in order to dim the kitchen lights, you have paid for the wrong model.

The market math behind it is worth a look too. Cactus counts more than 21 billion IoT devices against roughly 1.5 billion PCs, and estimates that about four in five edge devices cost under $200. "Edge AI has lately meant Macs and PCs," they write. This targets hardware with no GPU, no NPU, and a few dozen megabytes of RAM.

The mechanism, which is more interesting than the size

Needle 2 is not a shrunken transformer. Cactus calls it a Simple Attention Network and points at arXiv:2607.18363 for the design and ablations. Three choices carry most of the weight.

The Hadamard MLP replaces the usual dense up-and-down projections with a fixed orthonormal Walsh-Hadamard transform plus learned diagonals. The transform is a fixed matrix applied in n log n time with no weights to read at all, so the channel mixing that normally dominates a small model's memory traffic costs almost nothing.

The engram moves world knowledge out of the network and into hashed n-gram tables that get read a few rows per token. Eight million of Needle's 45 million parameters live there, retrieved by gather rather than exercised through a matmul, which means they cost zero arithmetic at decode time.

The bounded window. Attention runs a 256-token sliding window, and the tool declarations are pinned as permanent KV sinks. The thing a tool-calling model must never forget is structurally unable to be evicted, and RAM stays at a deterministic 28MB ceiling no matter how long a session runs. That last property is what puts microcontrollers in reach, because a curve that grows with conversation length is unshippable on an ESP32-P4.

Add it up and Cactus reports 70 MFLOPs per token, against 460 for LFM2.5 230M, 540 for FunctionGemma 270M, and roughly 6,000 for Apple FM. The table also carries two synthetic rows, and the second one is the honest bar: a same-shape dense-MLP transformer costs 164, but that model has 82M parameters, nearly double. A transformer at Needle's matched parameter count costs 87. So the architecture buys about 1.2x against a fair-sized transformer, not 2.3x. Cactus published that row itself, which is the sort of thing that makes me believe the rest of the table. On device silicon, moving a byte out of flash costs orders of magnitude more than a multiply-accumulate, so the real budget is FLOPs and bytes together. Both go down here.

The quantization deserves its own note, because the usual story is a lie. Small models fall apart under post-hoc quantization. Cactus never quantizes post-hoc: weights, activations, and the KV cache all train against Cactus Quants from pretraining through post-training. The CQ2-bit model you deploy is the model that was trained. That is a meaningfully different claim from "we squeezed it to 2 bits afterward and it mostly held."

Read the benchmark tables, not the summary

The launch page says Needle 2 "trades wins" with FunctionGemma 270M, LFM2.5 230M, and Apple FM. That is accurate, and it is doing some work. The actual pattern:

On Google's Mobile Actions (961 rows, ordered strict exact match) Needle 2 lands third at 63.7, behind LFM2.5 at 69.1 and FunctionGemma at 64.0, ahead of Apple FM at 57.6. Its function-name accuracy is best in class at 98.3, so it picks the right tool and loses points on arguments.

On the DroidCall test split (200 rows of Android intent-style calls) it lands second of three at 17.0, a half point behind FunctionGemma's 17.5 and ahead of LFM2.5's 11.0. Every model scores 0.0 on the 2-call rows, so that table says more about DroidCall than about any of them.

On both Seal-Tools splits it wins, against a three-model field with no Apple FM row. In-domain: 32.6 against LFM2.5's 26.9 and FunctionGemma's 16.3, on large candidate tool lists with a majority of multi-call rows. Out-of-domain, where entire tool domains are held out of training to test schema generalization: 28.7 against 17.0 and 15.6. Those are the hardest shapes on the page and the widest margins.

On BFCL v4 single-turn (3,641 rows) it finishes last overall at 42.6, against 61.7 for Apple FM and 60.8 for LFM2.5. Cactus explains why: BFCL's Java, JavaScript, and enterprise API surfaces sit entirely outside a training corpus built on consumer device actions. On Python simple calls it lands within a point of FunctionGemma, 61.2 against 62.3.

Cactus names both asymmetries in the evaluation section itself, unprompted, and says there is no clean way to level them at once. Precision skews toward the baselines, which run f16 against Needle's 2 bits. Scope skews toward Needle, which trains for agentic tool calling and nothing else while every baseline carries chat, prose, and world knowledge. Their words: "We accept the skew; it still paints the picture we intend." Publishing that paragraph directly above your own benchmark tables is rarer than it should be, and it is the reason I trust the numbers in them.

The number I would argue about

The README's refusal contract is unambiguous: "A request no declared tool can serve is refused with the empty call []. That is the whole contract for off-topic input; there is no free-text fallback."

BFCL has a category that measures exactly that behavior. It is called Irrelevance, and it scores whether a model correctly declines to call anything. Needle 2 scores 60.8. LFM2.5 230M scores 77.7. FunctionGemma 270M scores 72.1. Apple FM scores 28.3.

So the model whose entire safety story rests on refusing cleanly is third of four on the public measure of refusing cleanly. That is not disqualifying, and BFCL's irrelevance rows skew toward the general-purpose APIs Needle was never trained on. But if you are gating real actions on confidence, the escalation threshold is not a tuning detail. It is the control. The README describes confidence as "the minimum of two signals: a calibrated post-hoc head that scores the full prompt plus the call the model just produced, and the decoding probability of the call tokens," accepted "only when both agree," so the intended failure mode is escalation rather than wrong execution. Set your threshold high, log everything below it, and go look at what falls in the gap before you ship.

Put this into practice

The first useful run takes about ten minutes and needs no GPU.

1. pip install cactus-needle. The inference engine downloads once from Hugging Face and caches. Nothing to compile.
2. Decorate a function with @needle.tool. The signature gives argument types, the docstring becomes the tool description. Needle reads your descriptions to decide what to call, so writing them well is most of the work.
3. Call agent.run("...") to let Needle pick the call, execute your function, feed the result back, and return a final answer. Use agent.complete() instead if you want to drive the loop yourself and execute calls under your own controls.
4. Constrain the arguments. needle.Field supports ranges, patterns, lengths, enums, and item counts, and every one of them compiles into the decode grammar. A Field(gt=0, le=10000) on a payment amount is not a validation hint, it is a token the model cannot emit.
5. Read confidence on every response and pick a threshold for your product. Act at or above it, re-ask or escalate to a bigger model below it.
6. Declare more than five tools and retrieval engages automatically: only the five highest-scoring tools enter context each turn. Pass tool_index_path="tools.idx" to persist the embeddings between runs.
7. Try needle playground for a browser UI to poke at your schemas before you wire anything to real hardware.

For structured extraction, declare the record as the only tool and call extract() with a Pydantic model. With one declared tool the grammar admits exactly one call, so schema conformance is guaranteed rather than requested. Fine-tuning runs LoRA on the frozen base and merges at export, producing a single .cact file the same engine runs without recompiling, in minutes to a few hours on a Mac or PC.

Honest limitations

The license is contradictory across primary sources. The GitHub repo displays MIT. The Cactus launch page says "Needle is Apache 2.0-licensed." Those are both permissive and both fine, but if licensing matters to your legal review, get it in writing before you ship.

Five tools per turn is a hard ceiling, not a soft preference. Above five declared tools, retrieval selects five and rebuilds the grammar over just that subset. The README's phrasing is exact: "An unselected tool is unreachable, not merely unlikely." If your product needs a rare tool that never scores in the top five for the phrasings users actually type, the model will never reach it, and no amount of prompting fixes that.

The 256-token window evicts. Benchmarks ran with the window on and eviction included, which is the right way to benchmark, and it also means long chained interactions lose their middle.

Parallel multi-call is weak. BFCL parallel multiple: 22.5, against 51.5 for LFM2.5. Sequential dependent calls work well, and the README shows the pattern. Fanning out does not.

The offline story has a caveat at the edges. Inference does no network, which is the claim that matters, and the engine still fetches from Hugging Face on first use. The optional data synthesis step for fine-tuning requires an OPENROUTER_API_KEY and calls a cloud API, which is a curious dependency inside an offline-first toolchain. Skip it and bring your own JSONL.

The training corpus is consumer device actions. Smart home, mobile, wearables, TV, car, plus structured extraction. Point it at an enterprise API catalog and you are outside the distribution, which the Seal-Tools out-of-domain numbers suggest it handles better than its peers while still landing at 28.7.

What I would try next

Pebble runs this in production, locally in the Index 01 app. Founder Eric Migicovsky's quote on the launch page is about the Index Ring, which has no screen, so "when you speak to it, the action just has to happen, every time, with or without internet connection." That is the exact shape this model was built for, and it is a more convincing endorsement than any benchmark row.

The open question is whether the split holds as products get greedy. Needle's whole argument is that routing and knowing are separable jobs, and the moment someone asks their light switch a question the switch cannot answer, the confidence gate has to hand off cleanly to something bigger. Cactus calls that edge-cloud collaboration and treats escalation as rare. I would instrument that rate on day one, because the number of requests that fall below your threshold is the honest measure of whether a 45M-parameter model is enough for what you actually built.

Sources: cactus-compute/needle on GitHub; Needle 2 launch page; the paper Cactus cites for the Simple Attention Network, arXiv:2607.18363; weights on Hugging Face.