Independent AI intelligence Two editions daily · ET
Fervor AI
Menu

Analysis · November 5, 2026 · repo

Generated-Computers/Localslocal-aimulti-agentagent-infrastructure

Locals Runs AI Agents on Your Mac Up to 3x Faster, But Only When You Run Several at Once

A local model server built for the way agents send requests, and a quick way to tell whether your setup will see the speedup.

Ask how fast a local model runs on a Mac and you'll usually get one number: tokens per second, measured with one person typing one prompt. That number describes a chat. It says very little about agents, which rarely ask one question at a time.

A coding agent fires a planner, a few sub-agents and a stream of tool calls, often in parallel. Run three agents on one laptop and the model server is juggling a dozen small requests at once. On that workload, the usual benchmark is measuring the wrong thing.

Locals, an open-source project that showed up on Trendshift's daily board this week, is built around that gap. Its tagline is blunt: "Run many AI agents at once on your Mac, up to 3x faster." The interesting part is the fine print, because the project is unusually honest about when that 3x shows up and when it doesn't.

What Locals is

Locals is a model server for Apple-silicon Macs. A model server is the background program that loads a model into memory and answers requests from apps, the way Ollama or LM Studio do. Locals speaks the same request formats as the OpenAI, Anthropic and Ollama APIs, so most agent tools can point at it without code changes.

It's Apache-2.0 licensed, released v0.2.2 on November 4, and had roughly a thousand GitHub stars on November 5 (counts varied by source that day). It needs an M1 or newer Mac on macOS 14 or later. The README says a 16 GB Mac "gives the best results," and 8 GB machines should stick to the smaller models.

On Macs with at least 12 GB, the default model is Qwen3.5-4B, a 2.8 GB download; smaller Macs default to Qwen3.5-2B. Others include LFM2.5-2.6B, MiniCPM5-2B, LFM3-1.7B and Gemma 4 E4B, plus a larger mixture-of-experts model (one that wakes only a small slice of itself for each word) called Flint that needs 24 GB or more.

The speedup is a curve, not a number

Here is the benchmark that matters, from the README, on an M2 Pro with 16 GB running Qwen3.5-4B. Each pair is tokens per second for plain mlx-lm (the model runner from Apple's MLX team) and then Locals:

  • 1 request at a time: 61 vs 64
  • 2 at once: 82 vs 116
  • 4 at once: 87 vs 160
  • 8 at once: 87 vs 271
  • 16 at once: 157 vs 341

Read it top to bottom and the story is clear. With one request, Locals is about the same as mlx-lm, and the README says so directly: "One request alone, with nothing to reuse: about the same as mlx-lm." The gap opens as requests pile up, peaks at about 3.1x with eight in flight (244 rather than 271 when measured through the server API, which is how your agents will call it), then narrows to about 2.2x at sixteen as plain mlx-lm catches its second wind.

That shape is the whole article. If you use a local model as a chat window, Locals won't feel faster. If you run several agents against one model, it might feel like a different machine.

Why it's faster (in kitchen terms)

Picture a kitchen. Both mlx-lm and Locals already cook several orders at once (this is called continuous batching: new requests join the batch as old ones finish). Both keep the prep cook and the line cook busy at the same time. So the difference isn't in how orders get scheduled.

The difference is in the knife work. Locals' explainer page says the gain "is in how the model is computed for a batch of a few rows." In plainer terms, its GPU code (kernels, the small programs that do the math on the graphics chip) is written specifically for batches of a handful of requests, which is exactly the size agent traffic tends to be. Most model code is tuned either for one request or for big server batches. Agents live in the awkward middle.

The project backs this with an unusual ablation table, which switches off one piece at a time to show what each is worth: turn off the small-batch kernels and Locals at eight requests drops from about 273 to about 119 tokens per second. Most of the win comes from that one piece. A prompt cache, a few math steps merged into one, and a grammar check running on the GPU that keeps tool-call output in valid shape add the rest.

Where it isn't faster

This section is why I trust the project more than most benchmark-led READMEs. Locals lists its own losing cases under a heading called "Where Locals is not faster":

  • One request alone. Same speed as mlx-lm, as shown above.
  • Long prompts. When the work is mostly reading the prompt rather than writing an answer, the gain mostly disappears. The README's example: 8,000-token prompts, eight requests at once, 57 tokens per second.
  • Unusual model formats. The kernels target one common 4-bit format. A model in a different format ran at mlx-lm speed.
  • Newer chips. On M5, the README says the gain is 1.5x, not 3x.
  • One model at a time. "Locals runs one model at a time," so you can't mix a coder model and a chat model on one server.

Tuning ships for M2 Pro and M2 Max. Other chips tune themselves in the background, which the README says takes "10 to 20 minutes of idle time in all," and M1, M3 and M4 aren't benchmarked.

The long-prompt case deserves a second look. Agent tools like Claude Code send large system prompts and growing conversation histories, which is prompt-heavy work. The prompt cache should soften this for repeated prefixes, but the README doesn't publish a Claude Code benchmark, so treat the 3x as a best case for agents with short, frequent calls.

Put this into practice

The quickest way to find out if Locals helps you is to test the one thing that matters: how many requests your agents really send at once.

1. Install it. The README offers a one-line installer or a Python tool install. The second is easier to inspect and remove:

uv tool install --python 3.12 locals

(The other option is curl -LsSf https://raw.githubusercontent.com/Generated-Computers/Locals/main/install.sh | sh. Read the script before piping it to a shell.)

2. Start the server.

locals start

It runs in the background on port 8321, with an OpenAI-style base URL of http://127.0.0.1:8321/v1. The first run downloads the default model, about 3 GB, so expect a wait. After that, the README says everything runs on your Mac and the server only accepts requests from apps on your own computer.

3. Point an agent at it. Locals ships launchers that set up the connection for you:

locals launch claude     # Claude Code, via the Anthropic Messages format
locals launch aider
locals launch list       # see every supported launcher

For editors like Cline, Roo Code, Continue or Zed, pick the "OpenAI Compatible" provider, use the base URL above, type anything as the API key, and choose a model such as locals/qwen3.5-4b. locals launch editor prints those values. From Python:

from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8321/v1", api_key="local")

4. Measure your own concurrency. Run your normal agent workload once on your current server and once on Locals, and watch total tokens per second. If you mostly run one agent with long contexts, expect little change. If you run several agents, or one agent that fans out sub-agents and parallel tool calls, that's the workload the benchmark describes.

If you want a number before wiring up real agents, this short script sends the same small prompt at one, four and eight requests at once and prints combined throughput. Point base_url at whichever server you're comparing:

import time
from concurrent.futures import ThreadPoolExecutor
from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8321/v1", api_key="local")

def one_call(_):
    r = client.chat.completions.create(
        model="locals",
        messages=[{"role": "user", "content": "List five uses for a paperclip."}],
        max_tokens=256,
    )
    return r.usage.completion_tokens

for n in (1, 4, 8):
    start = time.time()
    with ThreadPoolExecutor(n) as pool:
        tokens = sum(pool.map(one_call, range(n)))
    print(n, "at once:", round(tokens / (time.time() - start)), "tok/s")

It's a rough test, not the project's method: short prompts flatter any server, and the first run includes warm-up. Run it twice and read the second pass. If the eight-at-once number isn't well above the one-at-a-time number on your Mac, the curve in the README doesn't apply to you.

5. Choose the model for the job, not the leaderboard. These are small models, a few billion parameters each. They're good at fast, structured, repetitive work like routing, summarizing tool output, or drafting simple edits. They are not a drop-in for a frontier model on hard refactors. A sensible setup is a frontier model for the planner and Locals serving the cheap, parallel helpers.

Honest limitations

The benchmarks are the project's own, mostly on one machine (an M2 Pro with 16 GB). The headline eight-request figure for Qwen3.5-4B (271 tokens per second) is measured in-process (calling the model directly, with no server in between). The project's benchmark notes put the same test through the server API, the way your agents actually call it, at 244. Still a big gain, but quote the lower one, and run your own test before you believe the curve.

The Apache-2.0 license covers Locals itself; each model you download carries its own license, so check it before shipping anything built on one.

"Nothing leaves your computer" is true for requests, but models still download from Hugging Face on first run, and some are big (the README lists 18.8 GB for one of them).

The one-model limit is real for agent setups that want a coder and a general model side by side.

Small models fail in small-model ways. Faster wrong answers are still wrong, and parallel agents can multiply mistakes as fast as they multiply throughput.

And it's young. The package is marked Beta, v0.2.0 shipped on October 21, and v0.2.2 on November 4. Expect rough edges and changing flags.

The real takeaway

The useful idea here outlives this one repo: local model speed is a curve over how many requests are in flight, and agents live further along that curve than chat does. Few model-server READMEs publish the whole curve, losing points included. This one does.

Before you swap servers, find out where your workload sits on that curve. If it's at one, keep what you have. If it's at four or eight, give Locals an afternoon.

Sources: Generated-Computers/Locals README and why-faster explainer, reviewed at v0.2.2 (released November 4, 2026); Locals releases; mlx-lm; star count from GitHub and the shields.io badge, read November 5, 2026; Trendshift daily board, read about 07:20 ET November 5, 2026.