Independent AI intelligence Two editions daily · ET
Fervor AI
Menu

Analysis · October 25, 2026 · repo

deepopen-com/deepopenLayaModernBERTlocal-aifine-tuningagent-infrastructure

DeepOpen Turns LLM Classification Calls Into a Small Model You Train Yourself

The open-source SDK answers "which of these options?" questions with probabilities instead of generated text. Its own benchmark report says where the accuracy really comes from.

A surprising share of the LLM calls in a production app are multiple-choice questions in disguise. Which team should get this ticket? Is this email spam? How urgent is this message, on a scale of one to four? You write a prompt, the model writes back a word, and your code parses that word into one of a handful of options.

That works. It's also a strange way to pick from a list. You pay per token for a model that can write sonnets, wait for it to generate text one piece at a time, and then hope it answers with an option that exists.

DeepOpen, which sat at #25 on Trendshift's daily board at about 15:10 ET on October 25, 2026, takes the other road. It never generates text. You hand it the input and the list of options, and it hands back a probability for each one. The interesting part is that its own documentation is unusually blunt about what that buys you, and what it doesn't.

What DeepOpen actually is

DeepOpen describes itself as "an intent-classification framework and non-autoregressive SDK for structured decisions." Two of those words need unpacking.

Intent classification means sorting a piece of text into one of a fixed set of labels: "refund request," "bug report," "sales question." Non-autoregressive means the model doesn't produce its answer word by word the way ChatGPT does. It reads the whole input once and scores every option at the same time.

Under the hood are encoder models, the reading-only cousins of chat models. An encoder turns text into numbers that capture its meaning, and a small layer on top turns those numbers into scores. DeepOpen's default checkpoints (saved, ready-to-load copies of a trained model) come from an upstream project called Laya: the English one is built on ModernBERT-large, with 421 million parameters, and a multilingual one is built on mmBERT-base. The English checkpoint is about 808 MB on disk. That's small enough to run on a laptop CPU.

The README lists three question types:

  • choice returns a label, a probability for each option, and a confidence score. Department routing is the example.
  • score returns a level on an ordered scale, like urgency.
  • noul returns the probability that a statement is true, like "this is a refund request."

The code is Apache-2.0, and the Laya model card declares Apache-2.0 for the weights too. The README still tells you to "review the separate model and dataset licenses for your intended use," which is fair advice for any fine-tuned model.

What a call looks like

Here is the README's quick start, lightly trimmed. It sorts a support message into one of four departments:

import deepopen

agent = deepopen.load("convaiinnovations/laya", device="cpu")
questions = {
    "department": {
        "type": "choice",
        "instructions": "Which department should handle this request?",
        "criteria": {
            "billing": "invoices, payments, refunds",
            "technical": "bugs, outages, system errors",
            "sales": "pricing, new contracts",
            "other": "everything else",
        },
    }
}
result = agent.predict(
    {"body": "I was charged twice. Please refund the duplicate payment."},
    questions,
)
print(result["answers"]["department"])

Notice what's missing. There's no prompt telling the model to "respond with only one word," and no parsing step for when it ignores you. The output can only be one of your four labels, because those are the only things the model scores.

Speed is the other draw. On an NVIDIA T4, the project measured 32.8 milliseconds for one warm question on the multilingual checkpoint, and 72.3 milliseconds for a batch of ten questions in one call.

Whose numbers these are

The headline results are the project's own, run on public test sets.

On BANKING77, a standard benchmark of 77 banking intents, DeepOpen reports 94.1234% accuracy (2,899 of 3,080 test examples). On CLINC150, a set of 150 assistant-style intents, it reports 98.0222%.

Both come with fine print the README states itself, and it's worth reading. The BANKING77 figure is "the best test result of three SAM seeds, selected after evaluation." (A seed is the random starting point for a training run. Training three times and reporting the best one flatters the result.) The three-run average is 94.0368%, and the README says there's "no statistically established improvement over the replayed baseline." The CLINC150 figure comes from an ensemble of four encoder models, three of them DeBERTa-v3-large, and it covers "150 in-scope intents only, without OOS detection," meaning it was never tested on messages that fit none of the categories. Real traffic is full of those.

Both scores also come from models trained on those datasets. That's the part that matters for you.

The catch, in its own words

The most useful file in the repo is the benchmark report, because it says plainly where the base models fall down.

On the project's own typed-decisions benchmark, the shipped base checkpoints scored 0.362 and 0.342. The majority-class baseline, which just guesses the most common answer every time, scored 0.461. The report's verdict: "The base checkpoints sit below the majority-class baseline... All of the capability on this benchmark comes from fine-tuning." The 0.766 accuracy listed elsewhere belongs to a checkpoint fine-tuned on that benchmark's own training split.

Moderation is weaker still. On held-out toxic-chat data (examples kept out of training, so the test is fair) the report measures 0.530 accuracy with a macro-F1 of 0.400. Macro-F1 averages how well it does on each class, so it punishes a model that ignores the rare one. The report calls the result "barely above chance on a balanced split" and adds: "hand-picked examples work, real traffic does not." Spam and phishing detection score 0.993, but the report notes both "were in the training mix." It also warns that the raw results file behind these application numbers isn't in the repo, so you can't audit them from the checked-in files.

Then there's confidence. The report says both base checkpoints "ship over-confident," and that the multilingual one "ships with no fitted temperatures at all." Its recommended fix is to refit one temperature per question type and option count on held-out data, which it calls "the single highest-value fix available." (Temperature fitting is a small adjustment that makes "90% confident" actually mean right about 90% of the time.)

The upstream model card sums it up in one sentence: "Laya is a fast base to specialise, not a zero-shot decision engine."

The real shift: from prompting to labeling

So here's the honest pitch. DeepOpen doesn't let you delete your classifier prompt and get the same accuracy for free. It moves the work.

With an LLM, the cost lives in every call: tokens, latency, and the occasional answer that isn't one of your options. With DeepOpen, the cost moves up front, into a labeled dataset, a fine-tuning run, and a held-out split for calibration. After that, each decision costs milliseconds on hardware you already pay for.

That trade is good when the question is fixed and the volume is high: routing thousands of tickets a day, tagging every inbound email, scoring every message in a queue. It's bad when the categories change weekly, when you have no labeled examples, or when the judgment needs reasoning across a long document. A frontier model handles those without training. DeepOpen won't.

Put this into practice

Treat DeepOpen as a candidate to beat your current LLM call on one specific question, not as a drop-in replacement. A sensible first afternoon looks like this.

  1. Install it on CPU. The README's CPU path is two commands, run in a fresh virtualenv with Python 3.10 or newer:

    python -m pip install torch --index-url https://download.pytorch.org/whl/cpu
python -m pip install "git+https://github.com/deepopen-com/deepopen.git"

    There's no PyPI release yet, so you're installing from the repo's main branch (version 0.3.4 as of an October 23 commit, with no tagged releases). Pin a commit if you plan to keep it. The first load call downloads the checkpoint, roughly 808 MB for the English model.

  2. Pick one fixed-choice question you already pay an LLM to answer. Ticket routing is the classic. Pull 200 to 500 real examples from your logs along with the label your team agrees is correct.

  3. Run the base model on them first. Expect it to disappoint; the report says it might not beat guessing the most common label. That number is your floor, and it tells you whether your labels are learnable at all.

  4. Fine-tune if the volume justifies it. The repo includes a typed-decisions fine-tuning notebook written for two T4 GPUs on Kaggle, plus Banking77 and CLINC150 reproduction guides (the latter two are in Chinese).

  5. Calibrate on data the model never trained on, then compare against your LLM on a held-out set: accuracy, the share of answers above your confidence cutoff, and cost per thousand decisions. Switch only if the small model wins on your traffic.

Keep the LLM as a fallback for low-confidence cases. That combination, a fast classifier for the easy majority and a big model for the leftovers, is often where the savings actually show up.

Honest limitations

Here's what to know before you commit.

  • It's young. No tagged releases, no PyPI package, and the version moved to 0.3.4 only days ago. APIs may change.
  • Zero-shot is weak. The project's own numbers put the base checkpoints below a majority-class baseline on its typed-decisions task. Budget for labeling and training.
  • The benchmarks are self-run and selected. The BANKING77 headline is the best of three seeds; the CLINC150 headline is a four-model ensemble that ignores out-of-scope messages.
  • Moderation isn't ready. 0.530 on held-out toxic chat is close to a coin flip. Don't put it in front of a safety decision.
  • Confidence needs fixing before you trust it. Both base checkpoints ship over-confident, so a threshold like "auto-route above 0.9" means little until you calibrate.
  • Some outputs are thin. The model card calls ordinal score questions "the weakest primitive" and says noul "can follow its option labels instead of the state."
  • Big option lists break it. Options share a fixed token budget, and the report advises keeping choice questions under about 20 options. Inputs are short too: the English checkpoint's default context is 512 tokens, the others 1,024.
  • Languages aren't equal. The English root checkpoint is English only; the multilingual checkpoint beat three times random accuracy on 45 of 51 tested languages, which still leaves gaps.

Where this leaves you

The useful question isn't whether a 421-million-parameter encoder is smarter than a frontier model. It isn't. The question is whether your app keeps asking a frontier model to pick from the same short list, thousands of times a day. If it does, DeepOpen gives you a way to find out, with your own data, whether a small trained model can take most of that load. If the answer on your traffic is no, you've spent an afternoon and learned your classifier prompt is earning its cost.

Sources: deepopen-com/deepopen README and BENCHMARKS.md at commit 6e8f2a6 (October 23, 2026); Laya model card on Hugging Face; Trendshift daily board, read about 15:10 ET on October 25, 2026. Star count about 3.1k per a cache-busted shields reading on October 25.