Benchmark v0.6.0 · Seven free models on Nous Portal
This report covers one pool of models: Nous Portal free models, benchmark v0.6.0. We ran 7 of them on the 18 tasks behind StaffBench's 9 roles, three runs each, every answer checked automatically. The most reliable passed 12 of 18 tasks on every run.
A full pass means the answer passed every automatic check with no serious mistake. The bars show how far each score could move with a different draw of tasks. With 18 tasks, that is a lot.
Each square is one model on one task. The dots are its three runs. Read across for a model, down for a task. Every task was passed at least once by some model.
| Chief of Staff | CFO | Engineer | Travel | Coach | Research | Systems | Product | Design | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Task number | 01 | 02 | 01 | 02 | 01 | 02 | 01 | 02 | 01 | 02 | 01 | 02 | 01 | 02 | 01 | 02 | 01 | 02 | Every run |
| Beam 1.0 501B | 2 of 3 | 3 of 3 | 3 of 3 | 3 of 3 | 3 of 3 | 2 of 3 | 3 of 3 | 3 of 3 | 2 of 3 | 3 of 3 | 3 of 3 | 3 of 3 | 1 of 3 | 1 of 3 | 3 of 3 | 3 of 3 | 3 of 3 | 0 of 3 | 12/18 |
| Ornith 1.5 122B | 3 of 3 | 3 of 3 | 3 of 3 | 1 of 3 | 1 of 3 | 3 of 3 | 3 of 3 | 3 of 3 | 2 of 3 | 2 of 3 | 2 of 3 | 3 of 3 | 2 of 3 | 3 of 3 | 3 of 3 | 3 of 3 | 2 of 3 | 0 of 3 | 10/18 |
| Step 3.7 Flash | 1 of 3 | 3 of 3 | 3 of 3 | 2 of 3 | 3 of 3 | 3 of 3 | 3 of 3 | 2 of 3 | 2 of 3 | 3 of 3 | 3 of 3 | 3 of 3 | 0 of 3 | 1 of 3 | 3 of 3 | 3 of 3 | 3 of 3 | 0 of 3 | 11/18 |
| Solar Mini4 | 2 of 3 | 3 of 3 | 2 of 3 | 1 of 3 | 2 of 3 | 3 of 3 | 3 of 3 | 2 of 3 | 3 of 3 | 3 of 3 | 3 of 3 | 2 of 3 | 2 of 3 | 2 of 3 | 3 of 3 | 2 of 3 | 3 of 3 | 0 of 3 | 8/18 |
| LongCat 2.5 Preview | 1 of 3 | 3 of 3 | 3 of 3 | 3 of 3 | 0 of 3 | 3 of 3 | 3 of 3 | 2 of 3 | 1 of 3 | 3 of 3 | 3 of 3 | 3 of 3 | 0 of 3 | 1 of 3 | 3 of 3 | 2 of 3 | 3 of 3 | 1 of 3 | 10/18 |
| Laguna S 2.1 | 0 of 3 | 3 of 3 | 3 of 3 | 1 of 3 | 2 of 3 | 3 of 3 | 3 of 3 | 3 of 3 | 1 of 3 | 1 of 3 | 3 of 3 | 2 of 3 | 2 of 3 | 1 of 3 | 3 of 3 | 3 of 3 | 3 of 3 | 0 of 3 | 9/18 |
| Laguna XS 2.1 | 1 of 3 | 3 of 3 | 1 of 3 | 1 of 3 | 3 of 3 | 3 of 3 | 3 of 3 | 3 of 3 | 0 of 3 | 2 of 3 | 3 of 3 | 1 of 3 | 1 of 3 | 0 of 3 | 1 of 3 | 2 of 3 | 3 of 3 | 0 of 3 | 7/18 |
| All models | 10/21 | 21/21 | 18/21 | 12/21 | 14/21 | 20/21 | 21/21 | 18/21 | 11/21 | 17/21 | 20/21 | 17/21 | 8/21 | 9/21 | 19/21 | 18/21 | 20/21 | 1/21 | |
passed runfailed run no scored runno model passed any run
An agent runs the same kind of job again and again, so a model that passes only sometimes is hard to trust. This counts tasks a model passed on every run against tasks it passed at least once.
Up is better, left is faster. The line joins models that no faster model beats on score. Beam 1.0 501B is alone on it: no model is both faster and better.
Each role has two tasks, so each bar rests on just six runs. Treat these as leads to check, not settings to apply. Every role has a full brief: what it gets, what it must return, and the traps it sets.
Your daily driver. Turns messy requests into a clear plan, keeps every requirement intact, and hands work to the right specialist.
COS-01 Turn a messy request into a dependency-aware planCOS-02 Reconcile three conflicting agent reports into one handoffOrnith 1.5 122B · 100%
Read the role brief →Models money decisions with stated assumptions, honest risk and no false certainty. Analysis, never advice or action.
FIN-01 Review a portfolio against fixed goals and limitsFIN-02 Compare two investment strategies under fixed scenariosBeam 1.0 501B and 1 more tied · 100%
Read the role brief →Ships and reviews code to a senior standard: correct, secure, tightly scoped and honestly verified.
ENG-01 Fix an authentication bug in a small repositoryENG-02 Security-review an extension install manifestStep 3.7 Flash and 1 more tied · 100%
Read the role brief →Plans trips and expeditions end to end: gear, routes, transfers, permits, weather, and when to turn back.
TRV-01 Pick a backpacking cook kit from a fixed catalogueTRV-02 Build a multi-day route and logistics planBeam 1.0 501B and 3 more tied · 100%
Read the role brief →Builds training that fits real life and adapts it when sleep, soreness or schedule changes. Coaching, not diagnosis.
FIT-01 Write a training plan from a baseline and goal dateFIT-02 Adapt a training plan after a weekly check-inSolar Mini4 · 100%
Read the role brief →Answers questions from sources you can check, with every claim traced and the uncertainty stated.
RES-01 Extract trip rules from a bundle of official sourcesRES-02 Choose a local AI model build that fits a 24 GB memory budgetBeam 1.0 501B and 2 more tied · 100%
Read the role brief →Watches the automations and services you rely on, tells stale from broken, and fixes things in a safe order.
OPS-01 Classify a data-collector incident from logs and health checksOPS-02 Diagnose source freshness and workflow drift in a scheduled report runOrnith 1.5 122B · 83%
Read the role brief →Turns rough ideas into a focused, measurable product and launch, without scope creep.
PRD-01 Turn a product brief into a measurable MVPPRD-02 Set launch and landing-page direction from a positioning briefBeam 1.0 501B and 3 more tied · 100%
Read the role brief →Critiques and specifies the experiences you ship: hierarchy, states, accessibility and clarity.
DES-01 Critique an HTML report with seeded design defectsDES-02 Specify interactions for a dashboard's loading, empty, stale and error statesLongCat 2.5 Preview · 67%
Read the role brief →Hard failure: a serious mistake, such as ignoring a hard limit or claiming an unsafe action was done. Valid JSON: answers in the required structured text format, with named fields. Provisional: at least one task has fewer scored runs than planned. Median time: the middle response time when sorted. Output tokens: the pieces of text in an answer.
| Rank | Model | Pass rate | 95% range | Vs leader | Status | Scored runs | Checks passed | Hard failures | Valid JSON | Median time | Output tokens |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Beam 1.0 501B | 81.5% | 67–94% | tied for first | confirmed | 54/54 | 96.1% | 7.4% | 100.0% | 6.7s | 1,064 |
| 2 | Ornith 1.5 122B | 77.8% | 65–91% | tied for first | confirmed | 54/54 | 91.1% | 11.1% | 92.6% | 25.4s | 1,238 |
| 3 | Step 3.7 Flash | 75.9% | 59–91% | tied for first | confirmed | 54/54 | 95.6% | 9.3% | 100.0% | 12.9s | 2,324 |
| 4 | Solar Mini4 | 75.9% | 63–87% | tied for first | confirmed | 54/54 | 94.1% | 9.3% | 98.1% | 25.2s | 3,720 |
| 5 | LongCat 2.5 Preview | 70.4% | 54–87% | tied for first | confirmed | 54/54 | 88.5% | 18.5% | 92.6% | 20.9s | 1,617 |
| 6 | Laguna S 2.1 | 68.5% | 52–83% | behind | confirmed | 54/54 | 90.4% | 11.1% | 96.3% | 16.4s | 1,510 |
| 7 | Laguna XS 2.1 | 57.4% | 41–74% | behind | confirmed | 54/54 | 83.7% | 9.3% | 92.6% | 11.1s | 1,137 |
Every model got the same tasks, the same test setup and the medium reasoning setting. Test setup: Hermes Agent, with tools, memory and automatic switching to another model turned off. Models ran on 23 Oct 2026.
sha256:3bd0dcc5a98ca7d5dfaba2016fe5b2c8a6a62e2d19c19f974d197cbfe8fac698