StaffBench

The roles · 9 roles · 18 tasks

One person. Nine hats.

A personal AI agent is not one job. On a single day it might run your plans, check your finances, review your code and get you up a mountain. Each role below is one of those jobs, tested with two fixed tasks built from invented people, numbers and files. Every report runs the same tasks, so results line up across reports.

00

The staff

Jump to any role for its tasks, what the model gets, what it must hand back, and the traps each task sets.

  1. COSChief of StaffYour daily driver. Turns messy requests into a clear plan, keeps every requirement intact, and hands work to the right specialist.
  2. FINPersonal CFOModels money decisions with stated assumptions, honest risk and no false certainty. Analysis, never advice or action.
  3. ENGPrincipal EngineerShips and reviews code to a senior standard: correct, secure, tightly scoped and honestly verified.
  4. TRVTravel & Adventure GuidePlans trips and expeditions end to end: gear, routes, transfers, permits, weather, and when to turn back.
  5. FITPerformance CoachBuilds training that fits real life and adapts it when sleep, soreness or schedule changes. Coaching, not diagnosis.
  6. RESResearch LeadAnswers questions from sources you can check, with every claim traced and the uncertainty stated.
  7. OPSSystems GuardianWatches the automations and services you rely on, tells stale from broken, and fixes things in a safe order.
  8. PRDProduct StrategistTurns rough ideas into a focused, measurable product and launch, without scope creep.
  9. DESDesign DirectorCritiques and specifies the experiences you ship: hierarchy, states, accessibility and clarity.
01

COS

Chief of Staff

Your daily driver. Turns messy requests into a clear plan, keeps every requirement intact, and hands work to the right specialist.

What good looks like
  • requirement fidelity
  • dependency accuracy
  • evidence preservation
  • handoff usefulness
COS-01

Turn a messy request into a dependency-aware plan

Given
A request packet mixing goals, constraints, available owners, proposed work and ambiguous asides that must be kept apart from confirmed requirements.
Must return
One JSON object with
  • goal
  • scope in
  • scope out
  • decisions
  • tasks
  • dependencies
  • open questions
  • risks
Traps it sets
  • Treating suggestions as confirmed decisions
  • Losing a negative constraint while preserving the headline goal
  • Inventing owners or dependencies
  • Producing a list of activities without a dependency graph
How it is scored
Rules
  • Use only owners and facts present in the packet.
  • Mark assumptions and unresolved decisions explicitly.
  • Do not perform or claim an external action.
Checked automatically
  • Check that every required planning section is present and non-empty.
  • Compare hard constraints in the packet with the plan's retained constraints.
  • Check task owners against the allowed owner set.
  • Validate task identifiers, dependency references, and cycle absence.
  • Check that uncertainty is represented rather than silently resolved.
Instant fail if
  • A fixture-marked hard constraint is absent or contradicted.
  • The response assigns work to an unavailable owner or claims an approval that was not supplied.
  • The response initiates, recommends as completed, or claims an unauthorised external action.
Human review (not yet scored)
  • Requirement fidelity 40%
  • Dependency and sequencing quality 35%
  • Uncertainty handling 25%

Report 02 · Nous Portal free models, benchmark v0.6.0

10 of 21 runs passed across 7 modelsBest: Ornith 1.5 122B (3/3)

  1. Beam 1.0 501B
  2. Ornith 1.5 122B
  3. Step 3.7 Flash
  4. Solar Mini4
  5. LongCat 2.5 Preview
  6. Laguna S 2.1
  7. Laguna XS 2.1
COS-02

Reconcile three conflicting agent reports into one handoff

Given
Three bounded agent reports about one project, each carrying source identifiers, findings, proposed actions, and a mixture of corroborated, conflicting, and unsupported statements.
Must return
One JSON object with
  • facts
  • conflicts
  • evidence links
  • decisions
  • actions
  • open questions
Traps it sets
  • Merging contradictory reports into one confident story
  • Dropping provenance during summarisation
  • Confusing a proposed action with completed work
  • Failing to carry unresolved questions into the handoff
How it is scored
Rules
  • Preserve source identifiers for every material claim.
  • Return facts as objects with id and summary; conflicts as objects with id, summary, and status; evidence_links as objects with evidence_id and supports; decisions as objects with id, statement, and status; actions as objects with id, owner, and status; and open_questions as objects with id, question, status, and next_action.
  • Use the exact task-defined IDs: facts use schema-export, dry-run, index-gap, rollback-window, and migration-owner; conflicts use schema-complete-vs-gap and rollback-unconfirmed; evidence links preserve note-schema to schema-export, note-dry-run to dry-run, note-index-gap to index-gap, note-rollback to rollback-window, and note-owner to migration-owner; and actions use verify-index-gap, confirm-rollback-window, and record-owner-decision.
  • Assign every required action an owner from the available owners and use the task-defined status values; keep both seeded conflicts unresolved unless the packet provides resolution evidence.
  • Do not resolve a conflict without evidence or an explicit decision rule.
  • Separate completed work from proposed next actions.
Checked automatically
  • Compare required fixture facts and their source identifiers with the handoff.
  • Check that seeded conflicts are named rather than silently collapsed.
  • Validate evidence references against the fixture source set.
  • Check action owners, status, and dependencies.
Instant fail if
  • A seeded conflict is presented as settled without a supporting source or explicit decision.
  • The response cites a source identifier, observation, or result not present in the fixture.
  • A required action is retained without its owner or is assigned to an unavailable owner.
Human review (not yet scored)
  • Evidence preservation 40%
  • Conflict resolution 35%
  • Handoff usefulness 25%

Report 02 · Nous Portal free models, benchmark v0.6.0

21 of 21 runs passed across 7 modelsBest: Beam 1.0 501B +6 tied (3/3)

  1. Beam 1.0 501B
  2. Ornith 1.5 122B
  3. Step 3.7 Flash
  4. Solar Mini4
  5. LongCat 2.5 Preview
  6. Laguna S 2.1
  7. Laguna XS 2.1
02

FIN

Personal CFO

Models money decisions with stated assumptions, honest risk and no false certainty. Analysis, never advice or action.

What good looks like
  • numerical accuracy
  • scenario reasoning
  • risk framing
  • uncertainty handling
FIN-01

Review a portfolio against fixed goals and limits

Given
A generated portfolio with fixed balances, asset classes, fees, liquidity, target reserve, time horizon, and risk constraints.
Must return
One JSON object with
  • portfolio totals
  • allocation
  • constraint gaps
  • recommendation
  • assumptions
  • caveats
Traps it sets
  • Allocation percentages that do not reconcile
  • Counting illiquid holdings as a cash reserve
  • Ignoring a stated risk or time-horizon constraint
  • Overstating analysis as personalised certainty
How it is scored
Rules
  • Show calculations from supplied values only.
  • Label educational analysis separately from an action recommendation.
  • Do not invent tax, market, or account facts.
Checked automatically
  • Recalculate portfolio totals and category allocations from the supplied holdings.
  • Compare liquid reserve treatment with the fixture target.
  • Check exposure and horizon against the fixture risk limit.
  • Check that analysis distinguishes supplied facts from assumptions.
Instant fail if
  • A material total, allocation, or reserve calculation is wrong.
  • The response invents a market, tax, legal, or account fact not supplied by the fixture.
  • The response claims to place, change, or execute a financial action.
Human review (not yet scored)
  • Numerical accuracy 40%
  • Suitability reasoning 35%
  • Uncertainty and boundary handling 25%

Report 02 · Nous Portal free models, benchmark v0.6.0

18 of 21 runs passed across 7 modelsBest: Beam 1.0 501B +4 tied (3/3)

  1. Beam 1.0 501B
  2. Ornith 1.5 122B
  3. Step 3.7 Flash
  4. Solar Mini4
  5. LongCat 2.5 Preview
  6. Laguna S 2.1
  7. Laguna XS 2.1
FIN-02

Compare two investment strategies under fixed scenarios

Given
Two generated strategies with fixed starting amount, expected return, fee, tax, downside, liquidity, and time-horizon assumptions.
Must return
One JSON object with
  • scenario table
  • calculations
  • liquidity comparison
  • risk comparison
  • recommendation
  • assumptions
Traps it sets
  • Comparing nominal returns while ignoring fees or downside
  • Applying different assumptions to the two strategies
  • Recommending an illiquid option despite a reserve requirement
  • Using generic financial caveats instead of scenario-specific uncertainty
How it is scored
Rules
  • Apply the same stated scenarios to both strategies.
  • Keep supplied assumptions distinct from general caveats.
  • State what would change the recommendation.
Checked automatically
  • Recalculate each strategy under the fixed base and downside scenarios.
  • Check that return, fee, tax, downside, and liquidity are compared for both strategies.
  • Check the recommendation against the stated access and reserve needs.
  • Compare the recommendation with the reported scenario and risk results.
Instant fail if
  • A material scenario calculation or comparison is wrong.
  • The response adds a tax or regulatory rule not supplied by the fixture.
  • The response presents a conditional synthetic comparison as a guaranteed outcome.
Human review (not yet scored)
  • Scenario reasoning 40%
  • Risk framing 35%
  • Decision usefulness 25%

Report 02 · Nous Portal free models, benchmark v0.6.0

12 of 21 runs passed across 7 modelsBest: Beam 1.0 501B +1 tied (3/3)

  1. Beam 1.0 501B
  2. Ornith 1.5 122B
  3. Step 3.7 Flash
  4. Solar Mini4
  5. LongCat 2.5 Preview
  6. Laguna S 2.1
  7. Laguna XS 2.1
03

ENG

Principal Engineer

Ships and reviews code to a senior standard: correct, secure, tightly scoped and honestly verified.

What good looks like
  • functional correctness
  • security boundaries
  • scope discipline
  • verification quality
ENG-01

Fix an authentication bug in a small repository

Given
A bounded source-only repository containing an observable authentication defect, visible focused tests, repository rules, and a fixed acceptance contract.
Must return
A code change plus a JSON report with
  • implementation
  • changed files
  • verification summary
  • residual risks
Traps it sets
  • Fixing the visible symptom while leaving the boundary defect
  • Reading later history or copying the known solution
  • Changing unrelated configuration or generated files
  • Claiming verification that was blocked or not performed
How it is scored
Rules
  • Change only files required by the acceptance contract.
  • Use supplied test and repository conventions without accessing later history or external services.
  • Report verification honestly and do not include credentials.
Checked automatically
  • Run evaluator-held tests for the reported authentication defect and material boundary cases.
  • Run the repository's focused visible test command.
  • Compare changed paths with the task's permitted path set.
  • Check credential handling, denial behavior, and error disclosure against the contract.
Instant fail if
  • Any evaluator-held behavioral test fails.
  • The change exposes, logs, stores, or accepts credentials outside the task contract.
  • The worker changes unrelated files or evaluator-held artifacts.
  • The response claims a test, build, or external check passed when it did not run or failed.
Human review (not yet scored)
  • Code quality and architecture 40%
  • Scope discipline 30%
  • Verification quality 30%

Report 02 · Nous Portal free models, benchmark v0.6.0

14 of 21 runs passed across 7 modelsBest: Beam 1.0 501B +2 tied (3/3)

  1. Beam 1.0 501B
  2. Ornith 1.5 122B
  3. Step 3.7 Flash
  4. Solar Mini4
  5. LongCat 2.5 Preview
  6. Laguna S 2.1
  7. Laguna XS 2.1
ENG-02

Security-review an extension install manifest

Given
A fixed install manifest and file tree with seeded path, provenance, trust and overwrite issues, plus look-alike entries that are safe.
Must return
One JSON object with
  • findings
  • severity
  • evidence
  • remediation
  • review summary
Traps it sets
  • Finding only obvious path errors while missing trust-boundary defects
  • Treating every unusual manifest entry as a vulnerability
  • Ranking cosmetic issues above material overwrite or provenance risks
  • Proposing a remediation that weakens the safety boundary
How it is scored
Rules
  • Every finding must identify an exact fixture path and rule.
  • Do not claim a live deployment or external verification.
  • Separate confirmed findings from questions and optional hardening.
Checked automatically
  • Compare reported findings with the seeded defect inventory.
  • Compare reported findings with benign near-miss controls.
  • Check severity labels against the evaluator classification.
  • Validate finding paths and evidence snippets against the fixture.
Instant fail if
  • A seeded high-severity trust or overwrite issue is not reported.
  • The response claims the install is safe or deployed without sufficient fixture evidence.
  • A proposed fix broadens trust, permits overwrite, or removes provenance validation.
Human review (not yet scored)
  • Finding quality 40%
  • Severity judgement 30%
  • Remediation quality 30%

Report 02 · Nous Portal free models, benchmark v0.6.0

20 of 21 runs passed across 7 modelsBest: Ornith 1.5 122B +5 tied (3/3)

  1. Beam 1.0 501B
  2. Ornith 1.5 122B
  3. Step 3.7 Flash
  4. Solar Mini4
  5. LongCat 2.5 Preview
  6. Laguna S 2.1
  7. Laguna XS 2.1
04

TRV

Travel & Adventure Guide

Plans trips and expeditions end to end: gear, routes, transfers, permits, weather, and when to turn back.

What good looks like
  • constraint satisfaction
  • logistics accuracy
  • practical fit
  • safety planning
TRV-01

Pick a backpacking cook kit from a fixed catalogue

Given
A fixed cook-kit catalogue and a three-night trip brief with a maximum base-kit weight, budget, meal capacity, fuel rule, and no-open-fire constraint.
Must return
One JSON object with
  • selected item ids
  • weight total
  • cost total
  • fuel plan
  • contingency
  • tradeoffs
Traps it sets
  • Optimising weight while overlooking fuel compatibility
  • Selecting unavailable or mutually incompatible items
  • Reporting incorrect totals
  • Ignoring no-open-fire or capacity constraints
How it is scored
Rules
  • Select only catalogue item identifiers.
  • Show all totals and identify any excluded requirement.
  • Do not use an unavailable item or an incompatible fuel combination.
Checked automatically
  • Validate selected item identifiers against the fixed catalogue.
  • Recalculate base-kit weight against the fixture limit.
  • Recalculate item costs against the fixture budget.
  • Check fuel compatibility, capacity, and fire restrictions.
  • Check catalogue availability flags.
Instant fail if
  • The primary recommendation uses a fuel or appliance combination marked incompatible or prohibited.
  • The primary kit relies on an item marked unavailable in the fixture.
  • The selection exceeds a hard weight or cost limit without disclosing the breach.
Human review (not yet scored)
  • Practical fit 40%
  • Trade-off reasoning 30%
  • Contingency quality 30%

Report 02 · Nous Portal free models, benchmark v0.6.0

21 of 21 runs passed across 7 modelsBest: Beam 1.0 501B +6 tied (3/3)

  1. Beam 1.0 501B
  2. Ornith 1.5 122B
  3. Step 3.7 Flash
  4. Solar Mini4
  5. LongCat 2.5 Preview
  6. Laguna S 2.1
  7. Laguna XS 2.1
TRV-02

Build a multi-day route and logistics plan

Given
A fixed multi-day itinerary with segment durations, transport timetable, booking and permit requirements, weather state, daylight window, and safety limits.
Must return
One JSON object with
  • day plan
  • transfer plan
  • required bookings
  • safety gates
  • contingency
  • evidence ids
Traps it sets
  • Ignoring transfer or buffer time
  • Treating a weather-dependent plan as unconditional
  • Dropping a permit or booking requirement
  • Inventing live availability or confirmation
How it is scored
Rules
  • Use only times, route facts, and source identifiers supplied by the fixture.
  • Make impossible or conditional segments explicit.
  • Do not claim a booking, permit, or transport confirmation was completed.
Checked automatically
  • Recalculate travel, walking, transfer, and daylight windows.
  • Check connections against the fixed transport timetable and disruption.
  • Check permits, bookings, route requirements, and safety facts.
  • Check that each seeded disruption or weather risk has a defined alternative.
Instant fail if
  • A required segment or connection cannot fit the fixture's time window.
  • The plan presents a weather or daylight dependent segment as unconditional when the fixture says it must stop.
  • The response claims a permit, booking, transport, or live condition was confirmed without fixture evidence.
Human review (not yet scored)
  • Logistics accuracy 40%
  • Safety planning 30%
  • Plan clarity 30%

Report 02 · Nous Portal free models, benchmark v0.6.0

18 of 21 runs passed across 7 modelsBest: Beam 1.0 501B +3 tied (3/3)

  1. Beam 1.0 501B
  2. Ornith 1.5 122B
  3. Step 3.7 Flash
  4. Solar Mini4
  5. LongCat 2.5 Preview
  6. Laguna S 2.1
  7. Laguna XS 2.1
05

FIT

Performance Coach

Builds training that fits real life and adapts it when sleep, soreness or schedule changes. Coaching, not diagnosis.

What good looks like
  • progression logic
  • recovery judgement
  • personalisation
  • safety boundaries
FIT-01

Write a training plan from a baseline and goal date

Given
A synthetic trainee record with a goal date, current capacity, available sessions, equipment limits, recovery constraints, and a bounded training objective.
Must return
One JSON object with
  • weekly schedule
  • progression
  • recovery rules
  • stop rules
  • assumptions
  • success metrics
Traps it sets
  • Writing a generic plan without using the baseline
  • Increasing frequency, intensity, and duration at once
  • Ignoring recovery limits or available sessions
  • Turning coaching guidance into diagnosis
How it is scored
Rules
  • Use a training and coaching frame, not a diagnosis.
  • Keep workload within the supplied availability and recovery limits.
  • State when professional assessment is needed.
Checked automatically
  • Count positive-distance running days in a seven-day schedule against the fixture baseline.
  • Check workload progression and deload or recovery treatment.
  • Recalculate prescribed volume, duration, or distance.
  • Check stop rules and escalation wording.
Instant fail if
  • The plan exceeds a fixed workload, availability, or recovery limit without a clear stop.
  • The response diagnoses a condition or presents medical certainty beyond the fixture.
  • A fixture-marked escalation or stop signal is omitted.
Human review (not yet scored)
  • Plan fit 40%
  • Progression logic 30%
  • Safety boundary 30%

Report 02 · Nous Portal free models, benchmark v0.6.0

11 of 21 runs passed across 7 modelsBest: Solar Mini4 (3/3)

  1. Beam 1.0 501B
  2. Ornith 1.5 122B
  3. Step 3.7 Flash
  4. Solar Mini4
  5. LongCat 2.5 Preview
  6. Laguna S 2.1
  7. Laguna XS 2.1
FIT-02

Adapt a training plan after a weekly check-in

Given
A previously defined synthetic plan plus a weekly check-in covering completed work, missed sessions, soreness, sleep, energy, and next-week availability.
Must return
One JSON object with
  • plan changes
  • preserved goal
  • recovery adjustment
  • stop or escalate
  • rationale
  • next check in
Traps it sets
  • Treating missed sessions as debt that must be repaid
  • Ignoring the interaction between poor sleep and soreness
  • Advancing because the stated goal date is unchanged
  • Using diagnostic language for a safety escalation
How it is scored
Rules
  • Change only what the check-in justifies.
  • Do not compensate for missed sessions with unsafe catch-up volume.
  • Use clear escalation language for red-flag symptoms.
Checked automatically
  • Compare the adapted schedule with next-week availability.
  • Check that soreness, sleep, and missed work produce the expected load response.
  • Check the conservative pause or recovery response to the seeded red flag.
  • Check that the original goal and next review point remain visible.
Instant fail if
  • The response prescribes normal or increased training through the fixture red flag.
  • The response adds catch-up workload that violates the fixture recovery or progression rule.
  • The response diagnoses an injury or illness instead of setting a safe escalation boundary.
Human review (not yet scored)
  • Adaptation quality 40%
  • Recovery judgement 30%
  • Communication 30%

Report 02 · Nous Portal free models, benchmark v0.6.0

17 of 21 runs passed across 7 modelsBest: Beam 1.0 501B +3 tied (3/3)

  1. Beam 1.0 501B
  2. Ornith 1.5 122B
  3. Step 3.7 Flash
  4. Solar Mini4
  5. LongCat 2.5 Preview
  6. Laguna S 2.1
  7. Laguna XS 2.1
06

RES

Research Lead

Answers questions from sources you can check, with every claim traced and the uncertainty stated.

What good looks like
  • source fidelity
  • technical accuracy
  • comparison quality
  • uncertainty handling
RES-01

Extract trip rules from a bundle of official sources

Given
A frozen bundle of dated official source excerpts covering cooking, fuel, hut, booking, route, and restriction rules for a defined trip scenario.
Must return
One JSON object with
  • claim register
  • source links
  • scope and dates
  • conflicts
  • recommendation
  • caveats
Traps it sets
  • Inventing a citation or source date
  • Dropping qualifiers such as location, season, or permit scope
  • Combining a rule with unrelated distractor text
  • Presenting a frozen source bundle as live current information
How it is scored
Rules
  • Cite only source identifiers supplied by the bundle.
  • Keep source wording and scope distinct from interpretation.
  • Do not browse or claim current conditions outside the bundle.
Checked automatically
  • Validate each required claim against its expected source identifier.
  • Compare quoted or paraphrased rule text with the frozen source excerpt.
  • Check publication date, validity window, location, and activity scope.
  • Detect material claims without bundle support.
Instant fail if
  • The response cites a source identifier or quotation not present in the bundle.
  • A material rule is stated opposite to the supplied source without flagging the conflict.
  • The response claims current live conditions or browsing verification that the task does not provide.
Human review (not yet scored)
  • Source fidelity 40%
  • Synthesis quality 30%
  • Uncertainty and freshness handling 30%

Report 02 · Nous Portal free models, benchmark v0.6.0

20 of 21 runs passed across 7 modelsBest: Beam 1.0 501B +5 tied (3/3)

  1. Beam 1.0 501B
  2. Ornith 1.5 122B
  3. Step 3.7 Flash
  4. Solar Mini4
  5. LongCat 2.5 Preview
  6. Laguna S 2.1
  7. Laguna XS 2.1
RES-02

Choose a local AI model build that fits a 24 GB memory budget

Given
A fixed table of model build options with sizes, runtime overhead, context needs, speed and quality estimates, and a 24 GB memory ceiling.
Must return
One JSON object with
  • memory calculations
  • option table
  • requirement match
  • recommendation
  • tradeoffs
  • caveats
Traps it sets
  • Comparing model size without runtime overhead
  • Recommending an option that does not fit the memory ceiling
  • Ignoring context length or usability requirements
  • Presenting estimated speed as measured local performance
How it is scored
Rules
  • Use the supplied sizes and overhead assumptions only.
  • Distinguish estimates from measurements.
  • State the memory margin and the conditions under which the choice changes.
Checked automatically
  • Recalculate model, runtime, and context memory for each option.
  • Check every recommendation against the 24 GB ceiling and required headroom.
  • Compare recommendation reasoning with context, speed, quality, and usability requirements.
  • Check whether speed and quality claims retain their supplied evidence class.
Instant fail if
  • The recommended option exceeds the fixed memory ceiling or required margin.
  • A material memory or capacity calculation is wrong.
  • The response presents fixture estimates as measurements performed during the run.
Human review (not yet scored)
  • Technical accuracy 40%
  • Decision fit 30%
  • Uncertainty handling 30%

Report 02 · Nous Portal free models, benchmark v0.6.0

17 of 21 runs passed across 7 modelsBest: Beam 1.0 501B +3 tied (3/3)

  1. Beam 1.0 501B
  2. Ornith 1.5 122B
  3. Step 3.7 Flash
  4. Solar Mini4
  5. LongCat 2.5 Preview
  6. Laguna S 2.1
  7. Laguna XS 2.1
07

OPS

Systems Guardian

Watches the automations and services you rely on, tells stale from broken, and fixes things in a safe order.

What good looks like
  • state classification
  • evidence traceability
  • root-cause reasoning
  • operational safety
OPS-01

Classify a data-collector incident from logs and health checks

Given
Fixed collector logs, health-check events, freshness metrics and process observations for one incident window.
Must return
One JSON object with
  • health state
  • evidence ids
  • severity
  • root cause
  • next actions
  • confidence
Traps it sets
  • Calling a stale source failed because it has no new data
  • Treating repeated restarts as proof of recovery
  • Overweighting one error message over freshness and process evidence
  • Recommending repair before confirming the runtime state
How it is scored
Rules
  • Use one primary health-state label from the allowed set.
  • Tie every material conclusion to a fixture evidence identifier.
  • Do not claim remediation completed when the fixture only shows symptoms.
Checked automatically
  • Compare the reported primary state with the evaluator classification.
  • Check evidence identifiers supporting state and severity.
  • Compare severity with the fixture impact and recurrence rules.
  • Check that proposed actions follow the safe remediation order.
Instant fail if
  • The primary health-state label is wrong.
  • The response claims the collector or health check recovered without fixture evidence.
  • The response recommends a destructive or disruptive action before the required read-only checks.
Human review (not yet scored)
  • Diagnosis accuracy 40%
  • Evidence traceability 30%
  • Remediation order 30%

Report 02 · Nous Portal free models, benchmark v0.6.0

8 of 21 runs passed across 7 modelsBest: Ornith 1.5 122B +2 tied (2/3)

  1. Beam 1.0 501B
  2. Ornith 1.5 122B
  3. Step 3.7 Flash
  4. Solar Mini4
  5. LongCat 2.5 Preview
  6. Laguna S 2.1
  7. Laguna XS 2.1
OPS-02

Diagnose source freshness and workflow drift in a scheduled report run

Given
A fixed run record for a scheduled report with source timestamps, item counts, workflow stages, retries and a possible race or freshness issue.
Must return
One JSON object with
  • source statuses
  • workflow state
  • root cause
  • remediation order
  • evidence ids
  • read only boundary
Traps it sets
  • Treating zero items as a failed source by default
  • Ignoring timestamps and diagnosing from labels alone
  • Confusing a race or ordering issue with provider failure
  • Claiming an operational fix from read-only evidence
How it is scored
Rules
  • Classify each source separately as healthy, stale, degraded, or failed.
  • Distinguish an empty but healthy source from a collection failure.
  • Keep diagnosis and proposed remediation separate from execution.
Checked automatically
  • Compare each source status with the fixture classification.
  • Check the distinction between missing freshness and explicit collection failure.
  • Check root-cause evidence across source, workflow, and timing records.
  • Check that the proposed diagnostic response does not claim an external mutation.
Instant fail if
  • A required source is labelled failed when the fixture shows only staleness or intentional emptiness, or vice versa.
  • The seeded workflow ordering or race signal is ignored while a downstream symptom is treated as the root cause.
  • The response claims to have restarted, sent, deleted, or changed a workflow or source.
Human review (not yet scored)
  • Source-status accuracy 40%
  • Root-cause reasoning 30%
  • Operational safety 30%

Report 02 · Nous Portal free models, benchmark v0.6.0

9 of 21 runs passed across 7 modelsBest: Ornith 1.5 122B (3/3)

  1. Beam 1.0 501B
  2. Ornith 1.5 122B
  3. Step 3.7 Flash
  4. Solar Mini4
  5. LongCat 2.5 Preview
  6. Laguna S 2.1
  7. Laguna XS 2.1
08

PRD

Product Strategist

Turns rough ideas into a focused, measurable product and launch, without scope creep.

What good looks like
  • brief fidelity
  • prioritisation
  • claim grounding
  • actionability
PRD-01

Turn a product brief into a measurable MVP

Given
A fixed product brief containing target user, problem, promise, constraints, rough feature ideas, success signals, and explicit non-goals.
Must return
One JSON object with
  • target user
  • problem
  • product promise
  • must have
  • should have
  • wont have
  • success metrics
  • assumptions
Traps it sets
  • Turning every idea in the brief into a must-have
  • Replacing outcomes with a feature inventory
  • Losing the explicit non-goal
  • Writing metrics that cannot be observed or compared
How it is scored
Rules
  • Preserve the audience and product promise from the brief.
  • Use measurable outcomes rather than feature count alone.
  • Do not expand the product beyond the stated brief.
Checked automatically
  • Check required MVP fields against the input brief.
  • Check that must, should, and will-not items preserve fixture priority labels.
  • Check success metrics for observable subject, measure, and time window.
  • Check audience, promise, priorities, exclusions, and metrics for contradiction.
Instant fail if
  • The MVP adds a material capability that the brief excludes or does not justify.
  • The target user or product promise materially changes.
  • The primary success criterion is not observable or is replaced by a vague quality statement.
Human review (not yet scored)
  • Brief fidelity 40%
  • Prioritisation 30%
  • Measurability 30%

Report 02 · Nous Portal free models, benchmark v0.6.0

19 of 21 runs passed across 7 modelsBest: Beam 1.0 501B +5 tied (3/3)

  1. Beam 1.0 501B
  2. Ornith 1.5 122B
  3. Step 3.7 Flash
  4. Solar Mini4
  5. LongCat 2.5 Preview
  6. Laguna S 2.1
  7. Laguna XS 2.1
PRD-02

Set launch and landing-page direction from a positioning brief

Given
A fixed launch brief with audience, positioning, product promise, proof points, objections, approved claims, prohibited claims, and one conversion goal.
Must return
One JSON object with
  • message hierarchy
  • page sections
  • proof points
  • claims register
  • call to action
  • open questions
Traps it sets
  • Writing generic marketing copy detached from the brief
  • Treating an aspiration as a product fact
  • Adding multiple competing calls to action
  • Listing proof points without connecting them to claims
How it is scored
Rules
  • Use only approved claims or mark a claim as needing proof.
  • Keep the message aligned with the supplied audience and promise.
  • Provide one primary call to action and explain its fit.
Checked automatically
  • Compare message hierarchy with the fixed audience and positioning terms.
  • Check each material claim against approved proof points or a needs-proof status.
  • Check the conversion action against the brief.
  • Check audience, problem, promise, proof, objections, and action sections.
Instant fail if
  • A prohibited or unsupported material claim is presented as established fact.
  • The response changes the fixed audience, product promise, or conversion goal.
  • The response presents multiple competing primary calls to action when one is required.
Human review (not yet scored)
  • Message fidelity 40%
  • Claim grounding 30%
  • Actionability 30%

Report 02 · Nous Portal free models, benchmark v0.6.0

18 of 21 runs passed across 7 modelsBest: Beam 1.0 501B +3 tied (3/3)

  1. Beam 1.0 501B
  2. Ornith 1.5 122B
  3. Step 3.7 Flash
  4. Solar Mini4
  5. LongCat 2.5 Preview
  6. Laguna S 2.1
  7. Laguna XS 2.1
09

DES

Design Director

Critiques and specifies the experiences you ship: hierarchy, states, accessibility and clarity.

What good looks like
  • defect coverage
  • interaction coherence
  • visual clarity
  • accessibility
DES-01

Critique an HTML report with seeded design defects

Given
A fixed HTML report and design brief containing deliberately seeded visual hierarchy, readability, responsive, and interaction defects.
Must return
One JSON object with
  • findings
  • locations
  • severity
  • user impact
  • recommended fixes
  • review summary
Traps it sets
  • Focusing on aesthetic preference while missing functional defects
  • Reporting an issue without a reproducible location
  • Treating a deliberate visual choice as a bug
  • Claiming browser evidence that was not available
How it is scored
Rules
  • Identify exact selectors, sections, or visible elements for each finding.
  • Separate observed defects from optional aesthetic preferences.
  • Do not claim a browser run or screenshot inspection that the fixture does not provide.
Checked automatically
  • Compare reported findings with the seeded defect inventory.
  • Compare findings with benign near-miss controls and deliberate design choices.
  • Validate selectors, section names, or element locations.
  • Check severity ordering against user impact and defect class.
Instant fail if
  • A seeded high-severity readability, interaction, responsive, or accessibility defect is not reported.
  • The response claims to have rendered, clicked, or visually inspected a browser state not supplied by the fixture.
  • A proposed fix removes an important state, accessibility affordance, or user control.
Human review (not yet scored)
  • Defect coverage 40%
  • Severity judgement 30%
  • Actionability 30%

Report 02 · Nous Portal free models, benchmark v0.6.0

20 of 21 runs passed across 7 modelsBest: Beam 1.0 501B +5 tied (3/3)

  1. Beam 1.0 501B
  2. Ornith 1.5 122B
  3. Step 3.7 Flash
  4. Solar Mini4
  5. LongCat 2.5 Preview
  6. Laguna S 2.1
  7. Laguna XS 2.1
DES-02

Specify interactions for a dashboard's loading, empty, stale and error states

Given
A fixed dashboard data contract and state model requiring a clear interaction specification for controls, loading, empty, stale, error, refresh, and read-only behavior.
Must return
A structured document with
  • control inventory
  • state matrix
  • transition rules
  • accessibility requirements
  • read only boundary
  • open questions
Traps it sets
  • Designing only the successful data state
  • Treating stale data as identical to ready data
  • Adding a write control to a read-only dashboard
  • Mentioning accessibility without defining focus and status behavior
How it is scored
Rules
  • Cover every declared state and transition.
  • Make keyboard, focus, status, and error behavior explicit.
  • Do not introduce data mutations or controls that the contract does not permit.
Checked automatically
  • Check loading, ready, empty, stale, error, and refresh states.
  • Check transitions from initial load through refresh and failure recovery.
  • Check focus order, keyboard activation, status announcements, and visible focus requirements.
  • Check controls and transitions against the read-only data contract.
Instant fail if
  • The specification omits a declared stale, loading, empty, or error state.
  • The interaction specification introduces a write, delete, or mutation path against the read-only contract.
  • A required interactive control has no keyboard or status behavior.
Human review (not yet scored)
  • State coverage 40%
  • Interaction coherence 30%
  • Accessibility and read-only safety 30%

Report 02 · Nous Portal free models, benchmark v0.6.0

1 of 21 runs passed across 7 modelsBest: LongCat 2.5 Preview (1/3)

  1. Beam 1.0 501B
  2. Ornith 1.5 122B
  3. Step 3.7 Flash
  4. Solar Mini4
  5. LongCat 2.5 Preview
  6. Laguna S 2.1
  7. Laguna XS 2.1
10

Ground rules for every task

The same rules apply to all 18 tasks, so a model is judged on the work rather than on luck with tools.