EJBench / v0.1By Gradient Dynamics

    Engineering Judgement Benchmark

    Train engineering judgement.
    Verify it with physics.

    Simulation-grounded tasks, engineering rubrics and verifiable rewards for teams training models and agents for the physical world. EJBench is our public benchmark for measuring that judgement.

    See EJBench results
    Simulation-grounded evidence
    FluxCore simulation showing surface temperature across prismatic battery cells
    01 / Battery thermal management

    Actual FluxCore simulation from a training design. The model reasons from numerical evidence; these images illustrate the underlying physics.

    272Frozen test tasks
    12Models evaluated
    228Tasks per leaderboard run
    CHTSimulation-grounded evidence

    01 / EJBench model performance

    Engineering judgement,
    measured.

    Our public benchmark puts engineering judgement to the test. Same frozen tasks. Same scoring rules.

    v0.1 · Updated 11 Nov 2026
    Overall results
    EJBench v0.1 model results, sorted by overall score. Scores are percentages. The rank column always shows the overall benchmark rank.
    Overall rankModel / organisationOverall score 0 — 100%95% CIAPI cost / task
    01GPT-6 AstraOpenAI
    71.0%
    65.1–77.3%$0.727
    02GPT-6.1 SolOpenAI
    68.9%
    62.4–74.7%$0.147
    03Claude Opus 5.5Anthropic
    64.1%
    58.1–70.1%$0.426
    04GPT-6.1 TerraOpenAI
    58.8%
    52.7–65.7%$0.036
    05Muse Spark 1.3Meta
    55.0%
    49.2–61.4%$0.186
    06Claude Sonnet 5.5Anthropic
    51.8%
    46.0–57.9%$0.559
    07Mistral Large 4Mistral AI
    50.3%
    44.0–57.2%$0.134
    08DeepSeek V4.1 FlashDeepSeek
    44.9%
    38.4–51.8%$0.052
    09Claude Haiku 5.5Anthropic
    37.6%
    31.0–44.1%$0.026
    10GPT-6 LunaOpenAI
    36.3%
    30.0–42.9%$0.007
    11Qwen3.5-9B + GRPO on 25% of trainGradient Dynamics · fine-tuned baseline
    7.8%
    4.2–12.0%—
    12Qwen3.5-9BQwen (open weights)
    5.9%
    2.7–9.7%—

    Overall scores use equal weighting across task types. Each model ran once. Intervals reflect task sampling, not run-to-run variation; close scores should be compared using paired task results. API costs use list prices at the time of each run; unreported costs are shown as —.

    Evaluation protocol & run details

    The leaderboard uses a fixed subset of 228 tasks from the 272-task test set. All 12 models scored all 228 tasks. API models used high reasoning effort; Qwen baselines used thinking on at temperature 0.6. Reply limits did not truncate any response.

    No external tools or new simulations are allowed. The benchmark harness can reveal stored simulation results where a task permits it. Every task scores 1 only when every required answer is correct; otherwise it scores 0. Confidence does not affect success.

    gpt-6-astrahigh · 2026-11-11
    gpt-6.1-solhigh · 2026-11-10
    claude-opus-5-5high · 2026-11-10
    gpt-6.1-terrahigh · 2026-11-10
    muse-spark-1.3high · 2026-11-10
    claude-sonnet-5-5high · 2026-11-10
    mistral-large-4-0high · 2026-11-11
    deepseek-flashhigh · 2026-11-10
    claude-haiku-5-5high · 2026-11-10
    gpt-6-lunahigh · 2026-11-10
    qwen9b-grpo-25pctthinking on, temperature 0.6 · 2026-11-10
    qwen9bthinking on, temperature 0.6 · 2026-11-09

    02 / Engineering RL environments

    Real engineering.
    A place to learn it.

    For AI labs and teams building engineering agents, we develop tasks grounded in physical systems, with evidence to work from and a measurable definition of success.

    The task

    Start with the engineering

    An engineering brief, operating conditions and competing constraints. Vary the design and duty so the model has to reason about the physics.

    The environment

    Give the agent room to act

    Define the available evidence, permitted actions and tool budget. EJBench already supports multi-step decisions over stored simulation results.

    The learning loop

    Run. Score. Repeat.

    Reset the task, capture the interaction and score the outcome consistently. Use the same checks to compare model versions and measure learning progress.

    Built around your training programme.

    Our starting point is simulation-grounded tasks, stored-result environments and deterministic reward functions. Work with us to scope custom task distributions, tool integrations, solver access and delivery into your training stack.

    03 / Rubrics, verifiers & rewards

    Define good engineering.
    Make it measurable.

    A useful training signal starts with a clear standard. We translate engineering requirements into explicit criteria, then check decisions against simulation-derived reference results.

    Rubric
    The requirements a successful answer must meet, including physical limits, required decisions and evidence.
    Verifier
    The checks that compare the submission with those requirements and identify which criteria were met.
    Reward
    The resulting score used for training. EJBench's released reward functions use the same deterministic scorer as its evaluation harness.

    For custom programmes, agree the hard constraints, quality criteria and reward structure with our engineering team before training begins.

    Illustrative engineering rubricDoes the decision match the physics?
    Thermal limits
    Temperature verdicts match the reference results.
    Hydraulic limits
    Pressure-drop verdicts match the reference results.
    Complete coverage
    Every requested operating condition is assessed.
    Valid submission
    The answer follows the task's required structure.

    Success requires every mandatory check to pass.

    Grounded in physics

    Our FluxCore solver provides conjugate heat transfer simulations that connect task requirements to numerical reference results.

    04 / For AI labs & engineering-agent teams

    Build the environment
    your model needs.

    Bring us the engineering capability you want to train. We can scope the tasks, evidence, rubrics and feedback around your model and workflow.

    Engineering RL environments

    Tasks, rubrics
    and rewards.

    Work with our team on an engineering environment for your training or evaluation programme, built around the physical systems and decisions that matter to your model.

    • Engineering tasks and simulation evidence
    • Explicit rubrics and verifiable scoring
    • Reward integration scoped to your training workflow

    Private evaluation.
    Optional publication.

    Companies can also evaluate their models with us on EJBench. Compare under the benchmark protocol, inspect results and discuss failure cases with our team.

    Your model appears on the public leaderboard only with your approval.

    Dataset & harness accessAsk for the v0.1 release and evaluation harness. The release is licensed CC BY-NC 4.0; contact us about commercial use. Evaluation data must not be used for training.