Roblox LLM Leaderboard

Click a column to sort, click a row for details.

The trade-off space

Hover for details. Ctrl + scroll or pinch to zoom, drag to pan, & double-click to reset view.

Frontier over time Composite score against release date. The line traces the best score available at any point in time.

Score vs cost Composite score against total eval spend (log scale).

Score vs output tokens Composite score against total output tokens (log scale).

Score vs model size Composite score against weight size on disk (log scale). Models without published weights are omitted.

Knowledge vs coding Weighted knowledge score against weighted coding score.

Compare Models

Two or three models side by side: score deltas, per-topic knowledge, and true head-to-head counts from the per-question records.

Score Anatomy

Breakdown of result compositions. Hover a segment to see exact counts.

Composite Score The five weighted terms of the overall score, stacked. Each bar ends at the model's score.

Question Topics Accuracy per docs area, with every topic given an equal share of the bar so a perfect score fills the chart. Segments blend freeform and multiple choice the way the Knowledge score does.

Failure Modes Breakdown of outcomes across all three difficulties.

Leetcode by difficulty Problems fully solved on Easy / Medium / Hard.

Results Explorer

Every question, and what each model actually answered. Judge reasoning, chosen letters, and generated code included. Click a row to open it.

Loading question data…

Hardest Problems

The questions the fewest models get right, ranked across every model on the board. Click a row to open it.

Loading question data…

Methodology

RobloxQA v2.0

Knowledge testing: 3,000 questions about Roblox and Luau, scored both as freeform answers (judged) and as multiple choice. Dataset ↗

Luau Leetcode

Single-shot code generation on 208 Leetcode problems translated to Luau (47 easy, 97 medium, 64 hard), run against unit tests with syntax validation and code-quality scoring. Dataset ↗

Composite score

Half knowledge, half coding, with weights favoring the harder signals.
Knowledge weighs freeform answers over multiple choice: 0.7 × freeform + 0.3 × multiple choice.
Coding scores each difficulty as 0.4 × unit tests + 0.25 × fully solved + 0.15 × partially solved + 0.1 × code quality + 0.1 × valid syntax, then blends difficulties as 0.2 × easy + 0.35 × medium + 0.45 × hard.

Sample sizes

Every percentage that counts problems carries its 95% Wilson interval. The two RobloxQA splits ask 3,000 questions each, so their intervals are tight. Leetcode is far smaller so it has a wider interval.
Unit tests passed and code quality are averages of per-problem values rather than proportions, and the composite scores are weighted blends, so none of those get an interval.

Topics

Every RobloxQA question is grounded in one page of the Roblox or Luau documentation, and “Question Topics” groups those pages into ten areas. The corpus is not evenly spread across them: engine class reference alone is 1,157 of the 3,000 questions, while Luau is 87. Hovering a segment gives the exact counts and percentages to distinguish those.

Cost

Based on OpenRouter's pricing, applied to the exact input and output token counts of every eval run.
Models run locally used an RX 7900XTX or RTX 3090, with cost computed from electricity use (tracked by HWiNFO) at $0.20 per kilowatt-hour.
Cost per point divides total spend by composite score.