Torpedo Software // Evals

Roblox LLM Leaderboard

Sounding out which models actually know Roblox, and what they cost to run. Every model here is benchmarked on RobloxQA v2.0 (Roblox & Luau knowledge) and Luau Leetcode (single-shot code generation, 208 problems).

Click a column to sort, click a row for details.

The trade-off space

Points are colored by provider. Hover for details, click to open the model's page.

Score vs cost Composite score against total eval spend (log scale).
Score vs output tokens Composite score against total output tokens (log scale).
Score vs model size Composite score against weight size on disk (log scale). Models without published weights are omitted.
Knowledge vs coding Weighted knowledge score against weighted coding score.
Leetcode by difficulty Problems fully solved on Easy / Medium / Hard.

Methodology

RobloxQA v2.0

Knowledge testing: 3,000 questions about Roblox and Luau, scored both as freeform answers (judged) and as multiple choice. Dataset ↗

Luau Leetcode

Single-shot code generation on 208 Leetcode problems translated to Luau (47 easy, 97 medium, 64 hard), run against unit tests with syntax validation and code-quality scoring. Dataset ↗

Composite score

Half knowledge, half coding, with weights favoring the harder signals. Knowledge weighs freeform answers over multiple choice: 0.7 × freeform + 0.3 × multiple choice. Coding scores each difficulty as 0.4 × unit tests + 0.25 × fully solved + 0.15 × partially solved + 0.1 × code quality + 0.1 × valid syntax, then blends difficulties as 0.2 × easy + 0.35 × medium + 0.45 × hard.

Cost

Based on OpenRouter's effective pricing. Torpedo Software models not on OpenRouter were run on an RX 7900XTX, with cost computed from electricity use at 20 cents per kilowatt-hour.