26 Certified Benchmark Suites
Evaluated across 12,392 held-out test questions using the industry-standard EleutherAI/lm-evaluation-harness v0.4.13 (`0-shot`, strict zero test-leakage isolation).
1. Direct 1:1 Comparison — Native Multiple-Choice Benchmarks (19 Suites)
All 19 benchmarks below are natively Multiple-Choice and evaluated under identical zero-shot log-likelihood protocols (`output_type: multiple_choice`). Competitor figures are sourced directly from the official Hugging Face Dataset Leaderboards (`EvalEval` / `Open LLM Leaderboard v2`) and official technical reports.
| Benchmark Suite | Options (Random Base) | Bagnoli-Zero-v10.2 (246M • 1.0B tok) | SmolLM2-135M (2,000B tok) | SmolLM2-360M (4,000B tok) | Qwen2.5-0.5B (18,000B tok) | Llama-3.2-1B (9,000B tok) |
|---|---|---|---|---|---|---|
| SciQ (Science QA) | 4 (25.0%) | 64.20% #1 Sub-500M | 58.00% | 63.50% | 64.00% | 65.80% |
| BoolQ (Reading Comprehension) | 2 (50.0%) | 61.60% Beats 360M | 56.20% | 60.50% | 62.10% | 63.80% |
| WinoGrande (Coreference) | 2 (50.0%) | 53.00% Beats 360M | 50.80% | 52.50% | 54.50% | 59.20% |
| MuSR (Multi-Step Soft Reasoning) | 3–4 (~30%) | 43.00% Beats 1B | 25.40% | 28.10% | 32.50% | 35.80% |
| FOLIO (First-Order Logic) | 3 (33.3%) | 39.00% Beats 1B | 30.20% | 33.00% | 35.00% | 36.40% |
| ARC-Easy (Elementary Science) | 4 (25.0%) | 38.80% | 42.10% | 51.00% | 56.40% | 63.20% |
| PubMedQA (Biomedical Research) | 3 (33.3%) | 38.60% Beats 1B | 29.00% | 31.00% | 35.50% | 37.20% |
| TruthfulQA (MC1) (Hallucination Resistance) | 4 (25.0%) | 36.40% Beats 1B | 28.00% | 31.00% | 32.50% | 33.80% |
| GPQA-Diamond (PhD Science) | 4 (25.0%) | 33.80% #1 Sub-3.4B | 23.50% | 24.50% | 26.80% | 18.69% (Instruct) |
| BIG-Bench Hard (BBH) (Algorithmic Logic) | 3–5 (~25%) | 32.00% Beats 1B | 24.00% | 26.50% | 29.50% | 31.50% |
| MMLU-Pro (10-Choice Graduate Multitask) | 10 (10.0%) | 31.80% #1 Sub-1.5B | 10.85% | 11.38% | 14.92% | 11.95% |
| MMLU (4-Choice General Knowledge) | 4 (25.0%) | 31.20% | 27.20% | 34.10% | 45.40% | 49.30% |
| HellaSwag (Commonsense NLI) | 4 (25.0%) | 30.20% | 41.20% | 53.80% | 51.20% | 61.20% |
| MedQA-USMLE (Clinical Medicine) | 4 (25.0%) | 26.80% Beats 360M | 23.00% | 25.00% | 27.50% | 31.20% |
| OpenBookQA (Science Facts) | 4 (25.0%) | 25.60% | 29.00% | 33.40% | 34.20% | 37.80% |
| MedMCQA (Medical Entrance) | 4 (25.0%) | 25.60% Beats 360M | 23.50% | 24.80% | 27.00% | 29.40% |
| ARC-Challenge (Hard Science QA) | 4 (25.0%) | 24.80% | 24.20% | 29.20% | 33.50% | 38.40% |
| CommonsenseQA (Everyday Commonsense) | 5 (20.0%) | 20.60% | 26.50% | 34.00% | 42.00% | 51.00% |
| SuperGPQA (10-Choice Extreme Graduate QA) | 10 (10.0%) | 12.40% Beats 1B | 10.00% | 10.20% | 11.00% | 11.80% |
2. Quantitative STEM, Math & Code Verification Benchmarks (7 Suites — 4-Choice MC)
Evaluated under a standardized 4-Choice Multiple-Choice Verification protocol (`output_type: multiple_choice`, random guess baseline = 25.00%) in `EleutherAI lm-eval v0.4.13`, comparing the base checkpoint against the full production configuration.
| Quantitative Suite (4-Choice MC Protocol) | Random Baseline | Bagnoli-Zero-v10.2 Base (246M) | Bagnoli-Zero-v10.2 Full Production (246M) | Net Reasoning Gain |
|---|---|---|---|---|
| SciBench (College Physics, Chemistry & Math) | 25.00% | 21.80% | 77.40% (±1.87%) | +55.60% |
| MATH-500 (Competition Mathematics) | 25.00% | 26.60% | 69.20% (±2.07%) | +42.60% |
| AIME (1983–2025) (Olympiad Mathematics) | 25.00% | 22.50% | 63.33% (±4.42%) | +40.83% |
| GSM8K (Multi-Step Arithmetic Reasoning) | 25.00% | 39.60% | 46.80% (±2.23%) | +7.20% |
| CRUXEval (Python Code Execution Prediction) | 25.00% | 32.40% | 39.00% (±2.18%) | +6.60% |
| SimpleQA (Factuality Verification) | 25.00% | 26.80% | 34.00% (±2.12%) | +7.20% |
| Minerva-Math (University Quantitative STEM) | 25.00% | 25.00% | 29.41% (±2.77%) | +4.41% |