σ-Bench

A quantitative-finance benchmark for AI coding agents.

Agents run their own experiments on real LSEG FX market data. Every number comes with an error bar.

M.Sc. Statistics

Status: Release 0, updated 2026-10-19. Task draft v0.1; no runs yet.

LEADERBOARD

No runs yet.

TASKS

Draft v0.1, 2026-10-19, not final: nine tasks on real CME FX futures data. Each has an exact output, metric and pass rule, and none should be solvable in one attempt. Rules and details: tasks.

E1 easy: Next-hour volatility, against seasonal HAR-RV.
E2 easy: Which barrier the price touches first.
E3 easy: The next bar's order flow.
M1 medium: Which signals of a fixed strategy to take.
M2 medium: When to exit fixed entries.
M3 medium: Pick one of 2,000 backtested strategies.
H1 hard: Net profit after costs, against random traders.
H2 hard: Order-flow effects that hold in every session.
H3 hard: Carry with crash protection, scored over months.

METHOD

σ-Bench rests on three parts:

Experiment loops. Agents run their own loops of hypothesis, experiment and evaluation, as in Dream-RSI [1]. A run is scored by how far its result improves and how much of that survives on hidden data, not by pass@k.
Data. Trade-by-trade CME FX futures from LSEG, with trade size and aggressor side, sampled as dollar bars [2] instead of 1-minute bars.
Tasks. Statistical inference on real data: martingale tests, optimal stopping, Markov regimes and meta-labeling.

Agents never receive credentials or live data access. They work on frozen data snapshots.

PRE-REGISTERED EXPERIMENTS

None yet.

LIMITATIONS

One author. I write the tasks, solutions and verifiers myself, so the set reflects my view of quant work and my conventions.
Private data. The tasks run on licensed LSEG data, which cannot be published. Task rules, methods and aggregate results are public; raw prices never are.

LOG

2026-10-15: Concept written.
2026-10-16: LSEG data access confirmed. sigmabench.dev registered.
2026-10-17: Release 0: this page, no runs yet. The benchmark is built on experiment loops with real FX data.
2026-10-18: Dataset fixed: trade-by-trade CME FX futures, sampled as dollar bars. Intraday archive started.
2026-10-19: Task draft v0.1: nine tasks, from easy to hard. Details: tasks.

REFERENCES

[1] T. Zheng et al. Dream-RSI: Recursive Self-Improvement through Evolving Worlds. arXiv:2609.14858, 2026.
[2] M. López de Prado. Advances in Financial Machine Learning. Wiley, 2018.