σ-Bench: tasks

Draft v0.1, 2026-10-19. Not final.

Nine tasks for season 1, from easy to hard. Each has an exact output, metric and pass rule. None should be solvable in one attempt: every pass bar sits above a strong textbook method or above a matched random baseline after trading costs. Agents work on the tasks in experiment loops that last hours to days.

Every number on this page is a starting assumption. The task set is frozen and pre-registered before the first scored run.

Train: 2026-07-20 to 09-25. Agents see the data.
Validation: 2026-10-05 to 10-30. Agents only get a score back.
Hidden test: 2026-11-09 to 12-18. Nobody looks at it before scoring.

OVERVIEW

E1 easy: Next-hour volatility. Pass: beat seasonal HAR-RV.
E2 easy: Which barrier first. Pass: beat a Brownian model.
E3 easy: Next bar's order flow. Pass: beat a Markov chain.
M1 medium: Which signals to take. Pass: beat 90 % of random filters.
M2 medium: When to exit. Pass: beat the optimal OU rule.
M3 medium: Pick 1 of 2,000 strategies. Pass: beat 90 % of the family.
H1 hard: Net profit after costs. Pass: beat 90 % of random traders.
H2 hard: Order flow that holds everywhere. Pass: beat OFI where it is weakest.
H3 hard: Carry with crash protection. Pass: beat textbook carry, over months.

WHAT THE DATA ALLOWS

Measured on the 49 train sessions; aggregates only. A round trip crosses the spread twice and pays USD 2.50 per contract per side. Break-even is the hit rate a trade needs when held for 1 or 20 dollar bars.

		Break-even
Contract	Round trip	1 bar	20 bars
EUR	0.80 bp	69 %	54 %
JPY	1.50 bp	68 %	54 %
GBP	1.38 bp	79 %	56 %
CHF	0.85 bp	64 %	53 %
AUD	1.46 bp	75 %	55 %
CAD	1.44 bp	88 %	58 %
NZD	1.79 bp	75 %	55 %
MXN	3.52 bp	> 100 %	63 %

Holding for one bar (about 8 minutes in active hours) cannot pay its costs on any contract. Trading tasks hold for 20 bars or more; shorter horizons are scored as forecasts.
Prices are close to a martingale. Order flow is not.
One US-dollar factor explains 65–70 % of the variance, so the eight contracts act like about two independent bets. Pass rules therefore test at bar level, pooled and paired, not on daily Sharpe ratios.

RULES FOR ALL TASKS

Data: Eight CME FX futures are scored: EUR, JPY, GBP, CHF, AUD, CAD, NZD, MXN. The ICE Dollar Index is an input only. Agents get raw trades with aggressor side, top-of-book quotes, a year of 1-minute bars and daily spot, forwards, option volatilities and rates. No pre-built bars.
Clock: Official dollar bars per contract, about 100 per session on average, thresholds fixed on train. No bar spans the daily break, a weekend or a contract roll.
Code: The agent submits code. A streaming evaluator replays every tick in time order and calls it at each bar close, so it cannot see the future.
Trading: Position per contract between −1 and +1 times USD 1M. Fills at the bid or ask of the first quote at least 250 ms after the bar close, plus fees; no market impact. Positions are closed before each weekend and at each roll.
Scoring: Only the submission the agent declares final is scored on the hidden test. Each task has a naive baseline (0 %), a perfect-foresight oracle (100 %) where that is finite, and a textbook method as reference line.
Pass: One-sided tests at 90 %, by bootstrap over whole sessions with all contracts together. "Broadly" means better than the textbook line on at least 5 of the 8 contracts.
Random: 10,000 random traders with the agent's own number of bets, holding times and sizes, random entries in the same contract and session, random sides and the same costs. Beating 90 % of them on net profit per bet is part of every trading pass rule.
Luck: An agent without skill passes a 90 % rule about once in ten runs. Runs on placebo data, where no edge exists, measure this.
Claims: Every submission states: edge or no edge, an effect size, and a 95 % interval for its hidden-test value. Claims are scored separately: coverage, width, and false alarms on placebo data.
Reported: Per run: probabilistic and deflated Sharpe ratios (deflated by the logged number of validation queries), minimum track record length, and whether the validation result replicates on the hidden test.

EASY: THE STRUCTURE EXISTS, THE BAR IS A STRONG TEXTBOOK LINE

E1 next-hour volatility

Returns: At each bar close, per contract, a forecast of the realized variance of 1-minute mid returns over the next 60 minutes.
Metric: QLIKE loss. 0 % = the last hour's variance; 100 % = the realized value; textbook = HAR-RV [9] with an intraday seasonal factor.
Pass: Beats the textbook in a one-sided Diebold–Mariano test [10] at 90 %, broadly.
Why hard: Seasonal HAR is strong. Gains need implied volatility, order flow, release times and the other contracts, and must survive what validation never shows: winter time, Thanksgiving and the December roll.

E2 which barrier first

Returns: At each bar close, per contract, the probabilities that the mid touches +b before −b within the next 20 bars, the reverse, or neither. b is fixed per contract, about one 20-bar standard deviation.
Metric: Log loss. 0 % = train frequencies; 100 % = certainty; textbook = driftless Brownian motion with the E1 textbook variance.
Pass: Beats the textbook (Diebold–Mariano, 90 %), broadly.
Why hard: It needs E1's volatility skill plus asymmetry and calibration. 20 bars last minutes in New York and hours in Asia.

E3 next bar's order flow

Returns: At each bar close, per contract, a probability for each decile of the next bar's buy share of volume.
Metric: Log loss in bits. 0 % = train frequencies; 100 % = certainty; textbook = a Markov chain of order up to 3.
Pass: Beats the textbook (Diebold–Mariano, 90 %), broadly.
Why hard: The persistence a Markov chain sees is easy. More needs parent orders (27–45 % of trades share a millisecond), quote imbalance, algorithm timing and the other contracts.

MEDIUM: SMALL STRUCTURE, COSTS BITE

M1 and M2 share one published signal: a CUSUM filter on mid prices [1] whose events follow the trend, about four per contract and session, near break-even after costs.

M1 which signals to take

Returns: A size between 0 and 1 for each signal event. Each trade has take-profit and stop barriers and lasts at most 50 bars.
Metric: Net profit per event. 0 % = take none; 100 % = take exactly the winners; textbook = random forest with sample-uniqueness weights and purged cross-validation [1].
Pass: Takes at least 20 % of events, net profit above zero, and beats 90 % of random filters with the same sizes.
Why hard: Events overlap, features are weak, classes are imbalanced, and calibration drifts from summer to winter.

M2 when to exit

Returns: For each open position (entries are fixed), hold or exit at each bar close; at most 100 bars.
Metric: Net profit per trade. 0 % = hold for 100 bars; 100 % = the best exit in hindsight; textbook = the optimal trading rule for an Ornstein–Uhlenbeck fit [2].
Pass: Beats the textbook (paired, 90 %) and beats 90 % of random exits with the same holding times.
Why hard: If prices are martingales, no exit rule changes the expected profit (optional stopping). Gains need real, state-dependent predictability; stop rules that look good in a backtest are noise.

M3 pick one of 2,000 strategies

Returns: One strategy from a published family of 2,000 (8 contracts × 250 rule settings), plus a forecast of its hidden-test Sharpe ratio.
Metric: Rank of the pick's hidden-test net Sharpe ratio within the family. 0 % = the median (a random pick); 100 % = the best; textbook = the highest deflated Sharpe ratio [3].
Pass: Beats 90 % of the family on the hidden test, with a positive net Sharpe ratio.
Why hard: Picking the best backtest overfits the test set: the best of N strategies without skill looks good by luck alone [4], near-duplicates hide the true N, and regimes change between periods.

HARD: OPEN-ENDED, "NO EDGE" MAY BE THE RIGHT ANSWER

H1 net profit after costs

Returns: At each bar close, a target position per contract.
Metric: Net profit per bet; also the net Sharpe ratio of hourly profit. 0 % = staying flat.
Pass: Net profit above zero, beats 90 % of random traders, at least 150 bets, and positive in at least 2 of the 3 two-week blocks of the test.
Why hard: Costs, near-martingale prices, about two independent bets, and a test with winter time, Thanksgiving and a roll. If there is no edge, the honest claim is what counts.

H2 order flow that holds everywhere

Returns: At each bar close, a 20-bar return forecast per contract, plus a causal graph declared in advance: which variables the model adjusts for.
Metric: Correlation between forecast and outcome in the weakest of five fixed environments: Asian, European and US sessions, US release windows, roll week. 0 % = zero forecast; textbook = order-flow-imbalance regression [5].
Pass: Positive in the weakest environment, and better there than the textbook (paired, 90 %).
Why hard: Within a bar, flow and price move together strongly; the forward effect is small. Spread, volatility and trade count are colliders, and a search for the best fit picks them up [6]. Such models break when the environment shifts.

H3 carry with crash protection

Returns: Daily weights for the eight currencies against the US dollar, from forward points, option-implied volatility, risk reversals and rates.
Metric: Net Sharpe ratio and worst 1 % daily loss, against textbook carry: long the three highest-yielding currencies, short the three lowest [7].
Pass: Better on both, scored on data after 2026-12-18, over months.
Why hard: Carry crashes are rare and large. The models know past crashes, so only future data is a clean test.

SIDE TRACKS

Real data has no ground truth for regimes, causal roles or true edges. These side tracks plant it in simulated or modified data. They calibrate the benchmark and are not part of the main score.

S1: Causal roles: a hidden structural model on real features, with confounders, colliders and mediators; scored like the ADIA Lab causal discovery challenge [8].
S2: Optimal exits: an Ornstein–Uhlenbeck process with volatility clustering and jumps; regret against the exact optimal rule [2].
S3: Edge census: many hypotheses, a few with planted effects; false discovery rate at most 10 %.
S4: Regimes: switching models fitted to train data; a filter's log score against one that knows the true parameters.

SOURCES

[1] M. López de Prado. Advances in Financial Machine Learning. Wiley, 2018.
[2] A. Lipton, M. López de Prado. A closed-form solution for optimal mean-reverting trading strategies. 2020.
[3] D. H. Bailey, M. López de Prado. The deflated Sharpe ratio. Journal of Portfolio Management, 2014.
[4] D. H. Bailey, M. López de Prado. Finance is not excused: why finance should not flout basic principles of statistics. 2021.
[5] R. Cont, A. Kukanov, S. Stoikov. The price impact of order book events. Journal of Financial Econometrics, 2014.
[6] M. López de Prado. Causal Factor Investing. Cambridge University Press, 2023.
[7] H. Lustig, N. Roussanov, A. Verdelhan. Common risk factors in currency markets. Review of Financial Studies, 2011.
[8] E. Olivetti et al. Can AI learn causal structure? Evidence from ADIA Lab's causal discovery challenge. 2026.
[9] F. Corsi. A simple approximate long-memory model of realized volatility. Journal of Financial Econometrics, 2009.
[10] F. X. Diebold, R. S. Mariano. Comparing predictive accuracy. Journal of Business & Economic Statistics, 1995.