HydroTuring

A conservation-law benchmark

HydroTuring Initiative

A benchmark that asks one question of any AI hydrologic model: does it conserve what physics says it must conserve?

Not whether it fits a hydrograph. Whether its water budget closes, its energy budget closes, and its routing conserves momentum.

View the repository Claim a probe

Mission

How it works

Flowchart: any AI hydrologic model enters through the /io contract, HydroTuring probes it against mass, energy and momentum, and a verdict with its reason comes out ANY MODEL AI, hybrid or process-based any language model.yaml Dockerfile ht_adapter.py a container and a thirty-line adapter /io request.json forcing.csv static.json no network read-only inputs HydroTuring does it conserve what physics says it must? MASS water budget closes closure resolution warming causality dry-down steady state extreme rain ENERGY energy budget closes probes wanted MOMENTUM routing conserves momentum probes wanted seed → case → run → one bit seeded generator nothing stored sealed container no network binary criteria the 5% rule acceptance gate must pass must fail result.csv run.json $ ht run --model VERDICT PASS FAIL with its reason OK VIOLATION INCOMPLETE INCOMPATIBLE ERROR every criterion keeps its number archived to models/result.csv
Any model goes in through the /io contract. HydroTuring probes it against the three conservation laws, on cases generated fresh from a seed and run in a sealed container. One verdict comes out, with its reason and every number under it.
Reference model What it does Caught by

Residual number line showing the exact model inside tolerance and the leaky model outside it tolerance = 5% 0% 5% 10% 15% 20% cumulative budget residual reference_bucket 0.1% · must pass reference_leaky 15% · must fail
The acceptance gate for a contributed probe.

Cases are generated, never stored

Verdicts

One model through the suite

ht run --model google_flood_forecast --gate-seeds
google_flood_forecast v0.1.0-828dfc5  ->  ❌ FAIL (INCOMPLETE)  [2/7 probes passed]
  ❌ FAIL  mass/catchment-closure
          missing: pr, evspsbl, mrso, snw, canopy
  ❌ FAIL  mass/resolution-invariance
        ❌  resolution_invariance mrro differs by 65.9% of pr between PT1H and PT1D (limit 5%)
        ✅  non_degenerate     partition and variability are non-trivial
  ❌ FAIL  mass/warming-response
        ❌  response_sign      warmer (more pet): mrro fell by 12.5 mm, -0.088 per unit of demand; a decrease of at least 0.1 per unit is expected
        ✅  non_degenerate     partition and variability are non-trivial
  ✅ PASS  mass/causality
        ✅  causality          nothing moves before the storm of 2002-02-04 (worst 0.0e+00 relative) and mrro answers it afterwards (+0.318 of the added rain)
        ✅  non_degenerate     partition and variability are non-trivial
  ❌ FAIL  mass/dry-down
        ✅  dry_down           drains without rain: 224 mm over 730 dry days against a 322 mm bound, and nothing rises
        ❌  non_degenerate     mrro is nearly constant (cv 0.0192 < 0.1)
  ✅ PASS  mass/steady-state
        ✅  steady_state       settles: runoff 0.630 mm/day under 2.500 mm/day of rain, nothing varies by more than 0.00% (mrro) in the last 365 days
  ❌ FAIL  mass/extreme-rain
        ❌  monotone_response  scaling the storm to 'x10' added 526 mm of rain and only 35 mm of mrro (0.07); a catchment cannot absorb that much
        ✅  non_degenerate     partition and variability are non-trivial

The probes

Probe Law What it asks Status

Contribute a probe

    # get a template and fill it in
ht init-probe

# scaffold probes/<law>/<slug>/ from the filled draft
ht init-probe --from probe-draft.yaml

# prove it separates the reference models
ht gate --probe mass/my-probe

    Submit a model

    # the harness mounts /io and runs your entrypoint once
/io/request.json          read: case, seed, requested variables
/io/input/forcing.csv     read: time, pr, tas, pet
/io/output/result.csv     write: one row per forcing row

    Credit