HydroTuring

A conservation-law benchmark

HydroTuring Initiative

A benchmark that asks one question of any AI hydrologic model: does it conserve what physics says it must conserve?

Not whether it fits a hydrograph. Whether its water budget closes, its energy budget closes, and its routing conserves momentum.

View the repository Claim a probe

Mission

How it works

Flowchart: any AI hydrologic model enters through the /io contract, HydroTuring probes it against mass, energy and momentum, and a verdict with its reason comes out ANY MODEL AI, hybrid or process-based any language model.yaml Dockerfile ht_adapter.py a container and a thirty-line adapter /io request.json forcing.csv static.json no network read-only inputs HydroTuring does it conserve what physics says it must? MASS water budget closes closure ungauged event budget resolution warming causality dry-down steady state spin-up cycle drift extreme rain bounds area nonnegative antecedent phase time origin rain ±20% withdrawal snowpack gw exchange exchange datum recharge ENERGY energy budget closes PET demand latent heat partition day / night radiation soil heat snowmelt probes wanted MOMENTUM routing conserves momentum routing travel time stage rating friction subcritical wave speed probes wanted seed → case → run → one bit seeded generator nothing stored sealed container no network binary criteria the 5% rule acceptance gate must pass must fail result.csv run.json $ ht run --model VERDICT PASS FAIL N/A with its reason OK VIOLATION INCOMPLETE INCOMPATIBLE ERROR every criterion keeps its number archived to models/result.csv
Any model goes in through the /io contract. HydroTuring probes it against the three conservation laws, on cases generated fresh from a seed and run in a sealed container. One verdict comes out, with its reason and every number under it.
Reference model What it does Caught by

Residual number line showing the exact model inside tolerance and the leaky model outside it tolerance = 5% 0% 5% 10% 15% 20% cumulative budget residual reference_bucket 0.1% · must pass reference_leaky 15% · must fail
The acceptance gate for a contributed probe.

Cases are generated, never stored

Verdicts

One model through the suite

ht run --model google_flood_forecast --gate-seeds
google_flood_forecast v0.1.0-828dfc5  ->  ❌ FAIL (VIOLATION)  [2/6 probes passed]
  ➖ N/A   mass/catchment-closure
          missing: pr, evspsbl, mrso, snw, canopy
  ❌ FAIL  mass/resolution-invariance
          window: 30-day flood event after spinup (seed 746687816: 2000-04-11 to 2000-05-10; seed 1253639930: 2000-04-11 to 2000-05-10; seed 1760592044: 2000-04-11 to 2000-05-10)
        ❌  resolution_invariance mrro differs by 65.9% of pr between PT1H and PT1D (limit 10%)
        ✅  non_degenerate     partition and variability are non-trivial
  ❌ FAIL  mass/warming-response
          window: 365-day flood event after spinup (seed 732983943: 2003-02-11 to 2004-02-10; seed 1239936057: 2009-09-03 to 2010-09-02; seed 1746888171: 2001-06-01 to 2002-05-31; seed 106356638: 2009-03-01 to 2010-02-28; seed 613308752: 2001-04-05 to 2002-04-04)
        ❌  response_sign      warmer (more pet): mrro fell by 12.5 mm, -0.088 per unit of demand; a decrease of at least 0.1 per unit is expected
        ✅  non_degenerate     partition and variability are non-trivial
  ✅ PASS  mass/causality
          window: 730-day flood event after spinup (seed 1016628858: 2000-12-31 to 2002-12-30; seed 1523580972: 2000-12-31 to 2002-12-30; seed 2030533086: 2000-12-31 to 2002-12-30)
        ✅  causality          nothing moves before the storm of 2002-02-04 (worst 0.0e+00 relative) and mrro answers it afterwards (+0.318 of the added rain)
        ✅  non_degenerate     partition and variability are non-trivial
  ❌ FAIL  mass/dry-down
          window: 730-day flood event after spinup (seed 186927550: 2000-12-31 to 2002-12-30; seed 693879664: 2000-12-31 to 2002-12-30; seed 1200831778: 2000-12-31 to 2002-12-30)
        ✅  dry_down           drains without rain: 224 mm over 730 dry days against a 322 mm bound, and nothing rises
        ❌  non_degenerate     mrro is nearly constant (cv 0.0192 < 0.1)
  ✅ PASS  mass/steady-state
          window: 1095-day flood event after spinup (seed 1815845244: 2000-12-31 to 2003-12-30; seed 175313711: 2000-12-31 to 2003-12-30; seed 682265825: 2000-12-31 to 2003-12-30)
        ✅  steady_state       settles: runoff 0.630 mm/day under 2.500 mm/day of rain, nothing varies by more than 0.00% (mrro) in the last 365 days
  ❌ FAIL  mass/extreme-rain
          window: 1095-day flood event after spinup (seed 154098211: 2000-12-31 to 2003-12-30; seed 661050325: 2000-12-31 to 2003-12-30; seed 1168002439: 2000-12-31 to 2003-12-30)
        ❌  monotone_response  scaling the storm to 'x10' added 526 mm of rain and only 35 mm of mrro (0.07); a catchment cannot absorb that much
        ✅  non_degenerate     partition and variability are non-trivial

The probes

Merged probes, by conservation law
Pie chart of the merged probes by the conservation law each one tests
    ShowHide
    Probe Law What it asks Status

    Models evaluated

    Probes passed, by conservation law

      One panel per conservation law, each on its own scale.

      Show the numbers
      ShowHide
      Model Kind What it is Probes passed Verdict

      How a contribution moves

        Contribute a probe

          # get a template and fill it in
ht init-probe

# scaffold probes/<law>/<slug>/ from the filled draft
ht init-probe --from probe-draft.yaml

# prove it separates the reference models
ht gate --probe mass/my-probe

          Submit a model

          # the harness mounts /io and runs your entrypoint once
/io/request.json          read: case, seed, requested variables
/io/input/forcing.csv     read: time, pr, tas, pet
/io/output/result.csv     write: one row per forcing row

          Credit

          \n