Pre-registered AI predictions for the highly demanding of rigor

Machine learning isn’t in Kansas anymore. Its not as simple as evaluating on a static held-out test set. Once a test set is public, it ends up in the next model’s training data and the score stops meaning what it meant before. ModelMirror builds evaluations that cannot leak by construction: models predict real-world events before they happen, the predictions are committed to a public, timestamped ledger, and the world supplies the answers.

How it works

• Pre-registration by timestamp. Predictions are committed to a public git repository before outcomes exist. What a model knew is fixed by when it wrote, not by restricting what it could read.
• A public ledger. Every prediction, evaluation, and outcome is a file in a public repository. You don’t need to trust us, you can verify.
• Underclaiming. Small samples are reported as small samples. We want to focus on sound methodology and careful interpretation, not virality.

Projects

• FedCourtsAI
  AI models predict U.S. Supreme Court events — cert grants, votes, and reasoning — on a public ledger that begins with the October 2026 term.
  Live. First forward predictions by three frontier models resolve in October 2026.
• AI and the billable hour
  A small economic model of what AI does to law firms that sell hours. Its central parameter, whether AI can close the expertise gap on the hardest work, is what FedCourtsAI tests.
  Analysis, not a benchmark or prediction. Illustrative parameters; static tables and charts, code on GitHub.

Contact

Issues on the relevant GitHub repository are the best channel. Occasional updates may be posted at @ModelMirrorAI.