Benchmark with AIRS-Bench: a Suite of Tasks for Frontier AI Research Science Agents

[codelion]

https://github.com/facebookresearch/airs-bench

[vje013]

Hi @codelion . Vlad from Darwin Adaptive Systems, we're an eval company running on gen algos and do work on open source eval standards. We would like to be assigned this issue.

Proposed scope for first PR:

• Adapter mapping AIRS-Bench's <problem, dataset, metric, SOTA> spec to OpenEvolve's (initial_program, evaluator, config) triple
• One task end-to-end first. Proposing U0MolecularPropertyPredictionQm9MeanAbsoluteError. Clean regression metric. Works with unbounded scoring.
• Example dir at examples/airs_bench/, mirroring existing example structure
• Clear extension path for the remaining 19 tasks

Two questions before I start:

1. Directory layout preference. Single examples/airs_bench/ with per-task subdirs, or one example per task?
2. airsbench as optional dependency, or expect users to install separately?

[codelion]

• directory layout is fine.
• for benchmarks we usually put the required dependencies in their own requirements.txt file in the example folder.