chemistry.toolsDocs
Search documentationSearch
Open workspace
GuidesTest and improve

Test parsers and TS strategies

Evaluate a custom trajectory parser or TS search strategy against a frozen benchmark dataset.

15 minRead as Markdown
Before you start

The shared runtime (`compute/runtime`) and a benchmark dataset.

Local smoke-test results and a scored remote benchmark evaluation.

Test changes to a trajectory reaction parser or a transition-state (TS) search strategy against a frozen benchmark dataset before running production jobs.

1. Check the benchmark CLI commands

The chemistry-benchmarks package (included in compute/runtime) provides ready-made benchmark adapters for trajectory parsers and TS strategies:

bash
ct-cli benchmark parser --helpct-cli benchmark ts --help
Benchmark familyCandidate contractHow it is scored
benchmark parserExtracts reaction events from MD trajectoriesEvent-matching tolerances and micro-F1 score
benchmark tsProposes a saddle-point geometry (run_ts_search(task))Strict Hessian + IRC verification (Search and verify a transition state)

2. Smoke-test your candidate locally

Before submitting a full cluster evaluation, run your candidate script on a single task locally to catch syntax or contract errors immediately.

For a trajectory parser with a local dataset built:

bash
ct-cli benchmark parser try parser.py --limit 1

For a TS search strategy (strategy.py), validate the script contract locally with check:

bash
ct-cli benchmark ts check compute/ts/examples/strategy.py

Tip: Keep unit tests fast by testing your helper logic with fake model handles locally, then run real GPU checkpoint evaluations on the cluster.

3. Freeze or select your dataset

Every benchmark evaluation runs against an immutable dataset snapshot:

  • TS benchmark datasets: Use ct-cli benchmark ts db to inspect local task sets and publish a frozen dataset snapshot with explicit verification rules.
  • Parser benchmark datasets: Use the parser dataset commands to build and publish labeled trajectories from a reference simulation set.

4. Submit a full benchmark evaluation

Submit your candidate script to run across the frozen dataset on your worker pool. The CLI packages your script as a versioned Program, schedules the evaluation, and returns an EVALUATION_UUID.

Inspect progress and item-level evidence using the standard benchmark commands:

bash
ct-cli benchmark status EVALUATION_UUIDct-cli benchmark evidence EVALUATION_UUID

When comparing two versions of an algorithm:

  • Publish a new candidate version each time you edit your parser or TS strategy code.
  • Inspect item-level failures: A parser exception or an unconverged TS search counts as a failed item in the benchmark summary—never filter failed tasks out of the denominator.
  • Track speed alongside accuracy: Follow Measure throughput and efficiency to make sure a more accurate strategy still meets your runtime budget.