# Test parsers and TS strategies

Evaluate a custom trajectory parser or TS search strategy against a frozen benchmark dataset.

Test changes to a trajectory reaction parser or a transition-state (TS) search strategy against a frozen benchmark dataset before running production jobs.

## 1. Check the benchmark CLI commands

The `chemistry-benchmarks` package (included in `compute/runtime`) provides ready-made benchmark adapters for trajectory parsers and TS strategies:

```bash
ct-cli benchmark parser --help
ct-cli benchmark ts --help
```

| Benchmark family       | Candidate contract                                       | How it is scored                                                                                        |
| ---------------------- | -------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- |
| **`benchmark parser`** | Extracts reaction events from MD trajectories            | Event-matching tolerances and micro-F1 score                                                            |
| **`benchmark ts`**     | Proposes a saddle-point geometry (`run_ts_search(task)`) | Strict Hessian + IRC verification ([Search and verify a transition state](/science/transition-states/)) |

## 2. Smoke-test your candidate locally

Before submitting a full cluster evaluation, run your candidate script on a single task locally to catch syntax or contract errors immediately.

For a trajectory parser with a local dataset built:

```bash
ct-cli benchmark parser try parser.py --limit 1
```

For a TS search strategy (`strategy.py`), validate the script contract locally with `check`:

```bash
ct-cli benchmark ts check compute/ts/examples/strategy.py
```

> **Tip:** Keep unit tests fast by testing your helper logic with fake model handles locally, then run real GPU checkpoint evaluations on the cluster.

## 3. Freeze or select your dataset

Every benchmark evaluation runs against an immutable dataset snapshot:

- **TS benchmark datasets:** Use `ct-cli benchmark ts db` to inspect local task sets and publish a frozen dataset snapshot with explicit verification rules.
- **Parser benchmark datasets:** Use the parser dataset commands to build and publish labeled trajectories from a reference simulation set.

## 4. Submit a full benchmark evaluation

Submit your candidate script to run across the frozen dataset on your worker pool. The CLI packages your script as a versioned Program, schedules the evaluation, and returns an `EVALUATION_UUID`.

Inspect progress and item-level evidence using the standard [benchmark commands](/test/benchmarks/):

```bash
ct-cli benchmark status EVALUATION_UUID
ct-cli benchmark evidence EVALUATION_UUID
```

When comparing two versions of an algorithm:

- **Publish a new candidate version** each time you edit your parser or TS strategy code.
- **Inspect item-level failures:** A parser exception or an unconverged TS search counts as a failed item in the benchmark summary—never filter failed tasks out of the denominator.
- **Track speed alongside accuracy:** Follow [Measure throughput and efficiency](/test/throughput/) to make sure a more accurate strategy still meets your runtime budget.