openai/prm800k

PRM800K underpins one of the most cited process-supervision papers in AI research and has fed dozens of open reward-model projects since its 2023 release. With 2,151 stars and 129 forks, it remains the most-referenced human-labeled dataset for step-level math reasoning verification.

Key capabilities:

• •800,000 step-level labels human annotators rated each reasoning step in model-generated MATH solutions as correct, neutral, or incorrect (+1, 0, -1).
• •Two labeling phases phase 1 collected alternative-step ratings; phase 2 used PRM-guided generation and stopped labeling at the first error, per the released instructions docs.
• •Custom MATH train/test split 4,500 of the original MATH test problems were folded into training to prevent overfitting, leaving 500 held-out test problems.
• •Scored samples for reproduction up to 1,860 scored samples per test problem let researchers reproduce the paper's best-of-N ORM/PRM evaluation with eval/eval.py.
• •Answer-grading logic included grader.py uses sympy-based expression matching, built on Hendrycks' MATH normalization, to check model answers against ground truth.
• •Git LFS delivery the label files and MATH splits ship as newline-delimited JSON via Git LFS, so a standard git clone needs LFS installed first.

Benchmarks:

The accompanying paper Let's Verify Step by Step (arXiv 2305.20050) reported that a PRM trained on this data solved 78.2% of a representative MATH test subset, beating an outcome-supervised reward model trained on the same problem count. That result helped establish process supervision as the training recipe behind today's reasoning models.

How to use it:

Clone with Git LFS enabled, then read the data folder's newline-delimited JSON files directly, or load a pre-processed copy from Hugging Face (tasksource/PRM800K or trl-lib/prm800k are commonly used mirrors for TRL-based training). Run python eval/eval.py --method prm to reproduce the paper's PRM evaluation, or --method orm for the ORM baseline.

Limitations:

The repository is archived and has not been pushed to since June 2023, so there are no active maintainers responding to the 7 open issues. That is expected for a finished dataset release rather than ongoing software, but it means schema questions or edge cases in the raw JSON go unanswered by the original team. Labels cover only MATH-style problems, so the dataset does not generalize to code, multi-turn dialogue, or other reasoning domains. Working with the raw format requires writing custom parsing logic; most practitioners use a Hugging Face mirror or a wrapper library instead of the raw JSON directly.