ContractBench-DROID Lite: review-context evaluation for robot policies.
A benchmark protocol for studying whether robot policy data carries safety context, replay records, and evidence needed to evaluate learned behavior around physical systems.
Review context
Measures whether robot episodes can be represented as bounded, reviewable action with explicit operating context.
Derived benchmark layer
Uses derived research metadata to evaluate public robot datasets without changing the underlying dataset.
Evidence-first evaluation
Treats replayability, review context, and evidence references as first-class benchmark outputs, not afterthoughts.
Robot learning benchmarks usually measure task success, but deployment teams also need to know whether learned behavior can be checked, constrained, replayed, and audited before it is trusted near physical systems.
ContractBench-DROID Lite introduces a review-context evaluation protocol for robot policy data. Instead of re-hosting raw robot data, it publishes derived metadata summaries, coverage metrics, and missing-metadata taxonomy.
The Lite v0 split uses public robot-data references from distinct collection contexts. The first baseline shows that review summaries can be represented consistently, while deployment-critical safety metadata remains largely absent from the public episode references.
Why this benchmark exists.
Robot learning benchmarks usually focus on task success. Deployment teams also need to know whether a policy output can be checked, constrained, replayed, and explained under real physical assumptions.
ContractBench-DROID Lite asks a deployment-facing question: does the data path expose enough structure to evaluate whether an action would be reviewable under explicit operating constraints?
Evaluation dimensions
Review completeness
Whether each episode carries enough action, observation, timing, and robot-context information for review.
Safety metadata coverage
Whether workspace bounds, contact limits, stop conditions, and operator context are available or must be inferred.
Replayability score
Whether observations, actions, timestamps, language goals, and calibration references are sufficient for reviewable deployment analysis.
Evidence coverage
Whether an episode can produce a reviewable record with validation status and failure annotations.
Benchmark construction.
The Lite protocol is designed as a derived evaluation layer over public robotics datasets. The first target is DROID-style real-world robot manipulation data, summarized into review and evidence fields.
1. Reference
Link upstream public episodes, dataset versions, and source identifiers.
2. Normalize
Map actions, observations, timing, and robot context into review fields.
3. Validate
Run coverage checks and mark missing deployment-critical metadata.
4. Record
Emit evidence summaries that can be cited and reviewed.
Lite v0 source set.
Lite v0 uses public DROID references without exposing raw episode identifiers on this page. The artifact stores references and derived metadata only; it does not re-host raw robot video, sensor streams, actions, trajectories, or annotation payloads.
| Source type | Public note |
|---|---|
| Public dataset subset | Derived references only |
| Distinct collection contexts | Raw data is not re-hosted |
| Review-oriented records | Metadata summaries only |
DROID is credited as the upstream dataset. The DROID project page describes 76k demonstration trajectories, 350 hours of interaction, 564 scenes, and 86 tasks.
Lite v0 baseline results
| Metric | Value | Interpretation |
|---|---|---|
| Review model validity | Complete | The public review model contains the expected top-level categories. |
| Evidence summary validity | Complete | The public evidence summaries contain the expected top-level categories. |
| Split reference validity | Complete | The split rows reference known public-data examples. |
| Review completeness | High | Most review fields are populated; unavailable fields remain explicit. |
| Safety metadata coverage | Low | Safety-critical fields are not present in the current public episode references. |
| Replay field coverage | Partial | Most replay-critical reference fields are present or inferable. |
| Average replay readiness | Partial | The current public references are partially replay-ready, with blockers. |
| Evidence reference coverage | Complete | Each summary has a source reference and review record. |
These values are generated from the Lite derived metadata layer. They measure reference and metadata coverage, not robot task success.
Missing metadata taxonomy
The repeated missing fields show the gap between robot-learning data and review evidence. This does not mean the upstream dataset is deficient; DROID was designed as a large-scale robot learning dataset, not an industrial evidence package.
ContractBench-DROID Lite measures the additional metadata layer needed when learned behavior is evaluated around physical-system review.
| Missing field | Count |
|---|---|
| camera_calibration | 3 |
| contact_limits | 5 |
| nominal_control_hz | 1 |
| operator_context | 5 |
| robot_world_transform | 5 |
| safety_zones | 5 |
| stop_conditions | 5 |
| workspace_bounds | 5 |
Reproducibility.
The research artifact contains derived research materials, evidence examples, a dataset card, baseline metrics, and a validator. The validator regenerates the result table from the derived metadata files.
The Xolver-authored derived materials are intended for CC BY 4.0 release after final approval. The artifact does not include or relicense upstream DROID raw data or annotation payloads.
Upstream attribution.
ContractBench-DROID Lite uses references to DROID: A Large-Scale In-the-Wild Robot Manipulation Dataset, and the public DROID annotation repository maintained on Hugging Face.
Citation
@misc{xolver_contractbench_droid_lite_2026,
title = {ContractBench-DROID Lite: Review-Context Evaluation for Robot Policies},
author = {Xolver Research},
year = {2026},
howpublished = {Xolver Research},
url = {https://xolver.ai/research/contractbench-droid/}
}Please also cite the upstream DROID dataset when using the ContractBench-DROID Lite references.
Research artifacts
Review model
Defines the robot context, action and observation coverage, timing, workspace, safety policy, and calibration assumptions at a public-summary level.
Evidence summary
Records upstream references, validation status, field coverage, replay readiness, and review notes for each episode.
Lite v0 split
A small public-data subset used for research validation.
Validator and metrics
A reproducible baseline that reports metadata coverage, replay readiness, and missing fields.
Scope and limitations.
ContractBench-DROID Lite evaluates the data and evidence path around robot policy behavior. It does not certify hardware deployment by itself.
Its purpose is to make the review gap measurable: safety context, replay records, and evidence must be visible before learned behavior can be responsibly evaluated near physical systems.
The Lite v0 split is intentionally small and should not be interpreted as representative of the full DROID dataset. The current task-family label is coarse because the first pass uses upstream episode references without re-hosting annotation payloads.
Partner with us on deployment-grade evaluation.
Xolver is building the measurement layer between learned robot policies and physical-system review. We work with teams that care about evidence, safety context, and accountable records.
Contact Xolver