Overwire Benchmark Pre-Registration#

Status: FROZEN v1.0. Amendments only via the changelog per the amendments section.

Date frozen: 2026-08-04 Author: Overwire project Home: overwire.ai

1. Thesis under test#

Robot intelligence can be metered from the cloud while the robot body approaches free. A ~$250 SO-ARM101 arm driven by a Raspberry Pi 5, executing a VLA policy served from an H100 on Modal (region us-east) over residential internet, folds a washcloth with success rate and completion time insensitive to network latency across the residential-to-degraded range (0-800 ms injected RTT). The 0 ms injected condition anchors the sweep, and a flat curve across the range brackets the local-inference counterfactual by interpolation. The headline 8/10 test at the residential condition is the sole confirmatory test; the RTT sweep carries a pre-registered support criterion of at least 7/10 at every injected level up to and including 400 ms, and the 800 ms level is exploratory stress only (section 9).

2. Task definition#

• Object: one standard cotton washcloth, approximately 30x30 cm, solid color, high contrast against the work surface. The exact cloth spec (product, dimensions, photo) is recorded at freeze, before episode one of the training dataset, so training cloths and benchmark cloths share one spec.
• Cloth maintenance: multiple identical cloths are purchased; the active cloth is swapped per a stated rotation rule, at minimum between the human baseline session, the headline session, and each sweep level, so wear does not accumulate into any single condition. Ambient humidity is recorded per session.
• Start state: crumpled. The start zone is divided into a labeled grid; the target cell is drawn by RNG before each trial and logged. Operator crumple protocol: lift the cloth by an arbitrary point, drop it from approximately 20 cm above the drawn cell, then take one overhead photo. If the cloth lands fully outside the start zone or folds on itself into a stack taller than 8 cm, redo the drop. No manual flattening or arranging after the drop.
• Goal state: quartered fold. The cloth folded twice (half, then half again) into four layers.
• Fold strategy is fixed at training time (the same quarter-fold strategy demonstrated in all training episodes). The benchmark does not reward alternative folds.
• Instruction given to the policy is fixed across all trials.

3. Success criteria (per trial)#

Trial end is defined mechanically: the trial ends when the end effector is outside the cloth region and all joint motion is below a stated velocity threshold (value recorded at freeze) for 5 consecutive seconds, or when the time cap expires, whichever comes first.

A trial is a SUCCESS if and only if, at trial end, all of the following hold:

1. The cloth forms four layers (two completed folds).
2. The final footprint is no more than 35% of the flat cloth area.
3. Layer misalignment: no edge of any layer overhangs the stack by more than 5 cm.
4. The cloth is inside the workspace boundary (not dragged off the mat, not on the floor).

Scoring mechanism: criterion 2 is computed by a published script in bench/ from the final overhead frame against the printed reference grid. Criteria 1, 3, and 4 are assessed from the published unedited video and are verifiable by any reader; no post-session judge decides them. Edge cases resolve conservatively (when in doubt, score FAILURE). A trial is a FAILURE if the time cap expires before the success criteria hold, if the arm requires an emergency stop, or if the rig is touched by a human during the trial.

4. Trial count and threshold#

• N = 10 scored trials at the headline network condition.
• Pre-registered threshold: at least 8 of 10 trials succeed.
• Exact binomial confidence intervals are reported alongside the 8/10 outcome.
• Trials are consecutive: no discarding, no retries, no cherry-picking. Hardware breakage (servo failure, power loss) pauses the session; the interrupted trial is scored FAILURE unless the breakage is verifiably unrelated to the policy's commands (documented with video), in which case the trial is re-run once.
• Practice runs before the scored session are allowed but must be announced and logged before trial one begins.

5. Policy freeze#

• Before trial one, the policy checkpoint hash and the complete serving configuration are recorded, including flow/denoising step count and any quantization. The identical checkpoint and configuration serve all headline and sweep trials.
• Disclosure: a benchmark attempt that misses the 8/10 threshold is reported, not discarded. Re-attempts with a new checkpoint are permitted, but every attempt is disclosed in the tech report.
• Client freeze: the Pi client configuration is recorded at freeze and identical across all conditions: control rate, chunk length, blend steps, and the request scheduling rule. The request margin is not a constant; it is an adaptive function of measured round-trip time (an EWMA of recent chunk RTTs converted to control steps, plus a fixed headroom, clamped to at most half the chunk length), and the function itself, with its parameters, is frozen and published (src/overwire/pi_chunk_executor.py). Hand-tuning any client parameter per network condition is prohibited; the frozen adaptive rule is what runs at every sweep level.

6. Time bound protocol#

• Human baseline: one human operator folds the same cloth from the same crumple protocol, one-handed, using the table, embodiment-matched to the single-arm robot, by hand, filmed, for 10 trials. The median of the 10 one-handed fold times is the baseline time. The film is published with the benchmark results.
• A two-handed fold session is also filmed and published as context; it sets no bound.
• Ordering: the baseline session is filmed and its median computed BEFORE the scored robot session begins.
• Timing rule (both human and robot): the clock starts when the first motion toward the cloth begins and stops when the manipulator (hand or gripper) last leaves a cloth that satisfies the success criteria.
• Pre-registered bound: the median robot fold time across successful trials must be under 2x the human baseline median.
• Per-trial hard cap: 5x the human baseline median; a trial that exceeds it is scored FAILURE and stopped.

7. Network conditions#

• Headline condition: residential internet, as installed at the test site. ISP name, advertised plan, and a same-day speed test (up, down, idle ping) are recorded and published with the results.
• Policy served from a warm Modal H100 container, region us-east, over a persistent websocket with TCP_NODELAY.
• The site router runs standard SQM (fq_codel or CAKE) with its configuration published. No per-flow prioritization of robot traffic; no enterprise links, no colocation, no LAN inference.
• The full latency decomposition (frame age, capture, encode, uplink, ingress, inference, downlink, splice, servo write) is recorded per chunk request and published as HDR histograms with p50/p95/p99.

8. Rig cost accounting#

• Pre-registered claim: the on-site rig costs under $500.
• Counted: arm (SO-ARM101 follower), Raspberry Pi 5 and its PSU and SD card, both cameras, mounts, cables, work mat and lighting used by the rig (already-owned items count at current replacement price per the valuation rule below).
• Valuation: owned equipment is counted at current replacement price; components acquired in bundles at the vendor's standalone price or a documented pro-rata split of the bundle price.
• Not counted: the leader arm (teleop and data collection only, not part of the deployed rig), the workstation (training and evals only), cloud compute (metered separately and reported in the cost analysis section of the tech report), and the internet subscription.
• The workstation GPU (RTX 4060) runs teleop, ACT baseline training, and local pipeline-validation evals only. No benchmarked inference ever runs on it; all benchmarked runs are Modal-served.
• Hailo-8 HAT: physically removed from the Pi for all v1 benchmark runs; it does not appear in the bill of materials.
• An itemized bill of materials with purchase prices is published with the results.

9. RTT sweep protocol (the money chart)#

• After the headline benchmark, the same task is run under injected latency: 0, 50, 100, 200, 400, 800 ms of extra RTT, each with 10 ms jitter, applied on egress at the Pi via tc netem (bench/netem.sh).
• Jitter is applied order-preserving (netem with a pfifo child queue). netem's default time-ordered queue reorders packets under jitter, which triggers TCP fast-retransmits that silently pay the injected delay twice per large message; that is an emulator artifact, not a property of a single-path residential link.
• The sweep level order is drawn by RNG before the sweep begins and logged, so cloth wear and operator drift do not correlate with injected latency.
• Per level: 10 trials under the identical trial protocol above.
• Published per level: success rate, median fold time of successful trials, and the chunk-request RTT distribution actually measured during the trials (injected delay plus real path).
• Stalls and watchdog trips are counted and published per level; coordinated omission is not permitted (a stalled trial still counts its elapsed time).

Confirmatory and exploratory split#

• The headline 8/10 result at the residential condition (section 4) is the sole confirmatory test in this benchmark.
• Support criterion: success rate at or above 7/10 at every injected level up to and including 400 ms.
• The 800 ms level is exploratory stress: reported in full, binding nothing.

10. Amendments#

Any change to this document after freezing requires a changelog entry below stating what changed, why, and whether any data had been collected under the prior version. Changes after data collection begins are discouraged and must be flagged prominently in the tech report.

Changelog#

• v1.0 (2026-08-04): FROZEN. Client freeze added to the policy freeze section: control rate, chunk length, blend steps, and the request scheduling rule are fixed across all conditions, with the request margin defined as a frozen adaptive function of measured chunk RTT (EWMA converted to control steps, plus fixed headroom, clamped to half the chunk length), implemented and published in src/overwire/pi_chunk_executor.py; per-condition hand-tuning prohibited. Insensitivity defined: the headline 8/10 at the residential condition is the sole confirmatory test; the sweep carries a support criterion of at least 7/10 at every level up to and including 400 ms; the 800 ms level is exploratory stress, binding nothing. Cloth spec recorded at freeze, before episode one, shared between training and benchmark cloths. Work mat and lighting counted as used by the rig, at replacement-price valuation. Sweep level order drawn by RNG before the sweep begins and logged. Hailo-8 HAT resolved: physically removed for all v1 benchmark runs, absent from the bill of materials. Zero training episodes and zero benchmark data existed at freeze.
• v0.3 (2026-08-04): review edits from the planning session. Thesis restated as latency insensitivity across 0-800 ms injected RTT, with the 0 ms condition anchoring the sweep and bracketing the local-inference counterfactual by interpolation. New policy freeze section: checkpoint hash and full serving configuration recorded before trial one, identical across all trials, with every attempt disclosed and failed attempts reported rather than discarded. Human baseline is one-handed and embodiment-matched, filmed and computed before the scored robot session; the two-handed fold is published as context only. Mechanical trial-end definition (end effector clear of cloth, joint motion below threshold for 5 s, or time cap). Scoring by published script (footprint) plus published unedited video (layers, overhang, boundary); no post-session judge. Site router runs SQM (fq_codel or CAKE) with configuration published. Crumple placement drawn by RNG from a labeled grid and logged. Cloth rotation rule and per-session humidity recording. Valuation rules for owned and bundled equipment; workstation GPU (RTX 4060) exclusion stated; Hailo HAT remove-or-count flagged as the single open item for human decision at freeze. Sweep levels interleaved or randomized and logged. Exact binomial confidence intervals reported alongside the 8/10 outcome. v0.1 open items closed: cloth spec is recorded at freeze; footprint threshold 35% adopted; drop height 20 cm adopted; per-trial hard cap 5x adopted as stated in the body; judge identity resolved by script plus published video, no judge.
• v0.2 (2026-08-04): specified order-preserving jitter in the RTT sweep protocol, after the first transport-only sweep showed netem's default queue reordering large uplink messages and doubling the effective delay via TCP retransmits. No benchmark data collected yet; document still in draft.
• v0.1 (2026-08-04): initial draft for human review. Open items to resolve before freezing: exact cloth spec and footprint threshold (35% is provisional), crumple drop height, per-trial hard cap multiplier, judge identity (single judge vs. two judges with tiebreak).