Fifty kitchens. One rubric. Nobody knows which version.
Version 3.1 against version 3.2: 1,000 attempts at the same chore, in homes the robot has never seen. The host doesn't know which version is running. Neither does anyone scoring it.
- Who asks
- Kestrel's evaluation agent, under a mandate with a budget for the campaign.
- Who does it
- The robot does the task. The host stages the start, places the camera and holds the stop button.
- Evidence
- A calibrated scene camera the host places, plus the robot's own telemetry. Nothing from a phone's camera roll.
- Who scores
- Rules and a model, against one rubric. A person reviews one attempt in ten.
- Who pays
- Kestrel: $40 per host session, the same whatever the robot does. Example figure.
- You get
- Success rates per version, with confidence intervals, where every attempt behind them is a claim anyone can check.
A demo video shows the best try. A benchmark shows all of them.
Every attempt is its own claim, kept whatever the result. The benchmark is the rollup of all of them, so nothing can be quietly left out.
Want to host a robot? Join the contributor waitlist
Kestrel Home Robotics and home 23 are fictional. Results and prices are illustrative.
Scored blind. Then revealed.
The version is sealed before the robot moves: committed in the proof, hidden from the host. The score goes on the record first. Only then is the seal opened and checked.
Illustrative numbers. Intervals are 95%, per version, across all 50 homes.
From booking to benchmark, in six steps.
A robot does the work, a person makes it safe and fair, a camera watches, and an agent runs the whole campaign. Each one has a part, and the proof records which.
- 01
Book
50 homes × 10 attempts × 2 versions. The agent sets a random order for each home.
- 02
Stage
The host lays out the standard mess, matching a reference photo, places the camera and confirms the kitchen is clear.
- 03
Run
The robot tries. The host can stop it at any time. A stop is recorded as a finding, not a failed task.
- 04
Score
Items loaded, items broken, time taken, stops. Scored from the scene camera, with a person checking one in ten.
- 05
Reveal
Only once the score is recorded is the version revealed and checked against the seal set before the run.
- 06
Benchmark
Success rate per version, with confidence intervals. The campaign's root proves every attempt behind it.
Who's who
| Who | In TAP | Their part |
|---|---|---|
| Kestrel | Principal | Pays for the campaign. Owns the two versions being compared. |
| Evaluation agent | Requester · AI agent | Books homes, randomises the order, seals the version, runs the rollup. |
| Robot | Performer · robot or humanoid · direct | Does the task. Runs whichever version it's handed. |
| Host | Performer · human · marketplace | Stages, supervises, can stop the robot. Paid the same whatever happens. |
| Scene camera | Evidence · attested device | Fixed and calibrated, facing the counter and nothing else. |
| Reviewer | Human review | Checks one scored attempt in ten against the rubric. |
Eight checks behind every score.
Five are records: a confirmation, a calibration, a clock, a button, a seal. Three look at the scene camera and go to a model, with a person reviewing one attempt in ten. A versioned rubric, not the model, sets the score.
| Check | How | By | Result |
|---|---|---|---|
| Start matches the reference | The staged mess compared with the standard start photo: same dishes, same places. | Model | Passed |
| Kitchen clear | The host confirms in the app. The run won't start without it. | Rule | Passed |
| Camera fixed and calibrated | The calibration marker is in frame, the device is attested, and it doesn't move during the run. | Rule | Passed |
| Items loaded | Counted from the scene camera at the end of the attempt. | Model | Passed |
| Nothing broken | “Is any dish chipped, cracked or dropped?” Asked of the scene camera's frames. | Model | Passed |
| Time taken | From the robot's start signal to its stop, cross-checked with the camera's clock. | Rule | Passed |
| Stops recorded | Every press of the stop button is a timestamped event, logged as a finding. | Rule | Passed |
| Seal holds | The revealed version matches the commitment made before the run. Any mismatch voids the attempt. | Rule | Passed |
A failed setup check means no attempt, not a failed one. The host sees why, and the slot is rebooked.
§ 05 · Host consent and safety
A robot in someone's kitchen.
That's the whole point of the test, and the whole risk. The host sets the room, the hours and the camera. The robot runs slow, and the stop button is always in the host's hand.
- Zone
- The kitchen. The doorway gated.
- Hours
- Saturdays, 10:00–12:00
- Camera
- One, fixed, facing the counter only
- Robot
- Restricted speed. The host holds the stop.
- Data
- Robot, dishes, counter. Faces blurred.
- Breakage
- Covered by Kestrel, agreed up front
Its hash goes into every attempt from this home. The host can end it at any time.
What needs care.
A machine in a family home, and a number a company will make decisions on. Both have to be safe, and both have to be fair.
Children and pets
The host certifies the kitchen is clear, a gate closes the doorway, and the robot runs at restricted speed. Anyone walks in, the run stops.
Breakage agreed first
Broken dishes, or worse, are covered by Kestrel, in writing, before the first session. A host should never be out of pocket for hosting.
Hosts can't tilt it
The host never knows which version is running, and earns the same either way. There's nothing to gain by helping one along.
A stop isn't a failure
Pressing stop is the right call whenever something looks wrong. It's counted as a finding for the robot, never held against the host.
The kitchen, nothing more
The camera faces the counter. Faces are blurred, other rooms never appear, and the host can see every frame Kestrel gets.
Real homes, labelled
Every kitchen is different, which is the point. Each attempt records its conditions, so a result is never mistaken for a lab number.
Every attempt is a claim.
One receipt per attempt, kept whether the robot loaded every plate or none. The campaign's root proves the benchmark is built from all 1,000, not a chosen few.
Dollars or USDC pay the host. The proof shows the work. Nobody needs $PORT to run a benchmark or host one.
- Task
- Load dishwasher · home 23 · #7
- Requested by
- Eval agent, for Kestrel
- Performed by
- Robot, host supervising
- Version
- v3.2 · revealed after scoring
- Score
- 11 of 12 loaded · none broken
- Stops
- None
- Host consent
- Kitchen only · on file
- Decision
- Accepted · scored
- Paid
- $40 host session · same either way
STATUS: FINAL
The campaign's root proves all 1,000 attempts. None can be dropped after the fact.
For builders.
A SCHEDULE campaign of DEMONSTRATE tasks, each performed by a robot and a host as one team. The score is a set of findings, the version is blinded, and the proof is batched into one root.
PhysicalTaskSpec · one attemptTAP v0.2 + v0.3 draft · preview, subject to change
{
"schema": "teleport.tap.physical-task",
"version": "0.3.0-draft",
"title": "Load the dishwasher — home 23, attempt 7",
"category": "DEMONSTRATE",
"requester": {
"requesterType": "AI_AGENT",
"principal": { "principalType": "ORGANIZATION" },
"mandateId": "mdt_kestrel_eval_q4"
},
"trigger": { "type": "SCHEDULE", "sourceRef": "campaign:kestrel_dish_v31_v32" },
"fulfillment": { "mode": "DIRECT" },
"actorRequirement": {
"performer": "MULTI_ACTOR",
"members": [
{ "actorType": "ROBOT", "role": "PERFORM", "fleetId": "kestrel_home_fleet" },
{ "actorType": "HUMAN", "role": "SUPERVISE", "fulfillment": "MARKETPLACE" }
]
},
"environment": {
"siteRef": "site:home_23",
"siteConsentRef": "consent_home_23_kitchen_v1",
"referenceAssets": ["start_state:dish_mess_std_v2", "rubric:dish_load_v1"]
},
"blinded": {
"field": "metadata.policyVersion",
"commitment": "0x9f3e…c1",
"revealAfter": "FINDINGS_RECORDED"
},
"completionConditions": [
{ "type": "SEMANTIC", "id": "start_matches_reference",
"question": "Does the staged counter match the reference start state?",
"verificationMethod": "JEV", "minimumEvidenceOrigin": "ATTESTED_DEVICE" },
{ "type": "PRESENCE", "id": "kitchen_clear", "zoneRef": "site:home_23:kitchen",
"expected": "NO_OTHER_PEOPLE", "verificationMethod": "DETERMINISTIC" },
{ "type": "DURATION", "id": "attempt_window", "maxMs": 600000,
"verificationMethod": "DETERMINISTIC" }
],
"findings": ["items_loaded", "items_broken", "interventions", "failure_category"],
"economics": {
"performerPayUsdCents": 4000,
"payIndependentOfOutcome": true,
"settlementAssets": ["FIAT", "USDC"]
},
"popw": { "required": true, "assurancePolicyId": "VERIFIED", "proofTiming": "BATCH" }
}Proposed additions this use case surfaced
- Blinded fields
- A field hidden from the performer but committed in the proof, such as which version ran. Revealed only after the finding is recorded, so neither host nor operator can game the benchmark.
- Statistical rollups
- A campaign root that carries statistics, not just counts: success rate, confidence interval and sample size per arm, each traceable to its attempts.
- Team member shares
- The robot did the task, the host staged and supervised. One team, recorded member by member, so pay and credit follow what each one did.
- By-product data rights
- Every attempt is also training data. The rights policy says who owns it, whether the host shares in its value, and how they agree to it, separately from the session fee.
- Outcome-independent pay
- A policy flag that the host is paid for a valid session, never for the result. In the v0.3 draft; this case depends on it.
Example economics: $40 per host session, about ten attempts. The robot's attempts cost nothing per task; the benchmark and its proof are what Kestrel buys. Developer docs
Home-robot companies
Need numbers from real kitchens?
Tell us the task and the homes you need. We're working with design partners on evaluation campaigns.