Human-in-the-Loop vs Human-on-the-Loop

Human-in-the-loop (HITL) means a human approves every action before it takes effect. Human-on-the-loop (HOTL) means the agent acts and a human supervises by exception — reviewing samples, watching drift, intervening on alerts. They are not stages on a maturity ladder to climb as fast as possible; they are cost and reliability tradeoffs to assign per task. The assignment rule: reversibility of the action × blast radius of failure × task length, weighed against the agent's proven reliability on your distribution — never its advertised reliability. Get the assignment right and supervision is affordable oversight; get it wrong and you either strangle throughput with approval queues or delegate irreversible mistakes to software that will make them at machine speed.

The reliability evidence makes "climb the ladder" the wrong instinct. METR's measurement research (arXiv 2503.14499) found that the length of software tasks AI agents complete with 50% reliability has been doubling roughly every seven months — impressive capability growth — but also that reliability falls off a cliff beyond the horizon: agents that succeed on short tasks fail the same task type when the task gets long.[1] Task length is a risk variable, not just a convenience variable. And the capability is jagged, in the sense documented by Dell'Acqua and colleagues' randomized field experiment with 758 BCG consultants: large gains inside the capability frontier (12.2% more tasks, 25.1% faster, over 40% quality), no gains — and misjudged confidence — outside it, with the boundary invisible to the consultants themselves.[3] An autonomy assignment that ignores either finding is a forecast, and forecasts are not controls.

Executive Summary

• HITL prices every action in human attention; HOTL prices failures in blast radius. The assignment per task decides which cost you pay.[2]
• Reliability evidence governs: 50%-success task horizons double ~7 months (METR, software benchmarks, domain variation caveat), and success collapses beyond the horizon — long chains need decomposition or checkpoints, not hope.[1]
• The assignment matrix plots reversibility × blast radius into four modes: HITL, sampled review, HOTL, and never-agent — with task length as an overlay.[3]
• Supervision capacity is finite: attention budgets, queue latency targets and anti-rubber-stamping checks are design artifacts, not management hopes.
• Autonomy is earned by evidence: N clean eval cycles + bounded blast radius + proven rollback → promotion at the review board; never promoted because a stakeholder is impatient.[4]

The three modes, defined precisely

Mode	Who initiates	Who approves	Latency	Human cost per action	Agent throughput	Error containment
HITL (in the loop)	Agent proposes	Human approves every action	Approval-bound (minutes to days)	High — every action	Low	Strongest: nothing executes unapproved
Sampled review	Agent acts	Human audits a risk-tiered sample	Real-time	Moderate — the sample only	Medium	Statistical: catches systematic drift, misses isolated errors
HOTL (on the loop)	Agent acts	Human intervenes on alert/exception	Real-time	Low — exceptions only	High	Weakest pre-failure: relies on monitors and rollback

A fourth row belongs on the table: out of the loop entirely — no monitoring, no review. It is a legitimate mode for trivial, reversible, low-blast-radius actions (drafting internal text with no downstream commitment), and a governance hole anywhere else.

The table's tradeoff line is the whole argument: as you move down the rows, agent throughput rises and human cost per action falls — and error containment weakens. There is no row that is better than the others; there are only tasks for which each row is right. The hybrid that most mature deployments converge on is sampled review with escalation: nothing queues for full approval, everything is sampled by risk tier, and anomalies escalate to humans — the mode the matrix below labels precisely.

The reliability evidence, read carefully

Three findings, three implications.

Task horizons double — but cliffs remain. METR's longitudinal measurement (on software task suites; their follow-up work shows horizons vary by domain) found the ~50%-success task length doubling about every seven months across model generations.[1] Implication: today's frontier is not next year's, so autonomy assignments carry a review date; and a task that is "too long" today may be reliable next year — the promotion protocol below exists for exactly this.

Success collapses beyond the horizon. An agent at 80% on 20-minute tasks may be near-zero on 8-hour versions of the same task, because errors compound along the chain. Implication: long workflows are not one autonomy assignment but several — decomposed into segments with human checkpoints at the joints, where errors are caught before they compound. The team-design analysis covers who staffs those checkpoints.

The frontier is invisible at the point of work. The 758-consultant experiment's consultants could not reliably tell which tasks were inside the capability band — and neither can your front line, however experienced.[3] Implication: autonomy assignments are organizational decisions made on evaluation evidence (your golden set, your distribution), written into the mission's autonomy contract — not personal judgments made in the flow of work, where optimism compounds.

Gartner's agent-reality guidance compresses all three into the operating rule: agent success hinges on reliability, not autonomy — scope autonomy to task criticality and reversibility, and expand it as evidence accumulates.[2]

The assignment matrix

Plot every agent action class on two axes: reversibility (can the action be undone at reasonable cost?) and blast radius (if wrong, how far does the damage spread — internal draft vs. customer-facing vs. contractual vs. safety-relevant?).

• Low reversibility, high blast radius → HITL. Irreversible actions with external consequences: contract terms, payments above thresholds, pricing to customers, anything regulated. A human approves every instance; agents prepare the decision package (evidence, sources, risks) so approval is fast and informed.
• High reversibility, high blast radius → sampled review + tight monitors. Customer-visible but correctable: content drafts, support replies, recommendations. Agents act; humans audit a risk-tiered sample; anomaly monitors watch drift; rollback is rehearsed.
• Low reversibility, low blast radius → sampled review or HITL by cost. Internal but permanent: record updates, data writes. The choice is arithmetic — approval cost vs. correction cost.
• High reversibility, low blast radius → HOTL or out of the loop. Internal drafts, internal search, formatting. Monitor lightly or not at all.

Two overlays complete the matrix. Task length: any action chain longer than the agent's demonstrated horizon on your distribution gets decomposed with checkpoints — the overlay converts "one long task" into "several short ones," which the reliability math rewards.[1] Volume: at high volume, HITL is not just slow but degrading — approval queues back up, and the mode silently converts itself into rubber-stamped sampled review with none of the sampling discipline. If volume forces a mode change, make it deliberately, with the monitoring that mode requires.

Attention budgeting: the constraint that bites

HITL converts compute cost into human-attention cost, and attention is the scarcest resource in the building. The budget is arithmetically honest: review minutes per thousand actions × volume × loaded hourly cost — a real, forecastable line item, which the cost stack analysis books next to inference. The design rules follow from the arithmetic:

Sample inversely to observed error rate. A new agent at high error rates gets heavy sampling; as the golden set and production monitors show error rates falling, sampling relaxes — attention flows to where errors actually live. This is the NIST AI RMF's Measure function in daily practice: continuous, proportional monitoring rather than one-off audits.[4]

Track review latency, and defend against rubber-stamping. Queue depth and median review time are team metrics; a "review" taking eight seconds is a signature. The countermeasure is a small second-reviewer sample that checks whether first reviewers actually caught what the golden set says they should have. Fake HITL — approvals without examination — is worse than honest HOTL, because it produces the audit trail of oversight with none of its substance.

Escalate on drift, not on schedule. Monitors watch the input distribution, output distribution, override rate and escalation rate; movement triggers human attention. The NIST framework's Manage function exists for this: acting on what measurement shows, proportionately and in time.[4]

The promotion protocol

Moving a task up the ladder — HITL → sampled → HOTL — is an evidence-gated decision, and the protocol is short enough to memorize:

1. N consecutive clean eval cycles on the golden set at the current mode (N set by risk tier — say, 4 cycles for internal actions, 8 for customer-facing).
2. Blast radius still bounded at the proposed mode — reversibility unchanged, monitors in place.
3. Rollback proven — rehearsed at the proposed mode, not assumed.
4. Proposal at the review board with the eval trend lines attached; approval recorded in the ledger with the date and evidence.

The protocol's power is what it forbids: promotion because a stakeholder is impatient, because a vendor's roadmap slide says so, or because the queue is long (the queue is a staffing and sampling problem, not an autonomy problem — though if the queue is chronically full at HITL for a high-reversibility action, that is itself evidence the action belongs in the sampled-review quadrant, and the protocol will find it). The governance model places this protocol inside the five-control stack, where the autonomy contract and audit trail live.

Key Takeaways

• HITL, sampled review and HOTL are per-task cost/reliability tradeoffs — not maturity stages; the matrix (reversibility × blast radius, with task-length and volume overlays) assigns them.[2]
• Reliability collapses beyond the task horizon; horizons double ~7 months but vary by domain — decompose long chains, re-review assignments as capability moves.[1]
• The frontier is invisible at the point of work: assignments are organizational decisions on evaluation evidence, written into autonomy contracts.[3]
• Attention is a budget: sample by observed error rate, track review latency, spot-check review quality — fake HITL is worse than honest HOTL.
• Autonomy is earned: clean cycles + bounded blast radius + proven rollback + board approval; every promotion recorded, every stall investigated.[4]

What Leaders Should Do Next

1. List every live agent action class and place it on the matrix. (Mission leads + governance partner, first 30 days.) Reversibility and blast radius are the axes; the exercise takes an afternoon and routinely surfaces actions nobody consciously assigned.
2. Instrument the attention budget. (Mission leads, first 30 days.) Review minutes, queue depth, median review latency, override rate — per agent, on the weekly review. Rubber-stamp signatures (sub-10-second approvals) get named and investigated.
3. Decompose the two longest agent chains in the portfolio. (Agent engineers, first 60 days.) Break them at natural joints; place human checkpoints where errors compound; re-measure reliability per segment against the golden set.[1]
4. Write the promotion protocol into the governance standard. (Review board, first 60 days.) N-by-risk-tier, evidence attached, ledger-recorded — and the first promotion decided by it, whatever the outcome, to establish that the protocol is real.
5. Quarterly: re-assign, don't accumulate. (Review board, quarterly.) Every autonomy contract carries a review date; the METR doubling trend means last year's assignments are this year's opportunities — and some of this year's assumptions are already stale.[1]

References

1. METR — Measuring AI Ability to Complete Long Tasks, arXiv:2503.14499 (2025, revised). arxiv.org — ~7-month doubling of 50%-success task length; reliability collapse beyond horizon; software benchmarks, domain variation per follow-up work.
2. Gartner — AI Agent Success Hinges on Reliability, Not Autonomy, 2025-2026. gartner.com — analyst guidance on reliability and reversibility-scoped autonomy.
3. Dell'Acqua et al. — Navigating the Jagged Technological Frontier, HBS WP 24-013 (2023), Organization Science (2025), 10.1287/orsc.2025.21838. pubsonline.informs.org — 758-consultant randomized field experiment; frontier-dependent gains, invisible boundary.
4. NIST — Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1, Jan 26, 2023. airc.nist.gov — Govern/Map/Measure/Manage; continuous, proportional risk management.