The Human-in-the-Loop Catalog

Source: github.com/langchain-ai/langgraph (MIT; Python and TypeScript)

Intent

Let the agent author mark places in the graph where execution should pause for human input, and let the runtime persist the paused state for resumption when input arrives.

Motivating Problem

LangGraph models agent execution as a graph of nodes operating on typed state. Within that model, the natural place for an approval gate is between specific node transitions: pause before invoking the tool-execution node, pause before committing to a destructive action, pause whenever the agent’s next move would cross a defined risk threshold. LangGraph exposes two ways to mark these gates.

How It Works

The dynamic interrupt() function (added in late 2024) is callable from inside any node. When invoked, it pauses execution at that exact point, persists the state via the configured checkpointer (in-memory, SQLite, Postgres, LangSmith Deployment, formerly LangGraph Platform / LangGraph Cloud), and surfaces the value passed in as the payload to the caller. Resume by calling graph.invoke(Command(resume=value), config=thread_config) with the same thread_id; the value becomes interrupt()‘s return value, and the interrupted node re-executes from its beginning, so code before the interrupt() call runs again. Dynamic interrupts let the pause condition depend on runtime state (“pause only if the transaction amount exceeds $1,000,” “pause only if the agent’s confidence is below 0.7”).

The static interrupt_before and interrupt_after options on graph.compile() pause unconditionally before or after named nodes. The docs now treat them as debugging breakpoints for stepping through a graph and recommend interrupt() for all human-in-the-loop gates.

LangChain v1 wraps interrupt() for tool-calling agents as HumanInTheLoopMiddleware, configured per tool through interrupt_on (see Declarative per-tool approval in agent SDKs below).

The durability comes from the checkpointer choice (Section B touches on durable backends; LangGraph itself defers to the persistence layer). For development the in-memory checkpointer is enough; for production the Postgres checkpointer is the recommended default, with LangSmith Deployment as the managed alternative.

When to Use It

Any LangGraph agent that needs approval gates. interrupt() for every human-in-the-loop gate, fixed or conditional; static interrupt_before / interrupt_after for stepping through the graph while debugging.

Alternatives --- Temporal Signals (Section B) when the wait may be measured in days and the application benefits from Temporal’s broader durable-execution surface. CrewAI human_input for the simpler check-the-box-on-a-task pattern. HumanInTheLoopMiddleware when the gates are per tool rather than per node.

Sources

• docs.langchain.com/oss/python/langgraph/interrupts

• docs.langchain.com/oss/python/langchain/human-in-the-loop

Example artifacts

Code.

from langgraph.graph import StateGraph, END

from langgraph.checkpoint.postgres import PostgresSaver

from langgraph.types import interrupt, Command

# Static breakpoint (debugging): pause before the 'tools' node

graph = (

StateGraph(AgentState)

.add_node("planner", planner_node)

.add_node("tools", tools_node)

.add_edge("planner", "tools")

.add_edge("tools", END)

.compile(

checkpointer=PostgresSaver(conn_string=DB_URL),

interrupt_before=["tools"], # pause every time, unconditionally

)

)

# Dynamic: pause from inside a node, only when a condition holds

def refund_node(state: AgentState):

if state["refund_amount"] > 500:

approval = interrupt({

"question": "Refund exceeds threshold. Approve?",

"amount": state["refund_amount"],

"customer": state["customer_id"],

})

if not approval["approved"]:

return {"status": "denied", "reason": approval.get("reason")}

# ... proceed with refund ...

return {"status": "refunded"}

# Resume after human approval:

thread_config = {"configurable": {"thread_id":
"alice-session-42"}}

result = graph.stream(

Command(resume={"approved": True}),

config=thread_config,

)

Source: github.com/crewAIInc/crewai (Python, MIT)

Intent

Let the crew author mark specific tasks as requiring human review of the output before the crew advances.

Motivating Problem

CrewAI’s model is task-shaped rather than graph-shaped: a crew is a list of tasks, each with an assigned agent. The natural granularity for approval in this model is the task: when this specific task completes, pause for a human to review the output and either accept it or send the task back with revision instructions.

How It Works

On a Task, set human_input=True. When the task completes, CrewAI prints the task output to the console and waits for the human to respond with either an acceptance or revision instructions. Revision instructions become input to the agent for a re-run of the task; acceptance allows the crew to advance to the next task.

The pattern works well for the regulated-workflow case where each task represents a discrete review-and-approve unit: a legal document drafted, a financial recommendation prepared, a customer email composed. It works less well for fine-grained tool-level approvals --- those need LangGraph’s interrupt model or the framework-agnostic approaches in Section G.

human_input is a console prompt. For non-console use, CrewAI documents two other routes: the @human_feedback decorator for Flows (CrewAI 1.8.0+), which routes on the reviewer’s feedback, and webhook-based HITL for crews deployed on CrewAI Enterprise, which can reach reviewers in Slack or Teams.

When to Use It

CrewAI deployments where the natural approval grain is the task. Co-pilot-style workflows where the agent prepares deliverables and the human accepts or revises.

Alternatives --- LangGraph interrupts when the approval grain is finer than the task. Manual orchestration when the workflow doesn’t fit CrewAI’s task model.

Sources

• docs.crewai.com/en/concepts/tasks

• docs.crewai.com/en/learn/human-in-the-loop

Example artifacts

Code.

from crewai import Agent, Task, Crew

legal_reviewer = Agent(

role="Legal Reviewer",

goal="Review contract drafts for compliance issues",

backstory="Senior corporate lawyer with 15 years of experience",

)

review_task = Task(

description="Review the attached contract for compliance issues.",

expected_output="A bulleted list of issues with severity ratings.",

agent=legal_reviewer,

human_input=True, # pause after this task for human review

)

crew = Crew(agents=[legal_reviewer], tasks=[review_task])

result = crew.kickoff(inputs={"contract": contract_text})

Source: OpenAI Agents SDK, Microsoft Agent Framework, LangChain v1, Vercel AI SDK (all open source)

Intent

Mark individual tools as requiring human approval when the agent is defined, and have the SDK stop before running such a tool, return the pending calls, and accept an approve or reject decision that resumes the same run, even in a different process.

Motivating Problem

Gating tools by hand on the raw tool_use loop (Section G) works, but every team rewrites the same pieces: a list of risky tools, code to stop the loop, a way to save the conversation, and a way to feed the decision back. Since 2025 the main agent SDKs have built this in. The risk decision moves into the tool definition, next to the code that does the damage, and the SDK owns the pause, the saved state, and the resume.

How It Works

The shape is the same across SDKs: a per-tool flag or policy, a run result that lists pending approvals instead of executing them, and a resume call that takes the human’s decisions.

• OpenAI Agents SDK: needs_approval=True (or an async function that decides per call) on a tool. Pending calls appear in result.interruptions as ToolApprovalItem entries; result.to_state() gives a RunState that can be saved with to_json() and restored with RunState.from_json(); state.approve(…) or state.reject(…) records the decision and Runner.run(agent, state) resumes. Local MCP servers take require_approval for the same effect.

• Microsoft Agent Framework (the successor to AutoGen): approval_mode=“always_require” on the @tool decorator in Python, or wrapping a function in ApprovalRequiredAIFunction in .NET. The response carries approval requests with the function name and arguments; the application answers with an approval response built from the request (to_function_approval_response in Python).

• LangChain v1: HumanInTheLoopMiddleware(interrupt_on={…}) maps tool names to True, False (auto-approve) or a config listing allowed_decisions (approve, edit, reject, respond) and an optional when predicate. It is built on LangGraph interrupt() and needs a checkpointer.

• Vercel AI SDK: toolApproval on generateText, streamText or ToolLoopAgent, per tool (for example refund: ‘user-approval’) or as one function for all calls; the older per-tool needsApproval flag is deprecated. In the UI, the tool part enters the approval-requested state and the client answers with addToolApprovalResponse({ id, approved }). Provider-executed tools ignore the setting.

Approval can also be conditional: OpenAI’s needs_approval function, LangChain’s when predicate, and the AI SDK’s per-tool approval function all decide from the actual arguments, so a refund under a threshold runs and a larger one pauses.

When to Use It

Any agent built on one of these SDKs where some tools are risky and others are not: the risk-tiered point on the approval spectrum (Chapter 2). Teams that want approval rules reviewed in the same diff as the tool code.

Alternatives --- The raw tool_use loop (Section G) when no SDK is in use. LangGraph interrupt() when the gate belongs at a graph node rather than a tool. Temporal or Step Functions (Section B) when the wait may last days and the saved run state needs a durable home; the SDKs serialize the state but leave storing it to the application. MCP elicitation (Section G) when the tool server, not the application, needs to ask.

Sources

• openai.github.io/openai-agents-python/human_in_the_loop/

• learn.microsoft.com/en-us/agent-framework/agents/tools/tool-approval

• docs.langchain.com/oss/python/langchain/human-in-the-loop

• ai-sdk.dev/docs/ai-sdk-core/tools-and-tool-calling

Example artifacts

Code.

from agents import Agent, Runner, RunState

from agents.decorators import tool

\@tool(needs_approval=True)

async def issue_refund(order_id: int, amount: float) -> str:

return f"Refunded {amount} on order {order_id}"

agent = Agent(name="Support agent", tools=[issue_refund])

result = await Runner.run(agent, "Refund order 4417 in full")

if result.interruptions:

state = result.to_state()

save(state.to_json())  # persist the paused run for a long wait

# Later, after the human decides:

state = await RunState.from_json(agent, load())

for item in result.interruptions:

state.approve(item)  # or state.reject(item)

result = await Runner.run(agent, state)

Source: github.com/temporalio/temporal (MIT; Go server with SDKs in many languages)

Intent

Provide durable execution for long-running workflows, with Signals as the mechanism by which external events (including human approvals) deliver into a running workflow regardless of how many process restarts have happened.

Motivating Problem

Production HITL for high-stakes workflows must tolerate every kind of interruption: process crashes, deploys, AWS region failures, the human approver going home for the weekend. The naive approach --- keep the workflow state in memory and wait --- fails the moment the process exits. Temporal’s answer is replayable workflows: every step’s inputs and outputs are recorded in an external history; on restart, the framework replays the history to reconstruct the workflow’s current execution point; external events (Signals) get delivered to the live workflow at whichever location it’s currently waiting.

How It Works

A workflow function is written in code (Python, TypeScript, Java, Go) using the Temporal SDK. The function looks like ordinary code with awaits and conditions; the framework transforms its execution into a record of inputs and outputs persisted to the Temporal server. When the workflow awaits workflow.wait_condition(), execution pauses; the framework remembers exactly where; the worker process can crash and restart without losing any progress.

Signals are typed messages an external system sends to a specific workflow run; the workflow defines a handler for each as a method decorated with @workflow.signal. For HITL: when an agent reaches an approval gate, the workflow calls await workflow.wait_condition(lambda: self.approval_received) and pauses. The approval UI, when the human clicks Approve, sends the Signal with await client.get_workflow_handle(workflow_id).signal(RefundWorkflow.submit_approval, payload). The handler sets self.approval_received = True; the wait_condition resolves; execution continues. When the approver needs a synchronous, validated acknowledgement (for example, rejecting an approval for a request that has already expired), use an Update (@workflow.update) instead of a Signal.

Operational maturity is the differentiator. Temporal runs in production at major financial services firms, healthcare companies, and large-scale tech platforms with workflows that span days to weeks. Built-in retry, timeout, and compensation primitives cover the workflow patterns that pure agent frameworks handle clumsily.

When to Use It

High-stakes HITL workflows where the wait may be measured in days and the cost of losing state is unacceptable. Multi-step workflows with complex retry, timeout, and compensation requirements. Workflows that span multiple actors (multiple humans, multiple systems) coordinating over extended periods.

Alternatives --- LangGraph + Postgres for lighter-weight agent-shaped workflows where Temporal’s broader power isn’t needed. AWS Step Functions for AWS-native deployments where the integration with other AWS services matters more than cross-cloud portability.

Sources

• github.com/temporalio/temporal

• docs.temporal.io/ai/cookbook/human-in-the-loop-python

• docs.temporal.io/develop/python/workflows/message-passing

Example

A high-value wire transfer workflow at a financial services firm. The agent prepares the transfer; the workflow calls await workflow.wait_condition(lambda: self.approved) and pauses. Two human approvers receive notifications. The first approver signs in, reviews the transfer, sends an approve_first Signal. The second approver does the same a few hours later. The wait_condition becomes true; the workflow proceeds to execute the transfer via the bank’s API. If the worker process crashes any time during the multi-hour wait, the workflow resumes from the persisted state on the new worker process with no loss of progress.

Example artifacts

Code.

# Python workflow (Temporal SDK)

from temporalio import workflow

from dataclasses import dataclass

\@dataclass

class ApprovalInput:

approver: str

approved: bool

reason: str = ""

@workflow.defn

class RefundWorkflow:

def __init__(self):

self.approval: ApprovalInput | None = None

@workflow.signal

def submit_approval(self, payload: ApprovalInput):

self.approval = payload

@workflow.run

async def run(self, refund_request: RefundRequest) -> RefundResult:

# Step 1: agent prepares the refund (an Activity, retried by
Temporal)

prepared = await workflow.execute_activity(

prepare_refund, refund_request,

start_to_close_timeout=timedelta(minutes=5),

)

# Step 2: pause for human approval - may wait hours or days

await workflow.wait_condition(lambda: self.approval is not None)

if not self.approval.approved:

return RefundResult(status="denied", reason=self.approval.reason)

# Step 3: execute the refund

result = await workflow.execute_activity(

execute_refund, prepared,

start_to_close_timeout=timedelta(minutes=10),

)

return result

# From the approval UI, when the human clicks Approve:

async def on_approve_clicked(workflow_id: str, approver: str):

client = await Client.connect("temporal:7233")

handle = client.get_workflow_handle(workflow_id)

await handle.signal(

RefundWorkflow.submit_approval,

ApprovalInput(approver=approver, approved=True),

)

Source: aws.amazon.com/step-functions (managed; SDKs in all AWS-supported languages)

Intent

Provide AWS-native durable workflow orchestration, with the task-token callback pattern as the mechanism for pausing a workflow until an external event (including human approval) triggers resumption.

Motivating Problem

For AWS-native deployments, Step Functions is the path of least resistance for durable workflow orchestration. The wait-for-callback pattern (.waitForTaskToken in the Step Functions integration syntax) is the equivalent of Temporal’s Signal: the workflow pauses, an external system holds a token, when the external system invokes SendTaskSuccess with the token, the workflow resumes.

How It Works

A state in the state machine is defined with the .waitForTaskToken modifier (e.g., “Resource”: “arn:aws:states:::sqs:sendMessage.waitForTaskToken”). When the workflow reaches this state, Step Functions invokes the underlying service (in this example, SQS) with a special $$.Task.Token parameter; the workflow then pauses indefinitely. The external system --- the approval UI, an SQS consumer, or any code path --- calls SendTaskSuccess(taskToken, output) when ready. The workflow resumes with the provided output as the state’s result.

Common HITL pattern: a state of type Task with the wait-for-callback variant sends a notification (via SNS, SES, or Lambda) to the approver, including the task token in the notification payload. The approval UI calls SendTaskSuccess when the human approves or SendTaskFailure when they reject. Step Functions handles the persistence; the human can take days, but the wait is bounded by the one-year execution quota, so set HeartbeatSeconds or TimeoutSeconds to avoid stuck executions. Wait-for-callback requires a Standard workflow; Express workflows support only the request-response pattern.

When to Use It

AWS-native deployments where Step Functions is already the chosen workflow orchestrator. Workflows that integrate with AWS services (Lambda, SQS, SNS, Bedrock) and want the managed-orchestration story. Cases where cross-cloud portability isn’t a requirement.

Alternatives --- Temporal when cross-cloud portability or feature breadth matters. LangGraph + Postgres for lighter-weight agent-shaped workflows.

Sources

• docs.aws.amazon.com/step-functions/latest/dg/connect-to-resource.html

Source: smith.langchain.com (LangChain Inc.; commercial SaaS plus self-hosted option)

Intent

Provide the observability story for LangChain and LangGraph applications: granular nested traces of every agent run, dataset-based evaluation, prompt versioning, human-feedback capture, and the audit surface for production agent systems.

Motivating Problem

LangChain and LangGraph are the most-used agent frameworks; LangSmith is LangChain Inc.’s commercial platform spanning observability, evaluation and deployment (LangSmith Deployment, formerly LangGraph Platform). The integration is the deepest of any tracing platform: every LangChain or LangGraph call is auto-traced; the trace tree mirrors the framework’s execution model exactly; tools for prompt versioning, evaluation, and human review of runs are first-class. For teams already on LangChain or LangGraph, LangSmith is the path of least friction; other frameworks can send traces through the LangSmith SDK or OpenTelemetry.

How It Works

Set LANGSMITH_API_KEY and LANGSMITH_TRACING=true; every LangChain or LangGraph operation auto-traces to the configured project. The UI shows the trace tree (thread → run → step → LLM call → tool call), filterable by time, status, latency, or token count. Drill-down on any run shows full inputs, outputs, intermediate state, and timing.

Beyond raw tracing: dataset features let teams capture sets of input/output pairs and run them through new agent versions for regression testing. Evaluators (built-in and custom) score outputs against criteria. Annotation queues route specific runs to human reviewers who tag them with thumbs-up or thumbs-down feedback. Prompt versioning tracks every iteration of a prompt with diff views and the ability to roll back.

When to Use It

Production LangChain and LangGraph deployments. Teams that want first-party tooling from the framework vendor. Workflows where prompt iteration and evaluation are heavy and the integrated platform is worth the commercial cost.

Alternatives --- Arize Phoenix for the self-hosted, source-available option with OpenTelemetry support. Langfuse for the open-source option with stronger feedback-capture features. Both work fine with LangChain via instrumentation. See also Volume 4, AgentOps (tracing event-triggered runs), and Volume 14, Agent observability platforms.

Sources

• docs.langchain.com/langsmith

Source: github.com/Arize-ai/phoenix (Elastic License 2.0; Python; runs locally or self-hosted)

Intent

Provide LangSmith-class observability features as source-available (ELv2) software that is free to self-host, with OpenTelemetry as the trace transport and the option to run entirely on-premises or in a local notebook.

Motivating Problem

Many teams cannot send their agent traces to a third-party SaaS for compliance, cost, or principle reasons. Phoenix is the self-hosted answer: the same trace-tree mental model, the same drill-down and filtering UI, the same evaluation and feedback features, but running on the team’s own infrastructure. The use of OpenTelemetry as the transport means any OTel-instrumented agent framework can send traces to Phoenix --- not just LangChain.

How It Works

Install arize-phoenix; either pip install and run px.launch_app() in a notebook for the local mode, or docker-compose for the self-hosted server mode. Configure the agent framework with the OpenInference instrumentation library for that framework (openinference-instrumentation-langchain, -openai, -llamaindex, -openai-agents, and many more); traces flow into Phoenix via the OpenTelemetry collector.

The UI has the same shape as LangSmith’s: trace tree views, filterable list views, drill-down into spans with full input/output capture. The evaluations surface lets teams run LLM-as-judge or rule-based evaluators against captured traces. The notebook integration is unusual --- Phoenix runs inside Jupyter or Colab for the prototype workflow, which makes the on-ramp shorter than competitors.

Operational: data stays in the deployer’s infrastructure (local disk for the notebook mode, configurable backend for self-hosted). No external SaaS dependency. Arize the company sells managed Phoenix and other AI observability products on top, but the self-hosted Phoenix is the same code. The Elastic License 2.0 is not an OSI open-source license: it allows free self-hosting but forbids offering Phoenix to others as a managed service.

When to Use It

Self-hosted requirements. OpenTelemetry-instrumented agent stacks where Phoenix’s OTel-native architecture matches. Local development workflows where launching the trace UI in a notebook beats managing a separate service. Teams that need the audit posture of “the traces never leave our infrastructure.”

Alternatives --- LangSmith for the LangChain-first managed option. Langfuse for the MIT-licensed alternative with stronger UI features for feedback capture. See also Volume 14, Agent observability platforms.

Sources

• github.com/Arize-ai/phoenix

• arize.com/docs/phoenix

Source: github.com/langfuse/langfuse (MIT; TypeScript and Python SDKs; self-hosted or cloud)

Intent

Provide an open-source observability and evaluation platform with first-class human-feedback capture (thumbs up/down, custom scores), prompt management, and dataset-based evaluation, deployable as self-hosted or as Langfuse Cloud.

Motivating Problem

Where Phoenix’s position emphasizes OpenTelemetry-native integration and notebook-first development, Langfuse’s position emphasizes UI quality and explicit human-feedback workflows. The dashboards are visually polished; the feedback capture is built around the production case of “users thumb up or down on agent responses” and the audit case of “a reviewer scores a sample of runs each day”; the prompt-management features include first-class versioning with diff views.

How It Works

Install the langfuse SDK; the current Python SDK is built on OpenTelemetry. Instrument with the @observe decorator or langfuse.start_as_current_observation(…) (the client comes from get_client()), or send OpenTelemetry spans from another instrumentation library; traces flow to the Langfuse server. The UI shows traces, scores, prompt versions, dataset runs, and dashboards. Score capture is a first-class operation: an application can call langfuse.create_score(…), or span.score(…) inside an observation, with arbitrary metric names and values, which surface in the dashboards as filterable metrics.

The feedback workflow is unusually well-developed. Production agents can log user feedback (thumbs-up/down clicks, ratings, custom comments) directly to Langfuse against specific traces. Internal reviewers can work through queues of traces in a structured review UI. Both feed into datasets used for evaluation and prompt iteration.

Operational: self-hosted via docker-compose, Langfuse Cloud for the managed offering. The MIT license covers the core product including evaluation and annotation features; enterprise security features under ee/ directories are separately licensed. ClickHouse, Inc. acquired Langfuse in January 2026; the project remains open source and self-hostable.

When to Use It

Teams that prioritize feedback capture and audit-friendly UI over framework integration depth. Deployments that must self-host. Workflows where user feedback and reviewer scores are the primary quality signal.

Alternatives --- LangSmith for the LangChain-native managed option. Phoenix for the OpenTelemetry-native self-hosted option. The three options serve the same need with different operational and feature emphases. See also Volume 14, Agent observability platforms.

Sources

• github.com/langfuse/langfuse

• langfuse.com/docs/observability/sdk/overview

Source: github.com/Arize-ai/openinference (Apache-2)

Intent

Define a vendor-neutral standard for representing LLM calls, tool calls, agent steps, and retrieval operations as structured trace spans, with attribute conventions that make the resulting traces portable across observability backends.

Motivating Problem

Before OpenInference, every observability product had its own trace schema; instrumenting an agent for LangSmith required different code than instrumenting it for Phoenix, and switching backends meant re-instrumenting. OpenInference defines a single schema with named span kinds (LLM, CHAIN, TOOL, RETRIEVER, RERANKER, EMBEDDING, AGENT, GUARDRAIL, EVALUATOR, PROMPT) and standardized attributes (input.value, output.value, llm.model_name, llm.token_count.prompt and .completion); instrumentation libraries that emit OpenInference-conformant spans work with any compliant backend.

How It Works

The OpenInference specification is a documented schema. The instrumentation libraries (openinference-instrumentation-langchain, -openai, -openai-agents, -llamaindex, -crewai, -smolagents, plus packages for the Claude Agent SDK, Microsoft Agent Framework and Strands Agents, and many more) wrap the major agent frameworks and emit conformant spans. The spans flow via OpenTelemetry transport to any compatible backend --- Phoenix natively, LangSmith via the OpenInference import path, Langfuse via the OpenInference integration, custom backends via direct OTel collection.

For agent applications: install the appropriate openinference-instrumentation-* package, configure the OpenTelemetry exporter pointing at the chosen backend, and the agent traces flow automatically. Multi-backend setups configure multiple exporters and the same spans go everywhere.

When to Use It

Any agent stack where observability backend portability matters. The default starting point for new instrumentation --- OpenInference is the broadest-supported standard as of mid-2026 and the path of least lock-in.

Alternatives --- vendor-specific instrumentation when the backend is fixed and the additional features (deeper integration, native sampling) are worth the lock-in. The OpenTelemetry GenAI semantic conventions for the broader cross-vendor approach. For correlating spans across cooperating agents, see Volume 9, Multi-agent tracing across LangSmith, Phoenix, Langfuse.

Sources

• github.com/Arize-ai/openinference

• github.com/Arize-ai/openinference/blob/main/spec/semantic_conventions.md

Source: github.com/open-telemetry/semantic-conventions-genai (CNCF; vendor-neutral)

Intent

Extend OpenTelemetry’s semantic conventions with LLM-specific attributes (model name, token counts, prompt and completion contents, agent step types) so that agent telemetry flows through the existing OpenTelemetry infrastructure used for general application observability.

Motivating Problem

Many enterprises have substantial existing investment in OpenTelemetry: their applications, databases, and infrastructure all emit OTel traces and metrics; their backends (Datadog, Grafana Tempo, Honeycomb, etc.) consume OTel-shaped data. Adding agent telemetry to this stack should not require a separate parallel observability infrastructure. The OpenTelemetry GenAI semantic conventions define the attribute names and span shapes that LLM-aware applications should emit so that the existing OTel pipeline handles agent telemetry alongside everything else.

How It Works

The GenAI semantic conventions specify attribute names: gen_ai.provider.name (the model provider; it replaced the deprecated gen_ai.system), gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, opt-in gen_ai.input.messages and gen_ai.output.messages for prompt and response content (replacing the removed gen_ai.prompt and gen_ai.completion), and many more. Agent operations are spans; LLM calls have a defined span kind; tool calls are spans with tool-specific attributes; the full agent trace is a tree of these spans with parent-child relationships standard in OpenTelemetry.

Status as of October 2026: every GenAI span and attribute, including the core LLM call attributes, is at Development status; none is Stable yet. In June 2026 (semconv v1.42.0) the GenAI conventions moved out of the main semantic-conventions repository into open-telemetry/semantic-conventions-genai, which has no tagged release yet. The OpenInference spec and the OTel GenAI conventions are actively cross-pollinating.

Because the spans are ordinary OpenTelemetry spans, trace context propagates across service and event boundaries: an event carries trace headers and the downstream consumer continues the trace, so one distributed trace can show the whole event, agent and tool chain. Implementations include Pydantic Logfire (whose pydantic-ai instrumentation emits the conventions automatically) and any OTLP backend such as Honeycomb, Grafana Tempo, Datadog or Jaeger.

When to Use It

Enterprises with existing OpenTelemetry infrastructure that want agent telemetry to flow through the same pipeline. Cases where vendor neutrality matters more than the depth of any specific backend integration. Future-proof instrumentation that can adapt as the standards mature.

Alternatives --- OpenInference for the more agent-specific spec with broader instrumentation library support today. Vendor-specific instrumentation for maximum integration depth at the cost of portability. See also OpenTelemetry GenAI semantic conventions, the next entry.

Sources

• github.com/open-telemetry/semantic-conventions-genai

• opentelemetry.io/docs/specs/semconv/gen-ai/

• opentelemetry.io/blog/2025/ai-agent-observability/

Source: github.com/streamlit/streamlit (Apache-2; Python)

Intent

Serve as the agent UI when the broader Streamlit application surface (dashboards, multi-page apps, non-chat UI) is valuable and the chat-component subset is sufficient for the agent interaction.

Motivating Problem

Many teams already use Streamlit for internal Python tools --- dashboards, data exploration, simple workflow apps. Adding an agent to the existing Streamlit application is significantly less work than introducing a second framework. The native st.chat_message and st.chat_input primitives (added 2023) provide enough surface for a serviceable chat UI, and the rest of the Streamlit application can surround the chat with the additional context the user needs.

How It Works

Install streamlit; use st.chat_input for the user input box, st.chat_message for displaying messages, and the same st.session_state for conversation history. The chat surface integrates with the rest of the Streamlit application (sidebar controls, multi-page navigation, data displays, file uploaders) as ordinary Streamlit components.

Streamlit doesn’t natively understand agent concepts. Step display, tool-call approval, and other agent-specific features have to be built using ordinary Streamlit primitives: st.expander for collapsible step views, st.button + st.session_state for approval gates. The trade-off: more flexibility and a broader application surface in exchange for more wiring.

When to Use It

Teams already on Streamlit who want to add an agent without introducing a second framework. Applications where the chat is one component of a broader app surface (dashboards plus agent, data exploration plus agent, internal tool plus agent).

Alternatives --- Gradio for the demos-and-prototypes case. A React chat UI built on the libraries in Volume 13 (Vercel AI SDK UI and generative UI, AI Elements and CopilotKit) when the agent is the product and needs a custom frontend. Streamlit fits best when the agent is a feature of something larger.

Sources

• docs.streamlit.io

Source: platform.claude.com/docs/en/agents-and-tools/tool-use/overview

Intent

Use the tool_use / tool_result handshake as the foundational pause-and-resume mechanism: when the model emits a tool_use, the application has full control over when and how to return a tool_result, including indefinite delays for human approval.

Motivating Problem

Most discussions of HITL focus on framework primitives, which obscures that the underlying capability is in the LLM API itself. The model emits a tool_use block; the API call returns; the application is in control. There is no requirement that the application execute the tool immediately. The application can wait an arbitrary time, store the tool_use to a queue, surface it to a human, wait for the human decision, and then return a tool_result that carries either the real output or the human’s refusal. The model’s subsequent API call resumes from the conversation state as if no pause occurred.

How It Works

The Anthropic API’s messages.create() returns a message that may include one or more tool_use blocks. The application inspects the tool_use, decides whether and how to execute the tool, and adds a tool_result block to the conversation history. The next messages.create() call passes the updated history; the model continues from where it paused, now with the tool_result in context.

For HITL: when the application receives a tool_use that requires approval, the application persists the entire conversation state (including the pending tool_use) to durable storage, notifies the appropriate human via the appropriate channel, and waits. When the human responds, the application either constructs the tool_result (if approved) or returns a tool_result whose content says the human denied the call (optionally with is_error: true), followed by any redirecting text in the same user message, and resumes the conversation. The API requires every tool_use to be answered by tool_result blocks at the start of the immediately following user message, so a plain user message in its place is rejected. The model has no way to tell whether the tool was executed in 50 milliseconds or 50 hours; the persistence and resumption are entirely the application’s concern.

The same pattern works across all major LLM API providers: OpenAI’s function calling, Google’s Gemini tool use, Mistral’s tool support. The mechanics differ slightly in syntax; the underlying pattern is identical. It applies to client-executed tools only: provider-executed server tools (web search, code execution) run on the provider’s side and cannot be paused this way.

When to Use It

Applications that want HITL without framework adoption. Custom frameworks that need to implement HITL as a primitive. Cases where the framework abstractions don’t fit the application’s exact persistence or approval routing needs. Teaching examples where understanding the foundational mechanism matters.

Alternatives --- the framework primitives in Section A when they fit the application’s shape. Worth understanding the foundational pattern regardless, because it clarifies what the frameworks are doing and where their abstractions leak.

Sources

• platform.claude.com/docs/en/agents-and-tools/tool-use/overview

• platform.claude.com/docs/en/agents-and-tools/tool-use/handle-tool-calls

Example artifacts

Code.

import anthropic

client = anthropic.Anthropic()

def run_with_hitl(conversation: list, tools: list) -> dict:

"""Run the agent until it produces a non-tool-use response or
requires approval."""

while True:

msg = client.messages.create(

model="claude-opus-4-7",

max_tokens=4096,

tools=tools,

messages=conversation,

)

conversation.append({"role": "assistant", "content":
msg.content})

# Find any tool_use blocks

tool_uses = [b for b in msg.content if b.type == "tool_use"]

if not tool_uses:

return {"status": "complete", "final_message": msg,
"conversation": conversation}

# For each tool_use, decide: execute, approve, or reject

tool_results = []

for tu in tool_uses:

if requires_approval(tu):

# Persist conversation state, notify human, return for resumption
later

save_to_durable_store(conversation, pending_tool_use=tu)

notify_approver(tu)

return {"status": "pending_approval", "tool_use": tu,
"conversation": conversation}

else:

result = execute_tool(tu.name, tu.input)

tool_results.append({

"type": "tool_result",

"tool_use_id": tu.id,

"content": result,

})

conversation.append({"role": "user", "content": tool_results})

def resume_after_approval(conversation: list, tool_use, approved:
bool):

"""Resume the agent run after the human approves or rejects."""

if approved:

result = execute_tool(tool_use.name, tool_use.input)

tool_result = {"type": "tool_result", "tool_use_id":
tool_use.id, "content": result}

else:

tool_result = {

"type": "tool_result", "tool_use_id": tool_use.id,

"content": "Tool call denied by human reviewer.", "is_error":
True,

}

conversation.append({"role": "user", "content":
[tool_result]})

return run_with_hitl(conversation, tools=available_tools)

Source: modelcontextprotocol.io/specification (open protocol; client feature since spec 2025-06-18)

Intent

Let an MCP server pause a request and ask the user, through whatever client the user is in, for structured input or a confirmation, then continue with the answer, without the server knowing anything about the client’s UI.

Motivating Problem

The tool_use loop puts the approval decision in the application that calls the model. Many tools, though, run on MCP servers the application did not write: a deploy server that needs to confirm the target environment, a booking server that needs a missing date, a payments server that needs the user to authorize a charge. Before elicitation, such a server could only fail the call and hope the model asked the user. Elicitation gives the server a standard way to ask the human directly and get a typed answer back.

How It Works

A client that supports elicitation declares the elicitation capability at initialization. In the 2025-06-18 and 2025-11-25 specs the server then sends an elicitation/create request with a message and a requestedSchema, a restricted subset of JSON Schema limited to flat objects of primitive properties (string, number, boolean, enum). The client renders the request however it likes and returns one of three actions: accept (with content matching the schema), decline, or cancel. The server must handle all three.

The 2025-11-25 spec split elicitation into two modes. Form mode is the structured request above. URL mode sends the user to an external URL for interactions that must not pass through the MCP client, such as OAuth flows, payment, or entering credentials; servers must not use form mode for passwords, API keys, tokens, or payment credentials. Clients declare which modes they support. In the 2026-07-28 spec, server-initiated requests (elicitation included) were replaced by the Multi Round-Trip Requests pattern: the server returns an InputRequiredResult whose inputRequests carry the elicitation, and the client sends the user’s answers back as inputResponses when it retries the original request.

The spec asks clients to show which server is asking, let the user review and change answers before sending, and make decline and cancel easy.

When to Use It

MCP servers whose tools need a human decision or a missing value mid-call: confirmations for destructive operations, choices among options only the server knows, credentials or payment through URL mode. Hosts that want one consistent approval surface for every connected server.

Alternatives --- Declarative per-tool approval in agent SDKs (Section A) when the application, not the tool server, decides which calls need a human. The plain tool_use loop above when there is no MCP server in the path. For MCP Apps, which let a tool return an interactive UI the host renders inline, see Volume 13.

Sources

• modelcontextprotocol.io/specification/2025-06-18/client/elicitation

• modelcontextprotocol.io/specification/2025-11-25/client/elicitation

• modelcontextprotocol.io/specification/2026-07-28/changelog

Appendix A --- HITL Face Reference Table

Cross-reference between the three faces of HITL (Chapter 1) and their representative substrates.

Appendix B --- The Series

This catalog joined the six prior volumes to form a seven-layer vocabulary for agentic AI; the series has since grown to nineteen volumes. The volumes are independent and the reading order is flexible; the cross-references make whichever path the reader chooses coherent.

• Volume 1 --- Patterns of AI Agent Workflows --- the timing of agent runs.

• Volume 2 --- The Claude Skills Catalog --- model instructions in packaged form.

• Volume 3 --- The AI Agent Tools Catalog --- the function-calling primitives.

• Volume 4 --- The AI Agent Events & Triggers Catalog --- the activation layer.

• Volume 5 --- The AI Agent Fabric Catalog --- the infrastructure substrate.

• Volume 6 --- The AI Agent Memory Catalog --- the state and context layer.

• Volume 7 --- The Human-in-the-Loop Catalog (this volume) --- the human-agent interaction layer.

Seven layers. Read top-down for the agent designer’s sequence: how runs compose, what model instructions are loaded, which tools the agent calls, what events trigger the agent, what fabric it runs on, what state and memory it operates against, and how humans approve, observe, and interact with it. Read bottom-up for the operator’s sequence: how humans engage with the system, where state lives, what fabric supports it, what events drive it, what tools it uses, what skills ship with it, what patterns its runs follow.

HITL is the layer that most often determines whether an agent system is acceptable to its eventual users --- not whether it works in the lab. Auditors, regulators, end users, and engineers each have specific needs at this layer; meeting them requires deliberate design across the three faces (approval, observation, interaction). A working architect makes choices at all seven layers; the choices interact; the interactions are what makes a production agent system actually viable.

Appendix C --- The HITL Anti-Patterns

Seven recurring mistakes that distinguish thoughtful HITL designs from improvised ones. Avoiding these is most of the practical wisdom in the field:

1. Treating HITL as binary. The choice is not on or off; it’s a five-point spectrum (Chapter 2) and a single application typically uses different points for different actions. A global setting is almost always wrong.

2. Conflating the three faces. Approval, observation, and interaction need different products and different designs. A platform that solves one of the three doesn’t automatically solve the others.

3. In-process pause for approval. The agent’s state lives in the running process; a deploy or a crash during the wait loses the in-flight task. Durable execution (Chapter 3) is the engineering answer; skipping it produces a system that works in demo and fails in production.

4. Trace volume without trace quality. Capturing every event without thought produces traces that are technically complete and operationally useless. Sampling, naming conventions, and selective input/output capture are what make traces searchable; the trace is the audit artifact, not the firehose.

5. Approval gates as the only safety mechanism. Approval gates depend on humans paying attention to specific actions at specific moments. Defense in depth (action-level validation, policy enforcement, sandbox isolation, observability) provides safety properties that human attention cannot. Approval is one layer of a stack, not a substitute for the stack.

6. Chat-only UI for high-stakes interaction. The chat metaphor caps the bandwidth of human-agent collaboration. For applications where the human is making consequential decisions (medical, legal, financial), embedded or generative UI is often the better fit; chat is the default, not the right default for every case.

7. Building observability last. Tracing instrumentation added after the system is already in production is significantly more expensive than instrumentation included from the start. The trace UI is also the primary debug surface; building it last means debugging the early development is harder than it needed to be.

Appendix D --- Discovery and Standards

Resources for tracking the HITL ecosystem as it evolves:

• OpenInference (github.com/Arize-ai/openinference) --- the open trace-format standard; the right starting point for new instrumentation.

• OpenTelemetry GenAI semantic conventions (opentelemetry.io/docs/specs/semconv/gen-ai/) --- the broader cross-vendor standard converging with OpenInference.

• Temporal’s human-in-the-loop cookbook (docs.temporal.io/ai/cookbook/human-in-the-loop-python): a worked example of durable approval with Signals.

• LangGraph’s interrupts guide (docs.langchain.com/oss/python/langgraph/interrupts): the conceptual reference for framework-level approval design.

• Microsoft Agent Framework tool-approval documentation (learn.microsoft.com/en-us/agent-framework/agents/tools/tool-approval) --- the Azure-side reference design for approval gates.

• Awesome lists for ecosystem discovery: awesome-llm-observability, awesome-ai-agents, and similar curated GitHub indexes.

Two pragmatic recommendations. First, instrument for portability: OpenInference or OpenTelemetry GenAI conventions both insulate against vendor lock-in at the observability layer, and the cost of switching backends is dramatically lower when the upstream traces are standards-compliant. Second, design for durability from day one: adding durable execution to a working agent system is significantly harder than building it in from the start, because the assumptions about persistence and state flow are different at every step.

Appendix E --- Omissions

This catalog covers 13 substrates across 7 sections. The wider ecosystem is significantly larger; a non-exhaustive list of what isn’t here:

• Conventional workflow engines (Airflow, Prefect, Dagster, Argo Workflows) when not specifically used for HITL. They can serve the durable-execution role with effort; the agent-specific tooling lives elsewhere.

• Enterprise approval products (ServiceNow, Salesforce Process Builder, Workday) when used outside the agent context. Production agent deployments in enterprises often integrate with these existing approval systems.

• Conventional observability platforms (Datadog, Grafana, New Relic, Honeycomb) when used without LLM-specific instrumentation. The OpenTelemetry GenAI conventions are the bridge that makes these usable for agent telemetry.

• Build-your-own chat UIs (custom React, Vue, or vanilla JS implementations). Many production agent UIs are bespoke rather than framework-based; the patterns in Sections E and F and in Volume 13 transfer but the implementation is application-specific.

• Voice and multimodal HITL surfaces (voice-call review, video annotation, screenshot-based review). These follow the same three-faces taxonomy but with different product categories.

• Specialized HITL products for narrow verticals (medical chart review, legal document approval, financial trade authorization) that haven’t consolidated into general-purpose categories.

Appendix F --- A Note on the Moving Target

LangGraph’s dynamic interrupt() function shipped in late 2024. Temporal’s HITL documentation became a first-class section of the docs in 2025. Langfuse passed LangSmith on some open-source observability metrics in 2025. CopilotKit pivoted toward headless components in 2024. Since 2025 the agent SDKs have added declarative per-tool approval, and MCP added elicitation (June 2025) and its URL mode (November 2025). In 2026 Microsoft moved AutoGen to maintenance mode in favour of Microsoft Agent Framework, Chainlit was archived, and the OpenTelemetry GenAI conventions moved to their own repository, still at Development status. The field is young enough that products and standards move quarterly; this catalog captures October 2026.

The deepest structural fact remains stable: the three faces of HITL (approval, observation, interaction), the five-point approval spectrum, the requirement for durable execution, the trace tree as auditable record, the three UI topologies. These are conceptual claims about the shape of the problem, and they don’t shift the way product names do. An architect who internalizes the conceptual structure can map any new product onto it in minutes; an architect who learns only the products has to relearn the field every twelve months.

Seven volumes complete at the time of writing (the series has since grown to nineteen). Patterns, Skills, Tools, Events, Fabric, Memory, Human-in-the-Loop. A working architect now has the vocabulary for the entire stack, top to bottom.

--- End of The Human-in-the-Loop Catalog v0.1 ---

— The Agentic AI Series —