The Wizard System

December 2023. We had a workflow automation platform that could chain together 168 integrations, process data through AI models, and execute multi-step tasks without human intervention. It worked well. It also terrified our enterprise clients.

The question they kept asking was simple: “What happens when the AI gets it wrong?”

We didn’t have a great answer. Our workflows ran end-to-end. If step three of a seven-step workflow produced bad output, steps four through seven would dutifully propagate that bad output downstream. No checkpoint. No review gate. No human in the loop.

The wizard system was our answer.

Five types of human intervention

We didn’t build one wizard. We built five, because human oversight isn’t a single interaction pattern. Different situations require different interventions.

Chat wizard. A conversational interaction embedded within a workflow step. The workflow pauses, opens a chat interface, and lets the user discuss the current state with the AI before proceeding. Use case: a content generation workflow that produces a draft and then lets the user refine it through conversation before moving to the publishing step.

Form wizard. Structured data collection via form fields. The workflow pauses and presents a form that the user fills out. Use case: a client onboarding workflow where the AI pre-fills most fields but needs human input for a few critical details – budget, timeline, specific requirements.

Studio wizard. A creative workspace for content or image review. The workflow pauses and presents generated content in an editing environment where the user can modify, approve, or reject it. Use case: a social media workflow that generates image variations and copy, then lets a designer pick the best options before scheduling.

Approval wizard. A binary gate. The workflow pauses, presents the proposed action, and waits for the user to approve or reject. No editing, no conversation – just a yes/no decision. Use case: a financial workflow that prepares an invoice and waits for a manager to sign off before sending.

Upload wizard. File upload during workflow execution. The workflow pauses because it needs a file that wasn’t available at the start. Use case: a document processing workflow that generates a summary and then needs the user to upload the final signed version for archiving.

Each type addresses a different failure mode of full automation. The chat wizard handles ambiguity. The form wizard handles missing data. The studio wizard handles quality judgment. The approval wizard handles accountability. The upload wizard handles missing inputs.

The technical architecture

Under the hood, the wizard system is built on RabbitMQ, the same message broker we use for inter-service communication across our entire platform. When a workflow execution hits a wizard step, three things happen:

First, the BullMQ job that’s running the workflow pauses. The current state – all variables, intermediate outputs, the execution position – is serialized and stored.

Second, a message is published to RabbitMQ with the wizard type, the required inputs, and a reference to the paused job. RabbitMQ routes this to the appropriate handler. We have five handler types: wizard, approval, users, organizations, and management. The wizard and approval handlers are the ones that interact with the wizard system directly.

Third, a task appears in the user’s dashboard. This is where the human sees the wizard – a chat interface, a form, a content review panel, an approve/reject button, or a file upload dialog, depending on the wizard type.

When the user completes the wizard interaction, the response travels back through RabbitMQ to the job system. The paused BullMQ job receives the human input, deserializes its state, and resumes execution with the new data incorporated.

The entire cycle is asynchronous. A workflow can pause at a wizard step and wait for minutes, hours, or days. The user doesn’t need to be online when the workflow reaches the wizard – they’ll see the pending task the next time they open their dashboard.

Why RabbitMQ, not WebSockets

An earlier version of the wizard system tried to use WebSocket connections for real-time wizard delivery. The idea was appealing: the moment a workflow hits a wizard step, the user sees it immediately in their browser.

It fell apart for two reasons. First, users aren’t always online – a workflow might run at 2 AM, and the wizard step sits with no connection to deliver to. Second, WebSocket connections are stateful, which means tracking per-user connections and handling reconnection. For a feature that’s fundamentally about pausing and waiting, statefulness was the wrong paradigm.

RabbitMQ solved both problems. Messages are durable – they persist until consumed. The pub/sub model is decoupled – the workflow publishes a message and moves on (to a paused state). The user consumes it whenever they’re ready.

Wizards in practice: the TVBS example

One of our enterprise clients, TVBS (a Taiwanese broadcaster), used Autopilot for content workflows. They had a multi-step process: scrape news sources, generate article summaries using AI, translate them, and publish to their content management system.

The problem was quality control. AI-generated summaries were good enough maybe 85% of the time. The other 15%, they contained factual errors, awkward phrasing, or missed the editorial angle that TVBS’s style required.

Without the wizard system, they had two options: run the workflow fully automated and fix errors after publication (bad for a news organization), or do everything manually and lose the automation benefit entirely.

With the wizard system, they inserted a studio wizard between the generation step and the publication step. The AI would produce the summary, and an editor would review it in the studio interface. Approve it, and the workflow continues to publishing. Edit it, and the modified version goes through. Reject it, and the workflow terminates.

The result was a workflow that ran automatically for the 85% of cases where the AI was right and paused for human judgment on the 15% where it wasn’t. The editors loved it because they reviewed only the content that needed reviewing, instead of manually processing every piece.

The task system connection

Wizard interactions don’t exist in isolation. They connect to our broader task system, where any action in a workflow can create a task that requires human review.

The distinction matters. A wizard is a specific moment in a specific workflow execution where human input is required. A task is a persistent item in a user’s task list that may or may not be connected to a running workflow. Wizards create tasks. Tasks get completed. Completions resume workflows.

This separation lets us do things like: assign wizard tasks to specific team members based on role, set SLAs for how long a wizard can wait before escalating, and aggregate pending wizards across all running workflows into a single review queue.

By late 2024, we added wizard publishing and an ownership system, so teams could create reusable wizard templates. An organization could define a standard approval wizard with specific form fields and review criteria, then use it across multiple workflows without rebuilding it each time.

HITL isn’t a compromise

There’s a persistent narrative in the AI industry that human-in-the-loop is a temporary measure. The assumption is that models will get good enough that human oversight becomes unnecessary. We don’t buy it.

Gartner projected that 90% of enterprise generative AI deployments will require formal human-in-the-loop governance by 2026. Our experience confirms this. Enterprise clients don’t want to remove human oversight as models improve. They want to make human oversight more efficient.

The wizard system isn’t a concession to model limitations. It’s a design principle. The value proposition isn’t “our AI is so good you don’t need humans.” It’s “our system is smart enough to know when it needs a human.”

The five wizard types map to five types of uncertainty:

• Chat wizard: the AI needs clarification
• Form wizard: the AI needs data it doesn’t have
• Studio wizard: the AI needs aesthetic judgment
• Approval wizard: the AI needs authorization
• Upload wizard: the AI needs a physical artifact

Each type of uncertainty requires a different interaction pattern. Trying to handle all five with a single “approve/reject” button would be like trying to handle all customer inquiries with a yes/no form.

What we’d build differently

If we were designing the wizard system from scratch today, we’d make two changes.

First, we’d add a confidence-based trigger – let the AI request a wizard dynamically when its confidence drops below a threshold, rather than configuring review gates statically.

Second, we’d build wizard analytics from the start. Approve vs. reject rates, wait times by wizard type, which workflows generate the most human interventions. We added some of this later, but having it from day one would have shaped our design decisions differently.

The wizard system taught us that the best automation isn’t the kind that eliminates humans. It’s the kind that involves them at exactly the right moments. Five wizard types, five types of human judgment, zero false confidence that the AI always knows best.