A customer asked a chatbot about a fee. It cancelled the trip.

Last month NBC Chicago looked at how often customer service chatbots get things wrong. One United customer was told their $200 travel credit was good for five years. It was good for about one. That’s a bad answer, and it’s the kind of mistake most people expect from a bot.

The other story in the piece is worse. A Chicago-area traveler said they asked United’s chatbot about a refund fee, and the chatbot cancelled their entire trip.

They asked a question. The system took an action. Those are two very different failures, and most of the conversation about AI in support treats them as one.

A wrong answer and a wrong action are not the same size

A wrong answer costs you an apology, maybe a goodwill credit, maybe (as Air Canada learned in court) the price of honoring what the bot said. It’s embarrassing. It’s also fixable, because nothing in the world changed yet.

A wrong action is different. A cancelled reservation can be resold to someone else within the hour. A refund issued to the wrong order is money out the door. An address changed on a shipment that already left the warehouse is a package in the wrong city. You can’t apologize those back into place.

Customers already know this, even if vendors don’t talk about it. Booking.com’s Global AI Sentiment Report surveyed more than 37,000 people: 89% want to use AI to plan travel, and only 12% are comfortable with AI making decisions on its own. Expedia’s research, reported by Skift in April, found just 8% trust AI to book for them. And a piece from Techweez this week makes the same point about the whole agent push: people want help, and they’re much warier of handing over control.

So the public is asking for something specific. Answer my question well. Don’t touch my stuff unless I clearly told you to.

How a question turns into a cancellation

We don’t know how United’s system is built, and we’re not going to guess at its internals. But the general failure is easy to describe because it’s common.

Many support bots run on one decision: what does this customer want? A model reads the message and picks an intent. “Refund fee for my trip” sits close to “cancel my trip” in plain language, so a model can land on the wrong one. If the system’s next step after picking an intent is to go do it, a misread becomes an action.

The fix isn’t a better guess. Any model will misread some messages. The fix is making sure a misread can’t reach anything that changes a customer’s account.

What that looks like in practice

Here’s how we think about it when we build Celeste, and it’s a fair set of questions to put to any vendor.

Reading and changing are separate permissions. Looking up a fee needs read access to the booking. Cancelling needs write access. A question should never be the thing that unlocks a write. If a tool changes something and nobody has said how risky it is, it waits for a person by default.

The model proposes. Code decides. The AI can suggest that a cancellation is what the customer wants. Whether a cancellation is allowed right now, on this topic, for this customer, is a rule in code that the model can’t argue its way past.

Every change gets confirmed in plain words first. Before anything is booked, moved, or cancelled, the customer sees exactly what is about to happen (“I’ll cancel your Oct 12 booking for two at 7pm. Reply yes to confirm.”) and nothing happens until they say so. If the bot misread “what’s the fee?” as “cancel it,” this is where the customer catches it, for free.

Autonomy is earned one topic at a time. Answering hours questions and handling cancellations shouldn’t go live on the same day just because they live in the same chat window. Celeste practices on a business’s own past conversations first, the owner reviews what she would have done, and each topic goes live on its own once it has shown it can be trusted there. Topics that change things can stay in “draft for a person to approve” for as long as the owner likes. How it works walks through the steps.

Why this matters more for a small business

United can absorb a viral story about a cancelled trip. A ten-person salon, a local tour company, or a two-person online shop mostly can’t. When your support bot speaks for you, it also acts for you, and customers won’t separate the two.

The good news is that the customer and the business want the same thing here. The customer wants fast, correct answers and wants to stay in charge of their own booking. The business wants to stop answering the same twenty questions by hand without risking a refund it never approved. Both are possible with the same design: let the AI answer freely where it has proven it’s right, and put a hard wall, built in code and not in a prompt, between a question and a change.

A bot that gets an answer wrong needs better knowledge. A bot that cancels a trip when someone asks about a fee needs a different architecture.