Your chatbot speaks for you, even when it makes things up

A cosmetic surgery practice in Germany put a chatbot on its website to book appointments and answer questions. Someone asked about the two doctors who run it. The bot said they were specialists in plastic and aesthetic surgery. They weren’t. They didn’t hold those titles.

A consumer group challenged it under unfair competition law, and on May 12 the Higher Regional Court of Hamm ruled against the practice. The doctors argued, in effect, that the chatbot was a third party. The court said no. What the bot says on your website is a commercial statement by your business, regardless of what data it was trained or grounded on. The court allowed an appeal to Germany’s Federal Court of Justice because the question is new, but the direction is the same one Air Canada found in 2024, when a tribunal made the airline honor a bereavement refund policy its bot had invented.

Look at what the Hamm bot got wrong. Not a hard medical question. A plain fact about the business. That’s the pattern worth paying attention to.

The dangerous answers are the easy ones

When people picture an AI support failure, they picture a tricky case: an angry customer, an edge-case refund, a legal threat. Those are real, but they’re also the conversations a human is most likely to be watching.

The failures that slip through are small, confident facts. Your hours. Your warranty length. Whether you ship to Canada. Whether the owner is board certified. A model is very good at producing an answer shaped like the right one, and a sentence like “all our products carry a 90 day warranty” looks exactly like a correct answer until someone tries to claim it.

Small businesses are more exposed to this than large ones. A roundup of 2026 studies on AI accuracy about companies found that 93 percent of companies had at least one fact wrong or missing in public AI answers, and that half of small businesses had at least one outright fabricated fact, against 32 percent of large companies. Contact details and company basics were among the most common misses. Those numbers are about public chatbots guessing from the open web, which you can’t control. But the lesson carries: when a model knows less about you, it fills the gap with something plausible.

Your own support bot is the one AI source about your business you do control. It shouldn’t be one more place that guesses.

”We configured it carefully” is not a defense

The Hamm court was explicit that careful setup didn’t change who was responsible. That matters, because the usual fix for invented facts is a sentence in the prompt. “Only answer using the provided information.” “Never make up policies.”

Prompt rules help. They don’t hold every time, because the same model reading the same rule can make a different call on the next run. A prompt is a request. It isn’t a check.

So the rule we’d suggest to anyone running an AI on their support inbox is simple to say and takes real work to build:

Any specific fact about your business has to point to a source you wrote or approved. If it can’t, a person sees it before the customer does.

That means the system has to notice a business claim in a draft, look for where it came from, and flag it when nothing backs it. And the rule that holds a flagged draft has to live in ordinary code outside the model, so the model can’t talk its way past it. The AI proposes. Hard code decides.

The other half of the rule is what happens next. “I don’t know” is fine as an answer to a customer (“the team will confirm that for you”) as long as the gap doesn’t stay a gap. Every unsourced question is a missing line in your knowledge base, and it should land in front of the owner as something to fill in once.

How we handle it

Celeste practices on your real traffic before any topic goes live. Each draft it writes gets checked for business claims no source backs, and those claims are listed on the draft in the inbox. Nobody approves that draft without acknowledging them first. When Celeste hits a question it has no source for, the gap goes to you as a question to answer, and your answer becomes knowledge it can cite next time.

Topics go live one at a time, when you’ve watched enough drafts in that topic to trust it. You’ll see which facts it reaches for before a customer ever does. More on the approach is on the product page.

The Hamm case may still go to a higher court, and the details may shift. The principle probably won’t. If a bot on your site says something about your business, you said it. Build it so it only says what you’d sign.