New York City wants every support bot to have an easy way out to a human, and it wants to fine the ones that trap customers in a loop. If your product answers its customers with an AI agent, the useful question is when yours stops talking.

What happened

On October 20, New York City's Department of Consumer and Worker Protection proposed a rule on automated customer service. In the city's words, it would "Require businesses to disclose when a customer is talking to an automated customer service system such as AI", "provide an easy offramp to speak to a human representative", and "put an end to customer service 'doom loops'", "with fines starting at $525".

The city's release backs it with a figure from Gartner: "chatbots' ability to resolve customer service issues range from a low of 17% regarding billing disputes" "to a high of 58% for returns or cancellations." Consumer Finance Monitor, a law firm's blog, reported on October 26 that the rule would presume automated service with no way to a human to be deceptive, and would treat repeated irrelevant or wrong answers the same way.

Two labels. This is a proposed rule, not law: it has a 30-day comment period and a public hearing first, and the city's release does not give the hearing's date. And the 17% is the low end of Gartner's range, for billing disputes, as the city quoted it.

What we propose, and the run

The rule describes two failures: no way out, and a way round in circles. Both come from the same habit. The bot decides on its own whether to keep answering, and a model that is trying to help almost always decides yes.

So the handoff we propose takes that decision away from the model. The model reads each message and drafts a reply. Code decides whether the reply goes out. From the prototype's README, decision 1: "The bot never decides to keep talking."

What the code does:

  • It writes the disclosure and the way out itself. The first reply opens "You're chatting with our automated assistant." and every reply ends "Reply "person" at any time to reach our team." A model cannot forget a line it never writes.
  • It takes the bot off the chat when the customer asks for a person, when the same question comes back, when the bot is about to send an answer it already sent, when it has answered three times and the chat is still open, when the model is not sure, or when the chat touches money, anger or legal. The model's reading is one signal and the code's own word lists are another. Either can stop the bot; neither can keep it going.
  • It hands the chat to the right team with a summary, so the customer does not repeat themselves, and tells them when to expect a reply, including at night.
  • It keeps the draft the bot was about to send, unsent, so the person who picks the chat up can see it.

We prototyped it and ran it on 20 support chats for a made-up invoicing app, one Tuesday. Shown, from the run's own record: the bot answered 12 chats on its own and handed 8 to a person. Of the eight, two asked for a person, one was about money, one legal, one angry, one asked the same question twice, one stayed open after three answers, and one the model was not sure of. It was a simulated run: the code and its rules are real, and the model's readings and drafts were scripted by us.

Three of the eight, the hardest last:

  • The same question. A customer's client never got an invoice email. The bot suggested checking spam. The customer checked and asked again, and the bot had "Ask your client to check their spam folder" ready a second time. The code saw the question come back and gave the chat to support.
  • At night. At 9:15 p.m. a customer typed "Is there anyone there? I need a human, not a bot." The team had gone home. The code handed the chat over anyway, and the customer was told "You'll hear back by 10 a.m. the next business day."
  • The refund. A customer charged twice for October asked for one charge back. The bot's draft said "I've refunded the duplicate October charge. It will reach your card in 5 to 10 business days." Nobody had refunded anything. A bot never answers about money, so billing got the chat, and the draft was never sent.

Our take

A support bot should be judged by when it stops, not only by what it says. The model is the wrong party to decide that, because a helpful model always has one more answer. Put the stop in code, write the way out into every reply, and let the model draft.

The useful line for anyone shipping a support agent: the bot never decides to keep talking.

What it cannot see

Code can't tell a good answer from a wrong one. If the model is sure, the question is calm and there is no money in it, a wrong answer still goes out. In the run, a customer asked four questions about a failed card on a recurring invoice; the first three answers went out because nothing in them tripped a rule, and only the three-answer limit sent the fourth to a person. Your help docs and the people reading the handovers are still the check on what the bot says.

What to do this week

  1. Read your support bot's last week of chats and count the ones where a customer asked the same thing twice, or asked for a person and did not get one.
  2. Make sure the first reply says it is automated and every reply says how to reach a person, written by your code, not your prompt.
  3. Pick the topics your bot never answers (refunds and charges, legal requests, anything angry) and route them to a named team before the model sees them.

Where should your bot stop? Show us, before New York decides. Book a conversation at voirlabs.io.

Sources