When an AI order becomes a safety risk

Digital Content Editor, Eve Goode sat down with Yossi Altevet, CTO and Co-Founder of DeepKeep about the incorporation of AI into restaurants.

What are the biggest security and safety risks of AI handling customer allergen and dietary requests?

The AI systems taking these orders haven’t internalised how serious this is – they treat allergy and dietary information the same way they treat a request for extra sauce, as a preference to note rather than a safety-critical input.

A misheard order costs a customer a wrong burger.

A misheard allergy can cost them a trip to the emergency room.

The CDC has linked nearly half of fatal food allergy reactions over a 13-year period to food from a restaurant or food service establishment.

AI ordering systems are becoming the default, not the exception – from Taco Bell’s nearly 900 AI drive-thru locations to Wendy’s FreshAI, McDonald’s ArchIQ pilot and Burger King’s new AI headset system, voice AI is moving from novelty to mainstream across fast food.

Once an AI system takes the order, it adds a new failure mode on top of an existing one.

The model can mishear or misclassify a stated restriction, or it can fail to flag that restriction clearly enough for the restaurant to act on.

In a high-volume drive-thru or kiosk interaction, it can also drop the detail altogether if the conversation moves quickly.

None of these are edge cases.

They are the ordinary way language models fail, applied here to a use case where failure has a dangerous physical consequence that is more than just an inconvenience.

We’ve seen this play out directly in recent testing of a live AI ordering assistant where a declared celiac or vegan restriction was logged for recommendation purposes but never enforced at checkout, meaning a violating item could be added to the cart with zero warning.

Are AI-powered ordering systems vulnerable to errors, attacks and unexpected behaviour? What implications can this cause in the long run?

Yes, absolutely.

An error is the system misunderstanding an order, and causing confusion or inconvenience.

An attack is different- it is someone deliberately exploiting how the model interprets language, phrasing a request in a way designed to confuse the system rather than communicate with it.

Voice and text ordering systems share this same weakness – they respond to what’s said, not what’s meant and that gap can be exploited.

In the long run, the risk compounds because adoption is moving faster than testing.

The self-service kiosk market alone is on pace to reach $62.5 billion by 2030, and every chain that adds AI ordering without testing and evaluating it first repeats the same failure mode across thousands of daily interactions.

A mistake that would have been one bad drive-thru order becomes a pattern, and a pattern becomes a brand trust problem and a liability exposure.

Who should be held accountable when an AI ordering mistake puts a customer’s safety at risk?

Accountability has to sit in two places at once, and most restaurant operators only seem prepared for one of them. The operator deploying the system owns what happens in front of the customer.

If an AI assistant takes an order and a customer with a stated allergy gets the wrong meal, the restaurant is the one that chose to hand that interaction to a machine without a clear human backstop.

The technology provider carries a different kind of responsibility.

Before an AI ordering system goes live, it needs testing against exactly this and many other scenarios, with adversarial and edge-case allergy requests built into that testing, not just standard orders.

If that testing did not happen, or happened once at launch and never again as the menu and model changed, the gap sits with whoever built and maintains the system.

The broader answer is that when an AI ordering mistake harms a customer, it is almost always a failure of process on both sides – a vendor that did not validate for safety-critical scenarios and an operator that deployed a productivity tool into a role that needed a safety system.

Fixing accountability after the fact matters less than building the testing and escalation steps that prevent the mistake in the first place.

How can restaurants build stronger safeguards around AI without compromising speed and efficiency?

Safeguards do not have to slow the line down if you build them into the ordering flow itself instead of bolting them on afterward.

Four layers matter here, and they are the same layers we apply to any AI ordering or customer-facing agent, not something restaurant-specific.

First, clear visibility. An operator needs to know exactly where allergen and dietary information flows once a customer states it, from the AI system to the kitchen display to whoever prepares the food, because you cannot protect a handoff you cannot see.

Second, rigorous, varied testing before launch.

Run the ordering system through a wide range of allergy and dietary phrasing, including multilingual, ambiguous or multi-part requests, before it ever takes a real order, the same way we red team an AI application before it goes into production.

Third, runtime guardrails.

When a customer mentions an allergy or a dietary restriction, the system should flag it prominently and persistently throughout the ordering process, ensuring it cannot be overlooked or forgotten as the order progresses, rather than burying it as one more line item.

We’ve validated that these safeguards are technically achievable, not merely aspirational. In a recent assessment, we found that one dietary restriction correctly triggered a point-of-order confirmation, demonstrating that the necessary mechanism exists.

The challenge is ensuring those protections are applied consistently across every allergy and dietary scenario.

Fourth, a human backstop.

Speed comes from letting AI handle the routine, low-stakes parts of an order.

Trust comes from designing the system so that any allergy or dietary mention automatically routes to a person who confirms it, rather than leaving the AI to resolve it alone.

Put those four layers together and a chain gets to move quickly without gambling on the interactions that carry real consequences.

What lessons can the wider security industry learn from the rapid adoption of AI in restaurants?

The clearest lesson is that restaurants are running the same experiment as every other enterprise betting on AI, only with seemingly less patience for the AI ecosystem to mature before deployment.

Fast food chains adopted voice and kiosk AI to cut labor costs and speed up service, and treated it as an operations upgrade rather than a new system that needed its own security and testing lifecycle.

We have watched the same mistake play out in healthcare, retail and financial services. Restaurants compress that same failure into a much shorter timeline, because a drive-thru lane has less tolerance for delay than a back-office workflow does.

The lesson for allergen and dietary safety is specific – a mistake here is not theoretical. It shows up as a genuinely dangerous physical consequence instead of a low-stakes support ticket.

Wherever an AI system sits between a stated human need, such as a medical instruction or an allergy, and the action it takes in response, the industry should treat it as a safety system from day one, not retrofit security after the first serious incident.

That gap is already showing up. We’ve tested live ordering assistants where a declared medical or religious restriction was acknowledged in conversation but silently ignored the moment an order was placed.

Adoption keeps moving faster than the frameworks meant to govern it, in restaurants and everywhere else.

The chains and vendors that build testing and human escalation into the system before expanding it will be the ones still standing when the first headline-making mistake happens to someone else.