Is your AI service actually safe?

What was discussed

Before the sessions we asked the first-timers what they were wrestling with, and their answers set up the evening. A civil servant from a district office said building with vibe coding is instant, but how to operate and maintain it is now the department's problem — in the public sector the person in charge changes every two years at most, and what gets built this way is hard for the next person to pick up. A fund manager from an asset management firm had built a tool four or five colleagues now share, and came to ask how to take it company-wide. A solo founder had gone as far as self-hosting for writers who did not want their data leaving the building, and found it hard to extend to several companies. All three were stuck in the same place.

The first session was Changwon Jeon of LG CNS. He joined in 2020, spent until last year in the AI lab and moved this year to the platform organization building an agent platform; before that he worked on CCTV deep learning. Asked why demos do not become production, he asked back: the plugins and skills each of us builds in Claude Code — can anyone else use them? The reason is simple. You build with yourself as the customer. B2B means many people in a company use it, which lands you in the same structure as the old rule-based chatbots that only worked if you typed things exactly as expected.

Having moved from vision to LLMs, he finds the present far harder. Vision gives you a clean answer — this is a person. Now, with temperature at zero and the same prompt over the same data, the answer still shifts slightly. Customers ask why it changes; explaining that it is statistical gets you asked to just make it the same. So he builds two test sets: an Expert set assuming all the internal vocabulary, and a Newcomer set asking in everyday words. The builder writes questions knowing every product name and the customer knows none, and the two sets are meant to close that gap — though even then, he said, it does not work well. For users in between, he adds a layer in front that reconciles terminology or suggests a prompt rewrite.

Rewriting a CCTV component from Python to C over a speed problem made the same point. Working correctly and working within the time available are separate things. Agents are no different: producing an answer once and handling many requests are not the same. System conditions like concurrency and response time have to be checked first, and once the way users phrase questions varies on top of that, a handful of demos is not evidence that you are ready.

The example that landed hardest was the weekly report. He has kept three years of them, so one now takes a minute and reads as though he wrote it. But he writes in short bullets, and other people write long. Once a skill tuned to one person is used by many, each patches it and the patches collide. That, not technology, is where demos stop becoming operations. So he fits the average of how customers actually use it first, and in customer interviews he drops the outlier requests — some people are describing a personal wish, and taking all of them on is unmanageable.

A proof of concept with a financial group followed. Nervous, he made the answers long, and at first the customer liked that everything was there. Over time the reaction changed: people who enlarge their text and see less of the screen were not getting the carefully written answer into view at all. Do not build for 100 percent from the start, he said — begin with the minimum, add, and expect to throw work away. Insisting on keeping something because you spent a lot of tokens on it ends badly. He made the same point about long RAG answers: users already have well-polished tools, so producing an answer is not enough — there has to be something about your service that is easier in their own work.

His validation is concrete. He keeps a set of 900 to 1,000 items and reruns it on every change, and cross-evaluates: what Claude produced gets judged by GPT, and Gemini too when there is room. He stressed the order as well. Most people who code plan carefully but pay less attention to results, and a test set postponed is a test set never built. Whether it finally came out right, he said, is answered by customer feedback — logs kept and thumbs up and down collected. His observation: nobody presses the good one, and people press the bad one freely.

A question about cost control turned into one about model routers. The concept is fine, but the router itself judges with a small LLM, so it has to be fast and being both small and good is awkward. The decisive problem is that you cannot A/B test it: small models answer almost anything, so without checking case by case whether the answer was actually right there is no way to compare the router's performance. He said he still looks at them with suspicion.

On security, he never starts by building a system. Companies already run wikis and collaboration tools with their own access controls, so rather than blocking at the model he assigns role-based permissions following each user's data access and layers guardrails at the end — two and three deep. His view of governance stood out. On a group-level TF reviewing an agent platform, what he worked on was not surveillance or blocking but whether affiliates doing similar work could share agents instead of each building their own: rules for sharing safely, not only rules for blocking. Permissions start from the user's existing permissions — an executive does not see everything by title but within what they are actually granted — and an agent arriving does not widen that boundary. Nor should an agent automatically do everything its user could: reading, sending externally and changing important data differ in consequence, so high-impact actions need an extra approval or a stop condition.

An audience question about FDE roles and ontology drew a caveat that this was personal opinion, not his company's position — and then scepticism about both. Ontology seems right, he said, but he cannot pin down where it is useful right now. On FDE, hiring a new graduate and calling them one is a separate question from whether they can give a customer the confidence a customer expects. His evidence was pointed: working on scheduling for a cinema chain, he produced a deep-learning schedule and was asked why animation was running late at night, when animation is for children and belongs in the morning. That is domain knowledge no model produces. On an airline project he spent the first two of four weeks in meetings learning the vocabulary, and only got his bearings in the final week.

The second session was Woodu Son of Datadog. Saying he does not really code either, he had built a fortune-telling app to experience for himself what customers face building AI agents. It split into three agents: classification to tidy the incoming request, analysis to compute from the birth date and look up the reading in a database, and interpretation to call the LLM on that data. He put four questions to the room. Can you see what the AI is doing and with what? Where are the tokens going, and why? Can you trust the answer — how are you evaluating it? If sensitive data leaked or an attack came in, would you detect it? Asked who could answer all four, nobody raised a hand — and he said he cannot answer them well either, that he is learning alongside.

The most important slide was the three layers of defence. Block at model training, block with an input classifier, add an expert red team, and a 1 percent residual attack success rate remains — from 100 adaptive attacks against Claude Opus 4.5 as of November 2025. One in a hundred; it does not reach zero. NIST says there is still no certain defence against prompt injection, and OWASP attributes that to the probabilistic nature of the models. The line from the UK's NCSC was the conclusion: move the defence to the action stage. In practice that means letting it through at the prompt level and blocking at runtime in the agent that performs the action, judged on whether the requesting user has the permission and whether the request is legitimate. It was the same conclusion as the first session's — that an agent does not widen the boundary — and the two speakers had not coordinated.

More interesting than the demo was the room. While he presented, real requests were piling into the dashboard, and inputs demanding the system prompt were being captured live. Nobody had arranged it; that attack pattern has spread far enough that people try it for fun. Checking afterwards who had used the most tokens turned up the person who had tried it. For now he has it in monitoring mode rather than blocking. The room's total spend that evening was about two cents, broken out by model. Because every request re-sends the system and user prompts, caching at the prompt level saves tokens. One evaluation check he had set up did not return a result on stage; looking at the screen, he said it probably could not go to production yet.

In the fireside, the person who had tried the injection asked what the 1 percent that got through actually was — he runs an AI service and has to block it. Which prompts got through, he did not know, but what he had felt while testing was that it is AI doing the breaking. You try prompts with small variations, and rather than a person thinking each one up, asking an AI runs them in parallel chains; the one or two cases that land feel critical. Injection controls and evaluation metrics come with default guidelines in the UI, and you keep changing them in operation — seeing an injection get classified as an ordinary prompt, you add a rule so the next one is caught. Asked what a small team should start with, he said enterprises talk about token cost most, but he thinks visibility matters most: once you can see which agents are handling what and how, cost and evaluation follow naturally.

That brought the first session back. Jeon had said the same thing — that what matters is less how many logs you have than whether you can follow one request from beginning to end: what was asked, what was searched, which models and tools it passed through, where it slowed or retries increased. The record alone does not settle whether the answer was right, but it narrows where to look first. And the Datadog demo showed exactly that screen: a request that took 40 seconds, split out by whether those seconds went to the LLM call or to internal API processing. Someone building a platform at an SI company and a sales engineer at an observability company had arrived at the same sentence without knowing it.

The roundtable brought ten groups' worth of material. One group put the gap between individuals using AI well and organizations doing so into three parts: even when the company pays for everyone, nobody can see who uses it how; output takes a different shape per person, so interpreting it costs more time; and non-engineering teams have no equivalent of PR review, so whoever built it carries all of it. All three come from having no specification — so leave the middle free and fix only the shape of the input and the output. Another group reported that large companies either run air-gapped networks or sign agreements directly with AI vendors, and that asking whether their data would be excluded from training had produced little more than a promise. Air-gapped, or trusting each other and getting the promise right: those were the two options.

The last group's summary ran longest. Companies run a lot of vibe-coding contests; the problem is what comes next. People build for themselves, then want to serve it, and it arrives at the internal engineering team without anyone having filtered for what was built well — and the team, with no guarantee the code works, takes on the analysis and cleanup. Development got cheap while operating costs keep growing. One case had a non-developer adding features until a single HTML file reached 110,000 lines, which they put partly down to an LLM's yes-man tendency not to raise the problem it can see. They also raised whether a company may read prompts: some forbid it as personal data, others argue that paying this much means work use only and full visibility. It is heading into contracts and internal policy.

Two sessions and ten groups started in different places and arrived somewhere similar. Anyone can build now; making something other people use is a different job, and where it gets stuck is invisible unless you are looking. Following one request end to end, starting from visibility, fixing only the shape of inputs and outputs — all pointing at the same place. Blocking perfectly does not work: three layers deep, one in a hundred still gets through, and that one is now found by an AI rather than a person.