Is your AI service actually safe?

What was discussed

Before the sessions we asked the first-timers what they were wrestling with, and their answers set up the evening. A civil servant from a district office said building with vibe coding is instant, but operating and maintaining it is now the department's problem. In the public sector the person in charge changes every two years at most, and what gets built this way is hard for the next person to pick up. All of them were stuck in the same place.

The first session was Changwon Jeon of LG CNS. He spent until last year in the AI lab and moved this year to the platform organization building an agent platform. Asked why demos do not become production, he asked back: the plugins and skills each of us builds, can anyone else use them? You build with yourself as the customer.

Having moved from vision to LLMs, he finds the present far harder. Vision gives you a clean answer. Now, with temperature at zero and the same prompt, the answer still shifts slightly. Customers ask why it changes; explaining that it is statistical gets you asked to just make it the same. So he builds two test sets, an Expert set assuming all the internal vocabulary and a Newcomer set asking in everyday words, though even then, he said, it does not work well.

The example that landed hardest was the weekly report. He has kept three years of them, so one now takes a minute and reads as though he wrote it. But he writes in short bullets, and other people write long. Once a skill tuned to one person is used by many, each patches it and the patches collide. That, not technology, is where demos stop becoming operations.

A proof of concept with a financial group followed. Nervous, he made the answers long, and the customer liked it at first, then the reaction changed. People who enlarge their text and see less of the screen were not getting the carefully written answer into view at all. Do not build for 100 percent from the start, he said. Begin with the minimum, add, and expect to throw work away.

His validation is concrete. He keeps a set of nearly 1,000 items and reruns it on every change, cross-checking by having GPT judge what Claude produced. He stressed the order too: a test set postponed is a test set never built. Whether it finally came out right is answered by customer feedback, and his observation was that nobody presses the good one while people press the bad one freely.

On security, he never starts by building a system. Companies already run wikis and collaboration tools with their own access controls, so rather than blocking at the model he assigns role-based permissions following each user's access and layers guardrails at the end. His view of governance stood out. What he worked on was not surveillance or blocking but whether affiliates doing similar work could share agents instead of each building their own. An agent arriving, he said, does not widen the boundary of what someone may reach.

The second session was Woodu Son of Datadog. Saying he does not really code either, he had built a fortune-telling app to experience for himself what customers face building AI agents. He put four questions to the room. Can you see what the AI is doing and with what? Where are the tokens going, and why? Can you trust the answer? Would you detect an attack? Asked who could answer all four, nobody raised a hand.

The three layers of defence was the slide the evening turned on. Block at model training, block with an input classifier, add an expert red team, and a 1 percent residual attack success rate remains. NIST says there is still no certain defence against prompt injection, and OWASP attributes that to the probabilistic nature of the models. The line from the UK's NCSC was the conclusion: move the defence to the action stage. Let it through at the prompt level and block at runtime in the agent, judged on permission and legitimacy.

More interesting than the demo was the room. While he presented, real requests were piling into the dashboard, and inputs demanding the system prompt were being captured live. Nobody had arranged it; that attack has spread far enough that people try it for fun. In the fireside one of them asked what the 1 percent that got through actually was. Which prompts got through, he did not know, but his sense was that it is AI doing the breaking. Rather than a person thinking each one up, asking an AI runs them in parallel chains.

Asked what a small team should start with, he said visibility matters most. Once you can see which agents are handling what and how, cost and evaluation follow naturally. That brought the first session back, because Jeon had said the same thing: what matters is less how many logs you have than whether you can follow one request from beginning to end. The Datadog demo showed exactly that screen, a request that took 40 seconds, split out by where those seconds actually went.

The roundtable brought ten groups' worth of material. One group put the gap between individuals using AI well and organizations doing so into three parts: nobody can see who uses it how; output takes a different shape per person, so interpreting it costs more time; and non-engineering teams have no equivalent of PR review, so whoever built it carries all of it. All three come from having no specification, so leave the middle free and fix only the shape of the input and the output. Anyone can build now; making something other people use turned out to be a different job.