Digital Content Editor, Eve Goode spoke with Oliver Simonnet, Lead Cybersecurity Researcher at CultureAI about preparing organisations for AI-driven attacks and the changing role of security teams.
What is the most immediate change security leaders should make to prepare for persistent AI-driven attacks?
Leaders should not wait for proof that AI is involved in attacks before making changes or adopting new security controls.
Detections should focus on malicious behaviour regardless of what produced it (human or AI).
They should also focus on obtain visibility into the AI tools and agents already being used within their organisations, treating LLMs and agent technologies as privileged high-risk infrastructure with the appropriate logging, patching/updating and access-controls enforced.
Which existing security controls are most likely to fail when attacks are operating at AI speed and scale?
It is likely that anything relying on a human judgement within an attack window will suffer.
Helpdesk verification, callbacks and other out-of-band processes assume that impersonation difficult, but modern deepfake technologies undermine this.
On the more technical side, signature-based detections are also going to struggle to keep up and provide value, as generating malware variants is now easy, cheap and fast enough that signatures become invalid increasingly fast.
How should organisations govern AI agents and non-human identities differently from traditional service accounts?
AI agents should be treated like new-starter employees as they can be unpredictable, especially when receiving input you may not control, causing its scope to drift and push it to do whatever it takes to complete its task.
They should have named owner, tightly defined scopes, follow least privilege with limited credentials and activity logging.
The boundary of what is in its scope should be explicit, as we’ve seen through recent rogue agent activity, most systems were accessed simply because they were reachable.
What role will human security teams play when both attacks and defensive responses become increasingly autonomous?
Humans will most likely move from actually doing the work to deciding what work should be done.
It is unlikely that anyone will be able to review an agent’s actions one at a timr when operating at machine speed, so human oversight would move to defining boundaries and reviewing outcomes.
What should CISOs be measuring now to determine whether their organisation can detect and contain an AI-driven attack quickly enough?
CISOs should measure what has actually been tested, not just what’s documented.
For example, whether an inventory of AI identities and their credentials is up to date, how long revoking an AIs access will actually take, how much AI activity is being logged somewhere in a form that can be queried, how many detections are behaviour-based rather than indicator-based and how the staff perform against convincing voice clone and other social engineering attacks.
Oliver Simonnet is the Lead Cybersecurity Researcher at CultureAI, focusing on human and AI-centric threats. With nearly a decade of experience, holding roles such as Principal Security Consultant, Global Head of Application Security and SWIFT Security SME his expertise spans multiple domains, including reverse engineering, payment system technologies and artificial intelligence. Known for his research on SWIFT payment systems, Oliver has published whitepapers, delivered industry talks and trained cybersecurity professionals worldwide