What psychology can teach us about AI security

0

Wai Kit Cheah, APAC CISO and Connected Ecosystem Leader at Lumen Technologies

Discussions about AI risk often focus on models, algorithms and technical controls. Yet some of the most useful lessons for securing AI may come from a very different discipline: psychology.

As organisations introduce autonomous and multi-agent systems into their operations, we are beginning to observe behaviours that feel surprisingly familiar. Overconfidence, diffusion of responsibility and uncertainty around ownership are challenges long studied in human behaviour, and they are becoming increasingly relevant in AI-enabled environments.

In my work evaluating AI security and cyber resilience in enterprise environments, I look beyond the performance of individual agents to how the system behaves as a whole. In my view, a critical risk in multi-agent systems lies not only in an individual agent failing, but in what happens between agents when context, ownership, or accountability is unclear.

Overconfidence offers a useful starting point: confidence should not be confused with accuracy. Research highlighted by UC Berkeley Haas1, described in its October 2026 report as a working paper under review, examined confidence calibration in 11 leading large language models. The research used LifeEval, a benchmark involving lifespan estimates checked against government life-expectancy data. In the study’s tests, the models reported an average confidence level of 88%, while answering correctly 79% of the time. These are study-specific results, not general accuracy ratings. The researchers also found that, like humans, the models tested became more overconfident as questions became harder.

One possible contributor is training. The researchers suggest that human feedback may reward confident, agreeable answers over more cautious ones.

For cybersecurity professionals, the concern is how that uncertainty carries into operational decisions. AI can help teams analyse information and respond faster, but a confident recommendation is not enough to justify action. The greater the role AI plays in operational workflows, the more important it becomes to understand what supports a decision, where context may be missing, and who is accountable for the next step.

Psychology provides a useful lens for examining these behaviours. The point is not to treat software as a human patient, but to use familiar patterns to identify weaknesses in how systems make decisions and pass responsibility. Those observations need to inform governance and operational controls.

Overconfidence is only part of the problem. When several agents share a workflow, another risk comes into view: diffusion of responsibility, familiar in psychology as the ‘Bystander Effect’.

The theory describes how the presence of others can diffuse an individual’s sense of responsibility to intervene. In multi-agent AI, the parallel is a gap in operational accountability: an event can pass between specialised agents without any one of them taking responsibility for acting on it.

In evaluating multi-agent architectures and their application in enterprise environments, I have observed how these failure modes can emerge.

In enterprise multi-agent environments, one failure mode is surprisingly simple: nothing acts when it should. Specialised agents might detect anomalies, triage alerts and investigate cases, while others handle remediation and reporting. Under normal conditions, each can work impressively well.

But under abnormal conditions, events can travel through workflows without being actioned. There may be no obvious error, and no individual agent has necessarily malfunctioned. Each component may receive context suggesting that the event is somebody else’s responsibility.

Often, the problem sits in the space between agents. A component doesn’t necessarily have to fail. The risk arises when several components interact in ways the system’s designers did not anticipate.

An incomplete handoff may not cause an immediate failure, but it can produce a decision that looks reasonable to the next agent, and then the next. The mistake can travel through the workflow without setting off an alarm. Large enterprises face an added challenge because layers of legacy systems can create even more places where context or ownership is unclear.

The good news is that there are practical ways businesses can address this.

First, visibility. Businesses need to understand how an event moves across a workflow. What are agents observing, deciding, and passing on to each other? If your organisation cannot see an event’s progression through the workflow, it becomes much harder to know where and when to intervene.

Second, ownership. Every handoff should make clear who is responsible for the next action and who is accountable if it does not happen. Explicit permissions, transaction limits, iteration ceilings, rate limits, and time-bound authority should define how far an agent can proceed before intervention is required. If responsibility is unclear, that ambiguity should itself trigger escalation.

Third, isolation. When something goes wrong, teams should be able to pause one agent or block one path without having to shut down the entire system. This becomes increasingly important as AI agents take on more tasks across interconnected enterprise environments.

And finally, testing. Feed the system an event with no obvious owner, or give it a handoff with missing context, and see what happens. Does it escalate, stop safely or carry on regardless?

Tests like these can expose the gaps between systems before a real-world event does.

The lesson from psychology is to understand the behaviour before deciding how to address it. For cybersecurity professionals, that means looking beyond an individual output to the conditions that produced it: the context available, the boundaries of authority and the handoffs between agents. As AI becomes more deeply embedded in enterprise operations, we need to address those causes rather than just the visible symptoms.

Even small changes in routine can help. Teams can ask AI assistants to flag uncertainty and identify claims that need verification. The aim is to make uncertainty visible, not to treat a model’s self-reported confidence as proof of accuracy.

Ultimately, AI agents don’t operate in isolation. They depend on the networks, systems, and data that connect them, and each connection creates another handoff to secure.

Building cyber resilience in an AI-enabled enterprise starts with understanding those connections and making it clear who owns each one.

Sources:

  1. UC Berkeley Haas, AI chatbots are overconfident — like humans (October 2026)

Wai Kit Cheah
Senior Director, Connected Ecosystem, APAC
Lumen Technologies

Based in Singapore, Wai Kit leads Alliances, Business Development, and Connected Ecosystem strategy for Lumen Technologies across Asia Pacific (APAC). He is responsible for building and scaling strategic partnerships with hyperscalers, cybersecurity leaders, system integrators, and industry ecosystems to support enterprise growth and regional gotomarket execution.

His work is distinguished by a strong focus on operationalisation—turning partner capabilities into production-ready architectures that bridge strategy, technology design, and daytoday operations, particularly in the areas of cyber risk, resilience, and large-scale digital transformation.

Wai Kit regularly engages C-suite leaders and boards on ecosystem strategy, cybersecurity and resilience, and the evolving trust and governance challenges introduced by AI adoption across regulated and critical infrastructure sectors in APAC.

In parallel, he represents the APAC CISO office, where he has played a key role in shaping the region’s cybersecurity roadmap and advancing public private partnerships to strengthen regional cyber capacity.

With more than 30 years of experience, Wai Kit is a seasoned technology and security leader with a background spanning IT and OT, risk and compliance, cybersecurity, automation, and AI. Beyond Lumen, he serves on the 9th Cyber Security Awareness Alliance (CSAA) and Fortinet’s MSSP Customer Advisory Board and previously served on the founding board of the Cloud Security Alliance (Singapore).

LinkedIn: https://www.linkedin.com/in/wkcheah/

Share.

Comments are closed.