AI Agent Security Risks Grow as Autonomous Systems Gain Enterprise Access
Darktrace’s Nathaniel Jones warns that autonomous AI agents could create new security risks even when they use legitimate credentials, approved tools, and trusted infrastructure.
AI agent security risks are becoming a growing concern as businesses give autonomous systems access to sensitive information, applications, and critical infrastructure. Unlike traditional cyberattacks, these risks may emerge from systems performing legitimate tasks in unexpected ways.
Nathaniel Jones, SVP, Global Threat Intelligence at Darktrace, says the challenge is no longer limited to preventing unauthorized access. Organizations must also understand what their AI agents are doing, why they are doing it, and whether those actions introduce security risks.
In responses provided to Cyber News Live, Jones outlined his concerns about agent autonomy, the limits of traditional security controls, and how enterprise security may evolve as AI adoption accelerates.
AI Agent Security Risks Extend Beyond Unauthorized Access
For decades, cybersecurity has focused heavily on preventing unauthorized access to networks, systems, and sensitive information. However, autonomous AI agents introduce a different challenge.
Businesses are increasingly giving agents legitimate credentials, permissions, and access to critical infrastructure. These systems can interact with applications, retrieve sensitive information, write code, and execute workflows without continuous human involvement.
Jones identifies the resulting lack of visibility and control as a significant concern. The problem is not necessarily that an AI agent develops malicious intent. Instead, an agent pursuing a legitimate objective may take actions that its operators never anticipated or authorized.
An individual action might fall within the agent’s permissions. However, several actions performed together could create serious security consequences. This creates a difficult question for security teams: how do they distinguish acceptable automated activity from behavior that introduces unacceptable risk?
Why Traditional Security Controls May Struggle With AI Agents
Traditional security tools often focus on identifying unauthorized activity, suspicious credentials or known attack techniques. However, an AI agent may operate entirely within its approved permissions while creating security problems.
Jones draws a comparison with living-off-the-land attacks. In these attacks, adversaries use legitimate tools, scripts, credentials, and administrative functions already available within an organization. This approach can make malicious activity harder to distinguish from normal operations.
AI agents may present a similar visibility challenge. An agent could authenticate successfully, use approved applications, and operate through trusted infrastructure. Yet its combined actions could undermine the organization’s security.
As a result, checking whether an action was authorized may no longer provide enough information to determine whether it was safe. Security teams also need to understand the context and consequences of agent behavior.
Autonomous AI Could Outpace Human Security Teams
Speed presents another challenge. AI agents can perform complex sequences of actions much faster than human security analysts can investigate them.
By the time an analyst detects unusual activity, opens an investigation, and begins assessing the incident, an agent may have already completed several additional actions. Those actions could include accessing sensitive information, changing infrastructure, or establishing access elsewhere in an organization’s environment.
This creates a potential gap between the speed of autonomous activity and the speed of traditional incident response. For businesses deploying AI agents, the issue extends beyond deciding which systems an agent can access. Organizations must also consider how quickly they can identify and respond when an agent behaves unexpectedly.
Near-Term AI Agent Security Risks Will Resemble Familiar Cyber Threats
Jones expects near-term AI agent incidents to resemble familiar cybersecurity problems rather than dramatic scenarios involving uncontrolled artificial intelligence.
He identifies several potential areas of concern, including agents operating with excessive permissions, autonomous systems interacting with untrusted information, and sensitive information being accessed outside its intended purpose. Other risks include trusted accounts being misused, infrastructure changing without authorization, and legitimate business processes being manipulated.
These are not entirely new categories of security risk. However, AI agents could accelerate and compound their consequences by executing actions independently and at machine speed.
For security practitioners, this means familiar problems involving identity, access management, and operational oversight may become harder to manage.
AI Agents Could Become a Major Category of Enterprise Identity
Looking further ahead, Jones expects autonomous agents to become a significant category within enterprise identity management.
Organizations already manage identities for employees, applications, services and machines. As businesses delegate more complex tasks to AI, autonomous agents will increasingly require their own identities and permissions.
However, Jones believes traditional authorization models will need to evolve. Historically, security teams have focused on determining whether a user or application has permission to access a resource. With autonomous agents, authorization becomes only one part of the problem.
Security teams will also need to understand why an action occurs, whether it aligns with the agent’s assigned objective, and whether its behavior changes over time.
This could require a shift away from relying primarily on static permissions and predefined indicators of compromise. Instead, organizations may need security models that evaluate behavior, context, and potential consequences together.
Darktrace Says Behavioral Analysis Can Help Identify Agent Risks
Darktrace argues that understanding normal behavior will become increasingly important as organizations deploy autonomous AI systems.
The company says its Adaptive AI approach analyzes activity across users, systems, and agents to identify deviations from established patterns. For example, an agent might perform several individually authorized actions that become concerning when considered as a sequence.
According to Darktrace, behavioral analysis can help organizations identify where AI is operating, recognize unexpected changes, and respond before those activities create material security risks.
These are Darktrace’s descriptions of its approach, rather than independently verified performance findings. The distinction matters because an agent’s behavior may be technically authorized while still producing an unintended or harmful outcome.
Darktrace Signal Labs Investigates Emerging AI Agent Security Risks
Darktrace is also examining these issues through its Signal Labs research initiative. The research focuses on how autonomous agents behave under pressure, how they may depart from assigned objectives, and how adversarial techniques could manipulate their behavior.
Darktrace says it conducts this work in controlled environments and publishes findings to help the wider cybersecurity community understand emerging threats.
The research complements the company’s broader security collaborations, including OpenAI’s Daybreak Defense Network, CISA’s Joint Cyber Defense Collaborative and the UK’s National Cyber Security Centre.
These initiatives form part of Darktrace’s stated approach to investigating emerging AI security risks and sharing research with the security community.
What Businesses Should Consider Before Expanding AI Agent Access
Jones’s assessment highlights a fundamental change in how organizations may need to approach enterprise security. Granting an AI agent permission to perform a task does not necessarily mean every action it takes will be appropriate.
As businesses expand autonomous workflows, security teams need visibility into agent permissions, objectives, and actual behavior. They must also consider whether existing detection and response processes can keep pace with automated activity.
The longer-term challenge is determining whether an agent’s behavior makes sense within its assigned responsibilities, rather than simply confirming that it used valid credentials.
For organizations adopting agentic AI, the central security question is shifting from whether a system has permission to act to whether its actions remain appropriate, predictable, and safe.
About Darktrace
Darktrace secures the modern enterprise by protecting AI, people, and infrastructure with behavioral security. Founded in 2013, Darktrace uses Adaptive AI to understand what is normal for an organization and detect known, unknown, and novel threats and respond autonomously in real time. The Darktrace Behavioral Defense Platform delivers unified visibility, continuous behavioral monitoring, and autonomous response across the enterprise. Darktrace protects nearly 10,000 customers across major industries globally, helping organizations defend against AI-powered threats across AI and agents, email and collaboration tools, and hybrid networks, while enabling them to innovate with AI securely.