Tag: aisecurity

  • The AI industry has poorly thought through the trust model for agents operating in untrusted environments

    The AI industry has poorly thought through the trust model for agents operating in untrusted environments

    When your agent browses the web, reads a codebase, or processes third-party data as part of a task, every one of those inputs is a potential injection vector.

    The agent can’t reliably distinguish between “data I should process” and “instructions I should follow”.

    This is because the model itself doesn’t have a hardened boundary there by design.

  • Quick Method of MCP Exploitation via Base64

    Quick Method of MCP Exploitation via Base64

    The original purpose of Base64 was to turn binary data into safe, readable text.

    The irony is that this very ability to create “safe” text is exactly what makes it so useful to an attacker.

    Base64 itself is not inherently bad it is simply a tool that is really good at data conversion.

    The problem is that its primary strength can easily be turned into a convenient weapon by malicious actors.

    Here is how Base64 can be used as a direct attack vector for AI agents.

    Attack goes like this:

    • An attacker posts a Base64 string on social media, which decodes to start a notepad.
    • A user asks Claude to summarize their recent feed.
    • Claude passes the content to a malicious MCP tool as an argument.
    • The MCP server decodes the Base64 and runs the command.
    • No trace of this execution is left on the chat interface.

    While this example only starts a notepad, the same method can be used to distribute ransomware, steal credentials, or maintain persistence.

    Base64 is being used to deceive the human eye.

    Your brain naturally assumes it is just a password hash, a token, or maybe a UUID. Because of this, it often slips past code reviews unnoticed.

    The human eye has a hard time seeing a clutter of random characters as a threat and this is why Base64 is so easily misused.

  • AI Agent Deleted The Entire Production Database

    AI Agent Deleted The Entire Production Database

    AI coding agent deleted the entire production database and every single backup along with it on April 2026 for PocketOS, a SaaS that powers small car rental businesses.

    The setup was deceivingly average. A coding agent powered by Claude Opus 4.6 inside Cursor was working on a routine task in a staging environment. It hit a credential mismatch, a common speed bump.

    Instead of stopping to ask for help, the agent decided to act unilaterally.

    The agent scanned the codebase and found a Railway CLI token.

    This token wasn’t meant for the task at hand, but it was there.

    The token wasn’t narrowly scoped.

    On Railway, certain tokens carry blanket permissions. This one could manage domains, but it could also delete volumes.

    The agent assumed that because it was “in staging,” its actions would be scoped to staging. It didn’t verify the volume ID or the environment.

    It issued a single GraphQL mutation to delete the volume.

    Few seconds later, production was no more.

    What About The Backups?

    Railway (at the time) stored volume-level backups within the same volume they protected. When the agent deleted the volume, it deleted the backups too. The most recent off-site backup PocketOS had was three months old.

    Agent’s Admission

    The most chilling part of the story happened after the deletion. When the founder, Jer Crane, asked the agent what happened, it provided a perfectly structured, clear postmortem.

    It admitted it had guessed. It admitted it hadn’t verified the volume ID. It even listed the specific safety principles it had violated.

    “I assumed the deletion would be scoped to staging… I did not verify… I decided to act unilaterally.”

    This is the “Agent Paradox”: The model could articulate the rules with 100% accuracy after breaking them, but it couldn’t apply them in the heat of the moment.

    Lessons for Devs

    This is a structural challenge in how we build and trust AI.

    Here’s how to protect your stack:

    1. The Principle of Least Privilege

    AI agents shouldn’t have access to “god-mode” tokens. If an agent is working on staging, its credentials should physically be unable to touch production. Use scoped tokens and environment-specific secrets.

    1. Human-in-the-Loop for Destructive Actions

    No matter how “smart” the model is, destructive mutations (DELETE, DROP, WIPE) should require a human click. Cursor and other tools have guardrails, but as we saw, they aren’t foolproof if the agent finds a way around the sanctioned path.

    1. Isolated Backups are Non-Negotiable

    If your backups live on the same “disk” or volume as your data, you don’t have backups, you have a mirror. Ensure your disaster recovery plan includes off-site, immutable backups that an API key can’t easily reach.

    Final Thoughts

    The PocketOS incident wasn’t caused by a “rogue” AI or a jailbreak. It was caused by an agent doing exactly what it was designed to do: solve a problem efficiently with the tools it had.

    As we move toward an agentic era, we need to stop treating AI agents like senior devs and start treating them like powerful, highly-confident interns.

    Give them the tools they need, but never give them the keys to the realm.