Tag: ai

  • The AI industry has poorly thought through the trust model for agents operating in untrusted environments

    The AI industry has poorly thought through the trust model for agents operating in untrusted environments

    When your agent browses the web, reads a codebase, or processes third-party data as part of a task, every one of those inputs is a potential injection vector.

    The agent can’t reliably distinguish between “data I should process” and “instructions I should follow”.

    This is because the model itself doesn’t have a hardened boundary there by design.

  • Experiences from Setting Up Fully Offline Local Only AI Assisted Workstation

    Experiences from Setting Up Fully Offline Local Only AI Assisted Workstation

    Let’s be real, local-first tooling is an essential capability when your internet is not stable, in a world where every cloud service out-there is frequently crashing, not to mention speed-privacy-ownership concerns while sending code, prompts, logs, or half-baked ideas to a remote API.

    To avoid all these noise and make my workflow more autonomous-quicker-stable-private I made a machine where I can code with an AI assistant Locally and Fully Offline.

    I do not think everyone should run a local coding model, this is still a serious tech task. But if you enjoy owning and playing with your stack, and you have the hardware for it, it can be a very satisfying experience.

    What I’ve used for the AI coding assistant that can run without the cloud

    This is a Debian machine with 64 GB GPU and OpenCode as the coding agent with llama.cpp .

    When you get your local llama-server running, OpenCode talks to it like it would talk to any OpenAI-compatible provider.

    The difference is that the whole loop stays on my local machine.

    Neat!

    Be warned – Local AI assistant is hungry for GPU memory

    Running Qwen3.6 27B Q8_0 with 256k context in reasoning mode loads around 50GB of the GPU memory and gives around 64 tokens/s for prompt+generation.

    That is quite good for a local model with that much context.

    There are some Caveats

    Qwen3.6 27B at 256k context is about three times slower compared to a hosted frontier model.

    You have to care about model storage, updates, server flags, GPU memory, and cooling.

  • Quick Method of MCP Exploitation via Base64

    Quick Method of MCP Exploitation via Base64

    The original purpose of Base64 was to turn binary data into safe, readable text.

    The irony is that this very ability to create “safe” text is exactly what makes it so useful to an attacker.

    Base64 itself is not inherently bad it is simply a tool that is really good at data conversion.

    The problem is that its primary strength can easily be turned into a convenient weapon by malicious actors.

    Here is how Base64 can be used as a direct attack vector for AI agents.

    Attack goes like this:

    • An attacker posts a Base64 string on social media, which decodes to start a notepad.
    • A user asks Claude to summarize their recent feed.
    • Claude passes the content to a malicious MCP tool as an argument.
    • The MCP server decodes the Base64 and runs the command.
    • No trace of this execution is left on the chat interface.

    While this example only starts a notepad, the same method can be used to distribute ransomware, steal credentials, or maintain persistence.

    Base64 is being used to deceive the human eye.

    Your brain naturally assumes it is just a password hash, a token, or maybe a UUID. Because of this, it often slips past code reviews unnoticed.

    The human eye has a hard time seeing a clutter of random characters as a threat and this is why Base64 is so easily misused.

  • Common Code Security Challenges with Vibe Coding

    Common Code Security Challenges with Vibe Coding

    Creating site or an app with the use of AI tools and without having a slightest clue of what you are doing is popularly known as “Vibe Coding”. This way of developing is a great way to creating critical security vulnerabilities and bugs.

    The basic cause of all these issues is that AI does not know about code security and it is not aware of the context so it fails to implement permission checks and often sends way too much data.

    Here is a list of some of the common code security issues with AI that we encountered.

    Lack of input validation

    One of the most common issues in vibe coding is the lack of input validation.

    AI’s can not plan ahead what they are doing, and neither are they taught to reason about their code’s security, nor about it’s quality, so it fails to implement permission checks and often sends way too much data.

    Transmit an error message to the attacker with sensitive information

    The API routes often transmit an error message to the attacker with sensitive information, such as, but not limited to paths, filenames and sometimes values from the database or stack used in the application.

    Remote code execution vulnerability

    AI tools often produce a remote code execution vulnerability, the backend code that runs another binary without proper input validation.

    This could be prevented by using proper input validation or even better not needing to run external binaries in the first place.

    Enumeration vulnerabilities – lots of it

    AI tools also produce lots of enumeration vulnerabilities that allowed anyone to request data about a user, including personal information such as email addresses, phone numbers, github and google access tokens, full names and more.

    AI responses that send everything

    The Vibe Coded backend code had no implementation of access controls and it was sending any user data to the client.

    This likely happened because the AI tool generated code that selected the whole user and did not remove the password and other fields from the response.

    AI fails to implement permission checks and often selects too much data, simply because it is not aware of the context.

    To wrap things up

    It is critical that you understand your own code, or the one you are getting from your AI assistants.

    Always have a second look at your code and make sure that you understand what it is doing.

  • Vibe Coding Is Not for Newbies

    Vibe Coding Is Not for Newbies

    Vibe Coding, a technique aimed at creating an optimal “vibe” or environment to enhance coding flow.

    It sounds attractive at first glance, but here is the reality.

    No developer should start vibe coding unless they’ve already built a complete project on their own from start to finish.

    Why is that?

    The answer lies in the fact that vibe coding requires solid foundational skills and discipline, developed only by independently finishing projects.

    This is not about newbies entering the industry from other fields, in those situations, exploration is fundamental.

    But for devs that have intention to advance, building the full stack of skills from personal experience first will keep you grounded and make your vibe coding really and profoundly productive.

  • Technical and Operational Measures essential for utilization of Autonomous AI Agents

    Technical and Operational Measures essential for utilization of Autonomous AI Agents

    To mitigate privacy risks it is essential for the organizations to implement broad technical and operational measures.

    Technical Measures

    Principle of Least Privilege

    Grant agents only the minimum permissions necessary to perform specific tasks, preventing the catastrophic consequences of excessive agency.

    Privacy-Enhancing Technologies

    Implement federated learning to train models without centralizing sensitive data, use differential privacy to add mathematical noise that protects individual privacy, and develop machine unlearning capabilities to address the right to be forgotten.

    Security Basics

    Encrypt data at rest and in transit, authenticate all requests, and regularly audit third-party services for security and compliance.

    Operational Measures

    Human-in-the-Loop (HITL)

    Integrate human oversight at critical decision points, especially for high-stakes decisions with financial, legal, or safety implications. This creates verifiable audit trails and restores accountability.

    Continuous Monitoring

    Implement ongoing auditing to detect model drift, track data provenance, and ensure compliance. Maintain tamper-proof, human-verifiable audit trails.

    Privacy-Centric Culture

    Train employees on privacy risks and set up clear policies for handling sensitive data and autonomous agents.

  • Data Lifecycle Vulnerabilities of Autonomous AI Agents

    Data Lifecycle Vulnerabilities of Autonomous AI Agents

    Autonomous AI Agents engage with data throughout its entire lifecycle and this creates multiple points of vulnerability.

    Massive Scale Collection

    To function effectively, these systems routinely handle terabytes or petabytes of data, including sensitive information like healthcare records, financial data, and biometric information. The sheer volume increases the probability of data exposure.

    Data Repurposing

    Information collected for one purpose may be used for completely different, unforeseen purposes without the user’s knowledge. A notable example involved a surgical patient who discovered that medical photos she had consented to for treatment were used in an AI training dataset without her permission.

    Data Persistence

    The persistent memory of autonomous agents and decreasing storage costs mean information can be stored indefinitely, potentially outlasting the person who created it. This is problematic because privacy preferences change over time – consent given in early adulthood may lead to data being used in ways an individual would no longer agree to later in life.

    Data Spillover

    Agents may unknowingly collect information about individuals who weren’t the intended subjects of data collection, such as bystanders who appear in photos or conversations.

    The independent nature of autonomous agents fundamentally transforms the security threat landscape through a concept known as “Excessive Agency” – agents having too much functionality, permissions, and autonomy.

  • The Built-in Hazards of Black Box Decision Making

    The Built-in Hazards of Black Box Decision Making

    Autonomous AI Agents have few built-in hazards due to their operational nature.

    Unpredictable Actions

    As AI models grow more complex, they can develop emergent abilities that were not explicitly programmed. These unintentional results can lead to unexpected and harmful outcomes, such as an agent optimizing server speed by deleting security monitoring software.

    Algorithmic Opaqueness

    Numerous AI models run as “black boxes”, making it hard to understand how they make decisions or use data. This counteracts accountability and makes it difficult to identify biases that could lead to discriminatory outcomes.

  • Current privacy regulations are not adequate for the unparalleled challenges posed by Autonomous AI Agents

    Current privacy regulations are not adequate for the unparalleled challenges posed by Autonomous AI Agents

    Legal frameworks like GDPR, CCPA/CPRA, and the EU AI Act were not designed for systems that can learn and act independently.

    The Accountability Flop

    Who is liable when an AI agent makes a pricey mistake? Conventional legal systems were not designed for entities that lack legal identity and cannot be held accountable for misconduct. Are we heading for the future where these cyber entities operate at scale with no one to answer for them?

    The Informed Consent Predicament

    GDPR requires explicit, informed consent for data processing, but obtaining genuinely informed consent from autonomous agents is just about impossible. Users would need to understand exactly which services and data the agent will access – information that’s often unknowable at the starting time. The agent, not the user, makes real-time decisions about data collection and processing.

    The Right to be Forgotten Situation

    GDPR’s Article 17 grants individuals the right to have their personal data deleted, but this presents profound technical challenges for AI systems. Personal information isn’t stored in discrete files but is embedded in the model’s weights and vector representations. Even if original training data is deleted, the patterns remain, making complete erasure technically difficult without expensive model retraining.

  • Rise of Autonomous AI Agents – Proactive AI systems that challenge our understanding of privacy

    Rise of Autonomous AI Agents – Proactive AI systems that challenge our understanding of privacy

    Autonomous AI Agents are sophisticated systems that represent a fundamental shift from reactive tools to proactive, decision-making cyber entities that can plan, reason, and act with minimal human supervision.

    They are no just responding to our commands but also anticipating our needs, makes decisions on our behalf, and operates independently across our digital ecosystem.

    These systems are usually used to manage our calendars, optimize our workflows, and help us in online shopping.

    While these agents promise unprecedented convenience and efficiency they also introduce an entirely new class of privacy risks that challenge our conventional understanding of data protection and accountability.

    This shift from reactive to proactive AI fundamentally changes the privacy landscape. When you ask a traditional AI to “play a movie”, it simply executes that command. An autonomous agent tasked with “organizing my afternoon” might access your calendar, order lunch, check traffic conditions, adjust your home temperature, and reschedule conflicts – all without explicit permission for each action.

    Personalization and automation that users desire are only achievable through intensive, continuous, and often shady data collection. And this represents a problem because it effectively disconnects user’s initial intent and the agent’s resulting actions.