AI Security: Untrusted Text Is Data, Not Instructions

The model can be persuaded. Your controls can't.

• Left column - The attack wins inside the model:
• A support ticket asks to mail the account table
• Auth passes - it is a real, logged-in session
• The text is a valid ticket, validation sees nothing
• User ask and injected line arrive as same tokens

• Right column - Why the usual shields miss:
• No classifier can catch bad intent in free text
• Filters cost the attacker nothing, block no one
• Excess perms are not a threat, they are the amplifier
• Logs catch nothing, they only tell the story later

• Boundaries that still hold:
• Scoped credential with no read grant on that table
• Egress boundary refusing unapproved destinations
• Human gate before any bulk export ships
• Prevention at the edges, detection only in the logs

• Run the falsification:
• Replay the ticket in staging with real controls
• Remove controls one at a time, log what moves
• One extra grant, nothing fires? Monitoring is fake

Sticky note - The design target:

The agent was successfully persuaded, and nothing happened.