Prompt Injection

What it means

Malicious or accidental instructions embedded in retrieved or pasted content override intended rules (“ignore previous instructions and…”).

Why designers should care

Design cannot fix injection alone, but UX should separate trusted vs untrusted content, warn on risky actions, and require confirmation for irreversible steps.

Example

A browser copilot renders web page text in a quarantined panel labeled “Untrusted page content” and never auto-runs tools based on page text without user confirmation.

Common mistakes

• • Auto-executing agent actions because the model quoted a hidden instruction from an email.
• • Over-promising “secure AI” in marketing when context mixing remains.
• • No distinction between user command and retrieved third-party text in the UI.

Related terms

• System Prompt: System prompts are fixed or dynamic instructions sent with every request to steer model behavior (persona, refusals, output format, tool policies).
• Guardrails: Guardrails include system prompts, classifiers, allow/deny lists, output validators, rate limits, and human review gates applied before or after generation.
• Tool Use: The model emits structured calls (name + arguments) that your app executes, then feeds results back into the conversation for the next step.
• Agent: The model decides sequences of actions (search, draft, click, call APIs) within policies until the task completes or hits a human checkpoint.
• Retrieval-Augmented Generation (RAG): RAG = search step + generation step: find chunks likely to contain the answer, inject them into the prompt, then produce a reply grounded in those sources.

Explore more

• Agentic UX framework