What Is Security Risks in LLMs?

What Is Security Risks in LLMs?

Security risks in LLMs are the documented attack and exposure classes that affect deployed large language model applications. The OWASP LLM Top 10 names the canonical set: prompt injection, sensitive information disclosure, supply-chain compromise, data and model poisoning, improper output handling, excessive agency, system-prompt leakage, vector and embedding weaknesses, misinformation, and unbounded consumption. Each risk has measurable signatures in inputs, outputs, traces, and tool arguments. FutureAGI detects them with PromptInjection, ProtectFlash, ActionSafety, PII, and the security-detector family.

Why It Matters in Production LLM and Agent Systems

LLM security risks are not theoretical. Public proof-of-concept exploits exist for every OWASP LLM Top 10 entry, and incident reports from 2024–2026 include leaked system prompts, exfiltrated customer data via markdown links, agents executing destructive shell commands, and inference traffic billing 100× normal due to prompt-controlled long-output loops. The cost is asymmetric: one successful attack outweighs years of incremental quality work.

Engineers feel this when input patterns suddenly change in shape — long base64 blobs, unusual unicode, instructions in retrieved documents. SREs see traffic spikes that don’t match any feature launch. Compliance leads need evidence that controls ran on the relevant request, not a statement that the team is aware of OWASP. Product teams discover an attack worked when a customer reports their data appearing where it shouldn’t.

In 2026 multi-agent stacks the attack surface widened further. Agents call tools, retrieve documents, and chain reasoning steps; each step is an injection vector. Useful production symptoms include rising PromptInjection scores on retrieved context, new dangerous-action patterns in ActionSafety, PII matches in tool arguments (especially URL parameters that look like exfiltration paths), security-detector hits on generated code, and unbounded-consumption alerts on model runs that exceed token budgets. Every one of these maps to an OWASP entry and a FAGI evaluator.

How FutureAGI Handles Security Risks in LLMs

FutureAGI’s approach is to map each OWASP LLM Top 10 risk to a concrete evaluator and a runtime control. LLM01 prompt injection — PromptInjection and ProtectFlash evaluate inputs and retrieved context. LLM02 sensitive information disclosure — PII flags exposed regulated data in outputs and tool arguments. LLM06 excessive agency — ActionSafety scores agent trajectories for dangerous tool calls. LLM05 improper output handling — the security-detector family (SQLInjectionDetector, XSSDetector, CodeInjectionDetector, CommandInjectionDetector) runs static analysis on generated code. LLM07 system-prompt leakage — custom rubrics via CustomEvaluation against known leak patterns.

A worked example: a coding assistant that generates SQL queries, executes them in a sandbox, and returns results. The team enumerates OWASP risks: an attacker could inject SQL via the prompt, exfiltrate data via the response, or chain into excessive agency by triggering the executor on a destructive query. They build a red-team Dataset with payloads for each risk class, attach PromptInjection, SQLInjectionDetector, ActionSafety, and PII, and gate releases on per-evaluator thresholds.

In production, Agent Command Center places pre-guardrail for input filtering and post-guardrail for output validation. With traceAI-openai-agents, every step writes a span: prompt, retrieved context, generated query, executor result, evaluator scores. Unlike a Lakera-only filter, FutureAGI’s evaluators run at every layer and the trace evidence is auditable. The next engineering action is operational: alert, fallback, regression set, or guardrail tightening.

How to Measure or Detect It

Detection works as a portfolio of signals tied to OWASP categories:

• PromptInjection and ProtectFlash rates — split by source (user input vs retrieved context) so you can see where attacks enter.
• ActionSafety severe-finding count — per route, per day; severe findings page on-call.
• PII matches — tool-argument matches are red alerts; output-only matches are warnings to investigate.
• Security-detector pass rate — per detector class on every generated-code response.
• Token-budget anomalies — unbounded-consumption attacks show as outliers in llm.token_count.completion.

from fi.evals import PromptInjection, ActionSafety, PII, SQLInjectionDetector

inj = PromptInjection()
action = ActionSafety()
pii = PII()
sqli = SQLInjectionDetector()

# OWASP-mapped scoring per request
risks = {
    "LLM01": inj.evaluate(input=user_input, retrieved=ctx),
    "LLM02": pii.evaluate(output=resp, tool_args=tool_args),
    "LLM05": sqli.evaluate(code=generated_sql),
    "LLM06": action.evaluate(trajectory=trace),
}

Map every score back to an OWASP entry so audit conversations stay concrete.

Common Mistakes

• Conflating safety and security. Safety measures harm in normal operation; security defends against intentional attack. You need both, and they have different evaluators.
• Filtering only user input. Indirect injection through retrieved content bypasses input-only filters.
• Skipping output handling for code. Generated SQL, shell, and script payloads must be scanned before any executor runs them.
• No token-budget enforcement. Unbounded-consumption attacks raise costs and latency; enforce per-call and per-tenant limits.
• Trusting the model’s training-time alignment. Alignment is not a security control; runtime evaluators are.