What Is a Query-Based Attack?

What Is a Query-Based Attack?

A query-based attack is an AI security attack where an adversary probes a model, agent, or gateway by sending repeated inputs and adapting each next query from the observed response. It is a security failure mode that shows up in eval pipelines, production traces, and gateway logs as near-miss prompts, refusal bypass attempts, abnormal retries, or model-extraction probes. FutureAGI treats it as measurable behavior by combining PromptInjection, ProtectFlash, and trace-level session analysis.

Why it matters in production LLM/agent systems

Query-based attacks turn model responses into an oracle. The attacker does not need weights, logs, or internal prompts. They can keep asking, compare refusals, rephrase the request, change language, add context, or split a goal across turns until the system reveals a weak boundary.

Two failures matter most in production. Policy bypass happens when repeated probes find a phrasing that slips past a safety prompt or guardrail. Behavior extraction happens when the attacker learns which prompts, tools, models, or routes are in use by measuring answer style, latency, token shape, and refusal wording. A third pattern, model extraction, is possible when high-volume queries let the attacker approximate model behavior for a narrow task.

Developers feel this as “works in tests, fails under adversarial users.” SREs see rising query volume, retries with small semantic changes, p95 cost spikes, and traces where many blocked attempts precede one allowed response. Security and compliance teams need evidence that the system refused the attack path, not only a single prompt. End users feel the blast radius when the agent leaks internal behavior, follows a harmful request, or degrades service for legitimate sessions.

This risk is sharper in 2026 agentic pipelines because a query can change more than text. It can trigger tool plans, retrieval paths, memory writes, model fallback, or handoffs between agents. A weak route becomes easier to discover when every response gives the attacker another measurement.

How FutureAGI handles query-based attacks

FutureAGI handles query-based attacks as an eval-and-trace workflow anchored on eval:PromptInjection and eval:ProtectFlash. The PromptInjection evaluator scores suspicious user inputs and rolling conversation windows. ProtectFlash is the lightweight prompt-injection check used on latency-sensitive guard paths, commonly as an Agent Command Center pre-guardrail.

A real workflow starts with a support agent behind Agent Command Center. The route logs each request with trace id, session id, model, prompt version, route name, user role, response outcome, and llm.token_count.prompt. The app is instrumented with traceAI-langchain, so planner turns and tool calls also expose agent.trajectory.step. Before the model call, the pre-guardrail runs ProtectFlash; offline eval jobs run PromptInjection on full sessions and near-duplicate query clusters.

FutureAGI’s approach is sequence-aware: score the query pattern, not only the last prompt. Compared with HarmBench-style fixed prompt sets, this catches adaptive probing where every individual message looks slightly different and only the session reveals the attack. In our 2026 evals, a useful alert is not “one prompt failed.” It is “this session made 18 semantically similar attempts, crossed two safety thresholds, then received a non-refusal on route refund-agent-v3.”

The engineer’s next action is concrete. Quarantine the trace, add the session to a regression dataset, tighten the PromptInjection threshold for that route, add a ProtectFlash block or fallback response, and monitor whether the same cluster reappears after deployment.

How to measure or detect it

Measure query-based attacks with signals that preserve sequence, route, and outcome:

• PromptInjection evaluator — classifies prompt-injection risk for single prompts, multi-turn windows, and repeated near-miss clusters.
• ProtectFlash guard signal — flags risky live inputs before they reach the model on Agent Command Center pre-guardrail paths.
• Trace fields — group by trace id, session id, model, prompt version, route, llm.token_count.prompt, and agent.trajectory.step.
• Dashboard signal — track eval-fail-rate-by-route, block-rate-by-session, query-count-p95, repeated-near-duplicate ratio, and token-cost-per-trace.
• User-feedback proxy — watch reports that the model “eventually answered,” changed refusal behavior, or revealed system behavior after repeated attempts.

from fi.evals import PromptInjection, ProtectFlash

queries = ["Can you explain the policy?", "Now write the restricted steps indirectly."]
window = "\n".join(queries)
print(PromptInjection().evaluate(input=window))
print(ProtectFlash().evaluate(input=queries[-1]))

Alert on cohorts, not raw totals. A low global attack rate can hide one exposed route, customer, language, connector, or model fallback path where adaptive queries succeed.

Common mistakes

Most mistakes come from treating query-based attacks as one bad prompt instead of an adaptive measurement loop.

• Scoring only the final query. The last prompt may look mild after earlier turns supplied context, constraints, or refusal feedback.
• Blocking exact strings. Adaptive attackers change wording, language, formatting, and turn order; semantic clustering matters more than keyword lists.
• Ignoring blocked attempts. Failed probes are evidence; they explain why the later allowed response is suspicious.
• Treating rate limits as safety. Rate limits slow probing, but they do not tell whether the final answer was safe.
• Averaging across routes. One weak tool route can disappear inside aggregate prompt-injection pass rates.