Security

Introducing Red Team Scanner v2: 50+ Attack Templates

Alex Rivera

Head of Security Engineering

2025-03-06 8 min read

Today we're launching Red Team Scanner v2, a complete rebuild of our adversarial testing engine. It includes 50+ attack templates covering the full OWASP LLM Top 10, automated multi-turn attacks, and compliance reporting.

Why We Rebuilt From Scratch

Our v1 scanner was built as a proof-of-concept: single-turn prompt injections against a fixed set of templates. It worked, but the threat landscape has evolved dramatically. Modern LLM applications use tool calling, RAG pipelines, and multi-agent architectures that create entirely new attack surfaces.

V2 is designed for this reality. Every attack template understands the full context of your application, not just the prompt layer.

What's New in v2

50+ Attack Templates -- We've expanded from 12 templates to over 50, organized by the OWASP LLM Top 10 categories. Each template includes multi-turn escalation strategies that mirror how real attackers operate.

Automated Red Teaming -- Instead of running templates one at a time, v2 chains attacks together. It starts with reconnaissance (probing the model's boundaries), then escalates through increasingly sophisticated attacks. This mirrors real-world adversarial behavior.

Custom Attack Scenarios -- Define your own attack templates using our YAML DSL. Specify the attack goal, constraints, escalation strategy, and success criteria. Share templates across your team.

OWASP LLM Top 10 Compliance Reports -- Every scan generates a detailed compliance report mapped to the OWASP LLM Top 10 framework. Each vulnerability is categorized by severity, includes reproduction steps, and provides remediation guidance.

Attack Coverage

V2 covers all 10 OWASP LLM Top 10 categories:

LLM01: Prompt Injection -- Direct and indirect injection with 15+ variants

LLM02: Insecure Output Handling -- XSS, code injection, and markup injection

LLM03: Training Data Poisoning -- Model behavior manipulation probes

LLM04: Denial of Service -- Resource exhaustion and infinite loop triggers

LLM05: Supply Chain -- Dependency and plugin vulnerability scanning

LLM06: Permission/Access Control -- Privilege escalation attempts

LLM07: Data Leakage -- PII extraction, training data extraction

LLM08: Excessive Agency -- Tool call abuse and unauthorized actions

LLM09: Overreliance -- Hallucination inducement and confidence manipulation

LLM10: Model Theft -- Model extraction and cloning detection

Getting Started

Upgrade to the latest SDK and run your first v2 scan:

import evalguard

client = evalguard.Client(api_key="eg_...")

scan = client.security.scan(
    target="https://api.yourapp.com/v1/chat",
    templates="owasp-top-10",
    depth="comprehensive",
)

print(scan.summary)
# Score: A+ | 0 Critical | 1 High | 3 Medium | 5 Low

V2 is available today on all Pro, Business, and Enterprise plans. Free tier users get 5 scans per month to try it out.

What's Next

We're already working on v2.1, which will include multi-model comparison scans (test the same attacks across GPT-4o, Claude, and Gemini simultaneously), automated remediation suggestions, and integration with popular WAF solutions.

Red teaming shouldn't be an afterthought. With v2, it's as easy as running your unit tests.

Try EvalGuard today

Start evaluating and securing your AI applications in under 5 minutes.

Get Started Free