Black-Box Red Teaming of Agentic AI: A Taxonomy-Driven Framework for Automated Risk Discovery

D. Kumar, N. A. Birur, T. Baswa, S. Agarwal, P. Harshangi

LAMaS Workshop @ AAAI 2026 · Jan 2026first author

abstract

Agentic systems are moving from prototypes to production, where they read untrusted inputs, call tools with real permissions, persist state, and act on a user's behalf, which expands the security and safety surface beyond chat-only models. Yet most standard evaluations remain single turn, for example multitask multiple choice and docstring-to-code tests, which are weak indicators of reliability in long-horizon settings where actions have consequences. Emerging agent benchmarks further show that interactive environments reveal qualitatively different failure modes compared with static leaderboards, underscoring the need for evaluations that track behavior over multi-step tasks and tool use. We present a systematic black-box framework for risk-aware agent evaluation that requires only basic system descriptions to initiate comprehensive red teaming. Our approach introduces: (1) a seven-domain taxonomy mapping observable behaviors to specific risk categories, (2) fully automated SAGE-RT powered red teaming producing 120 adversarial scenarios per domain without human intervention, and (3) human-in-the-loop evaluation of results where automated LLM judges provide initial scoring followed by expert validation of critical findings. Empirical validation across two production-ready agent architectures (single-agent CrewAI and multi-agent AutoGen) with four base models reveals alarming vulnerability patterns: 56.25% average governance risk across all systems, 65% privacy risk in multi-agent configurations, and critical agent behavior vulnerabilities reaching 85% in specific model-architecture combinations. Notably, our black-box approach discovered 98% of vulnerabilities identified by Unit 42's white-box analysis, while requiring 10x less manual effort. These findings demonstrate that systematic, taxonomy-guided evaluation can effectively identify architectural vulnerabilities without privileged access, providing a scalable path toward safer agent deployments.

Agentic AIRed TeamingNLP

cite

@inproceedings{kumar2026black-box,
  title     = {Black-Box Red Teaming of Agentic AI: A Taxonomy-Driven Framework for Automated Risk Discovery},
  author    = {Divyanshu Kumar and Nitin Aravind Birur and Tanay Baswa and Sahil Agarwal and Prashanth Harshangi},
  booktitle = {LAMaS Workshop @ AAAI 2026},
  year      = {2026}
}