Publications

Agentic systems are moving from prototypes to production, where they read untrusted inputs, call tools with real permissions, persist state, and act on a user's behalf, which expands the security and safety surface beyond chat-only models. Yet most standard evaluations remain single turn, for example multitask multiple choice and docstring-to-code tests, which are weak indicators of reliability in long-horizon settings where actions have consequences. Emerging agent benchmarks further show that interactive environments reveal qualitatively different failure modes compared with static leaderboards, underscoring the need for evaluations that track behavior over multi-step tasks and tool use. We present a systematic black-box framework for risk-aware agent evaluation that requires only basic system descriptions to initiate comprehensive red teaming. Our approach introduces: (1) a seven-domain taxonomy mapping observable behaviors to specific risk categories, (2) fully automated SAGE-RT powered red teaming producing 120 adversarial scenarios per domain without human intervention, and (3) human-in-the-loop evaluation of results where automated LLM judges provide initial scoring followed by expert validation of critical findings. Empirical validation across two production-ready agent architectures (single-agent CrewAI and multi-agent AutoGen) with four base models reveals alarming vulnerability patterns: 56.25% average governance risk across all systems, 65% privacy risk in multi-agent configurations, and critical agent behavior vulnerabilities reaching 85% in specific model-architecture combinations. Notably, our black-box approach discovered 98% of vulnerabilities identified by Unit 42's white-box analysis, while requiring 10x less manual effort. These findings demonstrate that systematic, taxonomy-guided evaluation can effectively identify architectural vulnerabilities without privileged access, providing a scalable path toward safer agent deployments.

Agentic AI · Red Teaming · NLP

As Large Language Models (LLMs) increasingly power decision-making systems across critical domains, understanding and mitigating their biases becomes essential for responsible AI deployment. Although bias assessment frameworks have proliferated for attributes such as race and gender, socioeconomic status bias remains significantly underexplored despite its widespread implications in the real world. We introduce SocioEval, a template-based framework for systematically evaluating socioeconomic bias in foundation models through decision-making tasks. Our hierarchical framework encompasses 8 themes and 18 topics, generating 240 prompts across 6 class-pair combinations. We evaluated 13 frontier LLMs on 3,120 responses using a rigorous three-stage annotation protocol, revealing substantial variation in bias rates (0.42%-33.75%). Our findings demonstrate that bias manifests differently across themes lifestyle judgments show 10x higher bias than education-related decisions and that deployment safeguards effectively prevent explicit discrimination but show brittleness to domain-specific stereotypes. SocioEval provides a scalable, extensible foundation for auditing class-based bias in language models.

Socioeconomic Status · Foundation Models · NLP

Frontier Large Language Models (LLMs) pose unprecedented dual-use risks through the potential proliferation of chemical, biological, radiological, and nuclear (CBRN) weapons knowledge. We present the first comprehensive evaluation of 10 leading commercial LLMs against both a novel 200-prompt CBRN dataset and a 180-prompt subset of the FORTRESS benchmark, using a rigorous three-tier attack methodology. Our findings expose critical safety vulnerabilities: Deep Inception attacks achieve 86.0% success versus 33.8% for direct requests, demonstrating superficial filtering mechanisms; Model safety performance varies dramatically from 2% (claude-opus-4) to 96% (mistral-small-latest) attack success rates; and eight models exceed 70% vulnerability when asked to enhance dangerous material properties. We identify fundamental brittleness in current safety alignment, where simple prompt engineering techniques bypass safeguards for dangerous CBRN information. These results challenge industry safety claims and highlight urgent needs for standardized evaluation frameworks, transparent safety metrics, and more robust alignment techniques to mitigate catastrophic misuse risks while preserving beneficial capabilities.

CBRN · Risk · Frontier Models · NLP

Partisan bias in LLMs has been evaluated to assess political leanings, typically through a broad lens and largely in Western contexts. We move beyond identifying general leanings to examine harmful, adversarial representational associations around political leaders and parties. To do so, we create datasets NeutQA-440 (non-adversarial prompts) and AdverQA-440 (adversarial prompts), which probe models for comparative plausibility judgments across the USA and India. Results show high susceptibility to biased partisan associations and pronounced asymmetries (e.g., substantially more favorable associations for U.S. Democrats than Republicans) alongside mixed-polarity concentration around India's BJP, highlighting systemic risks and motivating standardized, cross-cultural evaluation.

Partisan Bias · Large Language Models · NLP

As large language models (LLMs) and generative AI become widely adopted, guardrails have emerged as a key tool to ensure their safe use. However, adding guardrails isn't without tradeoffs; stronger security measures can reduce usability, while more flexible systems may leave gaps for adversarial attacks. In this work, we explore whether current guardrails effectively prevent misuse while maintaining practical utility. We introduce a framework to evaluate these tradeoffs, measuring how different guardrails balance risk, security, and usability, and build an efficient guardrail. Our findings confirm that there is no free lunch with guardrails; strengthening security often comes at the cost of usability. To address this, we propose a blueprint for designing better guardrails that minimize risk while maintaining usability.

Guardrails · AI Safety · Security · Usability · LLMs

We conduct a comprehensive investigation of implicit bias in large language models through systematic evaluation of over 50 different LLMs. Our study reveals concerning patterns of bias that persist across different model architectures, training methodologies, and deployment strategies. We provide insights into the sources of these biases and propose mitigation strategies for safer AI deployment.

Bias · Fairness · LLMs · Safety · Evaluation

This work presents a thorough evaluation of the SAGE-RT dataset's impact on model safety alignment. Through comparative studies across various model architectures and safety metrics, we demonstrate the dataset's effectiveness in improving alignment while maintaining model utility.

Safety Alignment · Dataset Evaluation · Model Safety · Comparative Study

We present VERA, a comprehensive framework for validation and enhancement of retrieval-augmented generation systems. Our approach addresses critical challenges in RAG system deployment including retrieval quality, generation consistency, and system reliability through systematic validation and enhancement techniques.

RAG · Retrieval Systems · Validation · Enhancement · NLP

Model quantization and fine-tuning have become essential techniques for deploying large language models in resource-constrained environments. However, their impact on model security has been largely overlooked. In this work, we conduct the first systematic study of how quantization and fine-tuning affect adversarial robustness in LLMs. Through comprehensive experiments across multiple model families and quantization schemes, we demonstrate that aggressive quantization can increase attack success rates significantly compared to full-precision models.

Quantization · Adversarial Attacks · Model Compression · Security · LLMs