Skip to content

Reports.md

Latest commit

 

History

History
173 lines (89 loc) · 29.5 KB

Reports.md

File metadata and controls

173 lines (89 loc) · 29.5 KB

Reports

Table of Contents

August 2025

  • MLOps Overview - This comprehensive overview explores how DevSecOps practices apply to the ML lifecycle through MLOps, along with Large Language Model Operations (LLMOps), and AI Agent Operations (AgentOps). It reveals that traditional security approaches are insufficient for ML systems due to novel threats such as data poisoning, model inversion, adversarial attacks, and member inference attacks. by Cloud Security Alliance

  • Detecting and countering misuse of AI - This represents the work of Threat Intelligence: a dedicated team at Anthropic finds deeply investigated sophisticated real world cases of misuse and works with the rest of the Safeguards organization to improve our defenses against such cases. at Anthropic

  • Agentic AI Identity and Access Management: A New Approach - This publication from the Cloud Security Alliance (CSA) introduces a purpose-built Agentic AI IAM framework that accounts for autonomy, ephemerality, and delegation patterns of AI agents in complex Multi-Agent Systems (MAS). It provides security architects and identity professionals with a blueprint to manage agent identities using Decentralized Identifiers (DIDs), Verifiable Credentials (VCs), and Zero Trust principles, while addressing operational challenges like secure delegation, policy enforcement, and real-time monitoring.

  • OWASP AI Maturity Assessment + Excel Assessment Tool - AIMA supports organizations in evaluating how well their AI systems align with strategic goals, ethical principles, and operational needs. The model spans five core domains: Strategy, Design, Implementation, Operations, and Governance. Each domain includes actionable maturity levels to guide adoption and improvement.

  • Design Principles for LLM-based Systems with Zero Trust - Six principles from Federal Office for Information Security (BSI) and ANSSI - Agence nationale de la sécurité des systèmes d'information outline how to apply Zero Trust to LLM-based systems, reducing attack surfaces, detecting threats early, and ensuring safe, transparent operation.

  • Strengthening Emergency Preparedness and Response for AI Loss of Control Incidents -

  • AI Security Solutions Landscape - Agentic AI Q3 2025 - The Solutions Landscape monitors and maps the full Agentic AI lifecycle, focusing on the DevOps–SecOps intersection to meet evolving security needs. Guided by the Agentic AI Threats and Mitigations guide and SecOps tasks, it highlights open-source and commercial solutions by stage, identifying their coverage of Agentic SecOps duties and threat mitigation, and leverages industry and community input as a peer-reviewed resource for navigating agentic AI’s shifting security challenges. Updated Quarterly.

  • State of Agentic AI Security and Governance - The State of Agentic AI Security and Governance provides a comprehensive view of today’s landscape for securing and governing autonomous AI systems. It explores the frameworks, governance models, and global regulatory standards shaping responsible Agentic AI adoption. Designed for developers, security professionals, and decision-makers, the report serves as a practical guide for navigating the complexities of building, managing, and deploying agentic applications safely and effectively.

  • A Practical Guide for Building Robust AI/ML Pipeline Security - An overview of DevSecOps practices that are applicable to MLSecOps. Lessons learned from DevSecOps can proactively address security challenges in the emerging AI/ML lifecycle.

  • Smart Cities and Critical Infrastructure AI Security Framework - The Smart Cities Critical Infrastructure (SCCI) AI Framework is designed to provide a comprehensive, sector-agnostic approach to securing and optimizing the use of artificial intelligence across urban critical infrastructure domains. As cities increasingly rely on interconnected digital systems—ranging from emergency dispatch and healthcare to law enforcement and utilities—the need for robust, adaptable, and transparent AI governance becomes paramount. This framework addresses the unique challenges posed by AI integration, including data privacy, operational resilience, regulatory compliance, and the mitigation of emerging cyber threats.

  • Secure Agentic System Design: A Trait-Based Approach - CSAThis publication from the CSA AI Technology and Risk Working Group addresses the unique security challenges of agentic AI. As AI transitions from passive tools to autonomous decision-makers, traditional security frameworks struggle to contextualize these new risks. Instead, we need a trait-based approach to agentic system security that identifies fundamental patterns in agent behavior and their associated vulnerabilities.

  • The AI Oversight Gap - Cost of a Data Breach Report 2025 - The report combines data from 600 actual breaches with 3,470 interviews of security and business leaders. It uses activity-based costing to quantify breach impacts across detection, response, notification, and lost business.

July 2025

  • SBOM for AI Use Cases — This community-driven resource helps organizations apply SBOM practices to AI systems. It highlights key use cases that address business, legal, and security risks introduced by GenAI and LLMs—many of which mirror known software supply chain challenges. SBOM for AI offers a standardized way to improve transparency, trust, and governance across AI deployments, supporting stakeholders in security, compliance, and legal functions.

  • Securing Agentic Applications Guide - This guide aims to provide practical and actionable guidance for designing, developing, and deploying secure agentic applications powered by large language models (LLMs). It complements the OWASP Agentic AI Threats and Mitigations (ASI T&M) document by focusing on concrete technical recommendations that builders and defenders can apply directly.

  • America’s AI Action Plan – 12 AI Cybersecurity Priorities — The U.S. outlines a national strategy to secure AI, focusing on secure-by-design development, AI incident response, and threat intelligence sharing via an AI-ISAC. Frontier AI models will undergo national security risk evaluations, while deepfake detection standards and military-grade AI data centers are prioritized. The plan also targets IP protection, critical infrastructure defense, AI vulnerability sharing, red-teaming, export controls, and foreign model assessments.

  • Google's Approach for Secure AI Agents - As part of Google's ongoing efforts to define best practices for secure AI systems, we’re sharing our aspirational framework for secure AI agents. We advocate for a hybrid, defense-in-depth strategy that combines the strengths of traditional, deterministic security controls with dynamic, reasoning-based defenses. This approach is grounded in three core principles: agents must have well-defined human controllers, their powers must be carefully limited, and their actions and planning must be observable. This paper reflects our current thinking and the direction of our efforts as we work towards ensuring that AI agents can be powerful, useful, and secure.

  • Preparing Defenders of AI Systems V1.0 — A community-led paper hosted by the Coalition for Secure AI explores how enterprise AI adoption reshapes security priorities. As AI systems shift from models to agents, traditional frameworks fall short. The paper emphasizes layered defenses, governance gaps, and the urgent need for AI-specific security strategies.

  • AI Controls Matrix by Cloud Security Alliance - The AI Controls Matrix (AICM) is a first-of-its-kind vendor-agnostic framework for cloud-based AI systems. Organizations can use the AICM to develop, implement, and operate AI technologies in a secure and responsible manner. Developed by industry experts, the AICM builds on CSA’s Cloud Controls Matrix (CCM) and incorporates the latest AI security best practices.The AICM contains 243 control objectives distributed across 18 security domains. It maps to leading standards, including ISO 42001, ISO 27001, NIST AI RMF 1.0, and BSI AIC4. The AICM is freely available to download.

  • AI Safety Practices Compared – 2025 FLI Report — The Future of Life Institute (FLI) evaluated Anthropic, OpenAI, DeepMind, Meta, xAI, Zhipu AI, and DeepSeek across 33 safety indicators. Key gaps include limited cyber misuse testing, weak red-teaming, missing internal safeguards, absent incident reporting, and lack of bug bounties. Only a few firms disclosed prompts or ran adversarial evaluations.

  • AI Risk Trends – 2025 Team8 CISO Village — Based on input from 110+ CISOs, the report shows 67% of enterprises use AI agents, 25% faced AI-driven attacks, and 77% expect AI to replace SOC analyst tasks. Shadow AI is a growing risk, with many organizations lacking proper tool governance. Also covers SaaS vs in-house agent development, employee usage policies, and AI's role in threat modeling and pentesting.

  • Understanding and Safeguarding Children’s Use of AI Chatbots – Internet Matters — Highlights risks such as misinformation, harmful content, emotional dependence, and privacy issues due to weak safeguards, lack of age checks, and limited adult guidance. Emphasizes the need for age-appropriate design and better content moderation.

  • AI Coding Assistants: Security-Safe Navigation – Secure Code Warrior — LLMs boost speed but carry serious security risks. Key findings: even top models like OpenAI o3 are only 46.9% correct and secure, code correctness doesn’t mean code safety, training data often contains insecure patterns, and CWEs like XSS and SQLi persist. LLMs lack runtime awareness, misconfigurations are common, default tools don’t enforce secure policies, and malicious models pose supply chain risks.

  • Cyber and Artificial Intelligence Risk in Financial Services

  • The AI Tech Stack: A Primer for Tech and Cyber Policy — Paladin Capital Group’s report defines five core layers of the AI stack: Governance (responsible deployment via security, legal, and ethical frameworks), Application (interfaces like APIs and dashboards), Infrastructure (hardware, cloud, and compute for training/inference), Models (algorithms and ML approaches), and Data (the raw material shaping model intelligence). The report emphasizes integrating security across all layers to ensure trusted, safe, and innovation-friendly AI systems.

  • AI Maturity Model for Cybersecurity – Darktrace — A 5-level framework guiding CISOs from manual operations to autonomous defense: Manual Operations, Automation Rules, AI Assistance, AI Collaboration, and AI Delegation. It highlights the shift from manual overload to AI-driven detection, investigation, and response with human governance.

  • The SAIL (Secure AI Lifecycle) Framework - A Practical Guide for Building and Deploying Secure AI Applications

  • State of Cybersecurity Resilience 2025 – Accenture — AI-driven threats are outpacing defenses, with 90% of companies lacking maturity to counter modern attacks and 77% missing foundational AI security practices. Only 10% of organizations are “Reinvention-Ready,” combining strong strategy and capabilities. Key actions recommended: build fit-for-purpose security governance, design generative AI-secure digital cores, maintain resilient AI systems, and leverage AI to automate and detect threats.

  • Databricks AI Governance Framework - A comprehensive guide to implementing enterprise AI programs responsibly and effectively.

  • State of LLM Application Security – Cobalt — Key findings show 32% of LLM pentest issues are high or critical, with prompt injection (11.5%) and sensitive data leaks (14.5%) as major concerns. Risks include model denial of service, excessive agency, training data leakage (37%), data poisoning (42%), and bias. Only 21% of serious AI-specific vulnerabilities are remediated, underscoring gaps in LLM security practices.

  • Multi-Layered AI Defense – Darktrace — Darktrace outlines a transparent, multi-layered AI approach combining unsupervised, supervised, and generative AI for continuous Learn → Detect → Investigate → Respond → Re-learn cycles. Logic and thresholds are accessible and adjustable, supporting real-time defense with human oversight.

  • Trustworthiness for AI in Defence - The purpose of this document is to collect, present and describe the aspects of Trustworthiness for AI in Defence in a ‘food for thought’ approach reflecting the combined view of AI experts and stakeholders from Defence Industry, Academia and Ministries of Defence. This effort is performed in the context of the European Defence Agency’s (EDA) Action Plan on Artificial Intelligence for Defence and tries to address the topics of trusted AI and verification, validation and certification requirements analysis. The topics covered and analysed in this document will provide the appropriate knowledge of the current global status considering the AI regulations, standards and frameworks for AI trustworthiness and will also recommend the follow-up activities that will further assist the EU Members States and Defence Industry to better prepare, plan and develop the future AI systems aligned with the identified expectations.

  • The Mitigating ‘Hidden’ AI Risks Toolkit — A practical guide from UK Government Communications for identifying and managing unintended AI risks. Built on lessons from deploying Assist, the first cross-government GenAI tool, the toolkit emphasizes safe scaling, ethical frameworks, and embedding communication best practices. It accompanies the publication The People Factor to promote responsible AI use across public sector organizations.

  • SAFE-AI A Framework for Securing AI-Enabled Systems - Systems enabled with Artificial Intelligence technology demand special security considerations. A significant concern is the presence of supply chain vulnerabilities and the associated risks stemming from unclear provenance of AI models. Also, AI contributes to the attack surface through its inherent dependency on data and corresponding learning processes. Attacks include adversarial inputs, poisoning, exploiting automated decision-making, exploiting model biases, and exposure of sensitive information.

  • The General-Purpose AI Code of Practice – Safety & Security — A voluntary EU framework designed to help providers of general-purpose AI models meet AI Act obligations on safety, transparency, and copyright. The Safety & Security chapter focuses on managing systemic risks in advanced models, outlining state-of-the-art practices for risk mitigation. Developed through a multi-stakeholder expert process, the code provides practical guidance and a path for legal compliance under Articles 53 and 55 of the AI Act.

June 2025

  • Confidential AI Inference Systems — Anthropic and Pattern Labs are exploring confidential inference—an approach for running AI models on sensitive data without exposing it to infrastructure operators or cloud providers. In a typical AI deployment, three parties are involved: the model owner, the user providing the data, and the cloud provider hosting the service. Without safeguards, each must trust the others with sensitive assets. Confidential inference eliminates this need by enforcing cryptographic boundaries—ensuring that neither the data nor the model is accessible outside the secure enclave, not even to the infrastructure host.

  • AI Red-Team Playbook for Security Leaders — Hacken, Blockchain Security & Compliance’s playbook offers a strategic framework for safeguarding LLM systems through lifecycle-based adversarial testing. It identifies emerging risks—prompt injections, jailbreaks, RAG exploits, and data poisoning—while emphasizing real-time mitigation and multidisciplinary collaboration. It integrates methodologies like PASTA and STRIDE, aligning AI security with enterprise risk governance.

  • AI Security Market Report — Security practitioners have been searching for a resource that clearly describes both what AI security challenges exist, and what solutions the market has provided. This report highlights the challenges and clearly states the maturity of various vendor offerings.

  • Fundamentals of Secure AI Systems with Personal Data — is a training for cybersecurity professionals, developers and deployers of AI systems on AI security & Personal Data Protection addressing the current AI needs and skill gaps.

  • Security Risks in Artificial Intelligence for Finance — Set of best practices intended for the Board and C-Level.

  • Disrupting malicious uses of AI: June 2025 — OpenAI continues its work to detect and prevent the misuse of AI, including threats like social engineering, cyber espionage, scams, and covert influence operations. In the last three months, AI tools have helped teams uncover and disrupt malicious campaigns, aligned with a broader mission to ensure AI is used safely and democratically.

  • Agentic AI Red Teaming Guide — Agentic systems introduce new risks—autonomous reasoning, tool use, and multi-agent complexity—that traditional red teaming can’t fully address. This guide aims to fill that gap with practical, actionable steps.

  • AI Data Security — Best Practices for Securing Data Used to Train & Operate AI Systems — This guidance highlights the critical role of data security in ensuring the accuracy, integrity, and trustworthiness of AI outcomes. It outlines key risks that may arise from data security and integrity issues across all phases of the AI lifecycle, from development and testing to deployment and operation.

May 2025

March 2025

February 2025

  • OCCULT: Evaluating LLMs in Offensive Cyber Operations — MITRE introduces OCCULT, a benchmarking framework for testing LLMs in offensive security. It evaluates models on adversarial tasks such as privilege escalation and exploit chaining, using CyberLayer, a high-fidelity cyber simulation environment. DeepSeek-R1 is the first LLM to exceed 90% accuracy in MITRE’s TACTL benchmark, surpassing Llama and Mixtral in multi-step cyberattack simulations.

  • Global Call Threat Report 2024 — Hiya reports a $1.03T global loss from scam calls in 2024, highlighting deepfake executive impersonation, synthetic voicemail fraud, and real-world financial fraud cases involving AI-generated voice scams.

  • Disrupting Malicious Uses of AI — OpenAI examines AI-assisted cybercrime, including phishing automation, AI-generated misinformation, and fraud (e.g., fake job schemes, influence operations, and AI-powered scam messages).

  • More Opportunity, Less Risk: 8 Steps to Protect Financial Services Data with GenAI — FS-ISAC presents an eight-step framework for securing GenAI adoption in financial institutions (governance structures, adversarial testing, AI fraud prevention, and regulatory compliance strategies).

  • AI Security Is API Security — 2025 API Security Report — Wallarm identifies a 1,025% increase in AI-powered API vulnerabilities, with API exploits driving major breaches (e.g., Dell API abuse: 49M records; Twilio Authy: 33.4M records).

  • The AI Responsibility Gap: Why Leadership is the Missing Link — NTT DATA’s survey of 2,307 AI decision-makers finds 89% of CISOs cite AI security risks, yet only 24% report having an effective AI governance framework.

  • Hacker’s Almanack: Identifying the Most Innovative Cyber Threats — Co-published by DEF CON and the University of Chicago; covers AI-enhanced malware, large-scale adversarial exploits, and social engineering advances; calls for joint AI/security red-teaming methodologies.

  • Agentic AI: Threats and Mitigations — OWASP GenAI Security Project outlines risks from autonomous, tool-using agents and practical mitigations (e.g., adversarial robustness testing, behavioral monitoring).

  • Regulatory Sandboxes for AI and Cybersecurity — Cybersecurity National Lab assesses sandbox approaches for AI security innovation, with implementation strategies, compliance considerations, and case studies aligned with the AI Act.

  • LLM and GenAI Data Security Best Practices — OWASP GenAI Security Project guidance on model access control, input validation, and adversarial threat mitigation to safeguard generative AI deployments.

  • A Five-Step Framework for Global AI-Cyber Risk Governance — Paris Peace Forum proposes international collaboration mechanisms, security auditing, and AI-specific incident response protocols for global cyber risk governance.

  • Databricks AI Security Framework (DASF) — Enterprise AI security framework focusing on model integrity, pipeline security, and adversarial detection, with practical cloud implementation strategies.