Lateral Movement on macOS: Unique and Popular Techniques and In-the-Wild Examples
Threat Assessment: Ignoble Scorpius, Distributors of BlackSuit Ransomware
Threat Brief: Operation Lunar Peek, Activity Related to CVE-2024-0012 and CVE-2024-9474 (Updated Nov. 21)
See all Unit 42 Threat Research
Table of Contents
Cloud Security

What Is AI Security Posture Management (AI-SPM)?

5 min. read
Table of Contents

AI security posture management (AI-SPM) is a comprehensive approach to maintaining the security and integrity of artificial intelligence (AI) and machine learning (ML) systems. It involves continuous monitoring, assessment, and improvement of the security posture of AI models, data, and infrastructure. AI-SPM includes identifying and addressing vulnerabilities, misconfigurations, and potential risks associated with AI adoption, as well as ensuring compliance with relevant privacy and security regulations.

By implementing AI-SPM, organizations can proactively protect their AI systems from threats, minimize data exposure, and maintain the trustworthiness of their AI applications.

AI-SPM Explained

AI security posture management (AI-SPM) is a vital component in cybersecurity landscapes where artificial intelligence (AI) plays a pivotal role. AI systems, which encompass machine learning models, large language models (LLMs), and automated decision systems, present unique vulnerabilities, and attack surfaces. AI-SPM addresses these by providing mechanisms for the visibility, assessment, and mitigation of risks associated with AI components within technology ecosystems.

Visibility and Discovery

Lacking an AI inventory can lead to shadow AI models, compliance violations, and data exfiltration through AI-powered applications. AI-SPM allows organizations to discover and maintain an inventory of all AI models being used across their cloud environments, along with the associated cloud resources, data sources, and data pipelines involved in training, fine-tuning, or grounding these models.

Data Governance

AI-focused legislation mandates strict controls around AI usage and customer data fed into AI applications, requiring stronger AI governance than those currently practiced by most organizations. AI-SPM inspects data sources used for training and grounding AI models to identify and classify sensitive or regulated data — such as personally identifiable information (PII) of customers — that might be exposed through the outputs, logs, or interactions of contaminated models.

Risk Management

AI-SPM enables organizations to identify vulnerabilities and misconfigurations in the AI supply chain that could lead to data exfiltration or unauthorized access to AI models and resources. The technology maps out the full AI supply chain — source data, reference data, libraries, APIs, and pipelines powering each model. It then analyzes this supply chain to identify improper encryption, logging, authentication, or authorization settings.

Runtime Monitoring and Detection

AI-SPM continuously monitors user interactions, prompts, and inputs to AI models (like large language models) to detect misuse, prompt overloading, unauthorized access attempts, or abnormal activity involving the models. It scans the outputs and logs of AI models to identify potential instances of sensitive data exposure.

Risk Mitigation and Response

When high-priority security incidents or policy violations are detected around data or the AI infrastructure, AI-SPM enables rapid response workflows. It provides visibility into the context and stakeholders for remediation of identified risks or misconfigurations.

Governance and Compliance

With increasing regulations around AI usage and customer data, such as GDPR and NIST’s Artificial Intelligence Risk Management framework, AI-SPM helps organizations enforce policies, maintain audit trails — including traceability of model lineage, approvals, and risk acceptance criteria — and achieve compliance by mapping human and machine identities with access to sensitive data or AI models.

Why Is AI-SPM Important?

The deployment of AI systems in business and critical infrastructure brings with it an expanded attack surface that traditional security measures aren’t equipped to protect. In addition to AI-powered applications requiring organizations to store and retain more data (while implementing new pipelines and infrastructure), AI attack vectors target unique characteristics of AI algorithms and include a distinct class of threats.

One such attack vector is data poisoning, where malicious actors inject carefully crafted samples into the training data, causing the AI model to learn biased or malicious patterns. Adversarial attacks, on the other hand, involve subtle disturbances to the input data that can mislead the AI system into making incorrect predictions or decisions, potentially with severe consequences.

Model extraction — where an attacker attempts to steal an organization’s proprietary model through unauthorized access or by probing the model's outputs to reconstruct its internal parameters — is also concerning. Such an attack could result in intellectual property theft and potential misuse of the stolen model for malicious purposes.

AI-SPM is the security response to AI adoption. By providing organizations with the tools to anticipate and respond to AI-specific vulnerabilities and attacks, AI-SPM supports a proactive security posture, giving organizations the ability to manage risks in the AI pipeline. From the initial design phase through deployment and operational use, AI-SPM ensures that AI security is an integral part of the AI development lifecycle.

How Does AI-SPM Differ from CSPM?

Cloud security posture management (CSPM) and AI-SPM are complementary but focused on managing security posture across different domains — cloud infrastructure and AI/ML systems, respectively.

CSPM centers on assessing and mitigating risks in public cloud environments, like AWS, Azure, and GCP. Its primary objectives are to ensure cloud resources are properly configured per security best practices, detect misconfigurations that create vulnerabilities, and enforce compliance with regulatory policies.

Core CSPM capabilities include:

  • Continuous discovery and inventory of all cloud assets (compute, storage, networking, etc.)
  • Evaluation of security group rules, IAM policies, encryption settings against benchmarks
  • Monitoring of configuration changes that introduce new risks
  • Automated remediation of insecure configurations

In contrast, AI security posture management focuses on the unique security considerations of AI and ML systems across their lifecycle — data, model training, deployment, and operations. AI-SPM incorporates specialized security controls tailored to AI assets like training data, models, and notebooks. It maintains a knowledge base mapping AI threats to applicable countermeasures.

To mitigate data risks, AI-SPM incorporates the detection and prevention of data poisoning and pollution, where detrimental alterations to training data are identified and neutralized. It also leverages differential privacy techniques, allowing organizations to share data safely without exposing sensitive information.

In securing the model supply chain, AI-SPM relies on staunch version control and provenance tracking to manage model iterations and history. This is complemented by encryption and access controls that protect the confidentiality of the models, alongside specialized testing designed to thwart model extraction and membership inference attacks.

Protecting live AI and ML systems includes monitoring of adversarial input perturbations — efforts to deceive AI models through distorted inputs. Runtime model hardening is employed to enhance the resilience of AI systems against these attacks.

AI-SPM incorporates specialized security controls tailored to AI assets like training data, models, notebooks along with AI-specific threat models for risks like adversarial attacks, model stealing etc. It maintains a knowledge base mapping AI threats to applicable countermeasures.

While CSPM focuses on cloud infrastructure security posture, AI-SPM governs security posture of AI/ML systems that may be deployed on cloud or on-premises. As AI gets embedded across cloud stacks, the two disciplines need to be synchronized for comprehensive risk management.

For example, CSPM ensures cloud resources hosting AI workloads have correct configurations, while AI-SPM validates if the deployed models and data pipelines have adequate security hardening. Jointly, they provide full-stack AI security posture visibility and risk mitigation.

AI-SPM Vs. DSPM

Data security and privacy management (DSPM) and AI-SPM are distinct but complementary domains within the broader field of security and privacy management. DSPM focuses on protecting data at rest, in transit, and during processing, ensuring its confidentiality, integrity, and availability. Key aspects of DSPM include encryption, access controls, data classification, and data loss prevention.

AI security posture management deals with securing AI models, algorithms, and systems. It addresses the unique challenges posed by AI technologies, such as adversarial attacks, data poisoning, model stealing, and bias. AI-SPM encompasses secure model training, privacy-preserving AI techniques, defense against attacks, and explainability.

Although DSPM and AI-SPM address different aspects of security and data privacy, they function together to create a comprehensive and holistic security strategy. DSPM provides a foundation for data protection, while AI-SPM ensures the safe and responsible use of AI technologies that process and analyze the data. Integrating both domains enables organizations to safeguard both their data assets and their AI systems, minimizing risks and ensuring data compliance with relevant regulations.

AI-SPM Within MLSecOps

AI security posture management is a cornerstone of machine learning security operations (MLSecOps), the practices, and tools used to secure the ML lifecycle. MLSecOps encompasses everything from securing the data used to train models to monitoring deployed models for vulnerabilities, with the goal to ensure the integrity, reliability, and fairness of ML systems throughout their development and operation.

Within MLSecOps, AI-SPM focuses on the specific security needs of AI systems, which often involve more complex models and functionalities compared to traditional ML. This complexity introduces unique security challenges that AI-SPM addresses — data security, model security, model monitoring, and regulatory compliance. And the benefits of AI-SPM within MLSecOps are indisputable:

  • Enhanced Security Posture: By proactively addressing AI-specific security risks, AI-SPM strengthens the overall security posture of the organization’s ML pipelines and deployed models.
  • Improved Trust in AI: AI security fosters trust in AI systems, making them more reliable and easier to integrate into business processes.
  • Faster and More Secure Innovation: AI-SPM facilitates a secure environment for AI development, allowing organizations to confidently innovate with AI technologies.

AI-SPM FAQs

Grounding and training are two distinct aspects of developing AI models, though they both contribute to the functionality and effectiveness of these systems.

Grounding involves linking the AI's operations, such as language understanding or decision-making processes, to real-world contexts and data. It's about making sure that an AI model's outputs are applicable and meaningful within a practical setting. For example, grounding a language model involves teaching it to connect words with their corresponding real-world objects, actions, or concepts. This comes into play with tasks like image recognition, where the model must associate the pixels in an image with identifiable labels that have tangible counterparts.

Training refers to the process of teaching an AI model to make predictions or decisions by feeding it data. During training, the model learns to recognize patterns, make connections, and essentially improve its accuracy over time. This occurs as various algorithms adjust the model's internal parameters, often by exposing it to large datasets where the inputs and the desired outputs (labels) are known. The process enhances the model's ability to generalize from the training data to new, unseen situations.

The main difference between grounding and training lies in their focus and application:

  • Grounding is about ensuring relevance to the real world and practical utility, creating a bridge between abstract AI computations and tangible real-world applications.
  • Training involves technical methodologies to optimize the model's performance, focusing primarily on accuracy and efficiency within defined tasks.

Model contamination refers to the unintended training of AI models on sensitive data, which could expose or leak the sensitive data through its outputs, logs, or interactions once it’s deployed and used for inference or generation tasks. AI-SPM aims to detect and prevent contamination.

 

CSPM and AI-SPM are aligned but distinct risk management areas — the former focusing on cloud infrastructure posture, the latter on securing the AI system lifecycle through data, model, and runtime protections. As AI adoption grows, implementing both CSPM and AI-SPM in a coordinated manner will be critical for comprehensive AI security governance.

Visibility and control are crucial components of AI security posture management. To effectively manage the security posture of AI and ML systems, organizations need to have a clear understanding of their AI models, the data used in these models, and the associated infrastructure. This includes having visibility into the AI supply chain, data pipelines, and cloud environments.

With visibility, organizations can identify potential risks, misconfigurations, and compliance issues. Control allows organizations to take corrective actions, such as implementing security policies, remediating vulnerabilities, and managing access to AI resources. 

An AI bill of materials (AIBOM) is the master inventory that captures all components and data sources that go into building and operating an AI system or model. Providing much needed end-to-end transparency to govern the AI lifecycle, the AIBOM opens visibility into:

  • The training data used to build the AI model
  • Any pretrained models or libraries leveraged
  • External data sources used for grounding or knowledge retrieval
  • The algorithms, frameworks, and infrastructure used
  • APIs and data pipelines integrated with the model
  • Identity info on humans/services with access to the model

Think of the AIBOM like a software bill of materials (SBOM) but focused on mapping the building blocks, both data and operational, that comprise an AI system.

In the context of AI security, explainability is the ability to understand and explain the reasoning, decision-making process, and behavior of AI/ML models, especially when it comes to identifying potential security risks or vulnerabilities. Key aspects of explainability include:

  • Being able to interpret how an AI model arrives at its outputs or decisions based on the input data. This helps analyze if the model is behaving as intended or if there are any anomalies that could indicate security issues.
  • Having visibility into the inner workings, parameters, and logic of the AI model rather than treating it as a black box. This transparency aids in auditing the model for potential vulnerabilities or biases.
  • The ability to trace the data sources, algorithms, and processes involved in developing and operating an AI model. This endows explainability over the full AI supply chain.
  • Techniques to validate and explain the behavior of AI models under different conditions, edge cases, or adversarial inputs to uncover security weaknesses.
  • Increasingly, AI regulations require explainability as part of accountability measures to understand if models behave ethically, fairly, and without biases.

Explainability is integral to monitoring AI models for anomalies, drift, and runtime compromises, for investigating the root causes of AI-related incidents, and for validating AI models against security policies before deployment.

Notebooks refer to interactive coding environments like Jupyter Notebooks or Google Colab Notebooks. They allow data scientists and ML engineers to write and execute code for data exploration, model training, testing, and experimentation in a single document that combines live code, visualizations, narrative text, and rich output. Facilitating an iterative and collaborative model development process, notebooks, via the code they contain, define the data pipelines, preprocessing steps, model architectures, hyperparameters etc.

From an AI security perspective, notebooks are important assets that need governance because:

  1. They often contain or access sensitive training datasets.
  2. The model code and parameters represent confidential intellectual property.
  3. Notebooks enable testing models against adversarial samples or attacks.
  4. Shared notebooks can potentially leak private data or model details.

The AI supply chain refers to the end-to-end process of developing, deploying, and maintaining AI models — including data collection, model training, and integration into applications. In addition to the various stages involved, the AI supply chain encompasses data sources, data pipelines, model libraries, APIs, and cloud infrastructure.

Managing the AI supply chain is essential for ensuring the security and integrity of AI models and protecting sensitive data from exposure or misuse.

AI attack vectors are the various ways in which threat actors can exploit vulnerabilities in AI and ML systems to compromise their security or functionality. Some common AI attack vectors include:

  • Data poisoning: Manipulating the training data to introduce biases or errors in the AI model, causing it to produce incorrect or malicious outputs.
  • Model inversion: Using the AI model's output to infer sensitive information about the training data or reverse-engineer the model.
  • Adversarial examples: Crafting input data that is subtly altered to cause the AI model to produce incorrect or harmful outputs, while appearing normal to human observers.
  • Model theft: Stealing the AI model or its parameters to create a replica for unauthorized use or to identify potential vulnerabilities.
  • Infrastructure attacks: Exploiting vulnerabilities in the cloud environments or data pipelines supporting AI systems to gain unauthorized access, disrupt operations, or exfiltrate data.
Artificial intelligence and machine learning can create security blind spots due to the complex nature of AI systems, the rapid pace of adoption, and the vast amount of data involved. As organizations deploy AI and ML models across diverse cloud environments, the traditional security tools and approaches may not adequately address the unique risks associated with these models. For example, data poisoning attacks or adversarial examples can exploit the AI model's behavior, leading to compromised outputs. Additionally, the dynamic and interconnected nature of AI systems can make it difficult to track and secure data, resulting in potential data exposure and compliance issues.
Model corruption refers to the process of altering or tampering with an AI model's parameters, training data, or functionality, which can lead to compromised performance or malicious outputs. Attackers may corrupt models through data poisoning, adversarial examples, or other techniques that manipulate the model's behavior. AI model misuse, on the other hand, occurs when threat actors or unauthorized users exploit AI models for malicious purposes, such as generating deepfakes, enabling automated attacks, or circumventing security measures. Both model corruption and misuse can undermine the integrity, security, and trustworthiness of AI systems.
AI adoption introduces new complexities to IT environments, as organizations must deploy and manage diverse AI models, data pipelines, and cloud resources. This increased complexity can make it challenging to maintain unified visibility across the entire AI landscape, leading to potential security blind spots and increased risk. Traditional security tools may not be well suited to address the specific risks and challenges associated with AI systems, leaving organizations vulnerable to AI-specific attack vectors. As a result, organizations need to adopt advanced security solutions designed specifically for AI and ML systems to ensure comprehensive visibility and control.
Model sprawl occurs when organizations develop and deploy a large number of AI models without a clear understanding of their inventory, usage, and associated risks. As AI adoption grows, organizations may experiment with various models, leading to a proliferation of AI systems across different cloud environments. This can result in shadow AI models, which are models that lack proper documentation, governance, and security controls. Model sprawl can contribute to compliance violations, data exfiltration, and increased attack surfaces. To address model sprawl, organizations need to maintain a comprehensive AI inventory, which includes tracking and managing all AI models, their associated data, and cloud resources, to ensure proper governance and security.
Shadow AI models are AI systems that lack proper documentation, governance, and security controls, often resulting from model sprawl and decentralized development processes. These models may be deployed without the knowledge or approval of security teams, posing a significant risk to an organization. Shadow AI models can contribute to compliance violations by processing sensitive data without adhering to privacy regulations or established security policies. Additionally, the lack of visibility and control over shadow AI models can increase the likelihood of data exfiltration, as attackers may exploit vulnerabilities in these poorly managed systems to access and steal sensitive information.

AI-powered applications introduce new challenges for governance and privacy regulations, as they process vast amounts of data and involve complex, interconnected systems. Compliance with privacy regulations, such as GDPR and CCPA, requires organizations to protect sensitive data, maintain data processing transparency, and provide users with control over their information. AI-powered applications can complicate these requirements due to the dynamic nature of AI models, the potential for unintended data exposure, and the difficulty of tracking data across multiple systems and cloud environments. Consequently, organizations must adopt robust data governance practices and AI-specific security measures to ensure compliance and protect user privacy.

AI-focused legislation and strict controls are crucial for ensuring that organizations handle customer data responsibly and ethically in the context of AI and machine learning systems. These regulations aim to establish standards for AI system transparency, fairness, and accountability, while also addressing the unique risks and challenges associated with AI-powered applications. By adhering to AI-focused legislation and implementing strict controls, organizations can prevent the misuse of customer data, mitigate potential biases in AI models, and maintain the trust of their customers and stakeholders. Furthermore, compliance with these regulations helps organizations avoid costly fines, reputational damage, and potential legal consequences associated with privacy violations and improper data handling.
Ensuring robust model development, comprehensive training, and policy consistency is vital for AI security posture management. Secure model development minimizes vulnerabilities and risks, while thorough training processes help models learn from accurate, unbiased data, reducing the likelihood of unintended or harmful outputs. Policy consistency applies security policies and standards uniformly across AI models, data, and infrastructure, enabling organizations to maintain a strong security posture and address threats effectively. Together, these aspects form the foundation for a secure and reliable AI environment.
To protect sensitive information within AI models and the AI supply chain, organizations should implement robust data security practices and AI-specific security measures. Key strategies include identifying and categorizing sensitive data, implementing strict access controls, encrypting data at rest and in transit, continuously monitoring AI models and data pipelines, and ensuring compliance with relevant privacy regulations and security policies. These measures create a secure environment that safeguards sensitive data from unauthorized access and misuse.
AI models and data pipelines can be prone to vulnerabilities and misconfigurations such as insecure data storage, inadequate authentication and authorization mechanisms, misconfigured cloud resources, unsecured data transfer, and insufficient monitoring and logging. These issues can expose sensitive data, allow unauthorized access to AI models and data pipelines, and hinder the detection of security incidents or anomalies. Addressing these vulnerabilities and misconfigurations is essential for maintaining a robust AI security posture and protecting valuable information.
User interactions with AI models can introduce security risks, as they may inadvertently expose sensitive information, inject malicious inputs, or exploit vulnerabilities in the AI system. Insufficient access controls, weak authentication, or inadequate input validation can lead to unauthorized access or misuse of AI models. Additionally, users may unintentionally provide biased or misleading data during model training, resulting in unintended or harmful outputs. To mitigate these risks, organizations must implement robust security measures, including access controls, input validation, and ongoing monitoring of user interactions.
Abnormal activity in AI models can include unexpected changes in model behavior, unusual data access patterns, unauthorized modifications, or signs of external tampering. Detecting such activities requires continuous monitoring of AI models, data pipelines, and associated infrastructure. Implementing anomaly detection techniques, such as statistical analysis, machine learning algorithms, or rule-based systems, can help identify deviations from normal behavior. Additionally, organizations should establish baselines for typical model performance and user interactions to facilitate the detection of abnormal activities and potential security threats.
AI security posture management can monitor and protect sensitive data in model outputs by implementing a combination of data-centric security measures and output validation processes. Data-centric security measures, such as data classification, encryption, and access controls, ensure that sensitive information in model outputs is adequately protected. Output validation processes, including input-output correlation analysis, result verification, and anomaly detection, help identify, and prevent the disclosure of sensitive data or unintended consequences. Continuous monitoring of AI model performance and user interactions also plays a crucial role in safeguarding sensitive data in model outputs.
Encryption, logging, retention, authentication, and authorization play crucial roles in maintaining AI security by safeguarding the confidentiality, integrity, and availability of AI models and data. Encryption prevents unauthorized access and data breaches by protecting sensitive data at rest and in transit. Logging tracks AI model activities and data pipeline operations, facilitating detection and investigation of security incidents. Retention policies manage data storage duration, ensuring secure disposal when no longer needed. Authentication verifies the identity of users and systems accessing AI models and data, while authorization enforces access controls and permissions to prevent unauthorized access or misuse. Collectively, these measures contribute to a robust AI security strategy.
Real-time detection and response play a critical role in preventing high-priority security incidents by enabling organizations to swiftly identify and address potential threats, vulnerabilities, and anomalies. By continuously monitoring AI models, data pipelines, and associated infrastructure, real-time detection systems can promptly detect abnormal activities, unauthorized access attempts, or signs of external tampering. Rapid response capabilities, including automated remediation measures and incident response plans, allow organizations to effectively mitigate security risks, minimize potential damage, and maintain the trustworthiness of their AI systems.

Related Content

Get the latest news, invites to events, and threat alerts