Utilizing AI Threat-Modelling Resources
This lab is designed to provide hands-on experience that directly maps to several objectives of the CompTIA SecAI+ (CY0-001) certification exam. The table below details how each major task or concept covered in this lab aligns with the official exam objectives.
| Task/Concept | CompTIA SecAI+ (CY0-001) Exam Objective |
|---|---|
| Task 1: OWASP Top 10 for LLM (Prompt Injection) | 2.6: Given a scenario, analyze an attack and implement compensating controls |
| 3.1: Given a scenario, utilize AI tools for security tasks | |
| Task 2: MIT AI Risk Repository (Algorithmic Bias) | 2.4: Given a scenario, implement data security controls for AI systems |
| 4.2: Explain risks associated with AI | |
| Task 3: MITRE ATLAS (Adversarial Evasion) | 2.1: Given a scenario, use AI threat-modeling resources |
| 2.6: Given a scenario, analyze an attack and implement compensating controls | |
| Task 4: CVE/CWE Investigation | 1.3: Explain the importance of security in the AI life cycle |
| 2.2: Given a scenario, implement security controls for AI systems | |
| Task 5: STRIDE Threat Modeling | 2.1: Given a scenario, use AI threat-modeling resources |
| 2.2: Given a scenario, implement security controls for AI systems |
Overview
This lab provides a comprehensive, hands-on experience in using industry-leading resources for artificial intelligence (AI) threat modeling. Students will learn to navigate and apply frameworks such as the OWASP Top 10 for Large Language Model Applications, the MIT AI Risk Repository, and the MITRE Adversarial Threat Landscape for Artificial-Intelligence Systems (ATLAS). The primary objective is to equip students with the practical skills necessary to analyze a given AI scenario and effectively apply these threat-modeling resources to identify, classify, and mitigate potential risks.
Learning Objective: Given a scenario, use AI threat-modeling resources.
VM Credentials
Username: student
Password: student
Key terms and descriptions
Adversarial Example
An input to an AI model that has been intentionally perturbed to cause the model to make an incorrect prediction
Data Poisoning
An attack where an adversary introduces malicious data into the training dataset, corrupting the model's integrity and performance.
Model Inversion
An attack that attempts to reconstruct the sensitive training data used to train a machine learning (ML) model
Prompt Injection
An attack that involves manipulating a LLM by providing malicious input (a "prompt") to override its original instructions
System Prompt
Hidden instructions given to an LLM before user interaction that define its behavior, personality, and constraints; often contains sensitive business logic or security controls
Jailbreaking
Techniques used to bypass an LLM's safety controls or content restrictions, making it produce outputs it was designed to refuse
Direct Prompt Injection
An attack where the user directly provides malicious instructions in their input to manipulate the LLM
Indirect Prompt Injection
An attack where malicious instructions are hidden in external data sources (documents, websites) that the LLM processes
OWASP Top 10 for LLM
A list of the top 10 most critical security risks specific to LLM applications, published by the Open Worldwide Application Security Project (OWASP).
MITRE ATLAS
A knowledge base of adversary tactics, techniques, and mitigations based on real-world observations of attacks against AI/ML systems; organized similar to MITRE ATT&CK, ATLAS provides standardized IDs (AML.*) for tactics (why attackers act), techniques (how they act), and mitigations (how to defend).
MIT AI Risk Repository
A comprehensive, living database of categorized AI risks, providing a structured vocabulary for risk assessment
Threat Modeling
A structured process of identifying potential threats, vulnerabilities, and attack vectors to an application or system
STRIDE
Microsoft's threat modeling methodology using six categories: Spoofing (authentication), Tampering (integrity), Repudiation (non-repudiation), Information Disclosure (confidentiality), Denial of Service (availability), and Elevation of Privilege (authorization); each category maps to a security property being violated.
Evasion Attack
An attack where the adversary manipulates the input data at inference time to cause a misclassification
Model Extraction (Theft)
An attack where an adversary probes a model to steal its underlying parameters or architecture, often by observing its outputs
Supply Chain Vulnerability (AI)
A risk associated with the components, data, or services used in the development and deployment pipeline of an AI system
Insecure Output Handling
A risk where an LLM's output is accepted without sufficient scrutiny, potentially leading to cross-site scripting or remote code execution
Denial of Service (Model DoS)
An attack that consumes excessive computational resources of an AI model, making it unavailable to legitimate users
Confidentiality
The security principle that prevents the unauthorized disclosure of information
Integrity
The security principle that ensures data has not been modified or tampered with
Availability
The security principle that ensures systems and data are accessible when needed
Common Vulnerabilities and Exposures (CVE)
A dictionary of publicly known information-security vulnerabilities and exposures
Adversarial Robustness
The ability of an AI model to maintain its performance even when subjected to adversarial examples
Red Teaming (AI)
A simulated attack exercise performed to test the security and safety of an AI system
Taint Analysis
A technique used to track the flow of untrusted data (taint) through an application to prevent security vulnerabilities