CT-GenAI — Testing with Generative AI glossary

91 terms the exam expects you to know, defined in plain language.

A

AI Output Validation(validators of ai-generated outputs, ai output validation): The process of verifying the correctness and quality of AI-generated artifacts such as test cases or data.
AI output validation and human oversight(ai output validation, human validation, human oversight): The process of verifying AI-generated artifacts for correctness, completeness, and compliance, requiring human review.

B

Bias in GenAI(bias, bias in genai outputs, biases): The tendency of GenAI models to amplify biases present in their training data, leading to unfair outputs.
Bias in LLMs(bias): Skewed or unfair outputs resulting from training data biases, relevant for fairness and compliance testing.

C

Chain-of-thought prompting(chain-of-thought, cot, chain-of-thought prompting, cot prompting): A prompting technique that improves reasoning for complex tasks by breaking them into steps.
Change Management for GenAI Testing Adoption(change management, driving adoption and training): The process of driving organizational adoption and training to realize benefits of GenAI in testing.
Change management(change management): The process of preparing and supporting employees for successful adoption of GenAI.
Chunking strategy(chunking, chunking strategy): A method for splitting documents into pieces, such as fixed-size or semantic chunks, affecting retrieval quality.
Cloud infrastructure: Scalable computing resources provided over the internet, requiring careful assessment of data privacy and latency for GenAI deployment.
Cloud-based GenAI(cloud-based genai): Generative AI delivered via cloud infrastructure, offering scalability but requiring data privacy evaluation.
Cloud-based GenAI infrastructure(cloud-based genai services, cloud-based deployment, cloud-based genai infrastructure): A deployment model where GenAI models are accessed via third-party APIs, offering low upfront cost and scalability but with data privacy and latency trade-offs.
Cloud-based infrastructure(cloud infrastructure): Infrastructure that offers scalability but requires careful privacy and latency evaluation.
Compliance (testing context)(compliance): Adherence to standards, conventions, or regulations in laws and similar prescriptions.
Conformity assessment(conformity assessment): A process required for high-risk AI systems under the EU AI Act, including testing for accuracy, robustness, and cybersecurity.
Context Grounding(grounding): The process by which the generative model uses retrieved documents to base its output on factual information.
Context Window(context window): The limited amount of text an LLM can consider at once, often ranging from 4K to 128K tokens.
Copyright Risks in GenAI(copyright risks, intellectual property risks): The risk that GenAI outputs may violate copyright or intellectual property rights.

D

Data Curation for AI Testing(data curation, curators of training data, data curation for ai testing): The activity of selecting, organizing, and maintaining training data for AI models used in testing.
Data curation(data curation): The process of maintaining the quality and relevance of training data for AI models.
Deterministic Setting(deterministic settings): A configuration (e.g., low temperature) ensuring reproducible LLM outputs when needed.

E

EU AI Act(eu ai act): A landmark regulation that classifies AI systems into risk categories and imposes requirements on providers and deployers.
Embedding Model(embedding model): A model that converts text into numerical vectors for semantic similarity search.
Embeddings(embeddings, embedding vector, embedding): Numerical representations of data stored in vector databases for similarity search.
End-to-End Behavior(end-to-end behavior): The overall pipeline functionality that testers evaluate, including correct context incorporation and retrieval failures.

F

Few-Shot Prompting(few-shot prompting, few-shot): A technique where the AI generates test cases guided by one or more provided example test cases.

G

GenAI Governance Framework(governance frameworks for genai use, guidelines for responsible use of genai, governance frameworks for genai, genai governance framework, governance framework): A set of policies for prompt reuse, output validation, and ethical considerations in using generative AI for testing.
GenAI Hallucination(hallucination, hallucinations): A phenomenon where GenAI models generate fabricated information that appears coherent but is factually incorrect due to statistical pattern prediction.
GenAI Hallucination Detection(hallucination): Identifying incorrect or fabricated information in GenAI outputs.
GenAI Risk Assessment(risk assessment): Evaluation of risks associated with AI-generated artifacts in testing.
Generation Stage(generation stage): The phase in RAG where the generative model produces an answer using the original query and retrieved context.
Generative AI(genai): AI technology that can generate content, often abbreviated as GenAI.
Generative Model(generative model): The component in a RAG pipeline that produces a response grounded in retrieved context and the original query.
Governance framework(governance framework): A system that ensures prompt reuse, validation, and ethical considerations in generative AI.

H

Hallucination(hallucinate, hallucinating, hallucination): The generation of plausible but incorrect information by an LLM, leading to false or invalid outputs.
Hallucination reduction: A benefit of RAG where the generative model produces fewer false or unsupported claims due to grounded context.
Human Oversight(human oversight): The necessary verification of LLM-generated artifacts by humans to ensure correctness and completeness.
Human Oversight Requirement(human oversight): The necessity for testers to critically evaluate and verify GenAI outputs before use in production.
Human Validation of GenAI Outputs(human validation, critical eye for ai-generated content): The necessity for testers to critically evaluate and refine AI-generated content to ensure quality and compliance.
Human validation(human validation, human oversight): Essential oversight by a knowledgeable tester to evaluate and refine AI outputs for quality and compliance.

I

ISO/IEC 42001(iso/iec 42001): An international standard specifying requirements for an AI management system.
ISTQB testing terminology glossary(istqb glossary, istqb testing terms): The authoritative glossary of testing terms used as the canonical reference for ISTQB exam answers across all syllabus levels.
Intelligent Test Analysis(intelligent test analysis): The use of GenAI to analyze test results and provide insights or summaries.
Intermediate Output Validation(intermediate outputs): Practice of ensuring each step's output is in a format easily parsed by the next prompt.

J

Jurisdiction-specific compliance(jurisdiction-specific compliance): The concept that regulatory compliance for GenAI systems varies by region and must be verified for applicable frameworks.

K

Knowledge base: A trusted repository of domain-specific documents from which the retrieval component sources context for generation.

L

LLMOps(llmops, large language model operations): The set of tools and practices for deploying, monitoring, and maintaining GenAI models in production, including performance tracking and prompt management.
Lack of Contextual Understanding in GenAI(lack of contextual understanding, no true comprehension): GenAI's inability to truly understand context, resulting in misinterpretation of requirements and inconsistent outputs.
Large Language Model(llm, llms, large language model): A neural network trained on vast text data that predicts the next token based on preceding context.

M

Meta Prompting(meta prompting): A technique where the model generates or improves its own prompts based on a high-level goal, leveraging the model's understanding of prompt engineering.

N

Natural Language Understanding(natural language understanding): The ability of an LLM to interpret and comprehend human language input for tasks like test requirements.
Next-Token Prediction(next-token prediction): The core process of an LLM generating text by predicting each subsequent token in a sequence.
Non-deterministic GenAI Outputs(non-deterministic, non-deterministic outputs): The property that the same prompt may yield different results each time due to the model's probabilistic nature.

O

On-premises GenAI infrastructure(on-premises deployments, on-premises infrastructure, on-premises genai infrastructure, on-premises deployment, on-premises genai): A deployment model where GenAI models run on local hardware, providing full data control and lower latency but requiring significant investment and maintenance.
One-Shot Prompting(one-shot prompting, one-shot): A prompting technique that provides a single example to guide the model's output format or reasoning.
Over-reliance on GenAI Output(over-reliance, over-reliance on genai output): The risk of using GenAI outputs without verification, potentially causing undetected defects.

P

Pattern Matching(pattern matching): The mechanism by which LLMs generate output by recognizing patterns rather than performing logical reasoning.
Privacy Risks in GenAI(privacy risks, privacy risk): The risk that GenAI may inadvertently leak sensitive information from its training data.
Prompt Chaining(prompt chaining): A technique that breaks a complex task into a sequence of smaller prompts, where each output becomes the input of the next, improving control and accuracy.
Prompt Engineering(prompt engineering): The systematic design of prompts to improve consistency and quality of LLM outputs.
Prompt Engineering for Testing(prompt engineering, testers as prompt engineers): The skill of designing effective prompts to generate test cases or test data using generative AI.
Prompt Quality Dependence(prompt): The reliance of GenAI output quality on the specificity and clarity of the input prompt.

R

RAG Knowledge Base(knowledge base): The curated collection of documents or data used as the retrieval source in a RAG pipeline.
RAG for Defect Analysis(defect analysis): Application of RAG to retrieve similar past defects for triaging new bugs.
RAG for Test Case Generation(test case generation, generate test cases): Application of RAG to retrieve relevant requirements and generate test cases using a generative model.
RAG for Test Data Creation(test data creation): Application of RAG to use domain-specific documents for generating realistic test data.
Response faithfulness: The degree to which the generated response is consistent with the retrieved context and does not contradict it.
Retrieval Component(retrieval component): The part of a RAG pipeline that converts a query into an embedding and searches a vector database.
Retrieval Stage(retrieval stage): The phase in RAG where the user query is converted to an embedding and searched against a vector database for relevant chunks.
Retrieval accuracy: The correctness and completeness of the document chunks retrieved by the RAG pipeline for a given query.
Retrieval-Augmented Generation (RAG)(rag, rag pipeline, retrieval-augmented generation): A pipeline that uses a vector database to retrieve relevant context to ground GenAI outputs and reduce hallucination.
Retrieval-Augmented Generation (RAG) pipeline(rag pipelines, rag, retrieval-augmented generation, rag pipeline, retrieval-augmented generation pipeline): An infrastructure pattern that combines a retrieval component with a generative model to ground outputs in domain-specific knowledge.

S

Semantic similarity(cosine similarity): A measure of closeness between embedding vectors, often computed as cosine similarity, used to retrieve relevant chunks.
Similarity Search(similarity search): The algorithm used by the vector database to find documents semantically similar to the query embedding.
Six-component prompt structure(role, context, task, format, constraints, and examples, six-component prompt structure, six-component structure, prompt structure, prompt components, six-component prompt structure framework): A systematic framework for crafting effective prompts composed of Role, Task, Context, Format, Examples, and Constraints.
Statistical Model(statistical models): An LLM is a statistical model that generates plausible continuations based on training data.
Synthetic Test Data Generation(synthetic test data, synthetic data): The use of GenAI to create artificial data that mimics real data for testing while preserving privacy.

T

Temperature Setting(temperature): A parameter controlling the randomness of an LLM's output, with higher values increasing variability.
Test Case Generation(test case generation): The capability of GenAI to automatically create test cases from requirements in natural language.
Test Data Synthesis(test data synthesis): The GenAI capability to generate test data, including synthetic data for privacy, for testing purposes.
Test Level Specification(test level): The explicit mention of the testing level (e.g., unit, integration, system) in a prompt to focus test case generation.
Test Manager Role Evolution with GenAI(evolution of the test manager role, test manager role with generative ai, test manager role evolution): A shift from direct oversight to governance, resource reallocation, and change management for GenAI integration.
Test Technique Specification(test technique): The explicit mention of a test design technique (e.g., equivalence partitioning, boundary value analysis) in a prompt.
Tester Role Evolution with GenAI(evolution of the tester role, tester role with generative ai, tester role evolution): A shift from manual test execution to strategic activities including prompt engineering, data curation, and AI output validation.
Token(tokens): A word or subword unit that an LLM processes as input or generates as output.
Top-k retrieval(top-k): A configurable parameter specifying the number of most relevant document chunks to retrieve for the generative model.
Total Cost of Ownership Evaluation(total cost of ownership): Assessment of whether on-premises infrastructure is cost-effective for small teams or short-term projects.
Training Data Curation(curators of training data): The selection and preparation of data used to train or prompt GenAI models.
Transformer Architecture(transformer architecture): A neural network architecture that processes text by attending to different parts of the input for generation.

V

Vector database(vector database, vector databases, vector store): A database type used in RAG pipelines to store and retrieve embeddings for similarity search, enabling grounded GenAI outputs.
Vector databases in RAG pipelines(vector database, vector databases): A database that stores embeddings and enables similarity search, used in RAG to retrieve relevant context for LLMs.

Z

Zero-Shot Prompting(zero-shot prompting, zero-shot): A technique where the AI generates test cases without examples, relying solely on its pre-trained knowledge.