Auditing the Algorithm: What AI Assurance Looks Like in Practice

Beyond Traditional IT Audit

Many Caribbean internal audit functions have responded to the AI governance imperative by extending their existing IT audit methodology to cover AI systems. While this is a reasonable starting point, it is insufficient. AI systems present audit challenges that traditional IT audit is not designed to address: statistical model behaviour, training data integrity, emergent system outputs, fairness and bias in probabilistic decision-making, and the opacity of complex machine learning models.

Effective AI assurance requires a distinct and specialised methodology — one that draws on data science, model risk management, fairness assessment, governance evaluation, and operational controls review. Dawgen Global has developed an integrated AI Assurance methodology that brings these disciplines together in a structured, repeatable audit framework.

The Dawgen Global AI Assurance Methodology

Our AI Assurance engagements are structured across five audit domains, each addressing a distinct dimension of AI governance risk:

Audit Domain	Key Questions Addressed
Governance & Accountability	Is there a clear governance framework? Are roles and accountabilities defined and operational? Does the board receive adequate AI oversight reporting?
Data Integrity & Lineage	Is training and operational data fit for purpose? Are data quality controls in place? Is there a documented and auditable data lineage?
Model Development & Validation	Was the model developed using appropriate methodology? Was it independently validated? Is there documented evidence of testing and sign-off?
Operational Controls & Monitoring	Are controls in place to detect model drift, bias, and performance degradation? Is there an incident detection and response process?
Fairness, Ethics & Compliance	Does the system comply with applicable regulations? Has it been assessed for discriminatory bias? Are explainability requirements met?

 

Domain 1: Governance and Accountability Audit

The governance audit begins at board level and works downward through the accountability chain. Auditors review AI governance policies for completeness, currency, and board approval. They assess whether the AI risk classification process has been applied consistently and whether the system under review was correctly classified. They interview system owners, business line executives, and risk functions to assess whether accountability assignments are understood and operational — not merely documented.

A critical audit procedure is the ‘accountability stress test’: presenting the audit team with a specific adverse AI outcome scenario and tracing the accountability chain to determine whether the governance framework would produce a timely, decisive response. Frameworks that cannot pass this test are not operationally effective.

Domain 2: Data Integrity and Lineage Audit

AI systems are only as good as the data they are trained on and the data they operate on. The data audit examines:

• Data provenance: where did the training data come from, and is it representative of the population the model will serve?
• Data quality controls: what validation, cleansing, and outlier-handling processes were applied?
• Historical bias: does the training data reflect historical patterns of discrimination that the model may perpetuate?
• Data lineage: can the complete path of data from source to model output be traced and documented?
• Ongoing data governance: how is the quality and representativeness of operational data maintained over time?

 

Data audit findings frequently represent the most significant risk exposures in AI assurance engagements. A technically sophisticated model built on biased or poor-quality data will produce biased or poor-quality outputs — regardless of its algorithmic elegance.

Domain 3: Model Development and Validation Audit

The model audit assesses whether the AI system was developed using appropriate methodology and whether it has been subject to meaningful independent validation. Key audit procedures include:

• Review of model documentation: is there adequate documentation of model design choices, training approach, and performance evaluation?
• Assessment of validation independence: was the model validated by a team independent of those who built it? Independence is a fundamental validation requirement.
• Performance benchmark review: what metrics were used to assess model performance, and are they appropriate for the intended use case? A model optimised for accuracy may produce unacceptably high false positive rates with severe consequences for affected individuals.
• Model explainability assessment: for high-risk models, can the auditor independently verify that the model’s outputs can be explained at the individual decision level?

Domain 4: Operational Controls Audit

An AI model that was sound at deployment can degrade over time as the world it was trained on diverges from the world it operates in. Operational controls must detect and respond to this degradation. Auditors assess:

• Monitoring infrastructure: are performance metrics being tracked in production, and are thresholds defined that trigger human review?
• Drift detection: are there statistical controls to identify when the model’s input data or output distribution has shifted materially from the deployment baseline?
• Human oversight effectiveness: where human review is required for AI decisions, are reviewers genuinely scrutinising AI outputs, or rubber-stamping them?
• Incident detection and response: have AI system failures been identified and responded to promptly, with appropriate escalation and root cause analysis?

Domain 5: Fairness, Ethics, and Compliance Audit

The final audit domain addresses the regulatory and ethical dimensions of AI deployment. This includes:

• Regulatory compliance review against applicable Caribbean and international AI regulations
• Fairness testing: statistical analysis of model outputs across protected characteristics to identify potential discriminatory bias
• Ethical impact assessment: review of pre-deployment ethical impact assessment documentation and process quality
• Customer rights compliance: assessment of mechanisms for customers to understand, challenge, and seek redress for AI decisions

AI assurance is not about finding fault with technical teams. It is about giving the board the independent evidence it needs to exercise governance confidently — knowing that the AI systems it has authorised are performing as intended and within acceptable risk parameters.

Reporting AI Assurance Findings

AI audit findings should be reported at multiple levels: technical findings to system owners and data science teams; governance findings to the CRO, CTO, and executive management; and material governance conclusions to the Audit Committee and Board. Dawgen Global’s AI Assurance reports are structured to serve all three audiences — with executive summaries that translate technical findings into governance implications that non-technical board members can act on.

Next in the Series — Article 6: Bias, Fairness, and the Duty of Non-Discrimination in AI Systems

About Dawgen Global