Free Intensive Nov 21–22Free 2-day hands-on Vibecoding Intensive on Nov 21–22 at 16:00 MSK (Sat–Sun) · Live Cohort 3 starts Nov 23Join Free →Cohort 3 (Nov 23)
ReleaseAPI / Product · ActionableAWS Machine Learning Blog→ original

Standard Bank cuts AI app review from two weeks to two hours

Before an AI application launches, a bank must prove to its risk committee that it is safe. That means two weeks of manual document review. Standard Bank Group worked with AWS to hand the comparison of an app's code and responses against the rules to a judge model. Across six assessments, a report is ready in two hours, and risk scores rose 14% after fixes. Teams in regulated industries should consider turning policies into automated tests.

AI-processed from AWS Machine Learning Blog; edited by Hamidun News
Standard Bank cuts AI app review from two weeks to two hours
Source: AWS Machine Learning Blog. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceAWS Machine Learning Blog ↗
Editor-in-Chief
Key metric #1An agent on AWS Fargate pre-fills a questionnaire of ~40 questions from the code
Key metric #2The judge answers yes/no to 300+ questions, each tied to a clause of a standard
Key metric #3The judge model is swapped by changing a setting: Bedrock has a single API

Imagine a team builds a generative AI chatbot in a few days, but cannot release it to the bank's customers. First, the risk committee must be sure the bot does not give dangerous answers, does not violate policy, and leaves an audit trail. Until now, that meant two weeks of manual work: people read documents, reviewed code, and asked the application tricky questions themselves.

Standard Bank Group, together with AWS Professional Services, automated exactly this part. Now a language model acting as a "judge" (the LLM-as-a-judge approach) checks the application's materials against a list of criteria drawn from international standards and the bank's internal rules. Across six assessments, a full risk report takes about two hours instead of two weeks, and the scenarios' risk scores improved by 14% after fixes.

For teams deploying AI in regulated industries, it is an example of how to keep sign-off from becoming the slowest step in the pipeline.

What exactly was built

The bank did not write a new AI policy. It already had a model risk management practice (explainability, fairness) and a process for rating each new use case by risk level before launch. The task was narrower: automate evidence collection and testing, and feed the results into the existing approval gates. The risk committee still makes the decision, but it now looks at uniform data gathered the same way for every use case.

Everything runs inside the bank's own AWS account. Documents, code, and test results sit in an Amazon S3 bucket, the AWS Well-Architected Tool manages the questionnaire, Step Functions orchestrates the pipeline, and a foundation model in Amazon Bedrock does the evaluation. Access to the model is controlled through IAM, as with the bank's other workloads. Bedrock does not pass prompts and responses to model providers and does not train models on them. Because Bedrock has a single API, switching to a newer judge model takes a settings change and no pipeline rebuild. That matters when models and tools change every three to six months.

How the review works

There are four steps. First, a pre-assessment agent on AWS Fargate scans the repository and documentation, maps files to roughly 40 questionnaire items (for example, "incomplete technical documentation" or "insufficient human oversight"), and fills in the answers with links to evidence. The development team checks the answers, corrects mistakes, and adds anything the agent did not find.

Next, the pipeline examines four kinds of material: system documentation, policies (roles, incident-handling procedures), source code with the system prompt, and the application's live behavior. For live tests, it calls the application through Lambda or an API. The team supplies ordinary user questions, and the pipeline adds a built-in set of jailbreaks and off-topic questions that the application must refuse. The same query is sent several times in its original and rephrased forms, and an embedding model checks that the meaning of the answers stays the same. All responses are saved to S3 so reviewers can see exactly what the application generated.

Then the judge, with no fine-tuning or customization, answers "yes" or "no" to narrow questions: does the material contain specific evidence? Each question is tied to a clause of an international standard for AI governance, risk, or security. The bank added questions from its own standards, bringing the total to more than 300. They cover AWS's eight dimensions of responsible AI: veracity and robustness, privacy and security, transparency, fairness, safety, controllability, governance, and explainability.

The scores come not from the model but from a fixed formula that the risk team can inspect. Risk likelihood equals the share of evidence not found, severity is set by a level the risk team can change, and the final rating falls on a scale from "Very low" to "Critical". Each missing detail is linked to a specific control and to a clause of a standard or policy, so a risk officer can explain any finding to a regulator. Each assessment runs several times, and unstable results are flagged in the review interface. Model calls pass through Bedrock Guardrails.

The result comes in four formats: a presentation for executives, a checklist of controls, a detailed technical report, and a CloudWatch dashboard that shows score trends across use cases.

Policy as a test, and caveats

The most practical idea here is a "question factory". It generates additional questions for the judge directly from the bank's internal documents. Suppose the incident-handling requirement in a policy changes: instead of circulating a memo, the check itself is updated, and all subsequent assessments apply the new rule automatically.

At the proof-of-concept stage, the assessment is optional and shows gaps early. At the build stage, it is mandatory and becomes part of the formal evidence for the committee. It currently runs in the production environment before launch. Repeat runs on a schedule, and whenever code or configuration changes, are planned for after live workloads are connected.

Some caveats are worth keeping in mind. This is a post by AWS itself, the sample of six assessments is small, and there is no independent confirmation. The system collects evidence; it does not certify legal compliance, and responsibility for regulatory decisions stays with the bank. The judge is still a language model, so the bank limits its work to narrow yes/no questions with repeated runs.

The practical takeaway: start from the processes you already have, turn policies into testable checks, and run them at the prototype stage, when fixes are cheapest. Then growth in the number of AI applications does not require a proportional growth in reviewer headcount.

Story

Key Questions Answered

What exactly was built?

The bank did not write a new AI policy. It already had a model risk management practice (explainability, fairness) and a process for rating each new use case by risk level before launch. The task was narrower: automate evidence collection and testing, and feed the results into the existing approval gates. The risk committee still makes the decision, but it now looks at uniform data gathered the same way for every use case. Everything runs inside the bank's own AWS account.

How the review works?

There are four steps. First, a pre-assessment agent on AWS Fargate scans the repository and documentation, maps files to roughly 40 questionnaire items (for example, "incomplete technical documentation" or "insufficient human oversight"), and fills in the answers with links to evidence. The development team checks the answers, corrects mistakes, and adds anything the agent did not find.

What are the key details on Policy as a test, and caveats?

The most practical idea here is a "question factory". It generates additional questions for the judge directly from the bank's internal documents. Suppose the incident-handling requirement in a policy changes: instead of circulating a memo, the check itself is updated, and all subsequent assessments apply the new rule automatically. At the proof-of-concept stage, the assessment is optional and shows gaps early. At the build stage, it is mandatory and becomes part of the formal evidence for the committee.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Private AI Builders Community · Vibe Club

Ship AI Faster with Founders, Engineers & Weekly Architecture Reviews

Inside Vibe Club (hamidun.com/club) — weekly live masterminds with Zhemal Khamidun, Tech Radar, full LMS access, and 370+ production Claude Code skills.

Join Vibe Club →Explore Hamidun Academy
What do you think?
Loading comments…