How to set up Model Governance as you operationalize Machine Learning?

Depiction of the various parts of ModelOps stack by Gartner in 2021
Types of data drift: concept drift, label drift, and feature drift
AI needs a new DevOps a.k.a ModelOps stack [4]

The current state of Model Governance in Banking

Model Governance in banks is mandated by SR-11–7 [3] and its OCC attachment for model risk management (MRM) within banking organizations. Over the years, banks have implemented policies and systems designed to safeguard against the potential adverse effects of models. After the 2008 financial debacle, banks had to comply with the SR 11–7 regulation, the intent of which was to ensure banking organizations were aware of the adverse consequences (including financial loss) of decisions based on AI. A typical bank may be running 100s or 1000s of quantitative or statistical models. The number keeps increasing every year by 10–15%. A single model failure can cause a Bank to lose billions of dollars. The advent of AI and ML models adds more challenges to Model Governance.

Financial institutions in the United States are regulated by a number of regulatory entities at local (yellow), state (yellow), federal (blue), and international levels (green) [2]

Rise of Industry-Agnostic Model Governance

Whereas SR-11–7 and the Model Governance practices it spawned were intended to ensure stability within the financial system, the growing number of privacy (GDPR, CCPA, AAA) and bias laws intend to ensure ethical and transparent use of data. In the past year, we’ve seen progress on AI regulations from European Commission’s proposal, NIST publishing principles on Explainable AI. It is heartening to see the Office of Science and Technology from Whitehouse creating a bill of rights for an AI-powered world. Local governments are waking up to this and now New York City law requires bias audits of AI hiring tools, to be enforced starting January 2023. Specifically, with respect to AI, GDPR (currently the most comprehensive regulation), mandates the following:

  • A comprehensive record of all processing of personal data that includes the purposes of the processing and a description of the categories of data subjects and personal data processed (30.1)
  • Right of data portability and deletion for individuals (17), and the right not to be subject to automated decision-making, including profiling (22)
  • Right to “meaningful information about the logic involved” in automated decisions (13.2.f)
  • The nomination of a “data protection officer” to monitor and advise on the fair and lawful collection and use of personal data (37.1)

Challenges to set up Model Governance

The goal of Model Governance is to identify and minimize the risks associated with the models deployed. The process has many steps covering the development, implementation, testing, and deployment stages. Following are some of the model governance challenges listed in [2] by a top US bank.

A simplified model governance lifecycle [2] in financial services showing model development, production deployment, regulatory oversight, and various feedback loops.
  • As the complexity of the AI models increases, it prolongs the time taken to validate them. It can take as much as 6–12 months to validate a sufficiently complex AI model. The large model size has been increasing 10x every year for the last few years. This is starting to look like another Moore’s Law [10].
Model complexity continues to grow at an exponential rate.
  • OCC audits are uncovering models that should not be running in production. Current governance practices translate to high compliance costs, for example in a given year US financial industry spends about $80 billion for model compliance. And between 2008–2016, US financial institutions paid close to $320 billion in regulatory fines [8].
  • Most teams do intermittent monitoring of their models in an ad-hoc manner. Intermittent monitoring fails to identify critical changes in the environment, data drifts, or data quality issues.
  • Issues don’t get detected or rectified promptly due to a lack of run-time monitoring and mitigation. They only get caught during model retraining or regulatory inquiries by which time the institution is already at risk of business loss, reputational damage, and regulatory fines.
  • Regulatory complexity and uncertainty make governance increasingly difficult. For US credit models alone, one has to make sure models are adhering to regulations like Fair Housing Act, Consumer Credit Protection Act, Fair Credit Reporting Act, Equal Credit Opportunity Act, Fair and Accurate Credit Transactions Act. It is possible for an AI model to be deployed in multiple territories where one jurisdiction has more conservative guidelines.
  • A growing number of model metrics are being proposed to quantify model bias using demographic parity, equalized odds, and other group fairness metrics [10]. Adding more metrics to an already non-scalable, manual model monitoring process does not help.

A new approach to Model Governance

These risks and the variety of AI applications and development processes call for a new Model Governance framework that is simple, flexible, and actionable. A streamlined Model Governance solution is a five-step workflow.

Model Governance workflow for the Modern Enterprise
  1. Validate: Conduct an automated assessment of feature quality, bias, and fairness checks to ensure compliance.
  2. Approve: Ensure human approval of models prior to launch to production and capture the model documentation and reports in one place.
  3. Monitor: Continuously stress test and performance test the models and set up alerts upon identifying outliers, data drift, and bias.
  4. Improve: Derive actionable insights and iterate on your model as your customers, business, and regulations change.

Automated Model Governance with ModelOps

The long list of Model Governance challenges motivates the need for new solution approaches. Below is a blueprint of a ModelOps solution that can enable Model Governance and move AI towards a Self-Regulated AI [2].

A blueprint of a ModelOps based solution to enable Model Governance leading to regulated AI Models [2]
  1. Integration of key self-regulatory analysis modules like: a) Explainability analysis for troubleshooting the models as well as to answer regulatory and customer inquiries. b) Fairness analysis that can help look at intersections of protected classes across metrics like Disparate Impact, Demographic Parity, etc.
  2. Reusable templates to generate automatic reports and documentation like: a) Ability to integrate custom libraries explaining models and/or fairness metrics. b) Customization and configuration of the reports specifying what inputs and outputs are to be presented.
  3. Runtime mitigation with human-in-loop alerting. The goal is to maintain model behavior more effectively during runtime. System capabilities include: a) Scenario-based mitigation for well-defined control paths that can be discovered during pre-deployment testing of the model on historical data or known established scenarios like holiday peak payment activity. b) System-level remediation through the use of alternative models such as using shadow AI models for population segments if the primary AI model shows detectable bias during monitoring.
  4. Robustness tests for AI models. Continuous monitoring provides the opportunity to collect vast amounts of runtime behavioral data. A ModelOps system can then slice and dice data [11] to identify the weaknesses, failure patterns, and risky scenarios in the data. Robustness tests can help reassure AI Governance teams by showing a variety of scenarios being covered through this analysis.
  5. Configurable risk policies and regulatory guidelines. Ability to configure a risk policy [8] for each of the model types to track through the lifecycle and setup of approval criteria will help Governance teams to ensure regulatory oversight of all the models getting deployed and maintained.
  6. Autonomous Governance. Taking this forward, we can have an intelligent system that can automatically govern models resulting in a self-regulated state of AI. This requires a Model Governance Controller looking at all the models being monitored, absorbing model validation reports, and looking for anomalous behavior to send warnings to the human to take look at the problematic models.

Conclusion

In this article, we’ve illustrated the challenges of Model Governance in both regulated and unregulated industries and demonstrated the need for an Industry Agnostic Model Governance for AI that can help us create self-regulated AI systems in the future. We argue that ModelOps, a new enterprise solution blueprint [2] being developed to operationalize AI is the best way to implement Model Governance capabilities in the organization. And then we listed the capabilities of a Model Ops system that can be useful for an effective and automated Model Governance in the enterprise.

References

  1. Wall Street Journal, April 1, 2019.
  2. Towards Self-Regulating AI: Challenges and Opportunities of AI Model Governance in Financial Services
  3. SR-11–7 Guidance on Model Risk Management and Governance
  4. AI needs a new developer stack
  5. Gartner’s definition of ModelOps
  6. What is ModelOps and how is it different from MLOps?
  7. The state of ModelOps in 2021
  8. Compliance by Design: Banking’s unmissable opportunity. BCG Whitepaper
  9. Large Language Models: A New Moore’s Law?
  10. The Measure and Mismeasure of Fairness: A Critical Review of Fair Machine Learning
  11. Introducing Slice and Explain™ — Automated Insights for your AI Models

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Krishna Gade

Krishna Gade

628 Followers

Founder, CEO Fiddler.AI — Building Trust into AI. Prior: @facebook , @pinterest , @twitter , @microsoft