Awesome Machine Learning Interpretability

A maintained and curated list of practical and awesome responsible machine learning resources.

If you want to contribute to this list (and please do!), read over the contribution guidelines, send a pull request, or file an issue.

If something you contributed or found here is missing after our September 2023 redeux, please check the archive.

Contents

• Community and Policy

  • Community Frameworks and Guidance
  • Conferences and Workshops (WIP - please contribute!)
  • Official Policy, Frameworks and Guidance

• Evaluation and Standards

  • AI Incident Trackers
  • Auditing and Evaluation Resources (WIP - please contribute!)
  • Benchmarks
  • Common or Useful Datasets

• General Resources

  • Comprehensive Software Examples and Tutorials
  • Free-ish Books
  • Glossaries and Dictionaries
  • Open-ish Classes

• Miscellaneous Resources

  • Challenges and Competitions (WIP - please contribute!)
  • Curated Bibliographies (WIP - please contribute !)
  • Generative AI Responsible Use Guidance
  • List of Lists

• Technical Resources

  • Domain-specific Software (WIP - please contribute !)
  • Machine Learning Environment Management Tools
  • Open Source/Access Responsible AI Software Packages
    • Browser
    • C/C++
    • Python
    • R

Community and Policy

Community Frameworks and Guidance

This section is for responsible ML guidance put forward by organizations or individuals, not for official government guidance.

• 8 Principles of Responsible ML
• Adversarial ML Threat Matrix
• AI Snake Oil
• AllenNLP Interpret: A Framework for Explaining Predictions of NLP Models
• Anthropic's Responsible Scaling Policy
• BIML Interactive Machine Learning Risk Framework
• Dealing with Bias and Fairness in AI/ML/Data Science Systems
• Debugging Machine Learning Models (ICLR workshop proceedings)
• Decision Points in AI Governance
• Distill
• FATML Principles and Best Practices
• From Principles to Practice: An interdisciplinary framework to operationalise AI ethics
• Google's Responsible AI Framework
• H2O.ai Algorithms
• Identifying and Overcoming Common Data Mining Mistakes
• MACHINE LEARNING QUICK REFERENCE: BEST PRACTICES
• MACHINE LEARNING QUICK REFERENCE: ALGORITHMS
• Microsoft Responsible AI Standard, v2
• Robust ML
• Safe and Reliable Machine Learning
• Sample AI Incident Response Checklist
• Troubleshooting Deep Neural Networks
• Warning Signs: The Future of Privacy and Security in an Age of Machine Learning
• When Not to Trust Your Explanations
• You Created A Machine Learning Application Now Make Sure It's Secure

Conferences and Workshops (WIP - please contribute!)

This section is for conferences, workshops and other major events related to responsible ML.

• ACM FAccT (Fairness, Accountability, and Transparency)
  • FAT/ML (Fairness, Accountability, and Transparency in Machine Learning)
• AIES (AAAI/ACM Conference on AI, Ethics, and Society)
• Black in AI

Official Policy, Frameworks and Guidance

This section serves as a repository for policy documents, regulations, guidelines, and recommendations that govern the ethical and responsible use of artificial intelligence and machine learning technologies. From international legal frameworks to specific national laws, the resources cover a broad spectrum of topics such as fairness, privacy, ethics, and governance.

• 12 CFR Part 1002 - Equal Credit Opportunity Act (Regulation B)
• A Regulatory Framework for AI: Recommendations for PIPEDA Reform
• AI Principles: Recommendations on the Ethical Use of Artificial Intelligence by the Department of Defense
• THE AIM INITIATIVE
• Aiming for truth, fairness, and equity in your company’s use of AI
• Algorithmic Accountability Act of 2019
• ALGORITHM CHARTER FOR AOTEAROA NEW ZEALAND
• Artificial Intelligence (AI) in the Securities Industry
• Article 22 EU GDPR
• Assessment List for Trustworthy Artificial Intelligence (ALTAI) for self-assessment - Shaping Europe’s digital future - European Commission
• Audit of Governance and Protection of Department of Defense Artificial Intelligence Data and Technology
• A Primer on Artificial Intelligence in Securities Markets
• Biometric Information Privacy Act
• Booker Wyden Health Care Letters
• California Consumer Privacy Act (CCPA)
• California Privacy Rights Act (CPRA)
• Can’t lose what you never had: Claims about digital ownership and creation in the age of generative AI
• Civil liability regime for artificial intelligence
• De-identification Tools
• DEVELOPING FINANCIAL SECTOR RESILIENCE IN A DIGITAL WORLD: SELECTED THEMES IN TECHNOLOGY AND RELATED RISKS
• Directive on Automated Decision Making
• EEOC Letter (from U.S. senators re: hiring software)
• Facial Recognition and Biometric Technology Moratorium Act of 2020
• Four Principles of Explainable Artificial Intelligence
• General principles for the use of Artificial Intelligence in the financial sector
• Gouvernance des algorithmes d’intelligence artificielle dans le secteur financier (French)
• IAPP Global AI Legislation Tracker
• IAPP US State Privacy Legislation Tracker
• Innovation spotlight: Providing adverse action notices when using AI/ML models
• Office of Management and Budget Guidance for Regulation of Artificial Intelligence Applications (Finalized Nov. 2020)
• NIST AI 100-1 Artificial Intelligence Risk Management Framework (NIST AI RMF 1.0)
• Principles of Artificial Intelligence Ethics for the Intelligence Community
• Proposal for a Regulation laying down harmonised rules on artificial intelligence (Artificial Intelligence Act)
• Psychological Foundations of Explainability and Interpretability in Artificial Intelligence
• Questions and Answers to Clarify and Provide a Common Interpretation of the Uniform Guidelines on Employee Selection Procedures
• Questions from the Commission on Protecting Privacy and Preventing Discrimination
• RE: Use of External Consumer Data and Information Sources in Underwriting for Life Insurance
• Singapore Personal Data Protection Commission (PDPC) Model Artificial Intelligence Governance Framework
• SUPERVISORY GUIDANCE ON MODEL RISK MANAGEMENT
• Artificial Intelligence/Machine Learning (AI/ML)-Based: Software as a Medical Device (SaMD) Action Plan (Updated Jan. 2021)
• Using Artificial Intelligence and Algorithms

Evaluation and Standards

AI Incident Trackers

This section is for repositories of information related to AI incidents and vulnerabilities. These resources should enable ML practitioners to learn from past mistakes, not just highlight failures.

• AI Incident Database (Responsible AI Collaborative)
• AI Vulnerability Database (AVID)
• AIAAIC
• George Washington University Law School's AI Litigation Database
• Verica Open Incident Database (VOID)

Auditing and Evaluation Resources (WIP - please contribute!)

We are seeking contributions related to auditing and evaluating ML systems for this section.

• Independent Audit of AI Systems
• Real-World Strategies for Model Debugging
• Testing and Debugging in Machine Learning

Benchmarks

This section contains benchmarks or datasets used for benchmarks for ML systems, particularly those related to responsible ML desiderata.

• benchm-ml
• Bias Benchmark for QA dataset (BBQ)
• HELM
• OpenML Benchmarking Suites
• TruthfulQA
• Winogender Schemas
• Real Toxicity Prompts (Allen Institute for AI)

Common or Useful Datasets

This section contains datasets that are commonly used in responsible ML evaulations or repositories of interesting/important data sources:

• Adult income dataset
• Balanced Faces in the Wild
• COMPAS Recidivism Risk Score Data and Analysis
• Data Repositories:
  • All Lending Club loan data
  • Amazon Open Data
  • Data.gov
  • Home Mortgage Disclosure Act (HMDA) Data
  • MIMIC-III Clinical Database
  • UCI ML Data Repository
• FANNIE MAE Single Family Loan Performance
• NYPD Stop, Question and Frisk Data
• Statlog (German Credit Data)
• Wikipedia Talk Labels: Personal Attacks

General Resources

Comprehensive Software Examples and Tutorials

This section is a curated collection of guides and tutorials that simplify responsible ML implementation. It spans from basic model interpretability to advanced fairness techniques. Suitable for both novices and experts, the resources cover topics like COMPAS fairness analyses and explainable machine learning via counterfactuals.

• COMPAS Analysis Using Aequitas
• Explaining Quantitative Measures of Fairness (with SHAP)
• Getting a Window into your Black Box Model
• From GLM to GBM Part 1
• From GLM to GBM Part 2
• IML
• Interpretable Machine Learning with Python
• Interpreting Machine Learning Models with the iml Package
• Interpretable Machine Learning using Counterfactuals
• Machine Learning Explainability by Kaggle Learn
• Model Interpretability with DALEX
• Model Interpretation series by Dipanjan (DJ) Sarkar:
  • The Importance of Human Interpretable Machine Learning
  • Model Interpretation Strategies
  • Hands-on Machine Learning Model Interpretation
  • Interpreting Deep Learning Models for Computer Vision
• Partial Dependence Plots in R
• PiML-Toolbox
• Saliency Maps for Deep Learning
• Visualizing ML Models with LIME
• Visualizing and debugging deep convolutional networks
• What does a CNN see?

Free-ish Books

This section contains books that can be reasonably described as free.

• An Introduction to Machine Learning Interpretability
• Explanatory Model Analysis
• Fairness and Machine Learning
• Interpretable Machine Learning
• Machine Learning for High-Risk Applications: Approaches to Responsible AI
• Responsible Machine Learning
• Trustworthy Machine Learning

Glossaries and Dictionaries

This section features a collection of either glossaries and dictionaries that are geared toward defining terms in ML.

• H2O.ai Glossary
• IAPP Glossary of Privacy Terms
• IAPP International Definitions of Artificial Intelligence
• IAPP Key Terms for AI Governance
• The Language of Trustworthy AI: An In-Depth Glossary of Terms (NIST AIRC) Glossary
• ISO: Information technology — Artificial intelligence — Artificial intelligence concepts and terminology
• VAIR (Vocabulary of AI Risks)
• Stanford University HAI Artificial Intelligence Definitions

Open-ish Classes

This section features a selection of educational courses focused on ethical considerations and best practices in ML. The classes range from introductory courses on data ethics to specialized training in fairness and trustworthy deep learning.

• An Introduction to Data Ethics
• Certified Ethical Emerging Technologist
• CS103F: Ethical Foundations of Computer Science
• Fairness in Machine Learning
• Fast.ai Data Ethics course
• Human-Centered Machine Learning
• INFO 4270: ETHICS AND POLICY IN DATA SCIENCE
• Introduction to Responsible Machine Learning
• Machine Learning Fairness by Google
• Trustworthy Deep Learning

Miscellaneous Resources

Challenges and Competitions (WIP - please contribute!)

This section contains challenges and competitions related to responsible ML.

• FICO Explainable Machine Learning Challenge
• National Fair Housing Alliance Hackathon
• Twitter Algorithmic Bias

Curated Bibliographies (WIP - please contribute !)

We are seeking curated bibliographies related to responsible ML.

BibTeX:

• Proposed Guidelines for Responsible Use of Explainable Machine Learning (presentation, bibliography)
• Proposed Guidelines for Responsible Use of Explainable Machine Learning (paper, bibliography)
• A Responsible Machine Learning Workflow (paper, bibliography)

Web:

• Fairness, Accountability, and Transparency in Machine Learning (FAT/ML) Scholarship

Generative AI Responsible Use Guidance

This section contains information related to the responsible use of generative AI.

• Andreessen Horowitz (a16z) AI Canon
• DAIR Prompt Engineering Guide
  • DAIR Prompt Engineering Guide GitHub
• Ethical and social risks of harm from Language Models
• Llama 2 Responsible Use Guide
• PAI's Responsible Practices for Synthetic Media: A Framework for Collective Action (Partnership on AI)

List of Lists

This section links to other lists of responsible ML or related resources.

• A Living and Curated Collection of Explainable AI Methods
• AI Ethics Guidelines Global Inventory
• AI Ethics Resources
• AI Tools and Platforms
• Awesome interpretable machine learning
• Awesome-ML-Model-Governance
• Awesome MLOps
• Awesome Production Machine Learning
• Awful AI
• criticalML
• Machine Learning Ethics References
• Machine Learning Interpretability Resources
• OECD-NIST Catalogue of AI Tools and Metrics
• private-ai-resources
• ResponsibleAI
• XAI Resources
• xaience

Technical Resources

Domain-specific Software (WIP - please contribute !)

This section curates specialized software tools aimed at responsible ML within specific domains, such as in healthcare, finance, or social sciences.

Machine Learning Environment Management Tools

This section contains open source or open access ML environment management software.

• dvc
• gigantum
• mlflow
• mlmd
• modeldb

Open Source/Access Responsible AI Software Packages

This section contains open source or open access software used to implement responsible ML.

Browser

• DiscriLens
• manifold
• TensorBoard Projector
• What-if Tool

C/C++

• Born-again Tree Ensembles
• Certifiably Optimal RulE ListS

Python

• acd
• aequitas
• AI Fairness 360
• AI Explainability 360
• ALEPython
• Aletheia
• allennlp
• algofairness
• Alibi
• anchor
• Bayesian Case Model
• Bayesian Ors-Of-Ands
• Bayesian Rule List (BRL)
• BlackBoxAuditing
• casme
• Causal Discovery Toolbox
• captum
• causalml
• cdt15
• checklist
• cleverhans
• contextual-AI
• ContrastiveExplanation (Foil Trees)
• counterfit
• dalex
• debiaswe
• DeepExplain
• deeplift
• deepvis
• dianna
• DiCE
• DoWhy
• ecco
• eli5
• explabox
• Explainable Boosting Machine (EBM)/GA2M
• explainerdashboard
• fair-classification
• fairml
• fairlearn
• fairness-comparison
• fairness_measures_code
• Falling Rule List (FRL)
• foolbox
• Grad-CAM (GitHub topic)
• gplearn
• H2O-3
  • Penalized Generalized Linear Models
  • Monotonic GBM
  • Sparse Principal Components (GLRM)
• hate-functional-tests
• imodels
• iNNvestigate neural nets
• Integrated-Gradients
• interpret
• interpret_with_rules
• Keras-vis
• keract
• L2X
• learning-fair-representations
• lime
• LiFT
• lit
• lofo-importance
• lrp_toolbox
• MindsDB
• MLextend
• ml-fairness-gym
• ml_privacy_meter
• Monotonic XGBoost
• Multilayer Logical Perceptron (MLLP)
• OptBinning
• Optimal Sparse Decision Trees
• parity-fairness
• PDPbox
• ProtoPNet
• pyBreakDown
• PyCEbox
• pyGAM
• pymc3
• pySS3
• pytorch-innvestigate
• rationale
• responsibly
• revise-tool
• robustness
• RISE
• Risk-SLIM
• sage
• SALib
• Scikit-learn
  • Decision Trees
  • Generalized Linear Models
  • Sparse Principal Components
• scikit-fairness
• scikit-multiflow
• shap
• shapley
• sklearn-expertsys
• skope-rules
• Super-sparse Linear Integer models (SLIMs)
• tensorflow/lattice
• tensorflow/lucid
• tensorflow/fairness-indicators
• tensorflow/model-analysis
• tensorflow/model-card-toolkit
• tensorflow/model-remediation
• tensorflow/privacy
• tensorflow/tcav
• tensorfuzz
• TensorWatch
• TextFooler
• text_explainability
• text_sensitivity
• tf-explain
• Themis
• themis-ml
• treeinterpreter
• woe
• xai
• xdeep
• yellowbrick

R

• ALEPlot
• arules
• Causal SVM
• DALEX
• DALEXtra
• DrWhyAI
• elasticnet
• ExplainPrediction
• Explainable Boosting Machine (EBM)/GA2M
• fairmodels
• fairness
• fastshap
• featureImportance
• flashlight
• forestmodel
• fscaret
• gam
• glm2
• glmnet
• H2O-3
  • Penalized Generalized Linear Models
  • Monotonic GBM
  • Sparse Principal Components (GLRM)
• iBreakDown
• ICEbox
• iml
• ingredients
• intepret
• lightgbmExplainer
• lime
• live
• mcr
• modelDown
• modelOriented
• modelStudio
• Monotonic XGBoost
• quantreg
• rpart
• RuleFit
• Scalable Bayesian Rule Lists (SBRL)
• shapFlex
• shapleyR
• shapper
• smbinning
• vip
• xgboostExplainer