Updated August 28, 2026
TL;DR: Start with Fairlearn for most Python classification and regression projects. Use AIF360 when you need a broad catalog of fairness metrics and mitigation algorithms, but isolate its dependencies. For generative AI, choose LangFair for use-case-level fairness measurement and LangTest for broader behavioral testing. These tools can expose disparities and test interventions; none can certify that a system is fair.
AI fairness work has changed since this list first appeared in 2024. Several familiar projects are now legacy software, while current teams need to evaluate both conventional predictive models and generative-AI applications. The harder lesson has not changed: a library cannot decide which harms matter, which population deserves protection, or which trade-off is acceptable. Those are product, policy, domain, and governance decisions.
This guide focuses on seven open-source projects that still have a credible role in a 2026 workflow. Some detect disparities, some implement mitigation, and some stress-test language systems. They are not interchangeable.
What Is AI Bias Detection?
AI bias detection is the process of testing whether data, model errors, rankings, decisions, or generated outputs differ across groups in ways that could create unfair harm. A credit model, for example, can have good overall accuracy while producing a higher false-negative rate for one group. An LLM can answer the same prompt differently after only a demographic attribute is changed.
Fairness is not a single score. Demographic parity compares outcome or selection rates. Equal opportunity compares true-positive rates. Equalized odds compares both true- and false-positive rates. Calibration asks whether a score has the same meaning across groups. These criteria can point in different directions, so the metric must follow the real-world harm and decision process – not the other way around.
Bias mitigation changes the system after a problem is identified. It may reweight or repair training data, constrain model training, adjust thresholds after training, redesign a ranking or workflow, or add human review. Every intervention should be re-evaluated for utility, calibration, intersectional effects, and new harms.
Important: these tools produce technical evidence. They do not prove legal compliance, eliminate the need for affected-stakeholder review, or establish that discrimination is absent.
How We Chose the Seven Tools
We checked public repositories, documentation, release histories, package metadata, and licenses as of August 28, 2026. To make the list, a project needed:
A clear fairness or bias-assessment capability – not only generic model monitoring.
Usable open-source code under a permissive license.
A documented installation path and enough current activity or ecosystem support to justify evaluation for a new project.
A distinct job in the workflow: metrics, mitigation, LLM evaluation, large-scale slicing, or behavioral testing.
Quick Comparison
Tool | Best for | Detects | Mitigates | License |
|---|---|---|---|---|
Fairlearn | Default Python choice for group fairness | Yes | Yes | MIT |
AIF360 | Broad research catalog and method comparison | Yes | Yes | Apache 2.0 |
HolisticAI | Fairness inside a broader trustworthiness audit | Yes | Yes | Metadata conflict: MIT / Apache 2.0 |
Aequitas | Transparent audits for binary decisions | Yes | Yes, through Aequitas Flow | MIT |
Fairness Indicators | Large-scale sliced evaluation in TensorFlow/TFX | Yes | No | Apache 2.0 |
LangFair | Use-case-level LLM bias and fairness measurement | Yes | No automatic mitigation | Apache 2.0, with listed MIT components |
LangTest | Broad NLP and LLM behavioral testing | Yes | Partial: data augmentation for select models | Apache 2.0 |
Which AI Bias Detection Tools Are Outdated in 2026?
The original version included the What-If Tool, FAT Forensics, Themis-ML, and FairTest. They remain useful historical or research references, but we no longer recommend them as default choices for a new 2026 production project. FairTest’s public installation instructions still target Python 2.7. The What-If Tool’s documented workflow centers on TensorBoard, TensorFlow Estimator, AI Platform Prediction, and TensorFlow Serving interfaces from an earlier generation. FAT Forensics and Themis-ML have older public release and dependency surfaces.
We have also moved Microsoft’s Responsible AI Toolbox out of the recommended seven. The last published raiwidgets release is 0.36.0 from July 2024. The current main branch now declares Python 3.9–3.11 – not 3.7–3.10 – but the release gap still fails the currentness test for a new project. Its fairness dashboard is powered by Fairlearn, while mitigation lives in separate tools such as Fairlearn and the responsible-ai-toolbox-mitigations project. Existing users can keep evaluating it, but new teams should expect compatibility work.
We will keep this lists updated. Our goal is to distinguish historical reference implementations from software a team should adopt now.
Fairlearn
Fairlearn computes disaggregated model metrics with MetricFrame, compares performance across user-defined sensitive groups, and includes mitigation methods for classification and regression. Its reductions approach can train models under fairness constraints, while ThresholdOptimizer applies post-processing when group-specific thresholds are appropriate and permitted.
The library fits naturally into scikit-learn-style workflows and makes performance-versus-fairness trade-offs visible. Fairlearn 0.14.0 was released in June 2026, making it the strongest default for many tabular ML projects.
Best for: teams that want a practical, well-documented starting point for assessing and mitigating group fairness in Python.
Watch for: Fairlearn focuses on group fairness. You still need defensible group definitions, sufficient sample sizes, intersectional analysis, and a monitoring plan. It is not an LLM-output evaluation toolkit.
License: MIT. Documentation · GitHub
AI Fairness 360 (AIF360)
AIF360, created by IBM Research and now hosted under Trusted-AI, remains the broadest toolbox in this list. It includes dataset- and model-level metrics plus preprocessing, in-processing, and post-processing algorithms. The project supports Python and R.
Use it when method breadth matters: reweighing, learned fair representations, adversarial debiasing, equalized-odds post-processing, and reductions are available in one ecosystem. Its notebooks are also useful for learning how alternative definitions change a result.
Best for: researchers and advanced teams comparing many fairness definitions and mitigation methods.
Watch for: the repository currently gives conflicting compatibility signals. Its README lists Python 3.10–3.13, while setup.py still identifies version 0.6.1, advertises Python 3.8–3.11, and pins scikit-learn below 1.6. That packaging constraint is the practical issue. Install only the extras you need and use a pinned, isolated environment.
License: Apache 2.0. Documentation · GitHub
HolisticAI
HolisticAI organizes its toolkit into five modules: bias, explainability, robustness, security, and efficacy. Optional extras let teams install only the capabilities they need. The bias module covers classification, regression, clustering, recommender systems, and mitigation methods, so the package can support a broader model-risk review than a fairness-only library.
Best for: teams that want bias metrics and mitigation alongside explainability, robustness, security, and efficacy checks.
Watch for: the latest PyPI release is 1.0.14 from March 2025, and the project has a license-metadata inconsistency. The repository’s top-level LICENSE file contains Apache 2.0, while the current pyproject.toml and PyPI metadata say MIT. Both are permissive licenses, but teams redistributing the package should confirm which notice applies to the exact version they use.
License: conflicting current metadata: MIT in package metadata; Apache 2.0 in the repository LICENSE. GitHub · PyPI
Aequitas
Aequitas was created by the Center for Data Science and Public Policy, originally at the University of Chicago. It takes scores, labels, and categorical group attributes, then reports disparities in selection rate, false-positive rate, false-negative rate, true-positive rate, and predicted prevalence. Its audit format is easier to communicate than a collection of disconnected notebook calculations.
Aequitas 1.0 introduced Aequitas Flow, adding experiments with fairness-aware methods and visualizations of performance/fairness trade-offs. Version 1.1.0 arrived in February 2026 and supports Python 3.10–3.13. The project’s center of gravity remains transparent auditing.
Best for: transparent disparity audits in structured, binary-decision settings.
Watch for: it is most natural for structured, binary-decision settings. Teams working on ranking, vision, or free-form generation will need other tools.
License: MIT. Documentation · GitHub
TensorFlow Fairness Indicators
Fairness Indicators computes and visualizes performance across user-defined slices for binary and multiclass classifiers. It supports confidence intervals, comparisons across thresholds, candidate-versus-baseline evaluation, and analysis over large datasets through TensorFlow Model Analysis. The documentation also describes a model-agnostic path when predictions are available outside TensorFlow.
Version 0.52.0 was released in July 2026, but Google’s documentation still labels the project Beta. Treat that label as a maturity warning, especially for long-lived production dependencies.
Best for: large-scale sliced evaluation when TensorFlow Model Analysis or TFX is already part of the production stack.
Watch for: Fairness Indicators is an assessment and visualization layer, not a mitigation library. Its pip package still bundles the What-If Tool, but that does not make the older What-If Tool workflow a current standalone recommendation. The surrounding TFMA/TFX dependency stack can also be heavy for teams that do not already use it.
License: Apache 2.0. Documentation · GitHub
LangFair
LangFair, developed by CVS Health, uses a Bring Your Own Prompts approach. Instead of assuming that a static benchmark represents the application, it evaluates repeated model outputs for the use case at hand. Its output-based design can assess text generation and summarization without access to internal model weights or activations.
This is the clearest fit here for chatbots, assistants, summarizers, and other generative-AI products. It works with LangChain-compatible interfaces and includes guidance for choosing use-case-level metrics. Version 0.8.0 was released in January 2026.
Best for: measuring bias and fairness in a specific LLM application using the prompts and outputs that users actually encounter.
Watch for: generative evaluations are stochastic. Run enough samples, preserve prompts and model settings, report uncertainty, and manually inspect examples. LangFair measures risks; it does not automatically debias the model.
License: Apache 2.0, except for three components explicitly identified as MIT-licensed in the project’s LICENSE file. Documentation · GitHub
LangTest
LangTest generates and runs more than 60 test types across categories such as bias, fairness, representation, robustness, and accuracy. It supports conventional NLP tasks – including named-entity recognition, translation, and text classification – as well as LLM tests for question answering, toxicity, factuality, sycophancy, summarization, and domain-specific scenarios.
Version 2.8.0 was released on August 17, 2026 and requires Python 3.12 or newer. LangTest can also augment training data based on failed tests for select models. That is useful, but it is not a universal fairness mitigation system: every generated example and retrained result still needs validation.
Best for: teams that want one behavioral testing harness for NLP models and LLM applications, including fairness, bias, representation, robustness, and accuracy.
Watch for: LangTest is broader than LangFair, but less specialized in formal group-fairness definitions. Use it to turn behavioral requirements into repeatable tests, not to replace a decision-specific fairness analysis with Fairlearn or AIF360.
License: Apache 2.0. Documentation · GitHub
How to Choose the Right Tool
Most Python classification or regression projects: begin with Fairlearn.
You need to compare many academic mitigation methods: use AIF360 in a pinned environment.
You want fairness inside a wider trustworthiness review: evaluate HolisticAI and confirm the license notice for your version.
You need a transparent binary-decision audit: use Aequitas.
Your evaluation pipeline already runs on TensorFlow/TFX: evaluate Fairness Indicators, noting its Beta status.
You are measuring fairness in a specific LLM use case: use LangFair with real prompts and repeated generations.
You need broad, repeatable NLP or LLM behavior tests: use LangTest.
For foundation-model-specific mitigation methods, see our separate guide to bias in foundation models and LLMs. Fairness testing measures disparities but does not block unsafe behaviour at runtime; for that layer see our overview of open-source AI guardrails.
A Practical Fairness Audit Workflow
Define the decision and potential harm. Write down who is affected, what outcome matters, and which mistakes carry the highest cost.
Choose groups and intersections. Do not stop at one attribute. Check combinations where sample size allows, and document missing or unreliable demographic data.
Establish utility baselines. Measure accuracy, calibration, ranking quality, or task success before comparing groups.
Compute several relevant fairness measures. Include confidence intervals and group counts. A large ratio based on a tiny slice is not reliable evidence.
Inspect examples and causes. Look for label bias, sampling gaps, proxy variables, threshold effects, feedback loops, and deployment-process failures.
Test mitigation as an intervention. Compare pre-, in-, and post-processing options, and quantify what improves and what degrades.
Monitor after launch. Populations, policies, labels, and model behavior drift. Re-run the audit on fresh data and track real outcomes – not only offline metrics.
FAQ
What is the best tool for AI bias detection?
Fairlearn is the strongest starting point for many Python classification and regression workflows because it combines disaggregated metrics with practical mitigation algorithms. AIF360 offers a broader research catalog. For language systems, use LangFair for use-case-level fairness measurement and LangTest for broader behavioral testing.
Is AIF360 still maintained?
AIF360 remains public and usable, but its compatibility signals conflict. The current setup metadata identifies version 0.6.1, advertises Python 3.8–3.11, and pins scikit-learn below 1.6, while the README lists Python 3.10–3.13. Treat it as a mature, compatibility-sensitive toolkit and use a pinned environment.
What is the difference between bias detection and bias mitigation?
Bias detection measures disparities and helps diagnose where they occur. Bias mitigation changes the data, training procedure, decision threshold, ranking logic, or surrounding process to reduce a defined harm. Mitigation must be re-tested because improving one fairness measure can reduce utility or worsen another.
Can open-source fairness tools test LLMs?
Yes. LangFair evaluates repeated outputs using prompts from the actual use case, while LangTest runs broader tests for bias, fairness, representation, robustness, and other behaviors. Conventional tools such as Fairlearn can help when LLM outputs become structured decisions with labels and demographic slices.
Which fairness metric should I use?
Choose the metric that matches the harm. Selection-rate measures fit allocation questions; false-negative or false-positive gaps fit unequal error costs; calibration matters when scores guide decisions. Report relevant metrics, group sizes, uncertainty, and why the definition fits the application.
❝
If you’ve found this article valuable, subscribe for free to our newsletter.
