Bias & fairness testing explained

94% accuracy tells you nothing about who the other 6% are. Bias testing is about that question: not how well a model performs on average, but for whom it fails.

A model can perform excellently across a test set as a whole while systematically performing worse for a specific group. That is not an edge case: it's exactly what bias testing is about.

Where bias comes from

Data bias

The training data reflects historical inequality, or under-represents certain groups.

Label bias

The "correct answers" the model was trained on are themselves already coloured by human prejudice.

Proxy bias

A seemingly neutral feature (postcode, school name) correlates strongly with a protected characteristic.

The last one is hardest to spot. A model that never receives gender or ethnicity as input can still exhibit discriminatory behaviour because other variables correlate strongly with it. "We don't use that feature" is therefore no guarantee.

How you measure it

// Two common fairness metrics

Demographic parityequal outcome odds per group

Equalized oddsequal error and hit rate per group

Neither metric is universally "the right one". Demographic parity can be undesirable when groups genuinely differ in the underlying situation; equalized odds requires reliable labels per group, which aren't always available. The choice depends on context, and that choice should be a deliberate, documented decision, not an accidental one.

Common mistake: testing only the average

A global accuracy score hides exactly the problem bias testing is meant to expose. Always test broken down by relevant subgroup.

Common mistake: bias testing as a one-off check

A model that scores fair today may no longer be fair after retraining on new data. Bias testing belongs with every model version, not just at launch.

Fairness isn't a trait a model has or doesn't have. It's a measurable property that varies by context, by metric and by subgroup, and one you have to check explicitly, not assume.

Why this is more than an ethical question

For AI systems in HR, credit scoring or other Annex III domains under the EU AI Act, non-discrimination isn't an optional aspiration but a legal obligation. Tools like Fairlearn and SHAP make bias measurable and the contribution of individual features traceable: exactly the kind of evidence a technical file needs to contain.

An example from practice

For a credit-scoring model we validated for a client, the model scored 91% overall accuracy. Broken down by postal code, though, the rejection rate for two neighbourhoods turned out to be double the average, with no difference in income or credit history to explain it.

// What the breakdown revealed

Overall accuracy91%

Rejection rate, average14%

Rejection rate, two specific postal codes27–29%

Postal code wasn't a model input. But a correlated feature (collateral type, strongly tied to neighbourhood) acted as a proxy. You never find that kind of link by inspecting input variables alone; you find it by breaking outcomes down by group and comparing.

How we approach this in practice

Always break down on multiple attributes at once. A model that scores fairly by gender and separately fairly by age can still systematically disadvantage one specific combination of both.
Explicitly look for proxy variables. Test whether non-protected attributes correlate strongly with protected ones, especially with location, name, or institution data.
Use Fairlearn or SHAP to document the contribution per feature. That shows not just that a gap exists, but which feature drives it.
Fix the chosen fairness metric upfront, not after the fact based on whichever result looks best.

// Know who your model isn't working for

Curious whether your model systematically disadvantages a group?

We run a bias analysis on your model and show concretely where the differences lie.

Book a call