Algorithmic bias means the results of an intelligent system consistently lean one way and unfairly affect some people or groups. This bias is usually not intentional and comes from the data or the system's design.
An example#
Suppose a company trains a model to screen résumés using the past ten years of hiring data. If mostly men were hired in those years, the model learns to score résumés that resemble men's more highly, without anyone giving such an instruction. The model has simply repeated the past.
Where bias comes from#
- Training data: insufficient or unfair representation of groups.
- Labeling: the prejudices of the people labeling.
- Feature selection: variables that act as proxies for gender, ethnicity or region.
- Feedback: the model reinforces its own results, such as recommending content that only gets more of the same.
Why it matters#
In hiring, lending, insurance, medicine and risk assessment, a biased model can repeat unfair decisions at scale. In language models, gender or cultural stereotypes may also appear in answers.
What can be done#
Examine the composition of the data, measure the model's performance separately for each group, document limitations, put human oversight on high-risk decisions, and review periodically.
Common mistake#
People assume an algorithm is neutral because it is mathematics. An algorithm learns whatever was in the data, with all its unfairness.