Bias and discrimination in AI


MANY people are unaware that the devices they use rely on artificial intelligence (AI) techniques to work.

AI is ubiquitously used by many devices and yet invisible to most people. Most applications and devices treat AI as a black box where the internal components and workings are hidden away from the general view.

For example, the fingerprint or face recognition to unlock smartphones, the smartphone’s camera that focus on the image and navigation maps. AI can be described as a computer programme that improves conventional coding methods by simulating how humans think or how nature works.

There are many sub-groups of AI, for example, machine learning, neural networks and genetic algorithms. Machine learning techniques use data and algorithms to mimic how people learn, increasing the model’s accuracy. Neural networks try to reproduce how human brains work in code. Neurons are coded in multiple layers and the connections between neurons are coded as weights.

On the other hand, genetic algorithms are influenced by Charles Darwin’s notion of natural evolution. This algorithm chooses the best solutions to serve as parents at each iteration. The parent will then give “birth” to “children” who have some of their “parents” traits and some random elements. This algorithm will then continue to choose the best solutions that meet the objective and evolve the solutions over hundreds to thousands of generations.

AI offers many benefits over traditional programming. Traditional programming requires extensive manual and labour-intensive work to code or update the application. However, with AI, the algorithm can automatically formulate and derive rules from the data.

Another advantage of AI is the capability of AI to overcome some of the linear assumptions of traditional techniques to solve. The result is a technique that can work out non-linear and complex problems such as face and voice recognition.

Despite the benefits of AI, there are issues with AI that are not widely mentioned and discussed. The problems are the biases and discrimination that the AI could produce. AI may produce discriminatory outcomes if the AI learns from discriminatory training data.

One of the principles of the information system is the Garbage In, Garbage Out (GIGO). No matter how sophisticated and terrific the system is, the output produced will also be atrocious if the data used is rubbish.

For example, it’s not always possible for speech recognition technologies to accurately translate spoken words, especially accents from non-first-language English speakers. The speech recognition training data is typically taken from the US, where most AI speech recognition is developed. This leaves out most of the population who are not from English-speaking countries and have accents not used for training AI speech recognition.

Some US cities utilise AI detection to detect and predict areas with a higher likelihood of crime. It is possible that the results could be biased and skewed from the data collected from those neighbourhoods. If the police pay special attention to neighbourhoods based on the ethnic breakdown of its communities, they increase their presence there; therefore, more crimes will be reported or picked up.

Those neighbourhoods are likely to be consistently overrepresented in police records. Suppose an AI system is trained on such a skewed sample. In that case, it will learn that criminal activity is more prevalent among those groups and neighbourhoods.

Another current usage is the human resource software to incorporate AI to rank and filter candidates for interviews. The AI may be biased against some groups, for instance, choosing candidates who attend prestigious and expensive universities. However, some racial or lower socioeconomic groups are unlikely to enrol in those universities. As a result, choosing job applicants based on whether they attended a prestigious institution could have discriminatory consequences.

Banks and other fintech lending platforms have also used AI to score credit assessments. The individuals applying for personal loans, besides traditional measures such as credit scores, income level and net worth, can also be scored according to their characteristics, for example, gender, ethnic group, age and employment history. Similarly, this might cause AI outputs to be biased against certain groups.

Are we pre-ordained to follow whatever the outputs from the AI models are? If the AI model predicts that you are likely to fail in a job, does it mean it is guaranteed you will fail and should not be selected for an interview?

The outcomes of AI models depend on probabilities. They may be based on biased training data sets due to how the data is gathered and used in these techniques. Due to the opaqueness of such results, it is frequently tricky for someone to understand why the AI system has made certain decisions or outputs.

As a result, it is challenging for people to determine whether they have experienced discrimination based on, for example, race or socioeconomic grouping. All this points to a need for strict human oversight of AI decisions.

  • Dr Siew Eu Gene is a senior lecturer at Monash University Malaysia’s School of Business.