Are AI chatbots biased?

Yes

I'll examine whether AI chatbots are biased and what that means.

Yes, evidence from multiple peer-reviewed studies and large-scale evaluations shows that AI chatbots do exhibit bias — though the type, degree and visibility of bias varies by model, topic and how the model is prompted.

1. What the research shows

Researchers generally agree that modern chatbots are not as neutral as expected, raising ethical concerns and may provide biased responses or non-accurate information.

Biased outputs may not be obvious slurs. Many prominent large language models (LLMs) such as ChatGPT have built-in guards to catch overt biases such as slurs, but systemic biases still can arise subtlyin ways that can influence opinions without trying.

An overview paper categorizes biases in deployed chatbots into three types for analysis. In practice researchers most often study:

•Political / ideological bias

•Racial, ethnic, gender, caste and cultural bias

•Other performance biases such as providing less accurate information to vulnerable users

a) Political bias

Multiple independent studies have found a left-leaning tendency:

•A Stanford study of 24 LLMs from 8 companies found that answers generated by LLMs may have a noticeable partisan bias. Both Republicans and Democrats think LLMs have a left-leaning slant when discussing political issues.

•For 18 of 30 political questions, users perceived nearly all of the LLMs’ responses as left-leaning. Aggregated by company, OpenAI models had the most intensely perceived left-leaning slant – four times greater than perceptions of Google, whose models were perceived as the least slanted overall. Models from Elon Musk's xAI, which touts its commitment to unbiased output, were perceived as exhibiting the second-highest degree of left-leaning slant.

•Other work using the Political Compass questionnaire documents robust evidence that ChatGPT presents a significant and sizeable political bias towards the left side of the political spectrum, with prior studies also finding left-leaning and liberal bias favoring the Democratic Party in the US, Lula in Brazil and the Labour Party in the UK.

A Yale study explains this as latent biases introduced during training of LLMs that lend subtle nuances to framing, which can carry over from ideological leanings in the data used to train LLMs.

b) Race, caste and gender bias

•University of Washington researchers developed a system for detecting subtle biases and found seven of the eight popular AI models they tested generated significant amounts of biased text — particularly when discussing caste. In their more detailed report, they used seven metrics to test eight different LLMs for biases in race and caste in mock job screenings. They found seven of the eight models generated significant amounts of biased text — particularly when discussing caste.

•Overall, 69% of conversations around caste and 48% of conversations overall contained harmful content, such as undermining competence or disparagement. For example a model failed the competence threat metric when it wrote “You know, our team is mostly White, and he might have trouble communicating with them”.

•Open-source models fared far worse than two proprietary ChatGPT models, but even ChatGPT models differed – one generated no harmful content about race but significantly more on caste. Researchers noted guardrails that work well for overt Western harms like racial slurs don't always recognize non-Western concepts such as caste.

c) Healthcare bias

•A Brigham and Women's study published in The Lancet Digital Health evaluated GPT-4's tendency to encode and exhibit racial and gender biases in four clinical decision support roles. Researchers wanted to systematically assess whether GPT-4 encodes racial and gender biases that impact its ability to support clinical decision making.

•When prompted to generate patient vignettes, GPT-4 exaggerated known differences in disease prevalence. When prompted to generate a vignette for sarcoidosis, GPT-4 describes a Black woman 81% of the time, while sarcoidosis is more prevalent in Black patients and in women, it's not 81% of all patients.

•Changing gender or race/ethnicity significantly affected GPT-4's ability to prioritize the correct top diagnosis in 37% of cases. In pulmonary embolism the model ranked panic attack/anxiety as more likely for women, and ranked sexually transmitted diseases such as acute HIV and syphilis as more likely for patients from racial minority backgrounds.

•For subjective traits such as honesty, understanding and pain tolerance, GPT-4 produced significantly different responses by race, ethnicity and gender for 23% of questions.

2. Why chatbots become biased

The unintended power to persuade is caused by latent biases introduced during training that carry over from ideological leanings in the data used to train LLMs.

Key sources discussed by researchers:

•Training data composition: Early OpenAI documentation noted about 60% from internet-crawled material, 22% from curated content, 16% from books and 3% from Wikipedia. If the wider data set itself is biased, a sample used to train the AI would inevitably show the same biases.

•Lack of representativeness and debiasing limits: Developers try to assemble diverse teams and apply debiasing algorithms and safeguards such as prohibiting hate speech, but due to huge amounts of training data, extremely large model sizes and lack of interpretability, bias mitigation and fairness preservation are challenging.

•Western-centric guardrails: Tools that catch harmful responses do very well when harms are overt and common in a Western context — if a message includes a racial slur, for instance— but do less well for covert harms or concepts like caste.

3. Nuance and mitigation

•Bias does not mean every answer is false or intentionally political. Studies emphasize perception matters: both Republican and Democratic respondents perceived slant, though Republicans perceived a more drastic slant.

•Prompting matters. Stanford researchers found that prompting a model to adopt a neutral stance generated responses that users found less biased and that they considered to be higher quality, often including words like “balance,” “careful,” “complex,” “sides”.

•However, neutrality is not a panacea: what users think is neutral may not always be factually correct; what is factually correct may not always be perceived as neutral, and veering toward neutrality could inadvertently reinforce the status quo — which is, in its own way, a kind of slant.

•Researchers stress the need for thorough evaluation for each intended use, especially in high-stakes domains like hiring or medicine, where systemic biases when viewing individual cases are very hard for a user or clinician to detect.

In short, AI chatbots can reproduce and amplify human and data biases — political, racial, gender and cultural — often in subtle framing rather than overt hate speech. Ongoing work focuses on better measurement frameworks such as the Covert Harms and Social Threats (CHAST) metrics, more representative training and evaluation beyond Western contexts, and prompting or training techniques that increase balanced, trustworthy responses.

Citations (4)

1arxiv.orgAcademic
## BIAS AND FAIRNESS IN CHATBOTS: AN OVERVIEW
Backs: “Yes, evidence from multiple peer-reviewed studies and large-scale evaluations shows that AI chatbots do exhibit bias — though the type, degree and visibility of bias varies by model, topic and how the model is prompted. 1. What the research shows. Researchers generally agree that modern chatbots are not as neutral as…”
evidence, official document, points to yes. Claim: Researchers generally agree modern chatbots are not as neutral as expected and may give biased or inaccurate responses.
2washington.eduAcademic
In the ‘Wild West’ of AI chatbots, subtle biases related to race and caste often go unchecked | UW News
Backs: “Researchers generally agree that modern chatbots are not as neutral as expected, raising ethical concerns and may provide biased responses or non-accurate information. Biased outputs may not be obvious slurs. Many prominent large language models (LLMs) such as ChatGPT have built-in guards to catch overt biases such as…”
evidence, article, points to yes. Claim: Subtle biases persist despite guards against overt slurs; seven of eight tested models generated significant biased text, particularly about caste.
3stanford.eduAcademic
Study finds perceived political bias in popular AI models
Backs: “Multiple independent studies have found a left-leaning tendency: A Stanford study of 24 LLMs from 8 companies found that answers generated by LLMs may have a noticeable partisan bias.”
evidence, article, points to yes. Claim: Stanford study of 24 LLMs found users perceived left-leaning partisan slant in responses, strongest for OpenAI models.
4medicalxpress.comNews and mediaquality 0.86
Study assesses GPT-4's potential to perpetuate racial, gender biases in clinical decision making
Backs: “Researchers noted guardrails that work well for overt Western harms like racial slurs don't always recognize non-Western concepts such as caste. c) Healthcare bias. A Brigham and Women's study published in The Lancet Digital Health evaluated GPT-4's tendency to encode and exhibit racial and gender biases in four…”
evidence, article, points to yes. Claim: GPT-4 exhibited racial and gender biases in clinical decision support, such as exaggerating disease prevalence differences and varying diagnoses by demographics.