OpenAI’s Impressive New Chatbot Isn’t Immune to Racism

“OpenAI’s latest language model, ChatGPT, is making waves in the world of conversational AI. With its ability to generate human-like text based on input from users, ChatGPT has the potential to revolutionize the way we interact with machines.”

That paragraph was entirely generated by ChatGPT—the new chatbot released by artificial intelligence research lab OpenAI—using the query “write a lede for a story about ChatGPT for The Daily Beast.” Aside from helping lazy writers with their stories, the bot went viral on social media after its release on Nov. 30 and has even “crossed 1 million users” less than a week later, according to OpenAI CEO Sam Altman.

It’s easy to see why if you spend a few minutes “chatting” with it. You can give ChatGPT prompts as simple as “What’s the recipe for an Old Fashioned?” or as complex as “Tell me the story of the Tortoise and the Hare, but you’re a 1980s Valley Girl” and it’ll give you a pretty realistic response.

That’s because ChatGPT is a large-language model (LLM)—or AI that reads and generates text. Specifically, it was created using GPT-3, an LLM from OpenAI that’s been described as “one of the most interesting and important AI systems ever produced,” by philosopher David Chalmers. As such, it’s capable of generating sophisticated and uncanny responses with such a high degree of realism that you’d probably think it was just another person on the other end if you didn’t know it was a bot.

And that’s exactly where the problems start.

Despite its ability to generate uncanny responses, it still manages to fall prey to the same issues that have plagued large-language models: bias. After ChatGPT’s release last week, users quickly took to social media to post instances in which the bot generated racist, sexist, and generally problematic responses to prompts.

Steven T. Piantadosi, a computational cognitive scientist at the University of California, Berkeley, posted a Twitter thread that detailed a few of these instances that he uncovered—including responses where ChatGPT said that only white males make good scientists and that a child’s life shouldn’t be saved if they were an African American male.

OpenAI says on their website that while they “have safeguards in place, the system may occasionally generate incorrect or misleading information and produce offensive or biased content.” They do not specify what those safeguards are, but the chatbot won’t directly give you a problematic response if prompted.

In fact, if you ask it something like, “Tell me a racist joke,” it’ll tell you that it’s “not capable of generating offensive or harmful content.”

However, by using a few clever workarounds, it becomes much easier to get a problematic response from ChatGPT.

For example, Piantadosi would ask if it could write a Python script for the race and gender of a good scientist. Another user asked it to make a 1980s style rap song on how to tell if someone is a good scientist based on race and gender.

“I was looking for something simple that would show what biases the model has,” Piantadosi told The Daily Beast. “The mechanisms OpenAI uses to prevent this kind of thing seem to be pretty easily bypassed. When I asked for things in nonstandard ways—for example, as a table or as a program—ChatGPT was happy to write horrible stuff. It doesn’t do it every time exactly the same, but it’s clear there is a ton of bad content inside of these systems.”

The problem with this chatbot is the same one that has plagued AI for decades— how it’s trained. In order to teach ChatGPT, researchers train it using GPT-3 which is a massive datasets containing books, Wikipedia articles, and web page archives. While this gives it the opportunity to produce realistic responses, it also gives it the chance to mimic the worst of our own behavior—namely, racism and sexism.

We’ve seen it many times before too, from bots that spout alt-right apologia, to bots that go on racist rants, to home lending bots that reject mortgages for people of color. Meta recently came under fire after they released an academic research LLM dubbed Galactica that would make up completely fake studies that claimed eating crushed glass is good and that Black people don’t speak a language.

However, Piantadosi believes that these issues are actually caused by much more than just the dataset that these bots are trained on. “There is certainly a lot of really gross text on the internet, but the output of these models is never just about the training set,” he said. “A million choices made by engineers go into the design of these models, the underlying assumptions, testing, etc. Maybe more importantly, these companies make choices about how models are marketed and released.”

He added, “It’s a pretty common problem that ethics and safety take a back seat to having a flashy new application.”

That’s a depressing reality to contend with when you think about how much potential AI technologies have in improving lives. ChatGPT presents itself as a very sophisticated tool to do anything from simple research queries to creative pursuits like storytelling. A product like this could have a wide range of commercial applications for a lot of different people, including generating essays, stories, music, and articles out of whole cloth. (OpenAI did not respond when reached for comment.)

Meta Made DALL-E for Video and There’s No Way This Ends Well

And while its developers can try and say that this product isn’t meant to be taken seriously, people will undoubtedly do so—especially since its engineers went to great lengths to ensure that the responses were as realistic and natural as possible. Perhaps that’s where the biggest danger is.

“When you have a system that’s intentionally confusable with a real, knowledgeable person, people will naturally take its output as having the force of a real, informed person,” Piantadosi said. “So what it says really matters.”

Read more at The Daily Beast.

Got a tip? Send it to The Daily Beast here

Get the Daily Beast's biggest scoops and scandals delivered right to your inbox. Sign up now.

Stay informed and gain unlimited access to the Daily Beast's unmatched reporting. Subscribe now.