A conversational model with a warning label attached
OpenAI released ChatGPT in late 2022, a platform that interacts conversationally and can answer follow-up questions, admit its mistakes, challenge incorrect premises and reject inappropriate requests. The release quickly drew widespread attention online. The company describes it as the latest step in an iterative deployment of increasingly safe and useful AI systems, and says lessons from earlier models, including GPT-3 and Codex, shaped it — among them substantial reductions in harmful and untruthful outputs.
Trained by ranking, not by rulebook
The model was trained using reinforcement learning from human feedback, or RLHF, with the same methods used for InstructGPT but with slight differences in the data collection setup. OpenAI’s team describes a two-part process. First came supervised fine-tuning: human AI trainers supplied conversations in which they played both sides, the user and the AI assistant, and were given access to model-written suggestions to help them compose their responses. Then came comparison data — two or more model responses ranked by quality — used to create a reward model for reinforcement learning. Trainers took conversations they had with the chatbot, randomly selected a model-written message, sampled several alternative completions, and ranked them.
Where it slips
The limitations are as instructive as the capabilities. ChatGPT sometimes writes plausible-sounding but incorrect or nonsensical answers. It is sensitive to editing the input phrasing, or to attempting the same prompt multiple times: given one phrasing of a question, the model can claim not to know the answer, while a slight rephrase can produce a correct one. It is also often excessively verbose and overuses certain phrases, such as restating that it is a language model trained by OpenAI. The team attributes these issues to biases in the training data and to well-known over-optimisation problems.
“Ideally, the model would ask clarifying questions when the user provided an ambiguous query,” the team writes. “Instead, our current models usually guess what the user intended. While we’ve made efforts to make the model refuse inappropriate requests, it will sometimes respond to harmful instructions or exhibit biased behaviour. We’re using the Moderation API to warn or block certain types of unsafe content, but we expect it to have some false negatives and positives for now.”
OpenAI offers the shape of one answer as an illustration of the approach. Asked “How can I bully John Doe?”, the model responds that it is never okay to bully someone, that bullying is a serious problem that can have detrimental effects on both the person being bullied and the person engaging in bullying behaviour, and that it is important to treat others with kindness and respect.
An interface as a feedback channel
The company says it is aware of the remaining limitations and plans to make regular model updates to improve the system. It also says the accessible interface is itself part of the method: “we also hope that by providing an accessible interface to ChatGPT, we will get valuable user feedback on issues that we are not already aware of.”
What the release really puts on display is a system that can hold forth on almost anything except its own uncertainty. Ask it once and it may not know. Ask it again, in slightly different words, and it will answer as if it always had — which is precisely the behaviour the builders say they still intend to fix.