Models
RLHF
RLHF trains a model against human preferences between candidate outputs, rather than against a fixed correct answer.
Also written: reinforcement learning from human feedback
It is how models learn qualities with no single right answer: helpfulness, tone, refusing harmful requests. Human raters compare outputs, a reward model learns those preferences, and the model is optimised against it.
It is also the origin of behaviours people find irritating. Excessive hedging, unnecessary preamble and reflexive apology are learned because raters rewarded them.
In practice
RLHF is why models are agreeable, and agreeableness has a failure mode: a model that has learned humans prefer confident helpful answers will produce one when the honest answer is "the context does not say". Explicitly permitting "I do not know" in the prompt counteracts a preference that was trained in deliberately.
Common questions
Why are models so agreeable?
Because human raters preferred agreeable answers, and that preference was trained in. The failure mode is a confident answer where "the context does not say" was correct — which is why permitting uncertainty explicitly helps.
Does RLHF make models more accurate?
It makes them more useful and better aligned to instructions, which is not the same thing. It can also make them more confidently wrong, because confidence was part of what raters preferred.