GPT-4

A benchmark result and a finished job are different

The technical report evaluated GPT-4 on exams and other tests while documenting reliability limitations. Such results help describe a release, but the test conditions are part of the result. The task, examples, scoring and allowed tools all affect what can reasonably be concluded.

Consider a customer support workflow. Answering one written question correctly is only one step. The system also needs the correct policy, the relevant order and a way to identify when information is missing. A better exam score does not establish that these other pieces are present. Our practical interpretation is to evaluate the complete workflow before handing it responsibility.

GPT-4 technical report

An example with a clear definition of success

Imagine a fictional store whose policy permits returns within 14 days but excludes personalized items. Supply the policy and ask for draft replies to three customers: an ordinary return on day ten, a personalized item and a message without a purchase date. A useful answer handles the exception and asks for missing information instead of inventing it.

Score policy accuracy separately from tone. Then check whether the response promises an action the system cannot perform. “I have refunded you” is different from “You may be eligible for a refund.” This example is a way to inspect instruction following; it is not a reported GPT-4 evaluation.

What image understanding adds

For a chart or screenshot, break the work into observable steps. First ask for the title, labels and visible values. Then ask for an interpretation. If the extracted values are wrong, a sophisticated explanation does not repair the answer. This approach helps distinguish a reading error from a reasoning error.

Compare a screenshot task with the same information supplied as text. The difference can tell you whether the image itself introduces difficulty. Keep the model version fixed and avoid giving one system extra context. When reading the historical launch, also distinguish capabilities demonstrated by researchers from capabilities broadly available to users that day.