Tokens, context and generation settings
Tokens are units used to represent text; they do not always correspond to whole words. A context window limits the information processed in one interaction. More context does not guarantee equal attention to every detail or a better answer.
Settings such as temperature affect sampling when available. A low temperature does not guarantee truth or perfectly deterministic results. Record the model, version, request, sources and settings when comparing experiments. Consider latency, usage limits and total cost.
When analysing a long document, compare working section by section with sending the whole text. Check whether relevant points are found and connected correctly, including those in the middle.
More context and more generation can increase costs without solving a quality problem.
Try it yourself
Design ten evaluation cases for a document assistant: include an answer that is present, one that is absent, conflicting sources, an outdated document and a document the user is not authorised to read. Define how to distinguish each failure type.