Several API-level parameters shape how the model converts probabilities into text. Temperature scales the logits before sampling: zero makes the model greedy and deterministic, picking the highest-probability next token, while higher values flatten the distribution for more creative output. Top-p, or nucleus sampling, restricts the token pool to the smallest set whose cumulative probability exceeds p, dynamically limiting candidates regardless of temperature. The two are often combined for fine-grained control. Other knobs include stop sequences that halt generation when a specified string appears, logit bias that adds a constant to selected tokens' logits to encourage or suppress them, max_tokens that caps output length, and repetition, frequency, or presence penalties that discourage verbatim reuse.
These controls matter because reproducibility with LLMs is hard. Sampling randomness, version updates, and non-deterministic infrastructure can change outputs across runs, so fixing seed and version is necessary to reproduce a response exactly. The OpenAI API exposes a system fingerprint identifying the backend configuration; comparing fingerprints across responses helps detect silent behavior changes.
For complex tasks, iterative prompt refinement—drafting a prompt, testing it on diverse inputs, diagnosing failures, and editing in cycles—adds constraints, examples, or clearer instructions until quality, format, and edge cases behave as desired. Prompt chaining sequences multiple LLM calls where each call's output becomes the next call's input, breaking complex workflows into modular, testable steps with intermediate validation. Skeleton-of-thought prompting first outlines a skeleton answer and then expands each point in parallel, while chain-of-density iteratively rewrites a summary to be more information-dense while keeping length fixed.