A hallucination is a fluent, plausible-sounding model statement that is factually incorrect, fabricated, or unsupported. Intrinsic hallucinations contradict the source prompt or context, while extrinsic hallucinations introduce plausible-but-unverifiable external facts such as fake citations or invented statistics. Common mitigations include grounding responses with retrieved documents, instructing the model to say "I don't know" when uncertain, requiring citations or quoted evidence for factual claims, and running a separate verifier to flag unsupported claims before showing them to users.
Prompt injection is an attack where adversarial input overrides or bypasses a model's system instructions, causing it to ignore original constraints, leak secrets, or perform unintended actions. Indirect injection hides malicious instructions in external content—web pages or documents—that the model later retrieves. Defenses include delimiter-based prompting that separates instructions from data, the "prompt sandwich" technique that places trusted instructions both before and after untrusted input, isolating untrusted input from instructions, requiring structured outputs, validating responses, applying allow-lists, and using a secondary classifier to filter suspicious inputs. Jailbreaking is related but distinct: it bypasses safety alignment via tactics like role-play framings, base64 encoding, or multi-turn escalation, and is typically defended with adversarial training, classifiers, and an explicit instruction hierarchy where system > developer > user > tool outputs.
Guardrails are safety mechanisms—input filters, output validators, policy prompts, or classifiers—that prevent the model from producing harmful, biased, or off-policy content. To avoid excessive refusal on safe queries, prompts should provide clear context, specify that the task is legitimate, and avoid trigger words that resemble policy-violating content when the request is benign. Privacy-preserving practices include PII redaction before sending data, opting out of training-data retention where available, and using local or isolated models for secrets. Rate limiting, token quotas, and anomaly detection protect availability and cost on public endpoints.
Evaluation closes the loop. Benchmarks require a held-out test set with ground-truth answers, quantitative metrics (accuracy, format compliance, BLEU, judge scores), and statistical comparison across variants. LLM-as-a-judge scales review by using a strong model to score outputs against rubrics, though it inherits biases such as position and verbosity that can be mitigated by blinding, multiple judges, and periodic human calibration. Other signals include groundedness for RAG, system-fingerprint comparisons across runs, red-teaming exercises that adversarially probe for unsafe behavior, and A/B tests of prompt variants on production traffic with clear guardrails and rollback paths. Overfitting in prompt design—tuning so tightly to specific examples that generality is lost—is caught by broad, diverse test sets and holdout data.