Beyond correctness, APIs must remain fast, reliable, and observable under load. Load testing measures behavior under expected and peak load to confirm latency, throughput, and error rates stay within targets. Stress testing pushes beyond capacity to find breaking points and observe how the system fails—gracefully or catastrophically. Soak testing runs moderate load for hours or days to expose slow leaks: memory growth, file handle exhaustion, and connection pool saturation. Spike testing applies sudden short bursts to verify recovery, and breakpoint testing incrementally increases load until the system fails to determine actual maximum capacity rather than a theoretical one. Popular tools include k6, JMeter, Gatling, Locust, and wrk. When evaluating results, percentiles matter more than averages: p50 is the median response time, p95 represents the worst case for 95% of requests, and p99 for 99%—the long tail is where most user pain lives. Performance assertions should be realistic enough to catch regressions without failing simply because environments vary slightly across runs.
Reliability targets translate into SLOs. An SLO (Service Level Objective) is a quantified reliability target such as "99.9% of requests succeed with p99 latency under 300ms over a 30-day window." An SLA (Service Level Agreement) is the contractual, customer-facing commitment that often carries financial consequences if missed. The error budget is the derived allowance for failure (for example, 0.1% downtime per month equals about 43 minutes); once exhausted, the team prioritizes reliability work over new features. A flaky test, which passes and fails unpredictably, signals underlying timing or state problems; retries in tests can mask real instability rather than diagnosing it, so they should be used with care. Observability—the ability to inspect logs, metrics, traces, and responses well enough to debug failures quickly—is what ties performance and reliability to debuggability.
Distributed tracing follows a single request across services, correlating timed spans by a shared trace ID to debug latency and failures end-to-end. A span is a single unit of work, with a name, start and end times, attributes such as http.status_code, and a parent span ID, typically following the OpenTelemetry data model. OpenTelemetry (OTel) is an open standard and SDK set for instrumenting applications to emit traces, metrics, and logs in a vendor-neutral format, often exported via OTLP to backends like Jaeger, Tempo, or Honeycomb. Operations practices complement this picture: a health check endpoint (often /health or /healthz) reports liveness and readiness; liveness asks "is the process alive?" (restart if not) while readiness asks "should I receive traffic?" (remove from the load balancer if not, for example when warming up or dependencies are down). Graceful shutdown drains in-flight requests, refuses new ones, finishes database transactions, and releases resources on SIGTERM, enabling zero-downtime deploys. Release strategies further limit blast radius: blue-green deployment runs the new version alongside the old and switches traffic atomically with the old version kept warm for instant rollback, canary deployment rolls out to a small fraction of traffic first and gradually increases, and feature flagging gates new behavior behind configuration so both old and new paths can be tested and rolled out or rolled back without redeploy.