Health Checks & Failover
Most important commands to remember
curl --max-time 2— bound each health or service request.curl --write-out— compare HTTP status and elapsed time for each check.
Commands and flags
| Command or option | Meaning |
|---|---|
--silent --show-error |
Hide progress while showing transport failures. |
--max-time 2 |
Limit each transfer to two seconds. |
--output /dev/null |
Discard the response body. |
--write-out '…' |
Print the labeled result after each request. |
%{http_code} / %{time_total} |
Report HTTP status / elapsed seconds. |
/live, /ready, / |
The fixture’s liveness, readiness, and proxied application paths. |
These path names are a lab contract, not universal automatic endpoints. Without fail-on-HTTP-error mode, read the printed HTTP status as well as curl’s transport diagnostics.
The concepts that matter
1. A health check asks one defined question
A health check tests a particular condition from a particular observer. A TCP connection can show that a listener accepts connections; an HTTP check can test a route and expected response.
Neither necessarily proves that a useful business operation will succeed. Define the question first: is the process making progress, can it accept traffic, or can a complete user journey finish? One endpoint cannot safely stand in for every level of health.
2. Liveness and readiness should trigger different actions
Liveness asks whether the process should keep running or needs recovery such as a restart. Readiness asks whether it should receive normal traffic now. Startup checks can give initialization its own allowance.
An application can be alive but temporarily unable to serve. Restarting every instance because a shared database is unavailable can amplify an outage. The check’s dependency depth and the controller’s reaction must match the failure you intend to repair.
3. Detection needs time and stable thresholds
Periodic checks do not notice failure instantly. Interval, timeout, and required consecutive failures determine detection delay. Success thresholds can prevent an unstable backend from repeatedly entering and leaving service.
There is a tradeoff between quick reaction and false alarms. A slow check can also consume resources while the system is already stressed. Active probes and passive observations of real request failures provide different evidence and may be combined.
4. Failover needs a viable destination
Removing an unhealthy backend from selection helps only if another backend can handle the traffic. Remaining capacity, shared dependencies, session state, and data consistency all affect the outcome.
Existing connections and in-flight requests may still fail after routing changes. A retry can also duplicate an uncertain operation. Failover is therefore a recovery workflow, not a promise that users never see errors. It needs controlled testing of both detection and the application’s behavior after the switch.
One small example
Optional: compare the existing backend checks with the proxied application request. These commands only observe the fixture; they do not disable a backend, restart a process, or change proxy policy.
curl --silent --show-error --max-time 2 --output /dev/null --write-out 'live: http=%{http_code} total=%{time_total}s\n' http://localhost:8081/live
curl --silent --show-error --max-time 2 --output /dev/null --write-out 'ready: http=%{http_code} total=%{time_total}s\n' http://localhost:8081/ready
curl --silent --show-error --max-time 2 --output /dev/null --write-out 'service: http=%{http_code} total=%{time_total}s\n' http://localhost:8080/Read each HTTP status and duration according to the fixture’s endpoint contract. A live success with readiness failure can be valid during a temporary inability to serve. A successful proxied request might use a different backend from the one checked directly.
A printed HTTP code of 000 commonly means no HTTP response was received; inspect curl’s error rather than interpreting it as an application status. This snapshot does not demonstrate automatic removal, recovery thresholds, or uninterrupted failover. Those require a separate controlled failure test against the configured proxy. No cleanup is needed.
Keep this idea: Health checks drive specific recovery decisions. Failover works only when detection, routing, remaining capacity, and application semantics agree.