Instruction 04-health-checks-dependencies-and-restart-policies: Reading

A process being present only proves that the operating system has not observed it exit. The process may be deadlocked, unable to reach its database, serving stale data, or still performing initialization. Reliable operations therefore separate startup, liveness, and readiness. A startup check allows a slow application time to initialize. A liveness check should test an internal condition that indicates whether restarting could help. A readiness check should test whether the instance can perform useful work now, including critical dependency access when appropriate. Readiness can change repeatedly during the lifetime of a process and should usually control traffic rather than process replacement.

Dependencies introduce another distinction: start order is not readiness. Starting a database before an application does not guarantee that the database has opened its sockets, completed recovery, applied migrations, or accepted credentials. A dependent application should use bounded connection attempts, explicit timeouts, and backoff. Orchestrator dependency conditions can improve initial sequencing, but applications still need runtime resilience because dependencies may fail after startup.

Restart policies are recovery tools, not substitutes for diagnosis. An on-failure policy is useful when a nonzero process exit may be transient. An always-style policy can be appropriate for appliance-like services, but it can also restart intentionally stopped workloads unless the supervisor distinguishes operator action. A no-restart policy is often best for one-shot jobs, migrations, and failures requiring human review. Every restart policy should be paired with logs, rate limits, and an alert for repeated failures. Immediate retries can overload storage, DNS, authentication services, or remote APIs. Backoff and retry ceilings reduce that risk.

Health checks should be narrowly scoped, inexpensive, deterministic, and protected by short timeouts. A check that performs a costly full transaction every few seconds can become its own denial-of-service source. Conversely, a check that only confirms an HTTP listener may report success even when the application cannot do useful work. Operators should document what each check proves, what it does not prove, how often it runs, and what automation consumes its result. The safest design connects each failure signal to a proportionate action: stop routing traffic for readiness failure, restart only for credible liveness failure, and escalate persistent or repeated failures to a human.