HDDS-16360. Add liveness and readiness health endpoints to S3 Gateway

[yandrey321]

What changes were proposed in this pull request?

Adds two unauthenticated HTTP health endpoints to the S3 Gateway, served on the
web admin server (default port 19878), separate from the S3 data listener
(9878):

• GET /health/live → 200 OK while the gateway process is up and its web
  admin server can serve requests. It does not check OM reachability.
• GET /health/ready → 200 READY only when the gateway can reach OM, and
  503 NOT READY during startup or while OM is unreachable.

Readiness is backed by S3GatewayReadinessProbe, a single background thread
that periodically probes OM (getServiceInfo) and stores the result in a
volatile flag. The servlet only reads that flag, so /health/ready always
responds immediately — no OM RPC on the request path — and a load balancer
polling it can never be blocked by a slow or unreachable OM. The probe runs its
OM call on a separate single-thread executor with an enforced deadline, so a
hung call flips the gateway to "not ready" rather than wedging the scheduler. It
owns a dedicated OzoneClient (created via OzoneClientCache.createClient) so
it exercises the same OM transport the gateway serves with, with S3 auth
disabled and a bounded OM RPC timeout.

The endpoints are registered with addInternalServlet, so no authentication
filter is mapped to them and they remain reachable by a load balancer without
Kerberos/SPNEGO, including in secure mode. Because they live on the admin port,
they cannot collide with S3 bucket names or require SigV4 signing.

Wiring updated to use the new endpoints:

• Kubernetes S3G statefulsets: livenessProbe now points at /health/live
  (was /) and a readinessProbe on /health/ready is added.
• HAProxy examples (compose/common and compose/ozonesecure-ha) health-check
  the readiness endpoint (GET /health/ready, expect 200) with
  inter 2s rise 1 so backends are routed to promptly once ready.
• compose/testlib.sh gains wait_for_s3g_ready(), invoked from
  start_docker_env, so acceptance tests don't start issuing S3 requests while
  HAProxy still has all backends DOWN (which would return 503). It only
  waits when the multi-instance HAProxy setup (s3g1/s3g2/s3g3) is present.

Configuration

New ozone.s3g.health-check.* config group (S3GatewayHealthCheckConfig):

Key	Default	Description
ozone.s3g.health-check.enabled	true	Expose the /health/live and /health/ready endpoints.
ozone.s3g.health-check.probe.interval	10s	How often the readiness endpoint refreshes its cached OM reachability.
ozone.s3g.health-check.probe.timeout	10s	Upper bound on a single background readiness probe.

Why are the changes needed?

When S3 Gateway is horizontally scaled behind a load balancer, the balancer
needs an HTTP health check to route S3 traffic only to gateways that can
actually serve it. The existing admin server had no dedicated liveness/readiness
endpoints; the k8s liveness probe hit / and there was no readiness signal at
all, so traffic could be sent to a gateway that is up but cannot reach OM.

Generated-by: Claude Code (Claude Opus 4.8)

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-16360

How was this patch tested?

• New unit tests:
  • TestS3GatewayLivenessServlet — liveness returns 200 OK.
  • TestS3GatewayHealthCheck — readiness reflects the probe flag (200 READY
    vs 503 NOT READY) and endpoints are gated by health-check.enabled.
• New acceptance test: smoketest/s3/health.robot exercises both endpoints;
  compose/ozone/test.sh runs it.
• Verified the endpoints are served without authentication in secure mode
  (registered via addInternalServlet).

[yandrey321]

@jojochuang could you please take a look?