Fixing the Scale-to-Zero Trap: Health Check Polling vs. Multi-Tenant Runtime Isolation

I have spent more than twenty years building and operating production software systems. In my current role as a software and platform architect, I design and deliver distributed systems for a large technology company. Across distributed engineering teams, the promise of scale-to-zero infrastructure remains one of the most attractive patterns for optimizing compute efficiency. However, when platform teams try to build serverless environments or multi-tenant Kubernetes platforms, they frequently fall into a pair of subtle engineering traps: accidental wakeups caused by probe traffic, and inadequate multi-tenant container isolation.

Scale-to-zero is not merely an autoscaling challenge that can be solved by tweaking scaling metrics. Achieving true operational efficiency in multi-tenant platforms requires addressing distinct operational boundaries. Platform engineers must prevent synthetic traffic like health probes from continuously waking up idle workloads. Furthermore, when workloads run on shared nodes, platforms must enforce strict capability allowlists rather than leaky denylists to prevent cross-tenant privilege escalation.

The Synthetic Probe Problem: Why Health Checks Break Scale-to-Zero

When designing serverless or function-as-a-service platforms on Kubernetes, scaling an idle deployment down to zero replicas is only half the battle. The moment a pod scales down to zero, your service ingress or service mesh must route traffic to an intermediate proxy or resolver capable of holding incoming connections and signaling the autoscaler to instantiate a new pod.

However, modern infrastructure is saturated with synthetic traffic. Ingress controllers, external load balancers, and internal monitoring systems routinely poll services with HTTP GET probes against endpoints like /healthz or /livez. If your ingress proxy treats every incoming HTTP request as a legitimate user signal, the very first health check probe sent after scaling down will trigger a cold start and bring the pod back up immediately.

According to vendor claims by KubeElasti, health check requests from load balancers or monitoring tools can prevent Kubernetes services from remaining in a scaled-to-zero state. When a service scales to zero, its endpoint points to an intermediate resolver. If that resolver blindly treats a health check as an application request, the platform enters an endless loop of scaling down and immediately waking up.

To solve this specific failure mode, KubeElasti claims that its ProbeResponse feature allows services to remain at zero replicas by intercepting health checks at the resolver level. By evaluating incoming requests against defined probe rules at the resolver, the proxy can return synthetic health responses directly to load balancers without forwarding the request to the operator or waking up the backend deployment.

Suppressing health check wakeups at the proxy layer resolves the traffic amplification problem, but it uncovers a deeper platform security problem: what happens when tenant workloads wake up and execute on shared cluster infrastructure?

The Isolation Illusion: Why Capability Denylists Fail in Multi-Tenant Runtimes

Once an incoming user request triggers a scale-up, your platform instantiates tenant pods on shared Kubernetes worker nodes. Many platform teams attempt to secure multi-tenant execution environments by applying capability denylists—explicitly blocking specific Linux capabilities like CAP_SYS_ADMIN or CAP_NET_ADMIN while assuming the remaining defaults are safe.

This approach is fundamentally flawed. Using a denylist for Linux capabilities in multi-tenant environments is structurally incomplete because it cannot constrain capabilities granted by default by the OCI runtime. For example, DAC_OVERRIDE is in the OCI default cap set and reaches the container regardless of any check on added capabilities, partially mooting the denylist for its own entries.

The operational dangers of permissive Linux capabilities on shared hosts are severe. A tenant container with CAP_SYS_TIME can modify the node’s wall clock, impacting TLS validity, lease renewals, and scheduling for other workloads. The Linux real-time clock is not namespaced—time namespaces virtualize only MONOTONIC and BOOTTIME, never REALTIME—so a tenant container holding CAP_SYS_TIME could call clock_settime(CLOCK_REALTIME) and rewrite the shared node wall clock.

Recognizing these structural limitations, open-source serverless frameworks have had to overhaul their security models. Fission v1.25.0 replaced a capability denylist with an allowlist to improve tenant isolation. Specifically, Fission replaced the denylist with a Pod Security Admission restricted allowlist containing only NET_BIND_SERVICE at both enforcement layers.

Instead of trying to enumerate every dangerous capability to block, multi-tenant platforms must enforce explicit capability drop-all policies combined with strict allowlists.

Implementing Safe Scale-to-Zero and Multi-Tenant Isolation

To demonstrate both principles, the following Kubernetes manifest illustrates an application deployment paired with strict security context configurations. It configures explicit capability stripping by dropping all capabilities and allowlisting only NET_BIND_SERVICE, sets unprivileged execution boundaries, and structures health probes to work alongside proxy-level interception.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tenant-service
  namespace: tenant-workloads
  labels:
    app.kubernetes.io/name: tenant-service
    kubeelasti.io/scale-to-zero: "true"
spec:
  replicas: 0
  selector:
    matchLabels:
      app: tenant-service
  template:
    metadata:
      labels:
        app: tenant-service
    spec:
      automountServiceAccountToken: false
      securityContext:
        runAsNonRoot: true
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: app
          image: registry.example.com/tenants/user-workload
          imagePullPolicy: IfNotPresent
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop:
                - ALL
              add:
                - NET_BIND_SERVICE
          livenessProbe:
            httpGet:
              path: /healthz
              port: http
          readinessProbe:
            httpGet:
              path: /healthz
              port: http

How to Verify This Configuration

To verify that synthetic health probes are properly intercepted while runtime isolation remains intact, perform the following verification steps:

1. Verify Scale-to-Zero Probe Interception: Deploy the manifest and ensure the replica count remains zero. Send an HTTP request directly to your ingress resolver endpoint:

   curl -i -H "Host: tenant-service.example.com" http://${RESOLVER_IP}/healthz
   

   Confirm that the resolver returns a successful HTTP response directly without incrementing the deployment replica count (kubectl get deployment tenant-service -n tenant-workloads).

2. Verify Capability Restriction Enforcement: Trigger an actual application request to force the autoscaler to launch a pod instance:

   curl -i -H "Host: tenant-service.example.com" http://${RESOLVER_IP}/api/v1/resource
   

   Once the pod is running, inspect the active Linux capabilities inside the running container by querying /proc/self/status:

   kubectl exec -n tenant-workloads deployment/tenant-service -- cat /proc/self/status | grep Cap
   

   Verify that effective and permitted capability bitmasks reflect only CAP_NET_BIND_SERVICE, confirming that default OCI capabilities such as DAC_OVERRIDE or CAP_SYS_TIME are absent.

3. Verify Host Clock Protection: Attempt to execute a time adjustment system call inside the tenant container:

   kubectl exec -n tenant-workloads deployment/tenant-service -- date --set="next hour"
   

   Confirm that the command fails immediately with an Operation not permitted error.

Operational Tradeoffs: When NOT to Do This

While probe-aware proxying and absolute capability restriction are critical for multi-tenant serverless clusters, adopting these patterns introduces architectural tradeoffs that are not always necessary.

Do not implement probe-aware resolver proxies or strict capability allowlisting if:

• You operate single-tenant dedicated clusters: If all workloads running on the cluster belong to a single trusted team, cross-container isolation breaches are not a threat vector. Managing custom resolver proxies and strict Pod Security Admission policies will add unnecessary operational complexity.
• Your workloads rely on legacy, unmodifiable container images: Third-party applications frequently require running as root or depend on default Linux capabilities like DAC_OVERRIDE to modify configuration files at runtime. Forcing drop: ["ALL"] on such legacy workloads will break pod initialization.
• You run imperative batch workloads: For batch processing systems where jobs are created imperatively via Kubernetes Job objects rather than driven by incoming HTTP traffic, scale-to-zero ingress resolvers provide no value.

Platform Architectural Recommendations

Platform engineering requires balancing operational convenience with defense-in-depth security. I hold the CKA, CKAD, and CKS Kubernetes certifications, and I recommend adopting a clear set of practices for shared container platforms:

• Decouple Health Probes from Traffic Autoscaling: Use probe-aware proxies at your ingress layer to answer synthetic health checks locally when service instances are scaled to zero.
• Abandon Capability Denylists: Replace all capability denylists across your admission controllers and Helm charts with explicit drop: ["ALL"] directives, adding back only specific capabilities like NET_BIND_SERVICE when required.
• Enforce Unprivileged Execution Boundaries: Set runAsNonRoot: true, allowPrivilegeEscalation: false, and read-only root filesystems as cluster-wide defaults via policy engines.

By combining probe-aware traffic routing with rigorous, allowlist-based runtime container security, platform teams can deliver true scale-to-zero efficiency without exposing host nodes or tenant workloads to catastrophic cross-tenant security compromises.

References

• Your Kubernetes health checks are accidentally waking your services. Here’s the fix.
• CVE-2026-50570 - GitHub Advisory Database