Designing High-Availability Container Platforms: ARC Zonal Shifts and Topology-Aware Routing
The ARC Zonal Shift architecture orchestrates infrastructure evacuation and traffic routing to bypass reactive health check failures. Achieving high availability in Kubernetes requires moving beyond standard reactive autoscaling. Traditional setups rely on…
At a Glance
The ARC Zonal Shift architecture orchestrates infrastructure evacuation and traffic routing to bypass reactive health check failures.
flowchart TD
ARC[ARC Zonal Shift]:::core -->|Triggers| Cordon[Cordon Nodes]:::core
ARC -->|Updates| LB[Load Balancer]:::external
LB -->|Routes Traffic| HealthyAZ[Healthy AZ]:::core
LB -.->|Blocks Traffic| ImpairedAZ[Impaired AZ]:::core
Karpenter[Karpenter Controller]:::core -->|Manages| Capacity[Cluster Capacity]:::core
HealthyAZ -->|Topology Aware| Service[Service Connect]:::core
Service -->|Optimizes| Latency[Network Latency]:::core
classDef external fill:#dbeafe,stroke:#2563eb,stroke-width:1px,color:#1e3a8a
classDef core fill:#dcfce7,stroke:#16a34a,stroke-width:1px,color:#14532d
The Fragility of Reactive Infrastructure
Achieving high availability in Kubernetes requires moving beyond standard reactive autoscaling. Traditional setups rely on Pod Disruption Budgets (PDBs) and horizontal pod autoscalers that wait for health checks to fail before taking action. During an Availability Zone (AZ) impairment, this approach often triggers cluster thrashing. By the time a controller detects a failure and attempts to reschedule pods, the cluster is already struggling to provision new capacity in healthy zones while managing failing pods in the degraded one. This feedback loop amplifies the outage rather than mitigating it.
To break this cycle, shift the authority for failover from application-level health checks to the cloud control plane. By integrating managed zonal shifts with topology-aware routing, you can evacuate traffic and block new capacity before the application even realizes the underlying infrastructure is struggling.
Automating Zonal Evacuation with ARC
The Application Recovery Controller (ARC) Zonal Shift treats an entire AZ as a single unit of failure. Enabling this integration in EKS allows you to cordon and taint nodes in an impaired zone programmatically. Unlike manual intervention, which is prone to human error during high-stress incidents, this automation ensures that the cluster state remains consistent across the remaining healthy zones.
When a shift is triggered, the integration performs three critical actions: it cordons worker nodes to prevent new scheduling, deregisters pod IPs from load balancers, and pauses voluntary disruptions like consolidation or drift in the affected zone. Crucially, this process does not force-evict existing pods. This design choice prevents a “thundering herd” of terminations, allowing existing processes to finish gracefully while ensuring no new traffic enters the degraded environment.
To implement this with Karpenter, ensure your controller is updated to version 1.12.0 or higher and explicitly enable the zonal shift flag.
1
2
3
4
5
6
7
8
9
# Example: Enabling Zonal Shift in Karpenter Controller
spec:
template:
spec:
containers:
- name: controller
args:
- --enableZonalShift=true
- --controller-name=karpenter
Optimizing Performance with Topology-Aware Routing
While ARC manages the “where” of your compute, topology-aware routing handles the “how” of your network traffic. In environments like ECS, using Service Connect allows you to keep traffic within the same AZ, which significantly improves performance. Keeping requests local to an AZ can reduce median network latency by approximately 24% in multi-AZ deployments.
The performance gains are measurable: intra-AZ communication latency typically stays between 300 and 400 μs, whereas cross-AZ calls frequently exceed 1.5 ms. However, you must be careful with your cluster density. Service Connect requires at least twice the number of endpoints as there are AZs to effectively activate this routing mode. If your endpoint count drops below this threshold, the system will automatically spill traffic over to other zones to maintain availability.
Decision Criteria and Operational Risks
Zonal shifting is a powerful tool for infrastructure-level failures, but it is not a substitute for application-level robustness.
When to use this approach:
- You operate multi-AZ workloads where cross-AZ latency is a performance bottleneck.
- You have strict Recovery Time Objectives (RTO) that cannot tolerate the latency of standard Kubernetes health check propagation.
- Your workloads are stateless or have robust, cross-zone data replication strategies.
When to avoid this approach:
- You have stateful services tied to zonal EBS volumes without automated cross-zone snapshot replication.
- Your cluster is too small to handle the sudden loss of capacity in one zone; if you only run in two zones, shifting one zone effectively cuts your capacity in half, which can lead to immediate cascading failures due to resource exhaustion.
Validating Resilience
A common failure mode is “resilience drift,” where a cluster is configured for zonal shifts but the underlying PDBs are too restrictive, preventing the cluster from ever actually evacuating the zone. You must validate your setup by running controlled experiments. Use the AWS Fault Injection Service (FIS) to trigger a simulated zonal impairment. Observe whether your nodes in the target zone receive the eks-arc-zonal-shift/impaired-zone taint and verify that your load balancer metrics show a drop in traffic to that zone. If the traffic does not drop, your service discovery or load balancer configuration is likely misaligned with the ARC control plane.
High availability is a continuous process of aligning your infrastructure’s behavior with the realities of cloud hardware. By moving away from reactive, health-check-driven failovers and toward proactive, control-plane-driven zonal management, you can build systems that remain stable even when the underlying infrastructure is not.