Post

Hardening Enterprise GitOps: Zero-Trust Private Repository Access for Argo CD on EKS

Storing long-lived SSH keys or Personal Access Tokens (PATs) inside Kubernetes secrets is an operational hazard. If an attacker compromises a namespace or dumps the etcd database, those static credentials grant direct, unrestricted access to your internal…

Hardening Enterprise GitOps: Zero-Trust Private Repository Access for Argo CD on EKS

Storing long-lived SSH keys or Personal Access Tokens (PATs) inside Kubernetes secrets is an operational hazard. If an attacker compromises a namespace or dumps the etcd database, those static credentials grant direct, unrestricted access to your internal source code. In an enterprise GitOps pipeline, persistent cluster-side credentials represent a massive attack surface that platform teams should actively dismantle.

The core problem is architectural: your GitOps engine needs to pull manifests from private Git servers, but your Kubernetes cluster shouldn’t have to store static, high-privilege credentials to do so. The Amazon EKS capability for Argo CD addresses this by removing repository credentials from cluster memory entirely. By combining AWS CodeConnections VPC integration with IAM-based ephemeral authentication, you can establish secure, private synchronization paths without storing a single SSH key in a Kubernetes Secret.

Routing Git Traffic via VPC Proxies and Ephemeral Auth

If you run the managed Argo CD capability on Amazon EKS, you will quickly find that the managed control plane cannot directly access resources inside your private Virtual Private Cloud (VPC). It cannot natively query a self-managed GitLab instance or GitHub Enterprise Server tucked behind private subnets or VPN gateways.

To bridge this gap without exposing your Git server to the public internet, you can deploy an AWS CodeConnections host inside your VPC. This host provisions cross-account Elastic Network Interfaces (ENIs) in your subnets, routing outbound HTTPS traffic on port 443 to your Git server. Currently, this setup supports self-managed GitLab and GitHub Enterprise Server for private repositories.

Instead of saving long-lived PATs or SSH keys in Kubernetes, the AWS Connector app requests temporary credentials for each individual interaction with the Git server. Argo CD uses its assigned IAM role to request access through this proxy, keeping static credentials completely out of the cluster’s etcd database.

Here is a complete, syntactically valid Argo CD Application manifest that references a private repository via an AWS CodeConnections HTTP proxy URL. Notice how the repository URL structure incorporates a 12-digit AWS account ID along with the specific connection ID:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
 name: payment-service-production
 namespace: argocd
spec:
 project: default
 source:
 repoURL: https://codestar-connections.us-west-2.amazonaws.com/git-http/123456789012/us-west-2/a1b2c3d4-5678-90ab-cdef-EXAMPLE11111/enterprise-org/payment-service.git
 targetRevision: HEAD
 path: config/environments/production
 destination:
 server: https://kubernetes.default.svc
 namespace: payment-system
 syncPolicy:
 automated:
 prune: true
 selfHeal: true

Validating the CodeConnections Integration in EKS

Applying a manifest is only the first step. You must verify that the authentication proxy and ENI routing actually function without silently falling back to cached credentials. Here is how I test and verify this integration:

  1. Apply the Application Manifest: Deploy the manifest to your EKS cluster:
    1
    
     kubectl apply -f application.yaml
    
  2. Inspect Argo CD Application Health: Query the status of the application via kubectl to ensure it syncs successfully:
    1
    
     kubectl get application payment-service-production -n argocd -o jsonpath='{.status.sync.status}'
    

    A successful connection returns Synced and Healthy without requiring any repository secret objects in the argocd namespace.

  3. Verify Connection Status in AWS: Confirm that the CodeConnections host and connection states remain Available using the AWS CLI:
    1
    
     aws codeconnections list-connections --host-arn-filter arn:aws:codeconnections:us-west-2:123456789012:host/h-1234567890abcdef0
    
  4. Audit Outbound Security Group Egress: Verify that ENI traffic is actively traversing port 443 by inspecting VPC Flow Logs for the cross-account ENIs assigned to your private subnets. You should observe accepted outbound HTTPS flows directed to your Git server’s internal IP address.

Common Failure Modes to Watch For:

  • Security Group Blocks: If the application status hangs in Unknown or fails to pull manifests, check your VPC security group rules. A common mistake is forgetting to allow outbound HTTPS (port 443) on the security group attached to the CodeConnections ENIs.
  • DNS Resolution Failures: If your private Git server uses a custom domain name, ensure that DNS hostnames and DNS resolution are enabled on your VPC, and that your route tables can reach your internal DNS servers.
  • IAM Permission Gaps: Ensure that the IAM execution role assigned to the Argo CD capability has explicit permissions to use the connection (codeconnections:UseConnection).

Comparing AWS Ephemeral Proxying with External HSM Key Control

This strategy of moving credentials out of cluster memory is part of a broader shift toward strict boundary isolation. If we look at how other cloud providers handle high-stakes security, we see a similar philosophy. For instance, Azure’s Managed HSM service generates and stores keys in a single-tenant, FIPS 140-3 Level 3 HSM. For organizations with extreme compliance mandates, Azure’s external key management allows you to keep those keys entirely outside Microsoft’s infrastructure, securing the connection between Azure and your external hardware using mutual TLS.

Comparing these two paradigms highlights how modern cloud platforms handle sensitive access:

Feature DimensionAWS CodeConnections (GitOps Focus)Azure External Key Management (Data/Crypto Focus)
Primary ObjectiveEliminate static Git PATs/SSH keys in cluster secretsKeep cryptographic root keys outside vendor infrastructure
Credential LifecycleEphemeral, short-lived tokens per Git fetchNon-exportable hardware keys maintained on-prem/external
Network SecurityCross-account ENIs inside customer VPC subnetsDedicated API endpoints over mutual TLS
Target WorkloadEKS Argo CD manifest synchronizationAzure Key Vault envelope encryption for data at rest

Both patterns address the same core vulnerability: cloud services holding static, high-privilege keys inside shared infrastructure. Whether you are delegating Git authentication to an ephemeral proxy or delegating cryptographic operations to an external FIPS 140-3 Level 3 HSM, the objective is identical: remove long-lived secrets from the application runtime layer.

Operational Realities and When to Skip This Architecture

I am a strong advocate for removing persistent secrets from cluster namespaces, but I would not blindly enforce AWS CodeConnections or external HSM patterns across every workload without considering the operational tax.

Delegating network routing to cross-account ENIs and external proxies introduces real operational trade-offs:

  1. Network Egress Complexity: Because traffic flows through ENIs inside your VPC, your platform team becomes responsible for subnets, Transit Gateway routing, and DNS host resolution across private zones. If your private GitLab DNS fails inside the VPC, Argo CD syncs drop immediately.
  2. Latency Bottlenecks: Ephemeral credential generation and cross-account network proxying add overhead to every Git fetch operation. For massive monorepos where Argo CD polls every few seconds, this overhead can degrade sync performance compared to local, direct SSH keys.
  3. External HSM Operational Risk: If you choose to extend your security architecture to external HSMs, remember that hardware availability, scaling, and mutual TLS proxy maintenance shift entirely to your ops team. If your on-premises HSM goes offline, every dependent cloud operation halts instantly.

Decision Criteria: When to Skip This Pattern If you are running small-scale, public, or non-sensitive internal Kubernetes workloads with standard public repositories, setting up CodeConnections hosts, cross-account ENIs, and customized IAM roles adds unnecessary complexity. Standard IAM Roles for Service Accounts (IRSA) with traditional secret management or basic public Git access is perfectly sufficient.

However, if you operate in regulated banking, healthcare, or public sector environments where storing static Git credentials in Kubernetes etcd violates compliance baselines, this architecture is not optional. Delegating authentication to VPC-level proxies and ephemeral credentials is the cleanest way to enforce true zero-trust GitOps at scale.

References

This post is licensed under CC BY 4.0 by the author.