Shift-Left Meets Shift-Node: Moving Container Supply Chain Verification to Runtime via NRI
We have spent the last half-decade obsessing over "shifting left." We scan our Git repositories, run container image scanners in CI/CD pipelines, sign our artifacts with Cosign, and deploy admission controllers like Kyverno or OPA Gatekeeper to block…
We have spent the last half-decade obsessing over “shifting left.” We scan our Git repositories, run container image scanners in CI/CD pipelines, sign our artifacts with Cosign, and deploy admission controllers like Kyverno or OPA Gatekeeper to block unsigned images at the Kubernetes API server.
As a platform architect who has spent more than twenty years building and operating production software systems, I have worked with engineering teams across Seattle, Montreal, London, Berlin and India, and have seen how platform decisions play out across distributed teams. I also hold CKA, CKAD, and CKS certifications and work as a Google Cloud Professional Cloud Security Engineer. This background has made me deeply skeptical of security silver bullets. The hard truth is that admission control at the API server level has a massive, glaring blind spot: it only validates the state of the world at the moment of admission.
Once the API server approves a Pod, it hands off execution to the kubelet, which in turn communicates with the container runtime via the Container Runtime Interface (CRI). If an attacker bypasses the API server—whether through direct kubelet API exploitation, local node access, or a container breakout vulnerability like Zip Slip—your admission webhooks are completely blind. To truly secure the container supply chain, we must verify what actually runs on the node, at the exact moment of execution.
Enter the Node Resource Interface (NRI)
To bridge this gap, we need container runtime security that operates synchronously at the node level. This is where the Node Resource Interface (NRI) comes in.
As a plugin interface, NRI is integrated into CRI-O starting with version 1.28, and it is built into containerd 1.7 and above, where it has been active by default since the release of version 2.0. Unlike traditional asynchronous runtime security tools that merely alert you after a malicious event has already occurred, NRI plugins enable synchronous container verification at the runtime level.
The mechanics are elegant but heavy. When an NRI plugin subscribes to the CreateContainer event, the runtime calls the plugin synchronously before the container starts. If the plugin returns an error, the container is rejected. This moves the policy enforcement point directly into the container runtime path, ensuring that no unverified image can slip through the cracks at the node level.
What the Hype Leaves Out (The Operator’s Tax)
This sounds like the ultimate security posture, but as operators, we must look past the CNCF blog posts and evaluate the actual operational cost. What does this architecture leave out, and what will it cost you at 3:00 AM?
First, let’s address the security boundaries. In the opinion of the CNCF, while runtime verification closes the gap between the API server and the container, it does not defend against a fully compromised node. If an attacker has already gained root access to the worker node, they can easily bypass or disable the NRI daemon, modify the containerd configuration, or run rogue containers directly via lower-level runtimes. NRI is not a defense-in-depth miracle; it is a gatekeeper. If the fence it stands on is knocked over, the gate is useless.
Second, there is the risk of runtime stability. When you insert synchronous hooks into the container lifecycle, any bug in your NRI plugin or the runtime’s NRI implementation can bring down your entire node. This is not theoretical. For example, containerd 2.2.6 includes fixes for NRI-related nil pointer dereferences in GetIPs during pod sandbox teardown or container exit. Imagine upgrading your container runtime only to have your pods fail to tear down because of a nil pointer dereference in the NRI hook. That is a real production outage risk that can leave your nodes in an unmanageable state.
Third, there is the developer friction. If you enforce strict, synchronous runtime verification, you will inevitably block legitimate workloads when external verification services are slow or down. If your NRI plugin has to call out to an external registry or KMS to verify a signature, and that service experiences latency, your pod startup times will skyrocket. If it times out, your pods fail to start.
Balancing Runtime Security with Developer Fatigue
To make runtime verification viable, we have to pair it with a pragmatic dependency management lifecycle. If we block containers at runtime due to newly discovered CVEs, we will break production. We need to resolve vulnerabilities before they hit the node, without drowning developers in alert fatigue.
This is where tools like Dependabot come in, but they must be configured carefully. GitHub designed Dependabot to issue security updates immediately upon the disclosure of a vulnerability with an available fix, operating completely outside your standard version-update groups and schedules. This ensures that critical security patches bypass your weekly or monthly dependency update schedules so you can rebuild and sign your images immediately.
By grouping non-security version updates to reduce noise, while allowing independent, immediate Dependabot security updates to flow through your CI/CD pipeline, you ensure that the images running on your nodes are constantly updated. This prevents your runtime NRI verification plugins from having to make hard, breaking decisions on the node when a new CVE is published.
Configuring and Verifying containerd for NRI
To implement NRI-based verification, you must first configure containerd to enable and recognize NRI plugins. Below is a complete, production-ready /etc/containerd/config.toml configuration for containerd 2.0+ that explicitly enables the NRI service and configures the plugin paths.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
# /etc/containerd/config.toml
version = 2
[plugins]
[plugins."io.containerd.grpc.v1.cri"]
disable_apparmor = false
restrict_oom_score_adj = false
[plugins."io.containerd.nri.v1.nri"]
disable = false
disable_connections = false
plugin_config_path = "/etc/nri/conf.d"
plugin_path = "/opt/nri/plugins"
socket_path = "/var/run/nri.sock"
To verify that containerd has successfully enabled NRI and is ready to process synchronous runtime verification hooks, follow these steps:
- Restart containerd: Apply the configuration changes by restarting the service:
1
sudo systemctl restart containerd - Verify the Socket: Ensure that containerd has successfully created the NRI socket at the specified path:
1
ls -la /var/run/nri.sock
- Check the Logs: Inspect the systemd journal to confirm that the NRI plugin has initialized without errors:
1
journalctl -u containerd | grep -i nri
You should see log lines indicating that the NRI controller has started and is listening on the socket.
- Deploy a Dummy Plugin: Place an executable NRI plugin in
/opt/nri/plugins/. When containerd starts or when a new container is created, you should see containerd log messages indicating that it has registered the plugin and is invoking its connection or container creation hooks.
Decision Criteria: When to Adopt (and When to Avoid) NRI Verification
Before rolling out NRI-based runtime verification across your clusters, evaluate your platform against these explicit decision criteria:
When to Use This Approach
- High-Security Compliance Environments: If you operate in highly regulated sectors (e.g., finance, healthcare, defense) where you must guarantee that only cryptographically signed, fully attested container images run on your nodes, regardless of how they were scheduled.
- Self-Managed or Bare-Metal Clusters: If you manage your own control planes and worker nodes (on-premises or on IaaS VMs) and have full administrative access to the underlying container runtimes.
- Dedicated Platform Engineering Support: If you have a dedicated platform or site reliability engineering team capable of debugging low-level container runtime issues, managing custom NRI plugins, and handling runtime upgrades.
When to Avoid This Approach
- Managed Serverless Kubernetes: If you run workloads on AWS Fargate or GKE Autopilot, you do not have access to the underlying node’s filesystem or the containerd configuration file. Trying to force-fit node-level runtime verification in these environments is a non-starter.
- Small Platform Teams: If your engineering organization does not have dedicated platform engineers to maintain, update, and debug container runtimes, stay away. A single nil pointer dereference in your runtime or NRI plugin—similar to the bugs resolved in containerd 2.2.6 —can take down your entire cluster’s scheduling capabilities. The operational overhead of maintaining custom NRI plugins outweighs the security benefits for most small-to-medium-sized teams.
- Environments with High Autoscaling Volatility: If your workloads scale up and down rapidly in response to real-time traffic spikes, adding synchronous blocking hooks to the container creation path introduces latency and a new point of failure. If your NRI plugin takes even 200 milliseconds to verify a signature, that latency compounds during massive scale-up events.
Moving supply chain verification from the distant API server down to the container runtime via NRI is a powerful step toward securing the node. However, we must be honest about the stability and performance costs. As containerd 2.0+ becomes the standard across enterprise clusters, we will likely see more turnkey, vendor-supported NRI plugins that reduce the risk of custom implementation failures.
But the open question remains: will the added resilience of runtime-level blocking be overshadowed by the inevitable operational outages caused by synchronous verification failures? Only time, and a few more patch releases, will tell.