On adopting Chainguard Images

Posted on June 28, 2024  •  5 minutes  • 986 words

In this post, I’ll discuss my experience transitioning to Chainguard Images and the rationale behind this decision.

The challanges

The conventional vulnerability management process often proves cumbersome and inefficient. Here’s a typical scenario from my past experience:

• Periodic scans: every xx months (usually once a quarter), security team will scan organization-wide and send vulnerability reports to all service owners.
• Manual update process: Service owners become bogged down with updating operating systems and application libraries.
• On-premise constraint: On-premises environments add complexity due to restricted permissions (e. g., backups, testing, rollback).
• Limited scalability: Traditional methods impede scalability. For instance, an organization I worked for once estimated an entire year for OS patching. It’s a true story.

Back in 2021, I aimed to solve this problem once and for all.

So basically, we have 2 kinds of workload: one on virtual machines (VM) and the other one is container workload (EKS, RedHat OCP, etc…). I decided to target containerized workloads for initial improvement due to their inherent manageability.

Evaluating Container Image Solutions

Distroless

I like GoogleContainerTools/distroless . It’s really good when it works for you but customization presents a significant hurdle ¹.

The beta-status rules_distroless might offer some future relief, but the underlying reliance on Bazel remains a barrier.

Another issue is that distroless is based on Debian. And while Debian offers a stable and well-tested foundation, its release cycle might not align perfectly with the need for rapid vulnerability patching. Debian adheres to a scheduled release cadence, which may not prioritize fixing specific CVEs (Common Vulnerabilities and Exposures) as quickly as we require. For scenarios demanding more immediate vulnerability mitigation, a rolling release operating system might be a more suitable choice.

Alpine-based

Another popular option is using Alpine OS. But the problem is software support & compatibility due to its use of musl libc instead of glibc.

For applications offering Alpine-based images (e.g., Redis), utilizing them is ideal. However, building applications on top of Alpine can be troublesome

• Language-specific challenges: Languages providing prebuilt binaries (e.g., Python with PyPI, Node.js with NPM) encounter issues when maintainers haven’t provided Alpine packages. This necessitates building from source, a time-consuming process requiring numerous build tools within the build image.
• Limited influence: Convincing maintainers to support Alpine by offering prebuilt packages might be feasible, but often developers must resort to building them themselves.

RapidFort

One good thing about RapidFort is that they do hardening for you (NIST 800-70 & FIPS certified, etc..).

The way they work is by instrumenting & profiling and remove stuff that are not used. I feel like I could replicate this setup myself by leveraging tool like Dockerslim.

RapidFort uses the traditional OS so you still have all the short-comings from depending on a vendor to fix it. You may relay it to a commercial vendor if that make you feel better but utlimately, they won’t fix everything but probably just high & critical CVE, leaving some noise in scanner still. If your goal is to get 0-CVE, I think it won’t work. In fact, what RapidFort advertises is just “Near Zero CVE”. When I wrote this post, their nginx image still have 1 critical and few high CVEs included. This is the ultimate reason we decided to keep looking.

Chainguard Images

Traditional container image solutions presented limitations, prompting me to explore Chainguard Images. Their key benefits include:

• Reduced attack surface: Chainguard Images minimize the attack surface by incorporating only essential components. This significantly lowers the number of potential vulnerabilities compared to industry alternatives.
• Streamlined patching: Daily patching ensures vulnerabilities are addressed promptly, eliminating the need to wait for upstream distributions.
• Enhanced security: Cryptographically signed images with SBOMs (Software Bill of Materials) and verifiable provenance bolster security posture.

A powerful blend

When you see this closely, Chainguard Images combines the strengths of Distroless and Alpine with additional benefits.

• Philosophy and expertise: Founded by Dan Lorenc and Matt Moore (creators of distroless), Chainguard Images inherits the distroless philosophy of minimizing attack surface. It represents their next iteration, prioritizing an improved developer experience.

• Better compatibility: Unlike Alpine’s musl libc, Chainguard Images leverage glibc (as in Distroless), ensuring broader software compatibility.

• Familiar tooling: Developers accustomed to Alpine’s tooling ecosystem (e.g., apk) will find a similar experience with Chainguard Images. This is likely due to Ariadne Conill, a core Alpine Linux maintainer, is the one bootstrapping Chainguard’s Wolfi OS and its key components: melange & apko .

• Simplified customization: Chainguard Images offer a significant advantage in customization. Unlike Bazel used by Distroless, Chainguard employs a more approachable YAML format for configuration, streamlining the process for developers & contributors.

The people

One of few things I love about Chainguard is its people. They are packed with talents.

• There were Ariadne Conill - a highly regarded, long time core contributor to Alpine Linux.
• There were Dan Lorenc & Matt Moore, the original creators of distroless.
• There were Ville Aikas - co-creator of Kubernetes.
• There were Jason Hall who cofounded GCP CloudBuild & Tekton, among lots of other things.
• And so many more like James Rawlings, Nghia Tran, Adrian Mouat, Josh Dolitsky, Carlos Panato, Dan Luhring, etc…..

And this give me confidence that they are poised for success.

In corporate environments, adopting new technologies often prioritizes long-term viability and a proven track record of success. This ensures a stable foundation for critical applications and minimizes the risk of disruption from short-lived solutions.

And I want to be a small part of its success by being an early contributor to Wolfi OS and advocate for them here in Vietnam (I gave a talk about Wolfi OS at FOSSASIA Summit earlier this year & another one at DevSecOps Leadership Forum Singapore 2024)

Last words

If you’re looking for a secure and efficient way to manage containerized workloads, Chainguard Images are definitely worth exploring. Their commitment to security and their impressive team inspire confidence in their ability to provide a long-term solution for the evolving container landscape.