Authors: Travis Cotton (Los Alamos National Laboratory)
Abstract: Building images for HPC clusters tends to be a monolithic process, requiring complete rebuilds
when new packages or configurations are added to them or updating existing images. It is also
generally a manual process, heavily involving a system administrator and system-specific custom
tooling. Rebuilding images from scratch can be time consuming and updating existing images
can introduce unwanted/unexpected changes to production systems.
These problems can be mitigated by using existing container models, creating and layering
images to create the final “production”-ready results. This allows for rapid turnaround and a
guarantee that existing layers remain unchanged while safely updating others. We can then
leverage this layer-based image building to allow for a more automated process using
Continuous Integration/Continuous Delivery (CI/CD) pipelines. Leveraging standard tools for
configuration management and version control combined with the OCI standard of layer-based
image building and CI/CD pipelines, we can create an automated and even distributed image
building workflow while still being customizable for specific sites and systems.
LA-UR-23-20251
Long Description: Building images for HPC clusters tends to be a monolithic process, requiring complete rebuilds
when new packages or configurations are added to them or updating existing images. It is also
generally a manual process, heavily involving a system administrator and system-specific custom
tooling. Rebuilding images from scratch can be time consuming and updating existing images
can introduce unwanted/unexpected changes to production systems.
These problems can be mitigated by using existing container models, creating and layering
images to create the final “production”-ready results. This allows for rapid turnaround and a
guarantee that existing layers remain unchanged while safely updating others. We can then
leverage this layer-based image building to allow for a more automated process using
Continuous Integration/Continuous Delivery (CI/CD) pipelines. Leveraging standard tools for
configuration management and version control combined with the OCI standard of layer-based
image building and CI/CD pipelines, we can create an automated and even distributed image
building workflow while still being customizable for specific sites and systems.
LA-UR-23-20251
Paper: PDF