PEAD Introduction

HPE Fortran Support

HPE/Cray CPE Roadmap

Programming Environment Management BoF with ORNL, Pawsey and Stony Brook representative.

HPE Documentation and Training Updates

HPC resources can only be used effectively when they are accessible to their users. Therefore, as CUG member sites continue to invest in HPE-Cray systems and software, it is critical to offer training sessions, training resources, and documentation to educate their seasoned scientific users about developments in their HPC resources and to bring newer users up to speed on HPC topics more broadly. There is often significant commonality in existing and needed training materials across CUG member sites. To improve the efficacy of their efforts, many HPC specialists working on training and education have expressed interest in collaborating in the curation, sharing, creation, and improvement of HPC and HPE-Cray specific training materials. This collaborative effort will improve learning outcomes for users at each site, and improve the scientific throughput. Therefore, we propose holding a BoF session to facilitate this collaboration.

The BoF session will consist of* introductory lightning talks from the convening members and vendors. We will then break into small-group discussions based on topic areas to identify redundancy, gaps, and foci for collaboration. An out-brief at the end of the session will summarize the findings from each small-group discussion.

The expected outcome is the production of a concise report documenting the BoF's ideas, progress, and recommendations for tools or resources needed for future efforts. This collaborative initiative aims to position CUG members as leaders in the development and sharing of HPC training materials.

*A full detailed agenda is attached.

HPE User Engagement Survey

PEAD Open Discussions

When we consider the grand challenges addressed by supercomputing, we likely imagine large machines, like Lawrence Livermore National Laboratory's El Capitan or Oak Ridge National Laboratory's Frontier, and parallel applications that can leverage those machines. Yet, these two pillars of HPC, HPC hardware and HPC software, are not enough to ensure excellent application performance. When unaware of the topology of the underlying hardware, even well-designed software applications can fail to achieve full performance on top-notch systems. Affinity, how software maps to and leverages local hardware resources, forms a third pillar critical to HPC.

Multiple factors motivate an understanding of affinity for HPC users. On the software side, applications are increasingly memory-bandwidth limited making locality more important. On the hardware side, proper affinity policies are crucial for effective software-hardware assignments on systems of greater complexity. Today's computer architectures offer increasingly complex memory and compute topologies. For example, the Accelerated Processing Unit (APU) architecture of El Capitan, built by HPE and AMD, incorporates CPU chiplets, GPU chiplets, and unified memory on a single package.

This tutorial teaches principles behind identifying a good affinity policy, like understanding hardware topology and the importance of memory and GPU locality. Attendees will learn how to control and apply these policies to create effective, locality-aware mappings for MPI processes and GPU kernels and to ensure better computing performance. The tutorial showcases the architectures of recent and emerging systems including Frontier and El Capitan.

Creating an image combining the components needed for a HPE Cluster compute node on HPCM can be a daunting task. This tutorial will show a recipe is curated, documented, created and deployed for a GPU enabled compute node. The image needed to perform user tasks requires the base OS, HPE Cray Supercomputing Programming Environment Software, Slingshot host software, OS Updates, GPU drivers and a Workload manager.

The tutorial assumes some familiarity with HPCM but this is not essential.

The top entries of the TOP500 list feature systems enabled with AMD Instinct GPUs, including world and Europe’s fastest supercomputers Frontier and LUMI, respectively. As these systems are already in production, application teams require the ability to profile applications to ascertain performance. To enable this, AMD has released in 2022 two new profiling tools: Omnitrace and Omniperf. These tools are a result of close collaborations between AMD development teams and computational scientists aimed at unpicking performance bottlenecks in applications and identifying improvement strategies. Omnitrace targets end-to-end application performance generating timelines that cover MPI, OpenMP, Kokkos, Python, etc. It enables the developer to identify relevant hardware counters to collect and generate information in performance-limiting kernels. Omniperf can then be used to seek further insight into these kernels through roofline analysis, memory chart analysis, and read-outs of many metrics including cache access, GPU utilization, and speed of light analysis. In this tutorial, we will present advanced features of these tools, with live demonstrations, and provide numerous hands-on examples for attendees to identify and mitigate bottlenecks in scientific and machine learning applications running on AMD GPUs. We will present the latest developments of the profiling tools among with examples and the relation to the hardware counters.

When we consider the grand challenges addressed by supercomputing, we likely imagine large machines, like Lawrence Livermore National Laboratory's El Capitan or Oak Ridge National Laboratory's Frontier, and parallel applications that can leverage those machines. Yet, these two pillars of HPC, HPC hardware and HPC software, are not enough to ensure excellent application performance. When unaware of the topology of the underlying hardware, even well-designed software applications can fail to achieve full performance on top-notch systems. Affinity, how software maps to and leverages local hardware resources, forms a third pillar critical to HPC.

Multiple factors motivate an understanding of affinity for HPC users. On the software side, applications are increasingly memory-bandwidth limited making locality more important. On the hardware side, proper affinity policies are crucial for effective software-hardware assignments on systems of greater complexity. Today's computer architectures offer increasingly complex memory and compute topologies. For example, the Accelerated Processing Unit (APU) architecture of El Capitan, built by HPE and AMD, incorporates CPU chiplets, GPU chiplets, and unified memory on a single package.

This tutorial teaches principles behind identifying a good affinity policy, like understanding hardware topology and the importance of memory and GPU locality. Attendees will learn how to control and apply these policies to create effective, locality-aware mappings for MPI processes and GPU kernels and to ensure better computing performance. The tutorial showcases the architectures of recent and emerging systems including Frontier and El Capitan.

Creating an image combining the components needed for a HPE Cluster compute node on HPCM can be a daunting task. This tutorial will show a recipe is curated, documented, created and deployed for a GPU enabled compute node. The image needed to perform user tasks requires the base OS, HPE Cray Supercomputing Programming Environment Software, Slingshot host software, OS Updates, GPU drivers and a Workload manager.

The tutorial assumes some familiarity with HPCM but this is not essential.

The top entries of the TOP500 list feature systems enabled with AMD Instinct GPUs, including world and Europe’s fastest supercomputers Frontier and LUMI, respectively. As these systems are already in production, application teams require the ability to profile applications to ascertain performance. To enable this, AMD has released in 2022 two new profiling tools: Omnitrace and Omniperf. These tools are a result of close collaborations between AMD development teams and computational scientists aimed at unpicking performance bottlenecks in applications and identifying improvement strategies. Omnitrace targets end-to-end application performance generating timelines that cover MPI, OpenMP, Kokkos, Python, etc. It enables the developer to identify relevant hardware counters to collect and generate information in performance-limiting kernels. Omniperf can then be used to seek further insight into these kernels through roofline analysis, memory chart analysis, and read-outs of many metrics including cache access, GPU utilization, and speed of light analysis. In this tutorial, we will present advanced features of these tools, with live demonstrations, and provide numerous hands-on examples for attendees to identify and mitigate bottlenecks in scientific and machine learning applications running on AMD GPUs. We will present the latest developments of the profiling tools among with examples and the relation to the hardware counters.

Once all the software has been installed, there are many tools available on a CSM-based HPE Cray EX system to monitor, test, and alert for proper health and operation of the system, tune the system software for a specific workload or a diverse job mixture, and troubleshoot any problems which arise. This tutorial will provide exposure to these tools and describe how to use them to tackle problems such as managed node boot failures, detecting and reacting to service unhealthiness related to storage or memory issues on management nodes, performance variability during job execution, finding nodes which underperform for CPU or network speed compared to other nodes, detecting faulty hardware on its way toward failure and after it fails, and other problems. There are some housekeeping activities which should be utilized to ensure continued system health. Don’t forget to run the tools to check and monitor system health. Differences in the toolset for the software stacks released with CSM 1.3, 1.4, and 1.5 systems will be highlighted while discussing these topics.

The tutorial assumes some familiarity with the CSM-based system infrastructure components (Kubernetes, Ceph, Etcd, authentication, authorization, etc.), networks, and management services.

Codee is a suite of software development tools to help improve the performance of C/C++/Fortran applications, providing a systematic, more predictable approach that leverages parallel programming best practices. Codee Static Code Analyzer provides a systematic predictable approach to enforce C/C++/Fortran performance optimization best practices for the target environment: hardware, compiler and operating system. It provides innovative Coding Assistant capabilities to enable semi-automatic source code rewriting, by inserting OpenMP or OpenACC directives in your codes to run on CPUs or offload to accelerator devices such as GPUs, so that novice programmers can write codes at the expert level. Codee provides integrations with IDEs and CI/CD frameworks to make it possible to Shift Left Performance. In this tutorial the participants will be introduced to Codee and to the first Open Catalog of Best Practices for Performance, using short demos and hands-on exercises with step-by-step guides for HPE/Cray systems like Perlmutter. The participants will learn to use the Codee tools, starting with simple well-known kernels and quickly jumping to large HPC codes like WRF.

This full-day tutorial provides a comprehensive introduction to critical components building a complex scientific workflow, encompassing storage I/O, data compression, in situ data processing, remote data access, and visualization. Through real-world applications and living demonstrations, attendances will learn how these tools enhance storage and I/O, enable the creation of streaming and file-coupled workflows with ease. The tutorial will also showcase utilizing these tools for remote data access over wide area network and conducting local analysis. The day will conclude with an integrated workflow of simulation, data compression, analysis, and visualization, showcasing configurations for file-based or in-situ analysis workflows.

The tutorial delves into four primary tools from the US Department of Energy Exascale Computing Project: 1) ADIOS, a publish/subscribe I/O abstraction, unifying storage and staging I/O, 2) MGARD, an error-controlled compression and refactor software for floating-point scientific data, 3) Fides, a visualization data model and schema facilitating easy ingestion into various visualization tools, and 4) ParaView, a scientific visualization framework supporting file-based post-processing and in situ visualization. Moreover, the tutorial spotlights the latest developments in ADIOS, covering remote data access services, data refactoring, derived quantities, queries and campaign management. These developments streamline the process of publishing and accessing data remotely.

Everything the attendees learn in this tutorial are directly usable on Cray systems, like Frontier and Perlmutter, where ECP applications like XGC, WarpX, S3D, E3SM and others have been using these tools at large scale.

Once all the software has been installed, there are many tools available on a CSM-based HPE Cray EX system to monitor, test, and alert for proper health and operation of the system, tune the system software for a specific workload or a diverse job mixture, and troubleshoot any problems which arise. This tutorial will provide exposure to these tools and describe how to use them to tackle problems such as managed node boot failures, detecting and reacting to service unhealthiness related to storage or memory issues on management nodes, performance variability during job execution, finding nodes which underperform for CPU or network speed compared to other nodes, detecting faulty hardware on its way toward failure and after it fails, and other problems. There are some housekeeping activities which should be utilized to ensure continued system health. Don’t forget to run the tools to check and monitor system health. Differences in the toolset for the software stacks released with CSM 1.3, 1.4, and 1.5 systems will be highlighted while discussing these topics.

The tutorial assumes some familiarity with the CSM-based system infrastructure components (Kubernetes, Ceph, Etcd, authentication, authorization, etc.), networks, and management services.

Codee is a suite of software development tools to help improve the performance of C/C++/Fortran applications, providing a systematic, more predictable approach that leverages parallel programming best practices. Codee Static Code Analyzer provides a systematic predictable approach to enforce C/C++/Fortran performance optimization best practices for the target environment: hardware, compiler and operating system. It provides innovative Coding Assistant capabilities to enable semi-automatic source code rewriting, by inserting OpenMP or OpenACC directives in your codes to run on CPUs or offload to accelerator devices such as GPUs, so that novice programmers can write codes at the expert level. Codee provides integrations with IDEs and CI/CD frameworks to make it possible to Shift Left Performance. In this tutorial the participants will be introduced to Codee and to the first Open Catalog of Best Practices for Performance, using short demos and hands-on exercises with step-by-step guides for HPE/Cray systems like Perlmutter. The participants will learn to use the Codee tools, starting with simple well-known kernels and quickly jumping to large HPC codes like WRF.

This full-day tutorial provides a comprehensive introduction to critical components building a complex scientific workflow, encompassing storage I/O, data compression, in situ data processing, remote data access, and visualization. Through real-world applications and living demonstrations, attendances will learn how these tools enhance storage and I/O, enable the creation of streaming and file-coupled workflows with ease. The tutorial will also showcase utilizing these tools for remote data access over wide area network and conducting local analysis. The day will conclude with an integrated workflow of simulation, data compression, analysis, and visualization, showcasing configurations for file-based or in-situ analysis workflows.

The tutorial delves into four primary tools from the US Department of Energy Exascale Computing Project: 1) ADIOS, a publish/subscribe I/O abstraction, unifying storage and staging I/O, 2) MGARD, an error-controlled compression and refactor software for floating-point scientific data, 3) Fides, a visualization data model and schema facilitating easy ingestion into various visualization tools, and 4) ParaView, a scientific visualization framework supporting file-based post-processing and in situ visualization. Moreover, the tutorial spotlights the latest developments in ADIOS, covering remote data access services, data refactoring, derived quantities, queries and campaign management. These developments streamline the process of publishing and accessing data remotely.

Everything the attendees learn in this tutorial are directly usable on Cray systems, like Frontier and Perlmutter, where ECP applications like XGC, WarpX, S3D, E3SM and others have been using these tools at large scale.

"OpenCHAMI or OCHAMI (Open, Composable, Heterogeneous, Adaptable, Management Infrastructure) was founded in 2023 as a collaboration between CSCS, the University of Bristol, NERSC, LANL, and HPE to foster development of open source software that was originally written to manage Cray/HPE’s exascale-class systems. The OpenCHAMI BoF at CUG will include presentations and demonstrations that share project insights and discuss futures towards enhancing collaboration in the community."

Digital Twins are the basis of a rapidly growing field of study that pairs physical systems with their digital representation to enhance understanding of the physical counterpart. Digital twins of data-centres are rapidly becoming a reality, using bi-directional feedback loops to link operational telemetry with models of associated sub-systems, which are combined with visualisation and analytics components. Such digital twins promise improved system behaviour prediction, optimised system operation, and ultimately better-informed decision-making for operating large-scale compute centre installations with associated power, cooling, and management infrastructure.

The aim of this BoF is to bring together the community of scientific users, data-centre operators, and vendors with an interest in data-centre digital twins. We will showcase on-going technical work, discuss experiences, and potentially identify new use cases, whilst spurring collaboration for a new way to better understand the operation of data-centres. The BoF is planned for a 90 minute session, structured as three 20 minute slots for technical talks (15 minutes talk + 5 minutes questions) followed by a 30 minute group discussion. ORNL will discuss its motivation for developing a digital twin framework, highlighting challenges and opportunities. EPCC will discuss the potential of using digital twins to improve overall system and data centre efficiency. HPE will discuss EX system architectures and networking in the context of digital twins. Finally, we will engage in a group discussion with the audience with the goal to share perspectives, experiences, and invite interested participants to further contribute in an ongoing collaborative effort on data-centre digital twins.

This birds-of-a-feather session will provide an opportunity for users to ask questions and share advice on managing and using HPE Slingshot systems, as well as to hear and provide input into HPE's Slingshot software roadmap. HPE Slingshot software scope covers capabilities both for the administrators who operate and manage the system and the fabric, and for HPC/AI application writers and users of the HPE Slingshot NIC’s Libfabric provider. Users will be encouraged to share desired use cases, learnings, and best-known methods.

Welcome to CUG2024

A long tradition of the Noongar people across the southwest of WA is the "Welcome to Country" ceremony, which is meant to call out to the spiritual ancestors of the land, to advise them that new friends have come to visit, and to look over and protect them from harm during their stay. This ceremony varies from person to person, where some share stories, songs, and dances reflecting the song lines and connections of the land and cultural heritage of the place where the gathering occurs. This has been done since time immemorial as a valued and respected ceremony as part of our events.

Get ready for a special treat during CUG2024! We are honored to have the incredible Ingrid Cumming, a custodian and traditional owner of the Wadjuk people, the specific language group and peoples that are from this part of Noongar country, who will welcome us all to her ancestor's land.

Welcome from the CUG President.

Keynote talk by SKA-Low Telescope Director

NVIDIA Grace Hopper Superchips are a scale-up architecture ideal for scientific computing workflows involving CPUs and GPUs. This session dives deep into HPC and AI workload performance results with a technical focus on the specific features of Grace-Hopper that accelerate each workload. Explore how Grace-Hopper's distinctive coupling of the CPU/GPU hardware and the accompanying software stack create a platform which increases developer productivity, accelerates existing applications, and facilitates new standard programming models in C++, Fortran, and Python. Attendees will hear about some of the early customers working with these innovative products as they apply this innovative, energy-efficient platform towards their scientific, generative AI, and industrial use cases.

ThinLinc is a scalable remote desktop solution for Linux, built with open-source components. It enables access to graphical Linux applications and full desktop environments. CUG member sites have used ThinLinc to provide users with access to applications like MATLAB or VMD, as well as a “High Performance Research Desktop”. This environment allows users to run their entire workflow, from data retrieval and preparation to job submission and post-processing. A “High Performance Research Desktop” also enables access to interactive applications and running jobs in a batch system. Cendio, the company behind ThinLinc, is a strong supporter of open-source projects and the main contributor to projects like TigerVNC, noVNC, and others.

Dive into the future of code development and see how Linaro Forge is reshaping what's possible in the world of parallel computing. Linaro Forge unveils the latest advancements: with Linaro DDT, MAP, and Performance Reports, we're setting new standards in scalability and ease-of-use. Discover how these tools have become the go-to solution for developers seeking to push the boundaries of code optimization and performance engineering.

Welcome CUG site

HPE corporate update by Gerald Kleyn

The HPE Cray Programming Environment (CPE) provides a suite of integrated programming tools for application development on a diverse range of HPC systems delivered by HPE. Its compilers, math libraries, communications libraries, debuggers, and performance tools enable the creation, enhancement and optimization of application codes written using mainstream programming languages and the most widely used parallel programming models.

CPE has expanded significantly in recent years to facilitate the development of scalable applications on accelerated platforms with AMD or NVIDIA GPUs, including emerging integrated node architectures. OpenMP 5.0 is available for these platforms, and OpenACC support for Fortran application codes is rapidly evolving. Fortran 2023 is already being implemented in CPE’s proprietary compiler. HPE Cray MPI was instrumental in surpassing the exascale performance barrier on ORNL’s Frontier supercomputer; it is also being enhanced to prototype “GPU-NIC Async” strategies in close collaboration with GPU vendors.

To enable greater deployment flexibility, and accommodate an expanded range of user needs, CPE will be delivered in the form of standalone components, in addition to traditional delivery mechanisms, in future. This presentation will provide an update on CPE, its features, support for GPUs, componentization, and other recent or planned enhancements.

NVIDIA's HPC Software enables developers to build applications that take advantage of every aspect of the hardware available to them: CPU, GPU, and interconnect. In this presentation you will learn the latest information on NVIDIA's HPC compilers, libraries, and tools. You will learn how NVIDIA's HPC software makes application developers productive and their applications portable and performant. This presentation will give an overview of NVIDIA's HPC SDK, optimized libraries for the GPU and CPU, performance tools, scalable libraries, python support, and more.

Slurm is the open-source workload manager used on the majority of the TOP500 systems.

This presentation will highlight changes in the Slurm 24.05 release, with a focus on improvements for high-scale and high-throughput systems that are directly relevant to the HPE/Cray community. Further content will discuss the longer-term roadmap and direction. Feedback from the CUG audience in particular is invaluable for establishing priorities for future development work.

With the evolution of scientific computational needs, there is a growing demand for enhanced resource access and sophisticated services beyond traditional HPC offerings. These demands encompass a wide array of services and use cases, from interactive computing platforms like JupyterHub to the integration of Continuous Integration (CI) pipelines with tools such as GitHub Actions and GitLab runners, and the automation of complex workflows in Machine Learning using AirFlow.

This paper addresses the challenges faced by HPC centers providing multi-purpose HPC infrastructure in scaling these services to meet the diverse needs of various scientific communities. Our proposed solution involves the adoption of a web-facing RESTful API, such as FirecREST, to streamline access to HPC resources for those services. We methodically demonstrate, through various use cases, how FirecREST can significantly simplify the configuration of complex services and enhance the efficiency of HPC service management for HPC providers. This approach not only addresses the immediate needs of HPC centers but showcases the benefits of a web-facing interface to access HPC resources.

This paper introduces algorithms for automating the grouping of compute nodes into clusters based on user-defined hardware requirements and simultaneously identifies potential hardware failures in HPC data centers. Addressing the challenges of dynamic workloads, the algorithms extract detailed hardware information through CSM APIs, automating node selection aligned with user-defined criteria. The automation streamlines node assignment, reducing human error and expediting the selection process.

Introducing a scoring mechanism improves the nodes selection, allowing informed decisions based on other criteria like penalizing a node with hardware not requested by the user. Beyond automated node selection, the algorithms contribute to proactive hardware failure detection by clustering nodes and comparing expected and actual configurations. This aids administrators in prompt issue resolution, aligning with industry trends in predictive maintenance.

The system's ability to pinpoint failures enables targeted maintenance, supporting administrators in efficiently diagnosing and resolving problems. Additionally, the algorithms contribute to a comprehensive hardware inventory and monitoring system, offering insights for long-term planning based on usage patterns.

In summary, the proposed algorithms provide a holistic solution for automated node selection, representing an advancement in optimizing resource allocation and strengthening computing infrastructure across multiple tenants.

This presentation introduces the versatile software-defined cluster (vCluster), a novel set of technologies for HPC infrastructure such as Cray HPE EX systems. This integration offers a service-oriented approach to computing resources, maintaining infrastructure independence and avoiding vendor lock-in. The vCluster technology bridges the gap between Cloud abstraction and the vertically integrated HPC stack, enabling large-scale infrastructures to support multiple scientific domains with specifically tailored services.

The presentation discusses the challenges faced by HPC centers in delivering customized services and the limitations of traditional HPC systems in terms of flexibility, composability, and upgradability. It highlights how Cloud abstraction can provide tailored solutions on HPC infrastructure, with service management decoupled from the infrastructure provider.

The core concept of vCluster revolves around a three-tiered structure: Infrastructure, Service Management, and Software-as-a-Service. This structure ensures immutable and consistent service deployment for diverse scientific communities. The vCluster model is particularly cost-effective for shared digital research infrastructure with diverse usage, such as batch, interactive, urgent computing, or machine learning workflows.

Exascale high performance computing (HPC) systems introduce new challenges related to fault tolerance due to the large component counts needed to operate at such scales. For example, the exascale Frontier system consists of approximately 60 million components. These counts warrant the investigation of new approaches for helping to ensure the functionality, performance, and usability of such systems. An approach explored by the ExaDigiT project is use of digital twins to help inform decisions related to the physical Frontier system. This paper discusses a subset of ExaDigiT’s Facility Digital Twin (FDT), the Network Digital Twin (NDT), which focuses on Frontier’s network as a target use case. We present the various strategies tested and early challenges faced towards the development of an exascale NDT, with the hope that such knowledge would benefit other practitioners who are interested in developing a similar digital twin.

The landscape of High-Performance Computing (HPC) is evolving. Traditional HPC simulations are merging with advanced visualization and AI techniques for analysis, resulting in intricate workflows that push the boundaries of current benchmarks and performance models. Here we focus on workflows that couple in near real-time digital twins, low-fidelity Artificial Intelligence (AI) simulations, alongside ongoing experiments or high-fidelity simulations to continuously drive the latter towards optimal results. It is expected that digital twin workflows will play a crucial role in optimizing the performance of next-generation simulations and instruments. This paper highlights performance limitations for the convergence of AI digital twins and HPC simulations by modeling and analyzing several I/O strategies at scale on HPE/Cray machines. We expose the limitations of relying on existing methods that benchmark individual components for these novel workflows, and propose a performance roofline model to predict the performance of these workflows on future machines and for more complex tasks. Additional layers of analytics and visualization further complicates the performance landscape. Understanding the unique performance characteristics of these intricate HPC-AI hybrid workflows is essential for designing future architectures and algorithms that can fully harness their potential.

This paper presents an in-depth examination of checkpoint-restart mechanisms in High-Performance Computing (HPC). It focuses on the use of Distributed MultiThreaded Check- Pointing (DMTCP) in various computational settings, including both within and outside of containers. The study is grounded in real-world applications running on NERSC Perlmutter, a state-of-the-art supercomputing system. We discuss the advantages of checkpoint-restart in managing complex and lengthy computations in HPC, highlighting its efficiency and reliability in such environments. The role of DMTCP in enhancing these workflows, especially in multi-threaded and distributed applications, is thoroughly explored. Additionally, the paper delves into the use of HPC containers, such as Shifter and Podman-HPC, which aid in the management of computational tasks, ensuring uniform performance across different environments. The methods, results, and potential future directions of this research, including its application in various scientific domains, are also covered, showcasing the critical advancements made in computational methodologies through this study.

Cray EX nodes have always supported a form of power capping that would allow customers to lower power usage of specific nodes as desired. With the introduction of the HPE Cray EX254n (NVIDIA Grace Hopper) and HPE Cray EX255a (AMD MI300a), this became critical as the overall rack power pushed beyond the maximum supported at some customers sites. With the HPE Cray EX254n in particular, the total TDP of the modules exceeds the maximum that can be delivered by the Cray EX infrastructure. This drove the decision to have to set a power limit on the Grace Hopper modules by default (a first for Cray EX). This presentation will walkthrough design goals of the blades, how power capping is implemented in the firmware, and how to configure the power limit in a running system. The presentation will also go through how to view the current limits configured via the node controllers Redfish API and in-band tools, where applicable, and how the in-band tools interact with the out-of-band configurations.

HPE Cray Power Monitoring (PM) Counters were first deployed on Cray XC30 systems, and several papers were presented at CUG in 2014 that described their use. PM Counters expose power energy and related meta-data collected out of band, directly to in-band consumers. Since introduction, PM Counters have been supported on all blades designed for use in Cray XC, and HPE Cray EX Supercomputer systems. PM Counters have continued to be important as system and application power and energy consumption continues to be a top priority for system vendors, application developers, and the wider HPC research community. Over the last decade, the design of PM Counters has remained very stable, with only minor updates to support evolving node architecture changes. This presentation will give a brief history and overview of PM Counters basics, it will then present details of PM Counters on the latest HPE Cray EX supercomputer blades announced at SC23, and then discuses opportunities and challenges in supporting PM Counters on NGI (Next Generation Infrastructure). The presentation will conclude with a reinforcement of the value of PM Counters in supporting research, development, and testing of energy efficient and sustainable HPC systems.

With the focus on sustainable data center operations the community is moving from chilled water cooling to warm water cooling solutions expecting that running at higher system inlet temperatures enables a more energy efficient facility operation. Since the overall efficiency is determined by the combination of facility infrastructure and system behavior, understanding the impact of different system inlet cooling temperatures on the system performance and efficiency is important. This inaugural presentation covers the analysis of the impact of the inlet cooling temperature on an Exascale compute blade when running HPL and HPCG. Data was collected using the HPE-LRZ PreExascale co-design project system (HPE Cray EX2500 with four modified Frontier blades optimized for higher cooling temperature support) installed at the Leibniz Supercomputing Centre. Our analysis will show that higher cooling temperatures increase node power consumption leading to a reduction in node performance and overall energy efficiency. Results are presented for different inlet temperatures showing that the overall node efficiency is reduced by around 6% for HPCG and 4% for HPL (25C vs 40C inlet temperature). Combing facility and system, warm water cooling is more efficient but the most efficient cooling temperature depends on the application and efficiency of the cooling infrastructure.

Amid conflicting demands for better application performance and energy efficiency, HPC systems must be able to identify opportunities to save power/energy without compromising performance, while ideally being transparent to the user. We identify three primary challenges for a successful energy saving solution - versatility to operate across processors of different types (e.g., CPUs and GPUs) and from different vendors, effectiveness in finding energy saving opportunities and making the right power-performance tradeoffs, and ability to handle parallel applications involving communication. We propose Everest, a lightweight runtime tool that switches to the computed (based on dynamic application characterization) ideal clock frequency for individual application phases/regions while meeting a specified performance target. Everest achieves versatility by relying on the minimum possible performance events for the needed characterization and power estimation. Region-awareness and accurate computation of MPI slack time allow Everest to find enhanced energy saving opportunities and thus save up to 20% more energy than existing solutions on CPUs. These energy savings rise to up to 30% and are more prominent on GPUs where Everest doubly benefits from its unique idle time characterization and by choosing to sacrifice an allowed/acceptable performance loss.

The Swiss National Supercomputing Centre (CSCS) is enhancing its computational capabilities through the expansion of the Alps architecture, a Cray HPE EX system equipped with approximately 5000 GH200 modules, in addition to the pre-existing 1000 nodes of a diverse combination of CPUs and GPUs. CSCS has developed an Extensible Monitoring and Observability Infrastructure (EMOI), designed to manage the substantial data influx and provide insightful analysis of the infrastructure`s behavior. This paper presents the architecture and capabilities of EMOI at CSCS, emphasizing its scalability and adaptability to handle the increasing volume of monitoring data generated by the Alps infrastructure. We detail the integration of the Cray System Management (CSM) and Cray System Monitoring Application (SMA) within EMOI. The paper describes our hardware infrastructure, leveraging Kubernetes for dynamic data collection and analysis tools deployment, and outlines our GitOps strategy for efficient service management. We also explore the distinctions in data models across various node architectures within the Alps system, focusing on power consumption data and its relevance concerning global supercomputing challenges. The insights and methodologies presented in this paper are anticipated to be beneficial not only to CSCS, but also to other HPE/Cray sites facing similar challenges in supercomputing infrastructure management.

HPC storage systems today are complex solutions. Contrary to typical compute environments, a single storage component operating subpar can have significant impact on productivity. Further, as a storage solution ages, capacity fills up and is used un-evenly. Understanding these changes and the reasons for performance bottlenecks are increasingly important. With the addition of a full RESTful monitoring API based on Swordfish, we now have the tools required to improve overall storage monitoring. Swordfish is a collaboration between DMTF and SNIA that extends the DMTF Redfish interface to provide a standardized, accessible way to represent and manage storage and file systems in both individual customer and cloud environments. A custom implementation of both Swordfish and Redfish in the ClusterStor software stack provides new ways of gaining insights into the inner workings of an HPE ClusterStor E1000 storage system or either of its forthcoming descendants, C500 and E2000. This paper is intended as an introduction into this new API, including examples and guidance on how this can be used to improve storage monitoring as well as understanding of how traditional HPC and/or modern AI/ML workflows behave and evolve over time.

In the rapidly evolving landscape of High-Performance Computing (HPC), the efficient management and analysis of telemetry data is pivotal for ensuring system robustness and performance optimization. As HPC systems scale in complexity and capability, traditional data processing methodologies struggle to meet the demands of rapid real-time analytics and large-scale data management. This paper introduces an innovative framework, Caddy, which employs a novel approach to HPC telemetry storage and interactive analysis. Built on the foundation of HPE's Slingshot interconnect and the Fabric AIOps (FAIO) system, Caddy aims to address the critical need for a memory-efficient, scalable, and real-time analytical solution for seamless monitoring over large HPC environments.

Fabric AIOps’ reliance on voluminous telemetry data generated from Slingshot’s nodes and switches poses significant challenges for traditional disk-based storage solutions and their ability to efficiently parse, analyze, extract real-time insights, identify potential bottlenecks, and ensure seamless operation. Telemetry analytics, particularly in cases of aggregation over large sections of the fabric, involves high disk I/O, network transfers, and record processing. Their interactivity and consistency are further impeded by constantly streaming telemetry and multiple simultaneous analytical queries.

In-memory analytics can offer significant speed-up over disk-based approaches and enable significantly accelerated aggregation computations, but limited memory capacity at scale remains a critical hurdle for traditional caching. This paper explores Caddy, a novel in-memory storage and summarization framework, emphasizing its ability to generate and accurately apply dynamic updates to low-dimensional representations of telemetry data as well as provide quick data access during query evaluations. Caddy facilitates instant insights, minimizes memory footprint, and retains accuracy in analytical processes, even under intense HPC demands.

System benchmarks over Caddy demonstrate that a Caddy-enabled FAIO live mode reduces query latency of a single-frame by ~2x for a switch-level telemetry aggregation. Caddy also reduced the memory footprint of in-memory telemetry drastically - for instance, Bins with temporal sizes of 10, 15, and 30 mins, the compression factors achieved were ~425x, 600x, and 1200x, respectively,

In the context of high-performance computing, a significant portion of users do not develop their own code from scratch but rely on existing software packages and libraries tailored for specific scientific or computational tasks. Many of these open source scientific software packages provide a variety of methods to efficiently use them in multicore, multinode, or large-memory systems. In this paper, we examine a set of applications that users run on Indiana University supercomputers, and determine for those applications the software parameter settings controlling CPU parallelism, GPU parallelism, and memory usage. We then investigate the common ways users employ these parameters and measure the degree of success in which they take advantage of available resources. By comparing data collected from XALT on the command line parameters used to the Slurm resource requests we are able to determine the degree to which users take advantage of the resources they request. This knowledge will inform us on how better to provide example usages for the software available on our systems, and will inform future software development efforts, guiding the design of more efficient, user-friendly, and adaptable tools that align closely with the specific needs of the HPC community.

The next-generation of radio astronomy telescopes are challenging existing data analysis paradigms, as they have an order of magnitude larger collecting area and bandwidth. The two primary problems encountered when processing this data is the need for storage and that processing is primarily I/O limited. An example of this is the data deluge expected from the SKA-Low Telescope of about 300 PB per year. To remedy these issues, we have demonstrated lossy and lossless compression of data an existing precursor telescope, the Australian Square Kilometre Array Pathfinder (ASKAP), using MGARD and ADIOS2 libraries. We find data processing is faster by a factor of 7 and give compression ratios from a factor of 7 (lossless) up to 37 (lossy with an absolute error bound of 0.001). We will discuss the effectiveness of lossy MGARD compression and its adherence to the designated error bounds, the trade-off between these error-bounds and the corresponding compression ratios, as well as the potential consequences of these I/O and storage improvements on the science quality of the data products.

This study presents scaling results and a performance analysis across different supercomputers and compilers for the Met Office weather and climate model, LFRic. The model is shown to scale to large numbers of nodes which meets the design criteria, that of exploitation of parallelism to achieve good scaling. The model is written in a Domain Specific Language, embedded in modern Fortran and uses a Domain Specific Compiler, PSyclone, to generate the parallel code. The performance analysis shows the effect of choice of algorithm, such as redundant computation and scaling with OpenMP threads. The analysis can be used to motivate a discussion of future work to improve the OpenMP performance of other parts of the code. Finally, an analysis of the performance tuning of the I/O server, XIOS is presented.

HPC architectures most often handle High Performance Data Analytics (HPDA) and Explorative Data Analytics (EDA) workloads where the working data set cannot be easily partitioned or is too large to fit into local node memory. This poses challenges for programming models such as OpenSHMEM[1] or MPI[2] where all data in the working set is assumed to fit in the memory of the participating compute nodes. Additionally, existing HPC programming models use expensive all-to-all communication to share data and results between nodes. The data and results are ephemeral and require additional work to save the data for analysis by other applications or subsequent invocations of the same application. Emerging disaggregated architectures including CXL GFAM enable data to be held in external memory accessible to all compute nodes, thus providing a new approach to handling large data sets in HPC applications. Most HPC libraries do not currently support disaggregated memory models. In this paper, we present how disaggregated memory can be accessed by existing programming models such as OpenSHMEM and the benefits of using disaggregated memory in these models.

Real-time processing of radio astronomy data presents a unique challenge for HPC centers. The science data processing contains memory-bound codes, CPU-bound codes, portions of the pipeline that consist of large number of embarrassingly parallel jobs, combined with large number of moderate- to large-scale MPI jobs, and IO that consists of both parallel IO writing large files to small jobs writing a large number of small files, all combined in a workflow with a complex job dependency graph and real-world time constraints from radio telescope observations. We present the migration of the Australian Square Kilometre Array Pathfinder Telescope's science processing pipeline from one of Pawsey's older Cray XC system to our HPE-Cray EX system, Setonix. We also discuss the migration from bare-metal deployment of the complex software stack to a containerized, more modular deployment of the workflow. We detail the challenges faced and how the migration unearthed issues in the original deployment of the EX system. The lessons learned in the migration of such a complex software stack and workflow is valuable for other centers.

The increasing complexity of HPC architectures requires a larger number of tests for thorough system evaluation post-installation and pre-software upgrades. HPC centers and vendors employ various methodologies for system evaluation throughout its lifespan, not only at the beginning during the installation and acceptance time, but also regularly during maintenance windows. The HPC Testathon 2024 co-organised with CUG2024 is the first of its kind event allowing HPC professionals to have the hands-on experience with different HPC system testing environments and tests. The event has been already supported by several CUG sites including KAUST, ORNL, LLNL, LANL, Pawsey as well as Microsoft. The goal is to share some of the most useful tests among participating institutions to leverage on others experience and further improve each site’s approaches. The event was built on success of previous workshops of the HPC System Test Working Group with contributions from many CUG sites and vendors.

This BoF will summarise HPC Testathon’s results, findings and recommendations. HPC Testathon subgroup leaders will present short overviews on various HPC Testing aspects, including scheduler and programming environment tests, node health tests, application-based regression tests, and isolated system components tests (interconnect, filesystem, memory, CPUs and GPUs). A panel discussion will follow the presentations, allowing other CUG participants to share their experiences and feedback.

HPC systems already produce terabytes of monitoring, usage, and performance data each day, ranging from that produced by low-level hardware telemetry and error reporting systems, to hardware performance counters, to job scheduling and system logs, with natural language text from administrator troubleshooting tickets and notes. Systems of the future will be even larger and more complex. This will increase the challenges of monitoring and characterizing user behaviors on these systems. Meanwhile, machine learning and artificial intelligence techniques have already started to demonstrate effectiveness for characterizing and extracting knowledge from large and complex datasets, but these efforts are just beginning to realize the full value of their potential across a wide variety of domains. For these reasons, we propose the First BoF on Artificial Intelligence and Machine Learning for HPC Workload Analysis. This BoF will provide a much-needed opportunity not only for discussing cutting-edge research ideas but also for bringing together researchers working across the disciplines of data science, machine learning, statistics, applied mathematics, systems design, systems monitoring, systems resilience, and hardware architecture. The proposed BoF aims to help the community advance in the direction towards better and more efficient monitoring and understanding of the usage of large-scale computing systems. The Machine Learning and Deep learning insights from the CUG systems of participating BoF authors will be shared. Our approach is to share best practices, and listen to the audience, constructing new paths forward for HPC workload analysis. Our objective is to get feedback from the audience and be highly interactive.

The requirements from emerging HPC/AI workflows are pushing HPC systems to have more cloud-like capabilities and cloud environments to have more HPC-like capabilities. Some key examples of this are workflows that combine data acquisition, real-time analysis, model training and inference, and traditional simulation. The requirements include heterogeneous hardware, tailored to the different components of the workflow, high availability of key support services, and elastic access to resources. They may also require technologies outside of traditional HPC, such as Kubernetes.

HPE recently announced as-a-service offerings with HPE Greenlake for Large Language Models (LLM) and supercomputing services. The CUG community helps drive the future of HPE Cray systems and is therefore a key community for considering how these cloud-based capabilities can complement the HPE Cray on-prem ecosystem. How cloud-based services are embedded into and accelerate emerging HPC/AI workflows is fundamental to architecting new solutions that extend the capabilities of HPE Cray systems.

The purpose of this BoF is to have an open discussion and collect community feedback. We will review our supercomputing as-a-service solutions and possible approaches to incorporating these services into example workflows. We will have an open discussion on workloads that have increasing affinity to cloud-based services due to availability, compute, latency, and security requirements. The goals of this BoF are to capture requirements, identify workflow design patterns suited for supercomputing services, and discuss the role of supercomputing services, whether deployed in a cloud or within the HPC Cray on-prem ecosystem.

CUG update from the board, overview of conference submissions and proceedings, Special Interest Group Updates, CUG Board Elections.

As part of the Election process the final ballot will be shared with site voting representatives (either site delegate or voting proxy) on Monday evening when electronic voting opens, electronic voting will close prior to this official voting session, any ballots cast in person will override prior electronic votes. During this session candidates will have the opportunity to make statements.

The session is opened to all attendees. This session fulfills official corporate updates and reporting requirements in addition to our annual election of board of director members.

NERSC’s Perlmutter scratch filesystem, an all-flash Lustre storage system running on Cray ClusterStor E1000 Storage Systems, has a capacity of 36 PetaBytes and a theoretical peak performance exceeding 7 TeraBytes per second across HPE’s Slingshot network fabric. Deploying an all-flash Lustre filesystem was a leap forward in an attempt to meet the diverse I/O needs of NERSC. With over 10,000 users representing over 1,000 different projects that span multiple disciplines, a file system that could overcome the performance limitations of spinning disk and reduce performance variation was very desirable. While solid state provided excellent performance gains, there were still challenges that required observation and tuning. Working with HPE’s storage team, NERSC staff engaged in an iterative process that through time, increased performance and provided more predictable outcomes. Through the use of IOR runs and OBDfilter scans, NERSC staff were able to closely monitor the performance of the file system at regular intervals to inform the process and chart progress. This paper will document the results of and report insights derived from over 6 months of NERSC’s continuous performance testing, and provide a comprehensive discussion of the tuning and adjustments that were made to improve performance.

Isambard-AI is a new, leadership-class supercomputer, designed to support AI-related research. Based on the HPE Cray EX4000 system, and housed in a new, energy efficient Modular Data Centre in Bristol, UK, Isambard-AI employs 5,448 NVIDIA Grace-Hopper GPUs to deliver over 21 ExaFLOP/s of 8-bit floating point performance for LLM training, and over 250 PetaFLOP/s of 64-bit performance, for under 5MW. Isambard-AI integrates two, all-flash storage systems: a 20 PiByte Cray ClusterStor and a 3.5 PiByte VAST solution. Combined these give Isambard-AI flexibility for training, inference and secure data accesses and sharing. But it is the software stack where Isambard-AI will be most different from traditional HPC systems. Isambard-AI is designed to support users who may have been using GPUs in the cloud, and so access will more typically be via Jupyter notebooks, MLOps, or other web-based, interactive interfaces, rather than the approach used on traditional supercomputers of ssh’ing into a system before submitting jobs to a batch scheduler. Its stack is designed to be quickly and regularly upgraded to keep pace with the rapid evolution of AI software, with full support for containers. Phase 1 of Isambard-AI is due online in May/June 2024, with the full system expected in production by the end of the year.

HPC is rapidly becoming more complex. Administrators must support a wide range of new workloads that mix AI/ML with HPC while pulling data from varied data sources and a compute environment consisting of assorted structures, including GPUs, CPUs and new accelerators. How do we meet this complexity as an industry while delivering better scalability and efficiency? Introducing the biggest change to HPC resource management in 30 years: Altair® Liquid Scheduling™.

Codee is a suite of software development tools to help improve the performance of C/C++/Fortran applications, providing a systematic, more predictable approach that leverages parallel programming best practices. Codee Static Code Analyzer provides a systematic predictable approach to enforce C/C++/Fortran performance optimization best practices for the target environment: hardware, compiler and operating system. It provides innovative Coding Assistant capabilities to enable semi-automatic source code rewriting, by inserting OpenMP or OpenACC directives in your codes to run on CPUs or offload to accelerator devices such as GPUs, so that novice programmers can write codes at the expert level. Codee provides integrations with IDEs and CI/CD frameworks to make it possible to Shift Left Performance. In this presentation we will also talk about how to use Codee in conjunction with the Cray tools, including compilers (CCE) and performance tools (e.g. CrayPat, Reveal).

With a focus on efficiency and execution to roadmap, HPE and AMD technologies are making Supercomputing a reality. We’ll discuss these technologies and well as explore the exciting future of AMD.

Session closed to only CUG sites, HPE representatives and sponsors not allowed.

Aurora, installed in 2023, is the newest system being prepared for production at the Argonne Leadership Computing Facility (ALCF). Throughout multiple years of preparation, the ALCF has tracked the progress of over 40 applications from the Exascale Computing Project and ALCF's Early Science Project in terms of ability to run on Aurora and performance on Aurora compared to other systems. In addition, the ALCF has been tracking bugs and issues reported by application developers. This broad tracking of applications in a standardized way as well as tracking of over 1100 bugs and issues via source code reproducers has been essential to ensuring the usability of Aurora. It has also helped ensure a smoother transition for applications that can run on past or current production systems, like Polaris, the ALCF's current production system, to Aurora. To gain insight into the current state of applications which were ported to Aurora on both Aurora and Polaris, a set of applications are compared in terms of single GPU and single node performance on Aurora and Polaris. On average the Figure-of-Merit performance for the set of applications was 1.3x greater on a single GPU of Aurora than a single GPU of Polaris. The intra-node parallel efficiency of the set of applications was similar between Aurora and Polaris.

The "IO Wall" problem, in which the gap between computation rate and data access rate grows continuously, poses significant problems to scientific workflows which have traditionally relied upon using the filesystem for intermediate storage between workflow stages. One way to avoid this problem in scientific workflows is to stream data directly from producers to consumers and avoiding storage entirely. However, the manner in which this is accomplished is key to both performance and usability. This paper presents the Sustanable Staging Transport, an approach which allows direct streaming between traditional file writers and readers with few application changes. SST is an ADIOS "engine", accessible via standard ADIOS APIs, and because ADIOS allows engines to be chosen at run-time, many existing file-oriented ADIOS workflows can utilize SST for direct application-to-application communication without any source code changes. This paper describes the design of SST and presents performance results from various applications that use SST, for feeding model training with simulation data with substantially higher bandwidth than the theoretical limits of Frontier's file system, or for strong coupling of separately developed applications for multiphysics multiscale simulation, or for in situ analysis and visualization of data to complete all data processing shortly after the simulation finishes.

Computational steering of experiments with the help of Artificial Intelligence (AI) has the potential to accelerate scientific discovery. Fulfilling this promise will require innovations in workflows and algorithms for experiment control. In this work we developed data management infrastructure that facilitates such innovation by enabling fine grain partitioning and optimization of complex workflows between computational and experimental facilities with dynamic data sharing. This is enabled by Common Metadata Framework (CMF) that tracks workflow data lineages and provides visibility across facility boundaries for relevant data subsets. We demonstrate benefits on a novel Scanning Transmission Electron Microscopy (STEM) workflow of nanoparticle plasmonic transitions in material science that crosses facility boundaries several times. The AI control model is seeded by a meta-model developed at a computational facility using data from other experimental sites to help impart prior knowledge, reduce measurement cost and sample degradation. The model is incrementally refined by successive measurements at an experimental facility. The evolution of model uncertainties is captured in CMF and fed back to computational facility for analysis of potential new physical phenomena. The relevant experimental results can be used to calibrate molecular dynamics simulations at the computational facility that in turn influence the AI model refinement trajectory.

The Cray Pprogramming Eenvironment (CPE) provided for Cray Shasta OS based clusters provides a small but solid set of tools for developers and cluster users. The CPE includes Cray MPICH, Cray Libsci, the Cray debugging tools, and support for a range of compilers – and not much else. Users expect a wide range of additional software on these clusters, and frequently request new software outside of what’s provided by the CPE or what can be provided via the system packages. Foregoing our old manual installation process, the LANL HPC Programming and Runtime Eenvironments Tteam has instead opted to utilize Spack as the installation mechanism for most additional software on all of our new HPE/Cray Shasta clusters. This brings with it several distinct advantages - Spack’s vast library of package recipes, well- defined software inventories, automatically- generated modulefiles, and binary packages produced through our CI infrastructure. It also brings with it substantial issues – remarkably higher manpower requirements, longer turnaround times for software requests, a more challenging build debug process, and questionable long-term maintainability. Our paper will detail our approach and the benefits and pitfalls of using Spack to install and maintain production software environments.

In High-Performance Computing (HPC), managing the user environment is a critical and complex task. It involves composing a mix of software that includes compilers, libraries, tools, environment settings, and their respective versions, all of which depend on each other in intricate ways. Traditional approaches to managing user environments often struggle with finding a balance between stability and flexibility, especially in large systems serving diverse user needs.

This work introduces a containers-first approach for HPC, enhancing stability and flexibility in user environments by seamlessly integrating container technologies on an HPE Cray EX system. This approach evolves the user environment management and delivery, enabling customized, fast, and transparent deployment through containers. It aims to isolate the user environment from system software, maintaining consistent workflows even during maintenance and upgrades. Furthermore, it increases environment flexibility, adaptability to specific user needs, and accelerates deployment of updates and fixes, while minimizing system dependency.

The containers-first approach is a step forward towards improved testing capabilities, with increased portability and reproducibility of scientific workflows. This paper outlines the principles and implementation of seamless integration of container technologies into HPC infrastructure, which can directly benefit operational efficiency, user productivity, and the overall management of HPC systems.

This work introduces a cloud-native deployment of the Slurm HPC Workload Manager, leveraging microservices, containerization, and on-premises cloud platforms to enhance efficiency and scalability. Utilizing Kubernetes and Nomad's APIs alongside DevOps tools, the system automates system operations, simplifies service configuration, and standardizes monitoring. However, implementing a cloud-native architecture poses challenges, including complex containerization and resource management issues that are intrinsic to HPC.

Our solution implements an automated deployment and management process of Slurm. Where some components are containerized, others are deployed natively and are managed via Kubernetes and Nomad, respectively. This approach ensures consistent, automated, reproducible, and resilient Slurm management. The full paper will detail this implementation and its benefits, underscoring the potential of modern software practices to evolve HPC systems to meet the performance and flexibility expected by our user base.

The Cray System Management (CSM) software stack has been enhanced within the past year to improve the operating experience for an HPE Cray EX system. Features to be discussed include more automated software installation and upgrade, system backup and using that backup for disaster recovery reinstallation, automation for concurrent rolling reboots of management nodes, enhancements in CSM Diagnostics tooling, improved SMA monitoring tools, Slingshot switch Orchestrated Maintenance and LACP traffic load sharing, containerized login environment, and new compute node features (default kernel configuration, Low Noise Mode improvements, OS Noise Detection, DVS node health monitoring, Dynamic Kernel Module Support, and containers on compute nodes), improved multi-tenancy support, and image management for the aarch64 architecture.

Some microservice changes will be discussed. The Power Control Service (PCS) replaces CAPMC for power information and status reporting, ensuring reliable power control. The Boot Orchestration Service (BOS) has been completely re-architected in version 2 to be more efficient and readily handle rolling upgrades. The Configuration Framework Service (CFS) is easier to debug with the inclusion of ARA (ARA Records Ansible) allowing better insight into each configuration layer. This is complemented by a new CFS debugging tool.

Come see the new and improved CSM.

This talk will give a brief overview of how HPCM is deployed on a HPE solution. It will describe what an admin node does, what Quorum HA is, why we use SU-Leaders. This will include when to use Quorum HA, SU Leaders, an introduction the “cm” command suite and a brief overview of the monitoring tools present with HPCM. With a new set of monitoring tools available in HPCM 1.11, the new pipeline will be described and how to enable the components needed.

The ability to effortless migrate compute clusters between sites or zones is common in the cloud world, and recently has also become a necessity for supercomputing facilities like the Swiss National Supercomputing Centre (CSCS), where its multi-region flagship infrastructure Alps serves multiple tenants and customers, some of them with very different development and operational requirements.

This presentation explores an innovative software solution based on HPE Cray System Management's API (CSM) that is designed to streamline the migration process of entire tenant clusters (as CSCS refers to them, versatile clusters, or vClusters). The proposed solution, part of CSCS' manta software ecosystem, serves as a powerful facilitator for organizations looking to adapt quickly to a multi-tenant model with different operational requirements, while ensuring minimal downtime and disruption during migration. The presentation will go through the core features, highlighting its sophisticated state transfer mechanisms, image preservation capabilities, and the seamless transfer of vCluster definitions and configurations, describing in detail all the design choices and constraints imposed by the environment, and HPE API.

Through real-world case studies, this presentation aims to showcase the practical applications of this solution, illustrating its efficacy in diverse scenarios applied within Alps.

In June 2022, the long-awaited exaflop compute barrier (1 quintillion floating-point operations per second) was surpassed on the TOP500 list by Frontier, an HPE Cray EX supercomputer at Oak Ridge National Laboratory (ORNL). Drawing peak power of 21.1 MW, Frontier demonstrated 1.1 exaflops/second of computational capability, much of which is supplied by more than 37,000 AMD Instinct MI250X graphics processing units (GPUs). With a single GPU drawing up to 560W thermal design power (TDP), each AMD MI250X draws 2x more power under load than the NVIDIA V100 GPUs used in Frontier’s predecessor at ORNL, the 200 petaflop IBM POWER9 supercomputer, Summit. There are many other major technological advances in the memory, compute, power, and infrastructure of Frontier that are new to production environments. Frontier's mission to enable ground-breaking research in U.S. energy, economic and national security sectors is fulfilled through leadership-class workloads, which are workloads that demand greater than 20\% of the supercomputer. These large workloads are vulnerable to defective and failing computing hardware. The rate of failing hardware is quantified through the mean time between failures (MTBF), which is the length of time between a hardware-level failure anywhere in the system. In this work, we describe the multi-staged approaches to stabilizing and maintaining the functionality of the hardware on Frontier. There are three strategies discussed; the first two utilize leadership-class tests to target improving the MTBF of Frontier, the third utilizes single-node validation to efficiently identify individual instances of defective hardware in Frontier. We provide summarized data from each of the three strategies, then classify the diverse set of failures and discuss trends in defective hardware, before discussing several key challenges to identifying defective hardware and improving the MTBF of Frontier.

Supercomputers are complex systems that bring together bleeding edge technologies. Take the example of the first exascale system, Frontier, installed at Oakridge National Laboratory. Frontier consists of more than 9400 compute nodes embedded in the HPE Cray EX4000 infrastructure. Each compute node consists of an AMD CPU, four MI250 GPUs interlinked by high-speed XGMI interfaces, as well as Slingshot-11 high-speed NICs. The system is comprised of over 150,000 node-level components interconnected in an extremely dense mechanical framework and is cooled via warm temperature liquid cooling. As impressive as all these technologies are on their own, the real value lies in bringing these technologies together in a system to achieve sustained performance over time. With that in mind, it behooves us to recognize that system quality attributes such as diagnosability and serviceability are critical to achieving high levels of availability throughout the service life of a system. Therefore, for HPE and our customers, providing a product-level hardware triage framework will help ameliorate the return to service time for failed components, provide a standardized approach to diagnosing hardware failures, reduce the number of no trouble found replacements, and minimize the need for SMEs from R&D to directly support systems in the field.

A user of an High Performance Computing (HPC) system relies on a multitude of components, both on the user-facing side, such as \texttt{modules}, and lower-level system software, such as Message Passing Interface (\mpi) libraries. Thus, all these different aspects must be tested to guarantee an HPC system is production ready. We present here a suite of tests that cover this larger space, which not only focus on benchmarking or sanity checks but also provide some diagnostic information in case failures are encountered. These tests cover the job scheduler, here \texttt{slurm}, the \mpi\ library, critical for running jobs at scale, GPUs, a vital part of any energy efficient HPC system, and the performance of compilers that are part of the Cray Programming Environment (CPE). Some tests were critical to uncovering a number of underlying issues with the communication libraries on a newly deployed HPE-Cray EX Shasta system that had gone undetected in other acceptance tests. Otherwise identified bugs within the job scheduler. The tests are implemented in a \reframe\ framework and are open source.

The National Nuclear Security Administration Department of Energy supercomputers at Los Alamos National Laboratory (LANL) are integral to supporting the lab’s mission and therefore need to be reliable and performant. To identify potential problems ahead of time while minimizing the interruption to the users’ work, the High Performance Computing (HPC) Division at LANL implemented a Continuous Testing framework and the necessary infrastructure to be able to automatically and frequently run a series of tests and proxy applications. The tests, which benchmark various system components, were integrated into the Pavilion2 testing framework and are launched in small Slurm jobs across each machine every weekend. The results of the tests are summarized in a comprehensive Splunk dashboard, enabling continuous monitoring of the health of the machines over time without having to parse through each run’s output and logs. This project is currently running on three of LANL’s newest fleet of Cray Shasta machines – Chicoma, Razorback, and Rocinante– and will eventually be implemented on all production clusters at LANL. This paper details the different components of the Continuous Testing framework, the resulting setup, and the impact it has on our HPC workflow.

OpenFabrics Interfaces (OFI) and Unified Communication X (UCX) are both transport protocols that underlie the HPE Cray MPICH library on HPC systems like ARCHER2. They can be selected at runtime by users. This paper presents the scalability and performance of OFI and UCX transport layer protocol implementations on ARCHER2, an HPE Cray EX system that features Slingshot 10 interconnect. We use ReproMPI microbenchmarks to study the performance of MPI collectives and run experiments using some of the most commonly used applications on ARCHER2. The results show that in most cases OFI and UCX performance is comparable on under 32 nodes (16384 cores) but for larger number of nodes OFI runs more reliably. Ultimately, when it comes to applications there is no one-size-fits-all solution and profiling can facilitate tuning for best performance.

We present a novel approach for co-developing the hardware of Network Interface Cards (NICs) ASICs (e.g., Slingshot Cassini-3) and their software stacks. Due to increasing gap between network link bandwidths and compute power of hosts, critical parts of the software stack of many applications are offloaded to NICs for efficient processing. By processing certain functions using specialized hardware blocks in NICs, compute resources can be better utilized for actual application processing. Therefore, use cases and design of NICs rapidly evolve with advancements in transmission capacity of network links. To reduce time-to-market for next-generation NICs with complex features implemented in hardware, we leverage the software ecosystem of programmable networks and compiler technology for development, testing and verification of the slingshot NICs.

There has been a rise in Machine Learning applications on High-Performance Computing Systems equipped with GPUs motivated by increasing demands for model and dataset sizes. Although there are no fundamental limitations preventing running these workloads in MPI runtimes, the in-house chip makers' communications collectives libraries have become the preferred deployments. In this presentation, we will how these libraries work and how we can take full advantage over the HPE Slingshot interconnect. We will review use and configuration of the two most common of these - NCCL and RCCL – to run using the HPE Slingshot NIC RDMA capability. We will explain the key parameters for the given environments and provide recommendations for the optimal settings.

The NVIDIA Collective Communications Library (NCCL) is widely used for multi-node AI applications as well as other scientific codes. Initial deployments of Slingshot 11 (SS11) did not support this library at high-performance leading to detrimental impacts on the performance of Deep Learning applications on SS11-based HPC systems. We describe collaborative efforts between NERSC, NVidia and HPE to develop capabilities for NCCL on SS11. This involved development of the Libfabric NCCL plugin and an extensive period of testing and refinement utilizing the Perlmutter HPC system at NERSC. In this presentation we will describe the development required as well as performance improvement measured on Perlmutter for both benchmarks and cutting-edge scientific AI applications.

A large language model (LLM)’s Key-Value (KV) cache requires enormous GPU memory during inference. For faster query processing and conversational memory of chat applications, this cache is stored to answer subsequent user queries. However, if the cache is buffered on the GPU, its large memory requirement prevents multiplexing and requires cache buffering in remote storage. In current systems, transferring and retrieving cache requires the CPU to coordinate with the GPU and push and fetch the data through the network, increasing the overall latency. This paper proposes lower overhead KV cache storage and retrieval with SmartNICs capable of triggered operations, such as Cassini. Triggered operations enqueue pre-defined data transfer instructions on the NIC. A GPU thread can trigger these instructions once the LLM computes the KV cache for a token. Cassini Network Interface Cards (NICs) then transfer the cache, bypassing the CPU and network stack and improving data-transfer latency. Our experiments show that data transfer with triggered operations provides 19× speedup in transfers ranging from 32KB to 5 Terabytes.

This paper explores the application of Large Language Models (LLMs) in three distinct scenarios, demonstrating their potential to aid user experience and efficiency. Firstly, we examine the application of LLM models such as OpenAI’s GPT or Llama in chatbots to assist in programming environments, providing real-time assistance to developers. Secondly, we explore using LLMs and Python to search internal document corpus for performance engineering, significantly improving the retrieval of relevant information from extensive technical documentation. Lastly, we investigate using LLMs as an interface to system batch schedulers, such as Slurm or PBS, replacing domain-specific languages and prompts with natural text. This approach democratizes access to complex systems, fostering ease of use and enhancing the user experience. Through these use cases, we underscore the versatility and potential of LLMs, highlighting their role as an aid to system operation and user experience.

In 2023, we saw a huge rise in the capability and popularity of large language models (LLMs). OpenAI released ChatGPT 3.0 to the public in November 2022, and since then, there have been many closed and open-source LLMs that have been released. It has been reported that training ChatGPT 3.0 took more than 10,000 GPU cards, making training a foundational LLM out of reach for many research teams and HPC centers, but there are many ways to use a pre-trained LLM using GPUs at scales available at an HPC center. For example, fine tuning a pre-trained model with site or application- specific data or augmenting the model with Retrieval Augmented Generation (RAG) to add specific knowledge to a pre-trained model. The availability of open-source LLMs has opened the field for individual researchers and service providers to do their own custom training and run their own chatbots. In this paper, we will describe how we deployed and evaluated open source LLMs on Quartz and Big Red 200, a Cray EX supercomputer, and provisioned access to these deployments to a select group of HPC users.

Large Language Models (LLMs) are showing impressive ability to reason and answer complex questions after a few contextual prompts. This has transformational potential for scientific productivity and new discoveries. However, LLMs commonly suffer from hallucination problems and their performance strongly depends on specific task and model. Several ad-hoc LLM benchmarking studies have been reported for specific tasks in medicine, biology and chemistry. The science community typically prefers open-source models, including derivatives of Llama-2, Galactica, etc. These models usually perform worse than the commercial GPT-4, requiring more scrutiny. In this work we developed an automated platform that co-optimizes LLM recommendation and evaluation for user specified tasks to make it easier for science practitioners to select and evaluate promising LLMs for their use case. Uniquely, it uses a recommender engine trained on characteristics of large number of training datasets, LLM models, and inference data to find a list of best candidate models for user specified prompt templates. Then, it evaluates and ranks these candidates using automated reviewer that performs multi-metric assessment and, uniquely, iteratively revisits the evaluation to validate its truthfulness employing the reflection approach and self-consistency technique. The evaluation platform seamlessly integrates with HPE MLDE (Machine Learning Development Environment).

Plenary session to hear from the CUG2025 site representative.

Get to know VAST Data and its revolutionary VAST Data Platform, a breakthrough approach to data intensive computing designed for the AI era.

The majority of research on the gas turbines used for aircraft propulsion was historically carried out via costly laboratory tests. This approach generally yields the overall heat transfer or loss of components, rather than providing sufficient detail to dissect the range of physical phenomena impacting the performance of a design. On the other hand, Computational Fluid Dynamics (CFD) yields the full, detailed air flow-field and has become a key gas turbine design tool. Yet, industrial design cycles rely on low-order models so that performing hundreds of engine-relevant analyses is tractable. The accuracy of these models, however, is notoriously challenged by the complex flow conditions within aircraft engines, including strong pressure gradients, separated flow and laminar to turbulent transition mechanisms.

The relentless increase in computing power has enabled high-fidelity simulations of the complex flow physics in gas turbines, at engine-relevant and including operational aspects, such as component wear. Conducting such cutting-edge, massively parallel, high-fidelity CFD is a challenging task requiring significant code development efforts, planning and foresight with regards to case setup, checkpointing and critically post-processing to make best use of the latest supercomputing architectures. The reward is new physical understanding that designer can exploit to eke out more efficiency and gold-standard datasets that can be used to develop low-order models usable by industry. The new knowledge and models support the development of a cleaner and more sustainable aviation sector. In this talk, we will discuss recent code porting and optimisation efforts to maximise usage of the latest HPC systems featuring AMD GPUs and Cray compilers, and present results of new, ambitious simulations enabled by HPC systems such as Setonix and Frontier, where we leverage extreme computing power to reveal new gas turbine physics.

Systems with tightly-integrated CPU and GPU on the same module or package will be deployed in 2024, based on NVIDIA GH200 and AMD Mi300a processors. There are some significant differences from current systems, of which the key ones for this presentation are:

- each processor consumes greater than 650W per component _under normal workloads_, which requires power capping of devices to stay within nominal power limit of 340 kW and 400 kW in the EU and US respectively;

- tight integration of memory between CPU and GPU opens new opportunities for application optimization, and changes some fundamental application design concepts.

Engineers at CSCS and HPE spent a week testing application performance and power consumption at the node and cabinet level on two cabinets in Chippewa Falls, in preparation for the installation of over 10,000 GH200 in the Cray EX system Alps at CSCS.

This presentation will focus the power measurement at the node and cabinet level, with a detailed description of measurement methodology and the results, followed by results from applications and micro-benchmarks, with discussion on:

- optimizing power consumption and performance;

- time and energy to solution;

- and the counter-intuitive effects of of power sloshing between CPU and GPU.

This session will focus on providing a demonstration of top ML frameworks, HPC applications, and tools for data science on NVIDIA's Grace Hopper and Grace CPU Superchip. These superchips are the cornerstones of versatile and power-efficient supercomputers worldwide that combine Grace CPUs, Hopper GPUs, and extreme scale networking technology, in a standards-compliant server. We 'll showcase recent results from key applications like PyTorch, JAX, WRF, GROMACS, and NAMD. We'll also provide lessons learned and experiences to help guide developers creating their own applications for NVIDIA Grace superchips. This session is a strong starting point for anyone looking to understand, and develop for, NVIDIA Grace Hopper or the Grace CPU Superchip.

GROMACS is a widely-used molecular dynamics software package with a focus on performance, portability, and maintainability across a broad range of platforms. Thanks to its early algorithmic redesign and flexible heterogeneous parallelization, GROMACS has successfully harnessed GPU accelerators for more than a decade. With the diversification of accelerator platforms in HPC and no obvious choice for a well-suited multi-vendor programming model, the GROMACS project found itself at a crossroads. The performance and portability requirements, as well as a strong preference for a standards-based programming model, motivated our choice to use SYCL for production on both new HPC GPU platforms: AMD and Intel. Since the GROMACS 2022 release, the SYCL backend has been the primary means to target AMD GPUs in preparation for exascale HPC architectures like LUMI and Frontier. SYCL is a cross-platform, royalty-free, C++17-based standard for programming hardware accelerators, from embedded to HPC. It allows using the same code to target GPUs from all three major vendors with minimal specialization, which offers major portability benefits. While SYCL implementations build on native compilers and runtimes, whether such an approach is performant is not immediately evident. Biomolecular simulations have challenging performance characteristics: latency sensitivity, the need for strong scaling, and typical iteration times as short as hundreds of microseconds. Hence, obtaining good performance across the range of problem sizes and scaling regimes is particularly challenging. Here, we share the results of our work on readying GROMACS for AMD GPU platforms using SYCL, and demonstrate performance on Cray EX235a machines with MI250X accelerators. Our findings illustrate that portability is possible without major performance compromises. We provide a detailed analysis of node-level kernel and runtime performance with the aim of sharing best practices with the HPC community on using SYCL as a performance-portable GPU framework.

In the realm of HPC, efficiently exploiting parallelism on large-scale systems presents a significant challenge, particularly when aiming for a low-barrier, nonprogrammatic approach. This presentation showcases the effective application of GNU Parallel for optimizing the use of four key resources at OLCF’s Frontier and NERSC’s Perlmutter supercomputers: CPUs, GPUs, NVMe storage, and large-scale Storage systems. Our work positions GNU Parallel not merely as a utility tool but as a mainstream, productive instrument for diverse HPC workloads. We illustrate GNU Parallel’s capacity to efficiently harness CPU resources across up to 9,000 Frontier compute nodes, with a manageable increase in overhead even as node utilization scales up. This demonstration includes a detailed analysis of overhead metrics, revealing that it remains ~5 minutes for up to 7,000 nodes and under 10 minutes for up to 9,000 nodes comprising up to 1.15 million parallel tasks. Our approach is further elucidated through a series of practical vignettes. These include diverse applications, ranging from daily tasks to massively scaled operations, all underpinned by real-world applications and practical use cases. These vignettes offer generalized solutions for common computational and I/O patterns observed in real-world applications, validated through scalable examples on Frontier and Perlmutter supercomputers.

Writing performant programs on modern supercomputers means targeting parallelism at multiple levels of scale: from vectorization in a single core, to multicore CPUs, to nodes with multiple CPUs, to GPUs and nodes with multiple GPUs. Traditionally, this has meant writing applications using multiple programming models; for example, an application might be written as a combination of MPI, OpenMP, and/or CUDA. This use of multiple programming models complicates an application’s implementation, maintainability, and portability.

Chapel is a parallel programming language that supports both scalable distributed-memory programming on multicore nodes as well as GPU computing with a single unified set of language features. In this talk, we discuss the achievements of Chapel’s growing GPU support in recent years. We will share user experiences porting benchmarks like miniBUDE and ChOp (Chapel-based Optimization); as well as a user application for spectral satellite image analysis for biodiversity detection in coral reefs. Chapel aims to be portable across both NVIDIA and GPU architectures and we will share results from Frontier and Perlmutter. We will conclude the talk with a summary of ongoing and planned work.

The Pawsey Centre for Extreme-Scale Readiness (PaCER) was a unique Australian program supporting researchers in developing and optimising codes on Setonix, Australia’s most powerful research supercomputer based on AMD Milan CPUs and MI250X GPUs. The focus of the PaCER program was on both extreme scale research (algorithms design, code optimisation, application and workflow readiness) and using the computational infrastructure to facilitate research for producing world-class scientific outcomes. PaCER was a partnership for collaboration with Pawsey and supercomputing vendors that provided early access to HPC tools and infrastructure, training and exclusive hackathons focused on performance at scale. PaCER supported 10 projects gathering 18 international and national institutions, supported more than 60 researchers and 10 Research Software Engineers. The biggest success of the project is the creation of a supercomputing software developers’ community, unique and first of its kind in Australia. In this talk, we will describe the collaboration model created by PaCER and cover significant achievements of the project, including the series of GPU programming mentored sprints. We will also share some of the lessons learned and future plans.

Security is an important, if sometimes overlooked, part of running an HPC system. We will describe the out-of-box experience with the Cray EX system from a security perspective. Several security issues (and mitigations for them) present in the Cray EX (HPCM) software stack and, perhaps, HPC systems in general will be described. The process for reporting, patching, and publishing these issues will be discussed as well as some thoughts on where to look for and how to reduce the risk posed by as yet undiscovered vulnerabilities. Finally, some general advice for securing Cray and other HPC systems will be provided.

Vulnerabilities are an unfortunate reality in modern computing - while there's plenty of discussion around the importance of detection and patching in HPC, there's not as much chatter about how to handle vulnerabilities discovered at one's institution. While this process is currently ad-hoc and depends in large part on the maintainers of the software in question, one HPC center's recent experience with the discovery of a critical vulnerability within Lustre within COS, reporting that vulnerability to HPE, and the subsequent handling of that information by both groups suggests that there's room for improvement in a variety of areas on both sides. In this presentation, NERSC Security will:

- go over the timeline of events related to a recent vulnerability discovery by their Computational Systems Group and its subsequent report and patching, - highlight and discuss multiple events, initiated by both NERSC and HPE, where the handling of sensitive information could be improved, and - propose a novel idea for an open, transparent, institutionally-agnostic entity focused on receiving and validating vulnerability reports from the HPC community to increase efficiency in vulnerability handling for both HPC centers and vendors alike.

In the realm of High-Performance Computing (HPC), our project addresses the escalating challenge of failures, particularly with the anticipated complexity surge in Exascale systems. Our innovative Semi-Supervised Failure Prediction service, applicable to various system management software tools, utilizes deep learning on telemetry data for real-time, end-to-end failure prediction. Our approach centers on deep learning models proficient in deciphering intricate patterns within extensive datasets. From data acquisition to prediction, our solution seamlessly integrates with system management software, analyzing critical metrics like CPU usage, memory status, and network activity. By learning from historical data, the model distinguishes between normal and failure states, providing real-time predictions before potential failures. With a semi-supervised learning approach using both labeled and unlabeled data, our model adapts effectively to diverse failure scenarios. Integrated with the AIOps service in system management tools, it offers organizations a proactive edge, enabling early intervention for cost reduction, minimized downtime, and improved data center efficiency. In our upcoming presentation, we will delve into a concise results overview, showcasing the benefits of our approach in enhancing predictive capabilities and providing organizations with strategic advantages in system management. This functionality is being considered for future inclusion in HPE system management products.

Unified alerting provides a single interface or platform in system management to create and manage alerts for various components within HPC systems. The system management monitoring stack collects different types of telemetry from various system components and store events and logs in OpenSearch and metrics in Timescale.

Unified alerting comprises two rule engines, OpenSearch Alerting and Grafana Alerting, and a single endpoint Alertmanager. It supports the creation and management of rules, as well as the viewing and silencing of alerts. A wrapper framework has been implemented on top of rule engines, providing users with a flexible method (a YAML file) to design and create new rules. This has capability to route alerts to email and Slack for notification. Furthermore, alerts can be stored in OpenSearch facilitating future access and analysis of historical alert data. Unified Alerting includes an easy-to-use plug-in/plug-out system, hence any sort of action-trigger solution can be integrated into it.

The entire unified alerting solution is manageable via the command-line interface (CLI), and Grafana dashboards are available for viewing alerts. This new alerting capability is included in HPCM 1.10 and will similarly be available in CSM later in 2024 and other future HPE monitoring solutions for HPC systems.

The new HPE Cray EX255a blade (two nodes, each with 4 AMD MI300a sockets) has power demands and additional sensors that require a more robust power subsystem and data collection capability. This necessitated careful evaluation and changes to how we manage, collect, and publish sensor data. We factored all sensor access parameters and operational characteristics for the EX225a node cards into a standardized file format. These files define default values for compilation. The files can be edited and unmarshalled into runtime accessible structures, enabling testing, tuning, and experimentation with alternative settings. Starting at the hardware access layer and working up the stack we optimized the code paths to enable collection of more sensors in our fixed time budget. This work is the foundation for future work that could enable the ability for higher-level management and monitoring software to customize data collection on behalf of users.

The Lightweight Distributed Metric Service (LDMS) has been deployed on some of the largest Cray systems over the past decade to enable low overhead capture of system and application metrics of interest. LDMS has evolved over time to provide new capabilities and associated configuration options to address the ever increasing size and heterogeneity of HPC systems. In the last quarter of 2023 a working group was formed to formalize “best practices” for LDMS deployment on large-scale HPC systems as well as to help guide future configuration management approaches and mechanisms in the LDMS open source/development project. We present the results from this working group as they apply to base-level configurations of samplers and aggregators, authentication mechanisms, and practices to simplify deployment, including use of pre-built Docker containers. For those interested in automated aggregator load balancing and resilience to host failure we describe capabilities of the LDMS distributed configuration manager (Maestro). Finally, we present planned extensions and the capabilities they provide.

This paper describes how a set of liquid-cooled compute and air-cooled storage resources can be reconfigured between Cray EX Systems. In each configuration, the resources function as a native set of managed nodes and/or directly attached storage resources to the associated host system. In a normal production configuration for example, users can and run compute jobs on the reconfigured compute nodes or read and write data to the corresponding filesystems. To administrators, these compute nodes are managed like conventional compute nodes and the storage is managed by the Neo software stack. The Slingshot network is connected to each of the possible host systems and the systems management networks interconnect via a layer 3 EVPN VXLAN tunnel. NERSC has implemented this reconfigurable architecture on a set of liquid-cooled and air-cooled resources termed POD. POD resources have been successfully transitioned between NERSC's development, staging and production systems and is currently configured and in use on NERSC's flagship system perlmutter.

In a perfect world, HPC system downtime would be easy to minimize. Just keep a perfect copy of the production cluster to prevent scaling surprises.

Spare exascale clusters are too expensive for this kind of operation. Most systems make do with proving changes on smaller testing and development systems (TDS), then rolling the changes out during a maintenance window. If patching regularly is a system requirement, this can introduce a lot of downtime, especially when unforeseen scaling issues crop up.

LANL is deploying a upgrade process where, with minimal rededication of resources, a parallel computing environment is defined, installed, tested in situ, and then scaled up without interfering with the current user environment. We show how to orchestrate this transition under Slurm and address a number of issues like user code rebuilding, rolling reboots and rollback.

Thanks to existing work on submission-time node reconfiguration, Shasta boot orchestration features, Slurm reboot functionality, and new work on retooling Slurm configuration files, LANL is implementing tools to roll out a new image live to active users without downtime.

The Swiss National Supercomputing Center (CSCS) has developed a strategy to provide Infrastructure as a Service (IaaS) to select customers co-investing in the Alps infrastructure. A critical component of this service is the ability to segregate and isolate the Alps network for various IaaS tenants, enabling them to integrate their own site networks. This necessitates the management of network multitenancy capabilities within the infrastructure. The Alps system, powered by HPE Cray EX machines and managed through Cray System Management (CSM) with Slingshot interconnect, utilizes VLANs for node segmentation at the High-Speed Network (HSN) level. Implementing network multitenancy with CSM requires a novel configuration approach in the node management network.

This presentation illustrates the need for network multitenancy and outlines our technical solutions for achieving network segmentation on a HPE Cray EX system. We delve into the technical configuration required for the node management network, exploring the challenges encountered and the innovative solutions we developed. This study contributes valuable insights into the complexities and methodologies of implementing effective network segmentation in a supercomputing environment.

ClusterStor Tiering is a suite of software features designed to enhance the usability and management of hybrid storage systems, combining both flash and disk components. Specifically crafted for monitoring and maintaining file layouts and free space on E1000 flash and disk tiers, Tiering offers a range of customizable capabilities through data management policies. Administrators can tailor fine-grained indexing controls, orchestrate file migrations between Object Storage Targets (OSTs) or pools, execute restriping processes, perform purges, and generate reports. These actions are intelligently triggered by preset timers or dynamically in response to system conditions, such as reaching capacity thresholds. Leveraging a scale-out architecture, tiering efficiently handles the movement of large data volumes. Utilizing the System Management Unit (SMU) for all functions, additional data mover nodes can be configured to augment throughput. Key functionalities supported by Tiering include scalable search, transparent tiering, parallel data movers, data purging, and reporting.

This presentation will delve into a comprehensive review of the Tiering software feature. It will provide valuable insights on achieving scalable searches, configuring policies for tiering, purging, or reporting, setting up parallel data movers, and will present performance results showcasing the efficacy of employing multiple data movers for higher bandwidth with data migration

Alps is the Swiss national supercomputing centre's multi-tenant software-defined infrastructure This paper describes the configuration and experiences to get VAST working as a performant filesystem option on the HPE Cray EX line of supercomputers and highlights the possibility to attach additional storage options over the edge routers of these systems.

The paper describes some different non-traditional HPC workloads seen on Alps that have trouble being met by a traditional HPC filesystem (Lustre). The paper discusses challenges seen by these workloads and includes network design choices, client tuning parameters, and results seen of the VAST storage appliance.

The paper covers the multi-tenant needs, and experiences and additionally some comparisons with other storage offers on Alps will be provided including Ceph and Lustre. Additionally, the network and client tuning decisions made for VAST are explored for their potential applicability to other storage attachments on the HPE Cray EX line, extending the relevance of the findings beyond the specific filesystem discussed.

After years of experience with Cray’s System Snapshot Analyzer (SSA), the HPC Call Home team worked to develop a new more flexible, scalable, open, and secure Call Home infrastructure to support our future HPC products. Becoming part of HPE allowed us to take advantage of and include HPE’s highly secure Remote Data Access (RDA) capabilities as part of that new infrastructure. A key design point was to make the new product useful even for sites that are not typically uploading data – which sounds rather odd for a “call home” tool set. Other points were the maintenance of a pluggable and highly configurable collection framework partnered with an efficient storage methodology. This paper will discuss the design and highlight where enhancements were made. Example collection plugins will be reviewed. Finally, the paper will seek to answer the question, “So why should I run SDU?”

The System Monitoring Working Group (SMWG) has been a CUG SIG (special interests group) to enable collaborations between HPE Cray and their customers on system monitoring capabilities. The Systems Monitoring Working Group includes representatives from many HPE Cray member sites. We meet to discuss and collaborate together on any issues related to system monitoring.

As system monitoring has become a more mainstream topic, much of the SMWG's original mission has either been addressed, or has become a regular topic within other forums. As such, we have decided to close the SWMG as a standalone entity.

This final BoF will be an opportunity for participants to informally share knowledge and experiences gathered during the lifetime of the SMWG, at the end of which we will officially close the working group.

Attendees of this tutorial will learn how to perform data analysis at scale using Arkouda and gain an understanding of how distributed computing can benefit their work.

First participants will be led through a hands-on example, using popular Python data science tools (e.g., NumPy, Dask, and PySpark) to explore data, read and write CSV, HDF5, and Parquet files, and perform graph analytics. Next, users will move to a Jupyter notebook within a Docker container. In this notebook, attendees will use Arkouda to do the same thing. Additionally, this tutorial will demonstrate a real application that performs motif finding on a connectome on a large system. Finally, Attendees will compare the performance and usability of Arkouda with the popular Python data science tools (e.g., NumPy, Dask, and PySpark).

Recently we have seen a change in the diversity of applications utilising high performance computing (HPC), from primarily computational simulation approaches, to a more varied application mix including machine learning and data analytics. With this diversification in workloads, there has also been a diversification in I/O patterns; the movements in, and requirements on, data storage and access. Data storage technologies in HPC have long been optimised for large scale bulk operations focussed on high bandwidth/low metadata operations. However, many applications now exhibit non-optimal I/O patterns for large scale parallel filesystems, with large amounts of small I/O operations, non-contiguous data access, and increases in read as well as write I/O loads.

Parallel filesystems, such as Lustre and Storage Scale, have been optimised and extended to provided higher metadata performance and to better handle small I/O operations. However, the underlying approach of POSIX-like I/O, with block-sized read and write operations, and file-level data storage, set limitations on the overall performance and functionality that such approaches can achieve.

This tutorial aims to provide application developers and system designers the knowledge and skills required to effectively and efficiently evaluate object storage potential for their applications and systems, and leverage new access mechanisms where appropriate. We will introduce the DAOS object store as an example, and provide experience through hands-on exercise and lecture sessions.

In the high-performance computing (HPC) realm, swiftly identifying job failures is critical to optimize resource allocation and ensure system efficiency. Given the high costs and extensive resource demands of HPC systems, the impact of job failures, particularly post-resource allocation, is significant. These failures, crucial in time-sensitive research domains, can derail progress and obstruct objectives. Proactive failure detection allows administrators to quickly enact corrective actions, like job resubmission or reconfiguration, reducing downtime and enhancing user satisfaction. Our approach includes predicting job failures at the initial stages, analyzing failure causes, and developing preventive strategies. By implementing a robust data collection process within the HPC system and utilizing the Slurm workload manager, we have streamlined the data handling procedures. Our methodology involves data preprocessing, feature engineering, and using machine learning models optimized with cross-validation, addressing class imbalances, and focusing on precision, recall, and F1-score metrics. This thorough approach aims to improve resource optimization and prevent future inefficiencies in HPC systems.

Scientific applications running on Exascale supercomputer generate massive data requiring efficient storage for future analysis. While simulations leverage thousands of nodes for writing, reading and analysis typically utilize limited resources, i,e, a handful of nodes. As a result, users commonly query only a particular plane of a multidimensional array or read data from files in strides, hoping that the overall I/O and data processing cost is reduced. This presentation delves into these non-contiguous and striding I/O patterns commonly employed in scientific data analyses. Using visualization as examples, we reveal the detrimental impact of non-contiguous file access on overall throughput, counteracting the speedup gained from analyses with reduced data volume. Recognizing the pattern of scientific data access – primarily written once and read frequently – we propose to refactor data at the time of writing into a format leading to efficient retrieval. We investigate several data organization and refactoring strategies, assessing their impact on reading performance, writing performance, and error incurred on post-analysis across several commonly used query and post-analyses tasks. Our experiments are conducted on the Frontier Supercomputer at Oak Ridge National Laboratory, providing insights for optimizing I/O operations in the Exascale computing era.

High Performance Computing (HPC) has a demand to streamline incident management workflows while keeping information synchronized between internal tickets and vendor support cases. Prior to HPE’s acquisition of Cray, NERSC created an integration between their ServiceNow incident management platform and the Crayport platform. This is now obsolete once HPE took over Cray and NERSC staff had no choice other than to input information manually every time a new incident was opened or required updating. Further, this manual entry needed to be performed in both ServiceNow and HPE’s platform.

This paper presents a novel integration between ServiceNow, a leading IT service management tool, and Hewlett Packard Enterprise's (HPE) support portal. This approach ensures a synchronized and efficient workflow between the two platforms, enhances visibility and tracking of issues, leads to quicker resolutions and improved service levels and creates an automated method to update incidents. We will detail the development journey, the technical challenges we overcame (including some ServiceNow-specific security concerns), and the operational benefits realized from this integration. Readers of this paper will gain insights into the strategic planning and technical considerations required for such integrations, making them well-equipped to handle similar transitions and integrations in their respective organizations.

The Lightweight Distributed Metric Service (LDMS) is widely used for monitoring HPC systems and is integrated in HPE’s Cray System Management architecture as well as HPE’s High Performance Cluster Management architecture. One of the important components of an HPC system to monitor is the high-speed interconnect. In the case of HPE Cray EX family of systems, that interconnect is the Slingshot high-speed network. LDMS utilizes “samplers” to gather data about network metrics, including some metrics that can only be determined by a sampler running on the Slingshot switches.

Slingshot switches run the Debian Linux distribution, making it possible to deploy LDMS samplers onto the switches to collect real-time metrics. However, these switches are not meant for general purpose development and lack the build tools needed to directly compile LDMS. This presentation will describe the approach that Sandia National Laboratories has taken to create a containerized build environment that we use to build the LDMS Slingshot sampler on any generic ARM system and deploy the build artifacts to our Slingshot switches. Additionally, we will discuss work to integrate this build process into into an automated Continuous Integration / Continuous Development pipeline to automate these builds in the future.