{"id":457,"date":"2023-11-21T08:00:24","date_gmt":"2023-11-21T08:00:24","guid":{"rendered":"https:\/\/christian-engelmann.de\/?page_id=457"},"modified":"2024-11-21T21:38:42","modified_gmt":"2024-11-21T21:38:42","slug":"symmetric-active-active-high-availability-for-hpc-system-services","status":"publish","type":"page","link":"https:\/\/www.christian-engelmann.info\/?page_id=457","title":{"rendered":"Symmetric Active\/Active High Availability for HPC System Services"},"content":{"rendered":"\n\n
Summary:<\/b> This solution provides high availability for high-performance computing system services with efficient redundancy strategies for head and service nodes. It relies on virtual synchrony, i.e., state-machine replication, utilizing a process group communication system for service group membership management and for reliable, totally ordered message delivery. The proof-of-concept prototypes offer 99.9997% high availability for the Torque resource manager and the Parallel Virtual File System Metadata Server.<\/i><\/td>\n<\/tr>\n<\/table>\n

This work paves the way for high availability in high-performance computing (HPC) by focusing on efficient redundancy strategies for head and service nodes. These nodes represent single points of failure and control for an entire HPC system as they render it inaccessible and unmanageable in case of a single node failure until manual repair. The approach relies on virtual synchrony, i.e., state-machine replication, utilizing a process group communication system for service group membership management and reliable, totally ordered message delivery. This replication method may be implemented internally, i.e., by modifying the service to be replicated to support redundant instances, or externally, i.e., by wrapping around an unmodified service, replicating input to multiple instances, and unifying output from these instances. Internal replication offers usually more performance, while external replication is typically easier to implement.<\/p>\n

One of the most important HPC system services running on the head node is the job and resource manager. If it goes down, all currently running jobs lose the service they report back to. They have to be restarted once the head node is up and running again. JOSHUA is a generic solution that provides a virtually synchronous environment for continuous availability without any interruption of service and without any loss of state. Replication is performed externally (Figure 1) via the Portable Batch System (PBS) service interface without the need to modify any service code. This makes it portable to most HPC job and resource managers as the PBS service interface is supported widely. The results, as well as, availability analysis of our proof-of-concept prototype implementation show that continuous availability can be provided by JOSHUA with an acceptable performance.<\/p>\n

The metadata service (MDS) of a networked parallel file system is another critical single point of failure. An interruption of service typically results in the failure of currently running applications utilizing its file system. A loss of state requires repairing the entire file system, which could take days on large-scale systems, and may cause loss of data. The developed proof-of-concept prototype for the MDS of the Parallel Virtual File System (PVFS) offers symmetric active\/active replication using virtual synchrony with an internal replication implementation (Figure 2). In addition to providing high availability, this solution is taking advantage of the internal replication implementation by load balancing MDS read requests, improving performance over the non-replicated MDS. The results show that MDS high availability can be achieved with an acceptable performance (Figures 3 and 4).<\/p>\n

Assuming a mean-time to failure of 5,000 hours for a single head or service node, the presented solutions improve service availability from 99.285% (2 nines) of a single node to 99.995% (4 nines) in a two-node system, and to 99.99996% (6 nines) with three nodes (Figure 5). Replication beyond three nodes may not provide further benefits, other than performance increases due to load balancing, since after three nodes, catastrophic incidents, such as earthquakes, hurricanes, and tornados, have a higher impact on availability than component failures.<\/p>\n\n\n
<\/td>\n\n


\nFigure 1: External replication of the PBS service<\/p>\n


\nFigure 2: Internal replication of the PVFS MDS service<\/p>\n<\/td>\n

<\/td>\n\n


\nFigure 3: PVS MDS service read performance<\/p>\n


\nFigure 4: PVS MDS service write performance<\/p>\n<\/td>\n

<\/td>\n\n
\nFigure 5: Service availability improvement\n<\/td>\n		<\/td>\n<\/tr>\n<\/table>\n

Research Projects<\/h4>\n