{"id":453,"date":"2023-11-21T08:00:23","date_gmt":"2023-11-21T08:00:23","guid":{"rendered":"https:\/\/christian-engelmann.de\/?page_id=453"},"modified":"2024-11-21T21:36:16","modified_gmt":"2024-11-21T21:36:16","slug":"redmpi-a-redundant-message-passing-interface-implementation","status":"publish","type":"page","link":"https:\/\/www.christian-engelmann.info\/?page_id=453","title":{"rendered":"redMPI: A Redundant Message Passing Interface Implementation"},"content":{"rendered":"\n\n
Summary:<\/b> RedMPI enables transparent redundant execution of Message Passing Interface (MPI) applications to protect against silent data corruption, i.e., undetected bit flips. It can be also used as a fault injection tool by disabling the online error correction and keeping replicas isolated, comparing error-free and erroneous execution using the online error detection mechanism to track propagation of corrupt messages.<\/i><\/td>\n<\/tr>\n<\/table>\n

As systems scale up in component count and nanometer process technology shrinks, soft errors are becoming the predominant source of interruptions in large-scale high-performance computing (HPC) systems. Double-error detection (DED) events that normally occur in a memory module with single-error correction (SEC) error correcting code (ECC) once within 1-2 million hours of operation can cause an error rate of 10-20 hours in a system with 100,000 modules. Moreover, vendors have warned that silent data corruption (SDC), i.e., undetected bit flips, are becoming a problem as well. In general, software redundancy is able to transparently mask reported errors, such as detected hard and soft errors, without recovery. It is also able to detect silent errors, like SDC, through comparison and recover them using voting if more than two replicas exist.<\/p>\n

RedMPI is a prototype that enables transparent redundant execution of Message Passing Interface (MPI) applications. It is based on two earlier prototypes, MR-MPI, developed by Oak Ridge National Laboratory and rMPI, developed by Sandia National Laboratory. RedMPI sits between the MPI library and the MPI application, utilizing the MPI performance tool interface, PMPI, to intercept MPI calls from the application and to hide all redundancy-related mechanisms (Figure 1). A redundantly executed application runs with r*m MPI processes, where r is the number of MPI ranks visible to the application and m is the replication degree. RedMPI supports partial replication, e.g., a degree of 2.5 instead of 2 or 3, for tunable resilience. It also supports a variety of message-based replication protocols with different consistency (Figure 2). Results indicate that the most efficient consistency protocol can successfully protect HPC applications even from high SDC rates with runtime overheads between 0% and 30%, compared to unprotected applications without redundancy.<\/p>\n

RedMPI can be also used as a fault injection tool by disabling the online error correction and keeping replicas isolated (Figure 3). A failure free execution can be compared to the redundant execution with an injected fault using the online error detection mechanism to track propagation of corrupt messages. Depending on the application properties, a single bit flip can corrupt all MPI processes of an application within a short period of time, or may be corrected by the application’s computational structure, such as by an iterative algorithm.<\/p>\n\n\n
<\/td>\n\n
\nFigure 1: redMPI Architecture\n<\/td>\n		<\/td>\n\n
\nFigure 2: Message+hash replication protocol\n<\/td>\n		<\/td>\n\n
\nFigure 3: Injecting bit-flips into the NAS LU benchmark\n<\/td>\n		<\/td>\n<\/tr>\n<\/table>\n

Research Projects<\/h4>\n