{"id":442,"date":"2023-11-21T08:00:00","date_gmt":"2023-11-21T08:00:00","guid":{"rendered":"https:\/\/christian-engelmann.de\/?page_id=442"},"modified":"2024-11-21T21:35:26","modified_gmt":"2024-11-21T21:35:26","slug":"proactive-fault-tolerance-framework","status":"publish","type":"page","link":"https:\/\/www.christian-engelmann.info\/?page_id=442","title":{"rendered":"Proactive Fault Tolerance Framework"},"content":{"rendered":"\n\n
Summary:<\/b> Proactive fault tolerance is a concept that prevents compute node failures from impacting running parallel applications by preemptively migrating parts of an application (task, process, or virtual machine) away from components that are about to fail. The proactive fault tolerance framework consists of a number of individual proof-of-concept prototypes, including process and virtual machine migration, scalable system monitoring, and online\/offline system health analysis.<\/i><\/td>\n<\/tr>\n<\/table>\n

In order to address anticipated high failure rates, resiliency characteristics have become an urgent priority for next-generation high-performance computing (HPC) systems. The notion of proactive fault tolerance emerged in recent years. It is a concept that prevents compute node failures from impacting running parallel applications by preemptively migrating parts of an application (task, process, or virtual machine) away from nodes that are about to fail. Pre-fault indicators, such as a significant increase in heat, can be used to avoid an imminent failure through anticipation and reconfiguration. As computation is migrated away, application failures are avoided and application mean-time to failure (AMTTF) is extended beyond system mean-time to failure (SMTTF). Since avoiding a failure through preemptive migration is significantly more efficient than recovery from failure via traditional reactive fault tolerance mechanisms, such as checkpoint\/restart, HPC system utilization becomes more efficient.<\/p>\n

The proactive fault tolerance framework (Figure 1) consists of a number of individual proof-of-concept prototypes, including process and virtual machine migration, scalable system monitoring (Figure 2), and online\/offline system health analysis. The novel process-level live migration mechanism (Figure 3) supports continued execution of applications during much of process migration. This scheme is integrated into an Message Passing Interface (MPI) execution environment to transparently sustain health-inflicted node failures, which eradicates the need to restart and requeue MPI jobs. Experiments indicate that 1-6.5 s of prior warning are required to successfully trigger live process migration while similar operating system virtualization mechanisms require 13-24 s. This self-healing approach complements reactive fault tolerance by nearly cutting the number of checkpoints in half when 70% of the faults are handled proactively. The scalable health monitoring system utilizes a tree-based overlay network to classify and aggregate monitoring metrics based on individual needs. The MRNet-based prototype is able to significantly reduce the amount of gathered and stored monitoring data, e.g., by a factor of 56 in comparison to the Ganglia distributed monitoring system. The online\/offline system health analysis uses statistical methods, such as clustering and temporal analysis, to identify pre-fault indicators in the collected health monitoring data and in traditional system logs.<\/p>\n\n\n
<\/td>\n\n
\nFigure 1: Proactive fault tolerance control loop\n<\/td>\n		<\/td>\n\n
\nFigure 2: Scalable system monitoring architecture\n<\/td>\n		<\/td>\n\n
\nFigure 3: Process migration overhead of NAS PB\n<\/td>\n		<\/td>\n<\/tr>\n<\/table>\n

Research Projects<\/h4>\n