{"id":462,"date":"2023-11-21T08:00:24","date_gmt":"2023-11-21T08:00:24","guid":{"rendered":"https:\/\/christian-engelmann.de\/?page_id=462"},"modified":"2024-11-21T21:33:22","modified_gmt":"2024-11-21T21:33:22","slug":"hybrid-full-incremental-system-level-checkpointing","status":"publish","type":"page","link":"https:\/\/www.christian-engelmann.info\/?page_id=462","title":{"rendered":"Hybrid Full\/Incremental System-level Checkpointing"},"content":{"rendered":"<table style=\"border:5pt solid grey; padding:10pt;\">\n<tr style=\"border:0pt; padding:0pt;\">\n<td style=\"border:0pt; padding:0pt; text-align:left;\"><i><b>Summary:<\/b> This operating system and runtime environment solution combines scalable group membership management, reuse of network connections, transparent coordinated checkpoint scheduling, a job pause feature, and full\/incremental checkpointing. The job pause allows compute nodes to remain active and roll back parallel applications to the last checkpoint. The hybrid checkpointing alternates between full and incremental checkpoints, where only data is captured that changed since the last checkpoint.<\/i><\/td>\n<\/tr>\n<\/table>\n<p>Checkpoint\/restart has become a requirement for long-running parallel jobs in large-scale high-performance computing (HPC) systems due to a mean-time-to-failure (MTTF) in the order of hours. After a failure, checkpoint\/restart mechanisms generally require a complete restart of a Message Passing Interface (MPI) job from the last saved checkpoint. A complete restart, however, is unnecessary since all but one compute node are typically still alive. Furthermore, a restart may result in lengthy job requeuing even though the original job had not exceeded its time quantum. Moreover, system-level checkpointing solutions capture full process images, even though only a subset of the process image changes between checkpoints.<\/p>\n<p>The developed proof-of-concept prototype includes enhancements in support of scalable group communication for membership management (Figure 1), reuse of network connections, transparent coordinated checkpoint scheduling, a job pause feature, and full\/incremental checkpointing (Figures 2 and 3). It is based on the Local Area Multicomputer MPI implementation (LAM\/MPI) and the Berkeley Lab Checkpoint\/Restart (BLCR) solution. The transparent mechanism for job pause allows live nodes to remain active and roll back to the last checkpoint, while failed nodes are dynamically replaced by spares before resuming from the last checkpoint. A minimal overhead of 5.6% is incurred in case migration takes place, while the regular checkpoint overhead remains unchanged. The hybrid checkpointing technique alternates between full and incremental checkpoints: At incremental checkpoints, only data changed since the last checkpoint is captured. This results in significantly reduced checkpoint sizes and overheads with only moderate increases in restart overhead. After accounting for cost and savings, benefits due to incremental checkpoints are an order of magnitude larger than overheads on restarts.<\/p>\n<table style=\"border:0pt; padding:0pt;\">\n<tr style=\"border:0pt; padding:0pt;\">\n<td style=\"border:0pt; padding:0pt;\" width=\"40\"><\/td>\n<td style=\"border:0pt; padding:0pt; text-align:center;; vertical-align:bottom\">\n<img loading=\"lazy\" decoding=\"async\" src=\"images\/hfichkpt\/low_lat.png\" height=\"188\" width=\"257\" border=\"0\" vspace=\"0\"><br \/>\nFigure 1: Membership stabilization after a failure\n<\/td>\n<td style=\"border:0pt; padding:0pt;\" width=\"40\"><\/td>\n<td style=\"border:0pt; padding:0pt; text-align:center; vertical-align:bottom;\">\n<img decoding=\"async\" loading=\"lazy\" src=\"images\/hfichkpt\/chkpt-file.png\" height=\"125\" width=\"230\" border=\"0\" vspace=\"0\"><br \/>\nFigure 2: Incremental checkpoint file structure\n<\/td>\n<td style=\"border:0pt; padding:0pt;\" width=\"40\"><\/td>\n<td style=\"border:0pt; padding:0pt; text-align:center;; vertical-align:bottom\">\n<img decoding=\"async\" loading=\"lazy\" src=\"images\/hfichkpt\/savings-16.png\" height=\"188\" width=\"315\" border=\"0\" vspace=\"0\"><br \/>\nFigure 3: Hybrid full\/incremental checkpoint savings\n<\/td>\n<td style=\"border:0pt; padding:0pt;\" width=\"40\"><\/td>\n<\/tr>\n<\/table>\n<h4>Research Projects<\/h4>\n<ul>\n<li><a href=\"?page_id=309\">2008-11: Reliability, Availability, and Serviceability (RAS) for Petascale High-End Computing and Beyond<\/a><\/li>\n<li><a href=\"?page_id=359\">2004-07: MOLAR: Modular Linux and Adaptive Runtime Support for High-End Computing<\/a><\/li>\n<\/ul>\n<h4>Peer-reviewed Conference Publications<\/h4>\n<ol>\n<li>Chao Wang, Frank Mueller, Christian Engelmann, and Stephen L. Scott. <b>Hybrid Checkpointing for MPI Jobs in HPC Environments<\/b>. In <i>Proceedings of the <a href=\"http:\/\/grid.sjtu.edu.cn\/icpads10\" target=\"grid.sjtu.edu.cn\/icpads10\">16th IEEE International Conference on Parallel and Distributed Systems (ICPADS) 2010<\/a><\/i>, pages 524-533, Shanghai, China, December 8-10, 2010. <a href=\"http:\/\/www.computer.org\" target=\"www.computer.org\">IEEE Computer Society, Los Alamitos, CA, USA<\/a>. ISBN 978-0-7695-4307-9. DOI <a href=\"http:\/\/dx.doi.org\/10.1109\/ICPADS.2010.48\" target=\"publication\">10.1109\/ICPADS.2010.48<\/a>. Acceptance rate 29.6% (77\/188). <a href=\"javascript:showAbstract('As the core count in high-performance computing systems keeps increasing, faults are becoming common place. Check pointing addresses such faults but captures full process images even though only a subset of the process image changes between checkpoints. We have designed a hybrid check pointing technique for MPI tasks of high-performance applications. This technique alternates between full and incremental checkpoints: At incremental checkpoints, only data changed since the last checkpoint is captured. Our implementation integrates new BLCR and LAM\/MPI features that complement traditional full checkpoints. This results in significantly reduced checkpoint sizes and overheads with only moderate increases in restart overhead. After accounting for cost and savings, benefits due to incremental checkpoints are an order of magnitude larger than overheads on restarts. We further derive qualitative results indicating an optimal balance between full\/incremental checkpoints of our novel approach at a ratio of 1:9, which outperforms both always-full and always-incremental check pointing.');\"><img decoding=\"async\" src=\"images\/txt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Abstract\"><\/a> <a href=\"publications\/wang10hybrid2.pdf\" target=\"publication\"><img decoding=\"async\" src=\"images\/pdf.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Publication\"><\/a> <a href=\"publications\/wang10hybrid2.ppt.pdf\" target=\"publication\"><img decoding=\"async\" src=\"images\/ppt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Presentation\"><\/a> <a href=\"?page_id=55#wang10hybrid2\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<li>Chao Wang, Frank Mueller, Christian Engelmann, and Stephen L. Scott. <b>A Job Pause Service under LAM\/MPI+BLCR for Transparent Fault Tolerance<\/b>. In <i>Proceedings of the <a href=\"http:\/\/www.ipdps.org\/ipdps2007\" target=\"www.ipdps.org\/ipdps2007\">21st IEEE International Parallel and Distributed Processing Symposium (IPDPS) 2007<\/a><\/i>, pages 1-10, Long Beach, CA, USA, March 26-30, 2007. <a href=\"http:\/\/www.acm.org\" target=\"www.acm.org\">ACM Press, New York, NY, USA<\/a>. ISBN 978-1-59593-768-1. DOI <a href=\"http:\/\/dx.doi.org\/10.1109\/IPDPS.2007.370307\" target=\"publication\">10.1109\/IPDPS.2007.370307<\/a>. Acceptance rate 26% (109\/419). <a href=\"javascript:showAbstract('Checkpoint\/restart (C\/R) has become a requirement for long-running jobs in large-scale clusters due to a mean-time-to-failure (MTTF) in the order of hours. After a failure, C\/R mechanisms generally require a complete restart of an MPI job from the last checkpoint. A complete restart, however, is unnecessary since all but one node are typically still alive. Furthermore, a restart may result in lengthy job requeuing even though the original job had not exceeded its time quantum. In this paper, we overcome these shortcomings. Instead of job restart, we have developed a transparent mechanism for job pause within LAM\/MPI+BLCR. This mechanism allows live nodes to remain active and roll back to the last checkpoint while failed nodes are dynamically replaced by spares before resuming from the last checkpoint. Our methodology includes LAM\/MPI enhancements in support of scalable group communication with fluctuating number of nodes, reuse of network connections, transparent coordinated checkpoint scheduling and a BLCR enhancement for job pause. Experiments in a cluster with the NAS Parallel Benchmark suite show that our overhead for job pause is comparable to that of a complete job restart. A minimal overhead of 5.6% is only incurred in case migration takes place while the regular checkpoint overhead remains unchanged. Yet, our approach alleviates the need to reboot the LAM run-time environment, which accounts for considerable overhead resulting in net savings of our scheme in the experiments. Our solution further provides full transparency and automation with the additional benefit of reusing existing resources. Executing continues after failures within the scheduled job, \\em \\textiti.e., the application staging overhead is not incurred again in contrast to a restart. Our scheme offers additional potential for savings through incremental checkpointing and proactive diskless live migration, which we are currently working on.');\"><img decoding=\"async\" src=\"images\/txt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Abstract\"><\/a> <a href=\"publications\/wang07job.pdf\" target=\"publication\"><img decoding=\"async\" src=\"images\/pdf.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Publication\"><\/a> <a href=\"publications\/wang07job.ppt.pdf\" target=\"publication\"><img decoding=\"async\" src=\"images\/ppt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Presentation\"><\/a> <a href=\"?page_id=55#wang07job\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<\/ol>\n<h4>Technical Reports<\/h4>\n<ol>\n<li>Chao Wang, Frank Mueller, Christian Engelmann, and Stephen L. Scott. <b>Hybrid Full\/Incremental Checkpoint\/Restart for MPI Jobs in HPC Environments<\/b>. Technical Report, ORNL\/TM-2010\/162, Oak Ridge National Laboratory, Oak Ridge, TN, USA, August 1, 2010. <a href=\"javascript:showAbstract('As the number of cores in high-performance computing environments keeps increasing, faults are becoming common place. Checkpointing addresses such faults but captures full process images even though only a subset of the process image changes between checkpoints. We have designed a high-performance hybrid disk-based full\/incremental checkpointing technique for MPI tasks to capture only data changed since the last checkpoint. Our implementation integrates new BLCR and LAM\/MPI features that complement traditional full checkpoints. This results in significantly reduced checkpoint sizes and overheads with only moderate increases in restart overhead. After accounting for cost and savings, benefits due to incremental checkpoints significantly outweigh the loss on restart operations. Experiments in a cluster with the NAS Parallel Benchmark suite and mpiBLAST indicate that savings due to replacing full checkpoints with incremental ones average 16.64 seconds while restore overhead amounts to just 1.17 seconds. These savings increase with the frequency of incremental checkpoints. Overall, our novel hybrid full\/incremental checkpointing is superior to prior non-hybrid techniques.');\"><img decoding=\"async\" src=\"images\/txt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Abstract\"><\/a> <a href=\"publications\/wang10hybrid.pdf\" target=\"publication\"><img decoding=\"async\" src=\"images\/pdf.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Publication\"><\/a> <a href=\"?page_id=55#wang10hybrid\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<\/ol>\n<h4>Talks and Lectures<\/h4>\n<ol>\n<li>Christian Engelmann. <b>Resilience Challenges and Solutions for Extreme-Scale Supercomputing<\/b>. Invited talk at the <a href=\"http:\/\/www.usna.edu\" target=\"www.usna.edu\">United  States Naval Academy<\/a>, Annapolis, MD, USA, February 18, 2016. <a href=\"javascript:showAbstract('The path to exascale computing poses several research challenges related to power, performance, resilience, productivity, programmability, data movement, and data management. Resilience, i.e., providing efficiency and correctness in the presence of faults, is one of the most important exascale computer science challenges as systems scale up in component count (100,000-1,000,000 nodes with 1,000-10,000 cores per node by 2022) and component reliability decreases (7 nm technology with near-threshold voltage operation by 2022). This talk provides an overview of recent and ongoing resilience research and development activities at Oak Ridge National Laboratory in advanced checkpoint storage architectures, process-level incremental checkpoint\/restart, proactive fault tolerance using prediction-triggered process or virtual machine migration, MPI process-level software redundancy, and soft-error injection tools to study the vulnerability of science applications.');\"><img decoding=\"async\" src=\"images\/txt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Abstract\"><\/a> <a href=\"publications\/engelmann16resilience2.ppt.pdf\" target=\"publication\"><img decoding=\"async\" src=\"images\/ppt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Presentation\"><\/a> <a href=\"?page_id=55#engelmann16resilience2\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<li>Christian Engelmann. <b>Resilience Challenges and Solutions for Extreme-Scale Supercomputing<\/b>. Invited talk at the  19th Workshop on Distributed Supercomputing (SOS)   2015, Park City, UT, USA, March 2-5, 2015. <a href=\"javascript:showAbstract('The path to exascale computing poses several research challenges related to power, performance, resilience, productivity, programmability, data movement, and data management. Resilience, i.e., providing efficiency and correctness in the presence of faults, is one of the most important exascale computer science challenges as systems scale up in component count (100,000-1,000,000 nodes with 1,000-10,000 cores per node by 2022) and component reliability decreases (7 nm technology with near-threshold voltage operation by 2022). This talk provides an overview of recent and ongoing resilience research and development activities at Oak Ridge National Laboratory in advanced checkpoint storage architectures, process-level incremental checkpoint\/restart, proactive fault tolerance using prediction-triggered process or virtual machine migration, MPI process-level software redundancy, and soft-error injection tools to study the vulnerability of science applications.');\"><img decoding=\"async\" src=\"images\/txt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Abstract\"><\/a> <a href=\"publications\/engelmann15resilience.ppt.pdf\" target=\"publication\"><img decoding=\"async\" src=\"images\/ppt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Presentation\"><\/a> <a href=\"?page_id=55#engelmann15resilience\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<li>Christian Engelmann. <b>Resilience and Hardware\/Software Co-design for Extreme-Scale Supercomputing<\/b>. Seminar at the <a href=\"http:\/\/www.bsc.es\" target=\"www.bsc.es\">Barcelona Supercomputing Center<\/a>, Barcelona, Spain, July 27, 2011. <a href=\"javascript:showAbstract('Oak Ridge National Laboratory (ORNL) provides the most powerful high-performance computing (HPC) resources in the world for open scientific research. Jaguar, a 224,162-core Cray XT5 with a LINPACK performance of 1.759 PFlop\/s, for example, is the world&amp;#39;s 3rd fastest supercomputer. 80% of its resources are allocated through a reviewed process to address the most challenging scientific problems in climate modeling, renewable energy, materials science, fusion and other areas. ORNL's Computer Science and Mathematics Division performs computer science and mathematics research to increase supercomputer efficiency and application scientist productivity while accelerating time to solution for scientific breakthroughs. This talk details recent research advancements at ORNL in two areas: (1) resilience and (2) hardware\/software co-design for extreme-scale supercomputing. Both are essential on the road toward exa-scale HPC systems with millions-to-billions of cores. Due to the expected drastic increase in scale, the corresponding decrease in system mean-time to interrupt warrants a rethinking of the traditional checkpoint\/restart approach for HPC resilience. New concepts discussed in this talk range from preventative measures, such as task migration based on fault prediction, to more aggressive fault masking, such as various levels of redundancy. Further, the expected drastic increase in task parallelism requires redesigning algorithms to avoid the consequences of Amdahl's law at extreme scale. As million-way task parallel systems don't exist yet, this talk discusses a lightweight system simulation approach for performance estimation of algorithms at scale.');\"><img decoding=\"async\" src=\"images\/txt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Abstract\"><\/a> <a href=\"publications\/engelmann11resilience.ppt.pdf\" target=\"publication\"><img decoding=\"async\" src=\"images\/ppt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Presentation\"><\/a> <a href=\"?page_id=55#engelmann11resilience\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<li>Christian Engelmann. <b>Beyond Application-Level Checkpoint\/Restart &#8211; Advanced Software Approaches for Fault Resilience<\/b>. Talk at the <a href=\"http:\/\/www.speedup.ch\/workshops\/w39_2010.html\" target=\"www.speedup.ch\/workshops\/w39_2010.html\">39th SPEEDUP Workshop on High Performance Computing<\/a>, Zurich, Switzerland, September 6, 2010. <a href=\"publications\/engelmann10beyond.ppt.pdf\" target=\"publication\"><img decoding=\"async\" src=\"images\/ppt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Presentation\"><\/a> <a href=\"?page_id=55#engelmann10beyond\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<li>Christian Engelmann and Stephen L. Scott. <b>Reliability, Availability, and Serviceability (RAS) for Petascale High-End Computing and Beyond<\/b>. Talk at the <a href=\"http:\/\/www.usenix.org\/events\/fastos10\" target=\"www.usenix.org\/events\/fastos10\">Forum to Address Scalable Technology for Runtime and Operating Systems (FAST-OS) Workshop<\/a>, in conjunction with the <a href=\"http:\/\/www.usenix.org\/events\/confweek10\" target=\"www.usenix.org\/events\/confweek10\">USENIX Federated Conferences Week (USENIX) 2010<\/a>, Boston MA, USA, June 22, 2010. <a href=\"javascript:showAbstract('This project aims at scalable technologies for providing high-level RAS for next-generation petascale scientific high-performance computing (HPC) resources and beyond as outlined by the U.S. Department of Energy (DOE) Forum to Address Scalable Technology for Runtime and Operating Systems (FAST-OS) and the U.S. National Coordination Office for Networking and Information Technology Research and Development (NCO\/NITRD) High-End Computing Revitalization Task Force (HECRTF) activities. Based on virtualized adaptation, reconfiguration, and preemptive measures, the ultimate goal is to provide for non-stop scientific computing on a 24x7 basis without interruption. The taken technical approach leverages system-level virtualization technology to enable transparent proactive and reactive fault tolerance mechanisms on extreme scale HPC systems. This effort targets: (1) reliability analysis for identifying pre-fault indicators, predicting failures, and modeling and monitoring component and system reliability, (2) proactive fault tolerance technology based on preemptive migration away from components that are about to fail, (3) reactive fault tolerance enhancements, such as checkpoint interval and placement adaptation to actual and predicted system health threats, and (4) holistic fault tolerance through combination of adaptive proactive and reactive fault tolerance.');\"><img decoding=\"async\" src=\"images\/txt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Abstract\"><\/a> <a href=\"publications\/engelmann10reliability.ppt.pdf\" target=\"publication\"><img decoding=\"async\" src=\"images\/ppt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Presentation\"><\/a> <a href=\"?page_id=55#engelmann10reliability\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<li>Christian Engelmann. <b>System Resilience Research at ORNL in the Context of HPC<\/b>. Invited talk at the <a href=\"http:\/\/www.inria.fr\/inria\/organigramme\/fiche_ur-ren.fr.html\" target=\"www.inria.fr\/inria\/organigramme\/fiche_ur-ren.fr.html\">Institut National de Recherche en Informatique et en Automatique (INRIA)<\/a>, Rennes, France, May 15, 2009. <a href=\"javascript:showAbstract('The continuing growth in high performance computing (HPC) system scale poses a challenge for system software and scientific applications with respect to reliability, availability and serviceability (RAS). With only very few exceptions, the availability of recently installed systems has been lower in comparison to the same deployment phase of their predecessors. As a result, sites lower allowable job run times in order to force applications to store intermediate results (checkpoints) as insurance against lost computation time. However, checkpoints themselves waste valuable computation time and resources. In contrast to the experienced loss of availability, the demand for continuous availability has risen dramatically with the trend towards capability computing, which drives the race for scientific discovery by running applications on the fastest machines available while desiring significant amounts of time (weeks and months) without interruption. These machines must be able to run in the event of frequent interrupts in such a manner that the capability is not severely degraded. Thus, research and development of scalable RAS technologies is paramount to the success of future extreme-scale systems. This talk summarizes our accomplishments in the area of high-level RAS for HPC, such as developed concepts and implemented proof-of-concept prototypes.');\"><img decoding=\"async\" src=\"images\/txt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Abstract\"><\/a> <a href=\"publications\/engelmann09system.ppt.pdf\" target=\"publication\"><img decoding=\"async\" src=\"images\/ppt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Presentation\"><\/a> <a href=\"?page_id=55#engelmann09system\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<li>Christian Engelmann. <b>High-Performance Computing Research at Oak Ridge National Laboratory<\/b>. Invited talk at the Reading Annual Computational Science  Workshop, Reading, United Kingdom, December 8, 2008. <a href=\"javascript:showAbstract('Oak Ridge National Laboratory (ORNL) is the largest energy laboratory in the United States. Its National Center for Computational Sciences (NCCS) provides the most powerful computing resources in the world for open scientific research. Jaguar, a Cray XT5 system at NCCS, is the second HPC system to exceed 1 PFlop\/s (10^15 Floating Point Operations Per Second), and the fastest open science supercomputer in the world. It recently ranked #2 in the Top 500 List of Supercomputer Sites with a maximal LINPACK benchmark performance of 1.059 PFlop\/s and a theoretical peak performance of 1.3814 PFlop\/s. Annually, 80 percent of Jaguar\u2019s resources are allocated through the U.S Department of Energy\u2019s Innovative and Novel Computational Impact on Theory and Experiment (INCITE) program, a competitively selected, peer reviewed process open to researchers from universities, industry, government and non-profit organizations. These allocations address some of the most challenging scientific problems in areas such as climate modeling, renewable energy, materials science, fusion and combustion. In conjunction with NCCS, the Computer Science and Mathematics Division at ORNL performs basic and applied research in HPC, mathematics, and intelligent systems. This talk gives a summary of the HPC research performed at ORNL. It provides details about the Jaguar peta-scale computing resource, an overview of the computational science research carried out using ORNL\u2019s computing resources, and a description of various computer science efforts targeting solutions for next-generation HPC systems.');\"><img decoding=\"async\" src=\"images\/txt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Abstract\"><\/a> <a href=\"publications\/engelmann08high.ppt.pdf\" target=\"publication\"><img decoding=\"async\" src=\"images\/ppt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Presentation\"><\/a> <a href=\"?page_id=55#engelmann08high\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<li>Christian Engelmann. <b>Resiliency for High-Performance Computing<\/b>. Invited talk at the <a href=\"http:\/\/acet.rdg.ac.uk\/events\/details\/cancun.php\" target=\"acet.rdg.ac.uk\/events\/details\/cancun.php\">2nd Collaborative and Grid Computing Technologies Workshop (CGCTW) 2008<\/a>, Cancun, Mexico, April 10-12, 2008. <a href=\"javascript:showAbstract('In order to address anticipated high failure rates, resiliency characteristics have become an urgent priority for next-generation high-performance computing (HPC) systems. One major source of concern are non-recoverable soft errors, i.e., bit flips in memory, cache, registers, and logic. The probability of such errors not only grows with system size, but also with increasing architectural vulnerability caused by employing accelerators and by shrinking nanometer technology. Reactive fault tolerance technologies, such as checkpoint\/restart, are unable to handle high failure rates due to associated overheads, while proactive resiliency technologies, such as preemptive migration, simply fail as random soft errors can&amp;#39;t be predicted. This talk proposes a new, bold direction in resiliency for HPC as it targets resiliency for next-generation extreme-scale HPC systems at the system software level through computational redundancy strategies, i.e., dual- and triple-modular redundancy.');\"><img decoding=\"async\" src=\"images\/txt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Abstract\"><\/a> <a href=\"publications\/engelmann08resiliency.ppt.pdf\" target=\"publication\"><img decoding=\"async\" src=\"images\/ppt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Presentation\"><\/a> <a href=\"?page_id=55#engelmann08resiliency\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<\/ol>\n<p><em><small>Symbols: <img decoding=\"async\" style=\"border-style: none;\" src=\"images\/txt.gif\" border=\"0\" alt=\"Abstract\" height=\"10pt\"> Abstract, <img decoding=\"async\" style=\"border-style: none;\" src=\"images\/pdf.gif\" border=\"0\" alt=\"Publication\" height=\"10pt\"> Publication, <img decoding=\"async\" style=\"border-style: none;\" src=\"images\/ppt.gif\" border=\"0\" alt=\"Presentation\" height=\"10pt\"> Presentation, <img decoding=\"async\" style=\"border-style: none;\" src=\"images\/bib.gif\" border=\"0\" alt=\"BibTeX Citation\" height=\"10pt\"> BibTeX Citation<\/small><\/em><\/p>\n<p><script language=\"JavaScript\">\nfunction showAbstract (text) {\n  var width  = 400;\n  var height = 400;\n  var left   = (screen.width  - width ) \/ 2;\n  var top    = (screen.height - height) \/ 2;\n  var win    = window.open('',\n                           'Abstract',\n                           'width='  + width  + ', ' + \n                           'height=' + height + ', ' +\n                           'left='   + left   + ', ' +\n                           'top='    + top    + ', ' +\n                           'toolbar=no, '     +\n                           'location=no, '    +\n                           'directories=no, ' +\n                           'status=no, '      +\n                           'menubar=no, '     +\n                           'copyhistory=no, ' +\n                           'scrollbars=yes, ' +\n                           'resizable=yes')\n  win.document.write(text);\n  win.document.close();\n}\n<\/script><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Summary: This operating system and runtime environment solution combines scalable group membership management, reuse of network connections, transparent coordinated checkpoint scheduling, a job pause feature, and full\/incremental checkpointing. The job pause allows compute nodes to remain active and roll back parallel applications to the last checkpoint. The hybrid checkpointing alternates between full and incremental checkpoints,&hellip;&nbsp;<\/p>\n","protected":false},"author":1,"featured_media":0,"parent":430,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"neve_meta_sidebar":"","neve_meta_container":"","neve_meta_enable_content_width":"off","neve_meta_content_width":100,"neve_meta_title_alignment":"","neve_meta_author_avatar":"","neve_post_elements_order":"","neve_meta_disable_header":"","neve_meta_disable_footer":"","neve_meta_disable_title":"","footnotes":""},"class_list":["post-462","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/www.christian-engelmann.info\/index.php?rest_route=\/wp\/v2\/pages\/462","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.christian-engelmann.info\/index.php?rest_route=\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/www.christian-engelmann.info\/index.php?rest_route=\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/www.christian-engelmann.info\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.christian-engelmann.info\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=462"}],"version-history":[{"count":4,"href":"https:\/\/www.christian-engelmann.info\/index.php?rest_route=\/wp\/v2\/pages\/462\/revisions"}],"predecessor-version":[{"id":1220,"href":"https:\/\/www.christian-engelmann.info\/index.php?rest_route=\/wp\/v2\/pages\/462\/revisions\/1220"}],"up":[{"embeddable":true,"href":"https:\/\/www.christian-engelmann.info\/index.php?rest_route=\/wp\/v2\/pages\/430"}],"wp:attachment":[{"href":"https:\/\/www.christian-engelmann.info\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=462"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}