{"id":309,"date":"2023-02-25T08:00:19","date_gmt":"2023-02-25T08:00:19","guid":{"rendered":"https:\/\/christian-engelmann.de\/?page_id=309"},"modified":"2023-02-26T17:05:38","modified_gmt":"2023-02-26T17:05:38","slug":"2008-11-reliability-availability-and-serviceability-ras-for-petascale-high-end-computing-and-beyond","status":"publish","type":"page","link":"https:\/\/www.christian-engelmann.info\/?page_id=309","title":{"rendered":"2008-11: Reliability, Availability, and Serviceability (RAS) for Petascale High-End Computing and Beyond"},"content":{"rendered":"<p>This project aims at scalable technologies for providing high-level RAS for next-generation petascale scientific high-end computing (HEC) resources and beyond as outlined by the U.S. Department of Energy (DOE) Forum to Address Scalable Technology for Runtime and Operating Systems (FAST-OS) and the U.S. National Coordination Office for Networking and Information Technology Research and Development (NCO\/NITRD) High-End Computing Revitalization Task Force (HECRTF) activities. Based on virtualized adaptation, reconfiguration, and preemptive measures, the ultimate goal is to provide for non-stop scientific computing on a 24\u00d77 basis without interruption.<\/p>\n<p>The taken technical approach leverages system-level virtualization technology to enable transparent proactive and reactive fault tolerance mechanisms on extreme scale HEC systems. This effort targets: (1) reliability analysis for identifying pre-fault indicators, predicting failures, and modeling and monitoring component and system reliability, (2) proactive fault tolerance technology based on preemptive migration away from components that are about to fail, (3) reactive fault tolerance enhancements, such as checkpoint interval and placement adaption to actual and predicted system health threats, and (4) holistic fault tolerance through combination of adaptive proactive and reactive fault tolerance.<\/p>\n<h4>Prominent Solutions<\/h4>\n<ul>\n<li><a href=\"?page_id=522\">Characterization of Faults, Errors, and Failures in Extreme-Scale Systems<\/a><\/li>\n<li><a href=\"?page_id=442\">Proactive Fault Tolerance Framework<\/a><\/li>\n<li><a href=\"?page_id=462\">Hybrid Full\/Incremental System-level Checkpointing<\/a><\/li>\n<\/ul>\n<h4>Funding Sources<\/h4>\n<ul>\n<li>Forum to Address Scalable Technology for Runtime and Operating Systems (FAST-OS) Program, <a href=\"http:\/\/science.energy.gov\/ascr\" target=\"science.energy.gov_ascr\" rel=\"noopener\">Office of Advanced Scientific Computing Research<\/a>, Office of Science, U.S. Department of Energy<\/li>\n<\/ul>\n<h4>Participating Institutions<\/h4>\n<ul>\n<li><a href=\"http:\/\/www.ornl.gov\" target=\"www.ornl.gov\" rel=\"noopener\">Oak Ridge National Laboratory<\/a><\/li>\n<li><a href=\"http:\/\/www.ncsu.edu\" target=\"www.ncsu.edu\" rel=\"noopener\">North Carolina State University<\/a><\/li>\n<li><a href=\"http:\/\/www.latech.edu\" target=\"www.latech.edu\" rel=\"noopener\">Louisiana Tech University<\/a><\/li>\n<li><a href=\"http:\/\/www.tntech.edu\" target=\"www.tntech.edu\" rel=\"noopener\">Tennessee Tech University<\/a><\/li>\n<\/ul>\n<h4>Peer-reviewed Journal Publications<\/h4>\n<ol>\n<li>Chao Wang, Frank Mueller, Christian Engelmann, and Stephen L. Scott. <b>Proactive Process-Level Live Migration and Back Migration in HPC Environments<\/b>. <i><a href=\"http:\/\/www.elsevier.com\/locate\/jpdc\" target=\"www.elsevier.com\/locate\/jpdc\" rel=\"noopener\">Journal of Parallel and Distributed Computing (JPDC)<\/a><\/i>, volume 72, number 2, pages 254-267, February 1, 2012. <a href=\"http:\/\/www.elsevier.com\" target=\"www.elsevier.com\" rel=\"noopener\">Elsevier B.V, Amsterdam, The Netherlands<\/a>. ISSN 0743-7315. DOI <a href=\"http:\/\/dx.doi.org\/10.1016\/j.jpdc.2011.10.009\" target=\"publication\" rel=\"noopener\">10.1016\/j.jpdc.2011.10.009<\/a>. <a href=\"javascript:showAbstract('As the number of nodes in high-performance computing environments keeps increasing, faults are becoming common place. Reactive fault tolerance (FT) often does not scale due to massive I\/O requirements and relies on manual job resubmission. This work complements reactive with proactive FT at the process level. Through health monitoring, a subset of node failures can be anticipated when one&amp;#39;s health deteriorates. A novel process-level live migration mechanism supports continued execution of applications during much of process migration. This scheme is integrated into an MPI execution environment to transparently sustain health-inflicted node failures, which eradicates the need to restart and requeue MPI jobs. Experiments indicate that 1-6.5 s of prior warning are required to successfully trigger live process migration while similar operating system virtualization mechanisms require 13-24 s. This self-healing approach complements reactive FT by nearly cutting the number of checkpoints in half when 70% of the faults are handled proactively. The work also provides a novel back migration approach to eliminate load imbalance or bottlenecks caused by migrated tasks. Experiments indicate the larger the amount of outstanding execution, the higher the benefit due to back migration.');\"><img decoding=\"async\" src=\"images\/txt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Abstract\"><\/a> <a href=\"publications\/wang12proactive.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/pdf.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Publication\"><\/a> <a href=\"?page_id=55#wang12proactive\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<\/ol>\n<h4>Peer-reviewed Conference Publications<\/h4>\n<ol>\n<li>Chao Wang, Frank Mueller, Christian Engelmann, and Stephen L. Scott. <b>Hybrid Checkpointing for MPI Jobs in HPC Environments<\/b>. In <i>Proceedings of the <a href=\"http:\/\/grid.sjtu.edu.cn\/icpads10\" target=\"grid.sjtu.edu.cn\/icpads10\" rel=\"noopener\">16th IEEE International Conference on Parallel and Distributed Systems (ICPADS) 2010<\/a><\/i>, pages 524-533, Shanghai, China, December 8-10, 2010. <a href=\"http:\/\/www.computer.org\" target=\"www.computer.org\" rel=\"noopener\">IEEE Computer Society, Los Alamitos, CA, USA<\/a>. ISBN 978-0-7695-4307-9. DOI <a href=\"http:\/\/dx.doi.org\/10.1109\/ICPADS.2010.48\" target=\"publication\" rel=\"noopener\">10.1109\/ICPADS.2010.48<\/a>. Acceptance rate 29.6% (77\/188). <a href=\"javascript:showAbstract('As the core count in high-performance computing systems keeps increasing, faults are becoming common place. Check pointing addresses such faults but captures full process images even though only a subset of the process image changes between checkpoints. We have designed a hybrid check pointing technique for MPI tasks of high-performance applications. This technique alternates between full and incremental checkpoints: At incremental checkpoints, only data changed since the last checkpoint is captured. Our implementation integrates new BLCR and LAM\/MPI features that complement traditional full checkpoints. This results in significantly reduced checkpoint sizes and overheads with only moderate increases in restart overhead. After accounting for cost and savings, benefits due to incremental checkpoints are an order of magnitude larger than overheads on restarts. We further derive qualitative results indicating an optimal balance between full\/incremental checkpoints of our novel approach at a ratio of 1:9, which outperforms both always-full and always-incremental check pointing.');\"><img decoding=\"async\" src=\"images\/txt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Abstract\"><\/a> <a href=\"publications\/wang10hybrid2.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/pdf.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Publication\"><\/a> <a href=\"publications\/wang10hybrid2.ppt.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/ppt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Presentation\"><\/a> <a href=\"?page_id=55#wang10hybrid2\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<li>Swen B&ouml;hm, Christian Engelmann, and Stephen L. Scott. <b>Aggregation of Real-Time System Monitoring Data for Analyzing Large-Scale Parallel and Distributed Computing Environments<\/b>. In <i>Proceedings of the <a href=\"http:\/\/www.anss.org.au\/hpcc2010\" target=\"www.anss.org.au\/hpcc2010\" rel=\"noopener\">12th IEEE International Conference on High Performance Computing and Communications (HPCC) 2010<\/a><\/i>, pages 72-78, Melbourne, Australia, September 1-3, 2010. <a href=\"http:\/\/www.computer.org\" target=\"www.computer.org\" rel=\"noopener\">IEEE Computer Society, Los Alamitos, CA, USA<\/a>. ISBN 978-0-7695-4214-0. DOI <a href=\"http:\/\/dx.doi.org\/10.1109\/HPCC.2010.32\" target=\"publication\" rel=\"noopener\">10.1109\/HPCC.2010.32<\/a>. Acceptance rate 19.1% (58\/304). <a href=\"javascript:showAbstract('We present a monitoring system for large-scale parallel and distributed computing environments that allows to trade-off accuracy in a tunable fashion to gain scalability without compromising fidelity. The approach relies on classifying each gathered monitoring metric based on individual needs and on aggregating messages containing classes of individual monitoring metrics using a tree-based overlay network. The MRNet-based prototype is able to significantly reduce the amount of gathered and stored monitoring data, e.g., by a factor of  56 in comparison to the Ganglia distributed monitoring system. A simple scaling study reveals, however, that further efforts are needed in reducing the amount of data to monitor future-generation extreme-scale systems with up to 1,000,000 nodes. The implemented solution did not had a measurable performance impact as the 32-node test system did not produce enough monitoring data to interfere with running applications.');\"><img decoding=\"async\" src=\"images\/txt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Abstract\"><\/a> <a href=\"publications\/boehm10aggregation.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/pdf.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Publication\"><\/a> <a href=\"publications\/boehm10aggregation.ppt.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/ppt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Presentation\"><\/a> <a href=\"?page_id=55#boehm10aggregation\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<li>Antonina Litvinova, Christian Engelmann, and Stephen L. Scott. <b>A Proactive Fault Tolerance Framework for High-Performance Computing<\/b>. In <i>Proceedings of the <a href=\"http:\/\/www.iasted.org\/conferences\/home-676.html\" target=\"www.iasted.org\/conferences\/home-676.html\" rel=\"noopener\">9th IASTED International Conference on Parallel and Distributed Computing and Networks (PDCN) 2010<\/a><\/i>, Innsbruck, Austria, February 16-18, 2010. <a href=\"http:\/\/www.actapress.com\" target=\"www.actapress.com\" rel=\"noopener\">ACTA Press, Calgary, AB, Canada<\/a>. ISBN 978-0-88986-783-3. DOI <a href=\"http:\/\/dx.doi.org\/10.2316\/P.2010.676-024\" target=\"publication\" rel=\"noopener\">10.2316\/P.2010.676-024<\/a>. <a href=\"javascript:showAbstract('As high-performance computing (HPC) systems continue to increase in scale, their mean-time to interrupt decreases respectively. The current state of practice for fault tolerance (FT) is checkpoint\/restart. However, with increasing error rates, increasing aggregate memory and not proportionally increasing I\/O capabilities, it is becoming less efficient. Proactive FT avoids experiencing failures through preventative measures, such as by migrating application parts away from nodes that are about to fail. This paper presents a proactive FT framework that performs environmental monitoring, event logging, parallel job monitoring and resource monitoring to analyze HPC system reliability and to perform FT through such preventative actions.');\"><img decoding=\"async\" src=\"images\/txt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Abstract\"><\/a> <a href=\"publications\/litvinova10proactive.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/pdf.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Publication\"><\/a> <a href=\"publications\/litvinova10proactive.ppt.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/ppt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Presentation\"><\/a> <a href=\"?page_id=55#litvinova10proactive\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<li>Narate Taerat, Nichamon Naksinehaboon, Clayton Chandler, James Elliott, Chokchai (Box) Leangsuksun, George Ostrouchov, Stephen L. Scott, and Christian Engelmann. <b>Blue Gene\/L Log Analysis and Time to Interrupt Estimation<\/b>. In <i>Proceedings of the <a href=\"http:\/\/www.ares-conference.eu\/ares2009\" target=\"www.ares-conference.eu\/ares2009\" rel=\"noopener\">4th International Conference on Availability, Reliability and Security (ARES) 2009<\/a><\/i>, pages 173-180, Fukuoka, Japan, March 16-19, 2009. <a href=\"http:\/\/www.computer.org\" target=\"www.computer.org\" rel=\"noopener\">IEEE Computer Society, Los Alamitos, CA, USA<\/a>. ISBN 978-1-4244-3572-2. DOI <a href=\"http:\/\/dx.doi.org\/10.1109\/ARES.2009.105\" target=\"publication\" rel=\"noopener\">10.1109\/ARES.2009.105<\/a>. Acceptance rate 25.0% (40\/160). <a href=\"javascript:showAbstract('System- and application-level failures could be characterized by analyzing relevant log files. The resulting data might then be used in numerous studies on and future developments for the mission-critical and large scale computational architecture, including fields such as failure prediction, reliability modeling, performance modeling and power awareness. In this paper, system logs covering a six month period of the Blue Gene\/L supercomputer were obtained and subsequently analyzed. Temporal filtering was applied to remove duplicated log messages. Optimistic and pessimistic perspectives were exerted on filtered log information to observe failure behavior within the system. Further, various time to repair factors were applied to obtain application time to interrupt, which will be exploited in further resilience modeling research.');\"><img decoding=\"async\" src=\"images\/txt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Abstract\"><\/a> <a href=\"publications\/taerat09blue.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/pdf.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Publication\"><\/a> <a href=\"?page_id=55#taerat09blue\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<li>Christian Engelmann, Hong H. Ong, and Stephen L. Scott. <b>Evaluating the Shared Root File System Approach for Diskless High-Performance Computing Systems<\/b>. In <i>Proceedings of the <a href=\"http:\/\/www.linuxclustersinstitute.org\/conferences\" target=\"www.linuxclustersinstitute.org\/conferences\" rel=\"noopener\">10th LCI International Conference on High-Performance Clustered Computing (LCI) 2009<\/a><\/i>, Boulder, CO, USA, March 9-12, 2009. <a href=\"javascript:showAbstract('Diskless high-performance computing (HPC) systems utilizing networked storage have become popular in the last several years. Removing disk drives significantly increases compute node reliability as they are known to be a major source of failures. Furthermore, networked storage solutions utilizing parallel I\/O and replication are able to provide increased scalability and availability. Reducing a compute node to processor(s), memory and network interface(s) greatly reduces its physical size, which in turn allows for large-scale dense HPC solutions. However, one major obstacle is the requirement by certain operating systems (OSs), such as Linux, for a root file system. While one solution is to remove this requirement from the OS, another is to share the root file system over the networked storage. This paper evaluates three networked file system solutions, NFSv4, Lustre and PVFS2, with respect to their performance, scalability, and availability features for servicing a common root file system in a diskless HPC configuration. Our findings indicate that Lustre is a viable solution as it meets both, scaling and performance requirements. However, certain availability issues regarding single points of failure and control need to be considered.');\"><img decoding=\"async\" src=\"images\/txt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Abstract\"><\/a> <a href=\"publications\/engelmann09evaluating.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/pdf.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Publication\"><\/a> <a href=\"publications\/engelmann09evaluating.ppt.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/ppt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Presentation\"><\/a> <a href=\"?page_id=55#engelmann09evaluating\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<li>Christian Engelmann, Geoffroy R. Vall&eacute;e, Thomas Naughton, and Stephen L. Scott. <b>Proactive Fault Tolerance Using Preemptive Migration<\/b>. In <i>Proceedings of the <a href=\"http:\/\/www.pdp2009.org\" target=\"www.pdp2009.org\" rel=\"noopener\">17th Euromicro International Conference on Parallel, Distributed, and network-based Processing (PDP) 2009<\/a><\/i>, pages 252-257, Weimar, Germany, February 18-20, 2009. <a href=\"http:\/\/www.computer.org\" target=\"www.computer.org\" rel=\"noopener\">IEEE Computer Society, Los Alamitos, CA, USA<\/a>. ISBN 978-0-7695-3544-9. ISSN 1066-6192. DOI <a href=\"http:\/\/dx.doi.org\/10.1109\/PDP.2009.31\" target=\"publication\" rel=\"noopener\">10.1109\/PDP.2009.31<\/a>. Acceptance rate 42.0% (58\/138). <a href=\"javascript:showAbstract('Proactive fault tolerance (FT) in high-performance computing is a concept that prevents compute node failures from impacting running parallel applications by preemptively migrating application parts away from nodes that are about to fail. This paper provides a foundation for proactive FT by defining its architecture and classifying implementation options. This paper further relates prior work to the presented architecture and classification, and discusses the challenges ahead for needed supporting technologies.');\"><img decoding=\"async\" src=\"images\/txt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Abstract\"><\/a> <a href=\"publications\/engelmann09proactive.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/pdf.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Publication\"><\/a> <a href=\"publications\/engelmann09proactive.ppt.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/ppt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Presentation\"><\/a> <a href=\"?page_id=55#engelmann09proactive\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<li>Chao Wang, Frank Mueller, Christian Engelmann, and Stephen L. Scott. <b>Proactive Process-Level Live Migration in HPC Environments<\/b>. In <i>Proceedings of the <a href=\"http:\/\/sc08.supercomputing.org\" target=\"sc08.supercomputing.org\" rel=\"noopener\">21st IEEE\/ACM International Conference on High Performance Computing, Networking, Storage and Analysis (SC) 2008<\/a><\/i>, pages 1-12, Austin, TX, USA, November 15-21, 2008. <a href=\"http:\/\/www.acm.org\" target=\"www.acm.org\" rel=\"noopener\">ACM Press, New York, NY, USA<\/a>. ISBN 978-1-4244-2835-9. DOI <a href=\"http:\/\/dx.doi.org\/10.1145\/1413370.1413414\" target=\"publication\" rel=\"noopener\">10.1145\/1413370.1413414<\/a>. Acceptance rate 21.3% (59\/277). <a href=\"javascript:showAbstract('As the number of nodes in high-performance computing environments keeps increasing, faults are becoming common place. Reactive fault tolerance (FT) often does not scale due to massive I\/O requirements and relies on manual job resubmission. This work complements reactive with proactive FT at the process level. Through health monitoring, a subset of node failures can be anticipated when one&amp;#39;s health deteriorates. A novel process-level live migration mechanism supports continued execution of applications during much of processes migration. This scheme is integrated into an MPI execution environment to transparently sustain health-inflicted node failures, which eradicates the need to restart and requeue MPI jobs. Experiments indicate that 1-6.5 seconds of prior warning are required to successfully trigger live process migration while similar operating system virtualization mechanisms require 13-24 seconds. This self-healing approach complements reactive FT by nearly cutting the number of checkpoints in half when 70% of the faults are handled proactively.');\"><img decoding=\"async\" src=\"images\/txt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Abstract\"><\/a> <a href=\"publications\/wang08proactive.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/pdf.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Publication\"><\/a> <a href=\"publications\/wang08proactive.ppt.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/ppt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Presentation\"><\/a> <a href=\"?page_id=55#wang08proactive\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<li>Geoffroy R. Vall&eacute;e, Kulathep Charoenpornwattana, Christian Engelmann, Anand Tikotekar, Chokchai (Box) Leangsuksun, Thomas Naughton, and Stephen L. Scott. <b>A Framework For Proactive Fault Tolerance<\/b>. In <i>Proceedings of the <a href=\"http:\/\/www.ares-conference.eu\/ares2008\" target=\"www.ares-conference.eu\/ares2008\" rel=\"noopener\">3rd International Conference on Availability, Reliability and Security (ARES) 2008<\/a><\/i>, pages 659-664, Barcelona, Spain, March 4-7, 2008. <a href=\"http:\/\/www.computer.org\" target=\"www.computer.org\" rel=\"noopener\">IEEE Computer Society, Los Alamitos, CA, USA<\/a>. ISBN 978-0-7695-3102-1. DOI <a href=\"http:\/\/dx.doi.org\/10.1109\/ARES.2008.171\" target=\"publication\" rel=\"noopener\">10.1109\/ARES.2008.171<\/a>. Acceptance rate 21.1% (40\/190). <a href=\"javascript:showAbstract('Fault tolerance is a major concern to guarantee availability of critical services as well as application execution. Traditional approaches for fault tolerance include checkpoint\/restart or duplication. However it is also possible to anticipate failures and proactively take action before failures occur in order to minimize failure impact on the system and application execution. This document presents a proactive fault tolerance framework. This framework can use different proactive fault tolerance mechanisms, i.e. migration and pause\/unpause. The framework also allows the implementation of new proactive fault tolerance policies thanks to a modular architecture. A first proactive fault tolerance policy has been implemented and preliminary experimentations have been done based on system-level virtualization and compared with results obtained by simulation.');\"><img decoding=\"async\" src=\"images\/txt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Abstract\"><\/a> <a href=\"publications\/vallee08framework.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/pdf.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Publication\"><\/a> <a href=\"publications\/vallee08framework.ppt.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/ppt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Presentation\"><\/a> <a href=\"?page_id=55#vallee08framework\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<\/ol>\n<h4>Peer-reviewed Workshop Publications<\/h4>\n<ol>\n<li>Thomas Naughton, Swen B&ouml;hm, Christian Engelmann, and Geoffroy Vall&eacute;e. <b>Using Performance Tools to Support Experiments in HPC Resilience<\/b>. In <i>Lecture Notes in Computer Science: Proceedings of the <a href=\"http:\/\/www.europar2013.org\/\" target=\"www.europar2013.org\/\" rel=\"noopener\">19th European Conference on Parallel and Distributed Computing (Euro-Par) 2013 Workshops<\/a>: <a href=\"http:\/\/xcr.cenit.latech.edu\/resilience2013\" target=\"xcr.cenit.latech.edu\/resilience2013\" rel=\"noopener\">6th Workshop on Resiliency in High Performance Computing (Resilience) in Clusters, Clouds, and Grids<\/a><\/i>, pages 727-736, Aachen, Germany, August 26, 2013. <a href=\"http:\/\/www.springer.com\" target=\"www.springer.com\" rel=\"noopener\">Springer Verlag, Berlin, Germany<\/a>. ISBN 978-3-642-54419-4. ISSN 0302-9743. DOI <a href=\"http:\/\/dx.doi.org\/10.1007\/978-3-642-54420-0_71\" target=\"publication\" rel=\"noopener\">10.1007\/978-3-642-54420-0_71<\/a>. Acceptance rate 87.5% (7\/8). <a href=\"javascript:showAbstract('The high performance computing (HPC) community is working to address fault tolerance and resilience concerns for current and future large scale computing platforms. This is driving enhancements in the programming environments, specifically research on enhancing message passing libraries to support fault tolerant computing capabilities. The community has also recognized that tools for resilience experimentation are greatly lacking. However, we argue that there are several parallels between &amp;#34;performance tools&amp;#34; and ``resilience tools&amp;#39;'. As such, we believe the rich set of HPC performance-focused tools can be extended (repurposed) to benefit the resilience community. In this paper, we describe the initial motivation to leverage standard HPC performance analysis techniques to aid in developing diagnostic tools to assist fault tolerance experiments for HPC applications. These diagnosis procedures help to provide context for the system when the errors (failures) occurred. We describe our initial work in leveraging an MPI performance trace tool to assist in providing global context during fault injection experiments. Such tools will assist the HPC resilience community as they extend existing and new application codes to support fault tolerances.');\"><img decoding=\"async\" src=\"images\/txt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Abstract\"><\/a> <a href=\"publications\/naughton13using.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/pdf.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Publication\"><\/a> <a href=\"publications\/naughton13using.ppt.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/ppt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Presentation\"><\/a> <a href=\"?page_id=55#naughton13using\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<li>Thomas Naughton, Geoffroy R. Vall&eacute;e, Christian Engelmann, and Stephen L. Scott. <b>A Case for Virtual Machine based Fault Injection in a High-Performance Computing Environment<\/b>. In <i>Lecture Notes in Computer Science: Proceedings of the <a href=\"http:\/\/europar2011.bordeaux.inria.fr\/\" target=\"europar2011.bordeaux.inria.fr\/\" rel=\"noopener\">17th European Conference on Parallel and Distributed Computing (Euro-Par) 2011<\/a>: <a href=\"http:\/\/www.csm.ornl.gov\/srt\/conferences\/hpcvirt2011\" target=\"www.csm.ornl.gov\/srt\/conferences\/hpcvirt2011\" rel=\"noopener\">5th Workshop on System-level Virtualization for High Performance Computing (HPCVirt)<\/a><\/i>, pages 234-243, Bordeaux, France, August 29 &#8211; September 2, 2011. <a href=\"http:\/\/www.springer.com\" target=\"www.springer.com\" rel=\"noopener\">Springer Verlag, Berlin, Germany<\/a>. ISBN 978-3-642-29737. DOI <a href=\"http:\/\/dx.doi.org\/10.1007\/978-3-642-29737-3_27\" target=\"publication\" rel=\"noopener\">10.1007\/978-3-642-29737-3_27<\/a>. <a href=\"javascript:showAbstract('Large-scale computing platforms provide tremendous capabilities for scientific discovery. These systems have hundreds of thousands of computing cores, hundreds of terabytes of memory, and enormous high-performance interconnection networks. These systems are facing enormous challenges to achieve performance at such scale. Failures are an Achilles heel of these enormous systems. As applications and system software scale up to multi-petaflop and beyond to exascale platforms, the occurrence of failure will be much more common. This has given rise to a push in fault-tolerance and resilience research for HPC systems. This includes work on log analysis to identify types of failures, enhancements to the Message Passing Interface (MPI) to incorporate fault awareness, and a variety of fault tolerance mechanisms that span redundant computation, algorithm based fault tolerance, and advanced checkpoint\/ restart techniques. While there is much work to be done on the FT\/Resilience mechanisms for such large-scale systems, there is also a profound gap in the tools for experimentation. This gap is compounded by the fact that HPC environments have stringent performance requirements and are often highly customized. The tool chain for these systems are often tailored for the platform and while the majority of systems on the Top500 Supercomputer list run Linux, these operating environments typically contain many site\/machine specific enhancements. Therefore, it is desirable to maintain a consistent execution environment to minimize end-user (scientist) interruption. The work on system-level virtualization for HPC system offers a unique opportunity to maintain a consistent execution environment via a virtual machine (VM). Recent work on virtualization for HPC has shown that low-overhead, high performance systems can be realized [1, 2] Virtualization also provides a clean abstraction for building experimental tools for investigation into the effects of failures in HPC and the related research on FT\/ Resilience mechanisms and policies. In this paper we discuss the motivation for tools to perform fault injection in an HPC context, and outline an approach that can leverage virtualization.');\"><img decoding=\"async\" src=\"images\/txt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Abstract\"><\/a> <a href=\"publications\/naughton11case.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/pdf.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Publication\"><\/a> <a href=\"publications\/naughton11case.ppt.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/ppt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Presentation\"><\/a> <a href=\"?page_id=55#naughton11case\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<li>George Ostrouchov, Thomas Naughton, Christian Engelmann, Geoffroy R. Vall&eacute;e, and Stephen L. Scott. <b>Nonparametric Multivariate Anomaly Analysis in Support of HPC Resilience<\/b>. In <i>Proceedings of the <a href=\"http:\/\/www.oerc.ox.ac.uk\/ieee\" target=\"www.oerc.ox.ac.uk\/ieee\" rel=\"noopener\">5th IEEE International Conference on e-Science (e-Science) 2009<\/a>: <a href=\"http:\/\/www.oerc.ox.ac.uk\/ieee\/workshops\/workshops\/computational-science\" target=\"www.oerc.ox.ac.uk\/ieee\/workshops\/workshops\/computational-science\" rel=\"noopener\">Workshop on Computational Science<\/a><\/i>, pages 80-85, Oxford, UK, December 9-11, 2009. <a href=\"http:\/\/www.computer.org\" target=\"www.computer.org\" rel=\"noopener\">IEEE Computer Society, Los Alamitos, CA, USA<\/a>. ISBN 978-1-4244-5946-9. DOI <a href=\"http:\/\/dx.doi.org\/10.1109\/ESCIW.2009.5407992\" target=\"publication\" rel=\"noopener\">10.1109\/ESCIW.2009.5407992<\/a>. <a href=\"javascript:showAbstract('Large-scale computing systems provide great potential for scientific exploration. However, the complexity that accompanies these enormous machines raises challeges for both, users and operators. The effective use of such systems is often hampered by failures encountered when running applications on systems containing tens-of-thousands of nodes and hundreds-of-thousands of compute cores capable of yielding petaflops of performance. In systems of this size failure detection is complicated and root-cause diagnosis difficult. This paper describes our recent work in the identification of anomalies in monitoring data and system logs to provide further insights into machine status, runtime behavior, failure modes and failure root causes. It discusses the details of an initial prototype that gathers the data and uses statistical techniques for analysis.');\"><img decoding=\"async\" src=\"images\/txt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Abstract\"><\/a> <a href=\"publications\/ostrouchov09nonparametric.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/pdf.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Publication\"><\/a> <a href=\"publications\/ostrouchov09nonparametric.ppt.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/ppt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Presentation\"><\/a> <a href=\"?page_id=55#ostrouchov09nonparametric\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<li>Thomas Naughton, Wesley Bland, Geoffroy R. Vall&eacute;e, Christian Engelmann, and Stephen L. Scott. <b>Fault Injection Framework for System Resilience Evaluation &#8211; Fake Faults for Finding Future Failures<\/b>. In <i>Proceedings of the <a href=\"http:\/\/www.lrz-muenchen.de\/hpdc2009\" target=\"www.lrz-muenchen.de\/hpdc2009\" rel=\"noopener\">18th International Symposium on High Performance Distributed Computing (HPDC) 2009<\/a>: <a href=\"http:\/\/xcr.cenit.latech.edu\/resilience2009\" target=\"xcr.cenit.latech.edu\/resilience2009\" rel=\"noopener\">2nd Workshop on Resiliency in High Performance Computing (Resilience) 2009<\/a><\/i>, pages 23-28, Munich, Germany, June 9, 2009. <a href=\"http:\/\/www.acm.org\" target=\"www.acm.org\" rel=\"noopener\">ACM Press, New York, NY, USA<\/a>. ISBN 978-1-60558-587-1. DOI <a href=\"http:\/\/dx.doi.org\/10.1145\/1552526.1552530\" target=\"publication\" rel=\"noopener\">10.1145\/1552526.1552530<\/a>. <a href=\"javascript:showAbstract('As high-performance computing (HPC) systems increase in size and complexity they become more difficult to manage. The enormous component counts associated with these large systems lead to significant challenges in system reliability and availability. This in turn is driving research into the resilience of large scale systems, which seeks to curb the effects of increased failures at large scales by masking the inevitable faults in these systems. The basic premise being that failure must be accepted as a reality of large scale system and coped with accordingly through system resilience. A key component in the development and evaluation of system resilience techniques is having a means to conduct controlled experiments. A common method for performing such experiments is to generate synthetic faults and study the resulting effects. In this paper we discuss the motivation and our initial use of software fault injection to support the evaluation of resilience for HPC systems. We mention background and related work in the area and discuss the design of a tool to aid in fault injection experiments for both user-space (application-level) and system-level failures.');\"><img decoding=\"async\" src=\"images\/txt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Abstract\"><\/a> <a href=\"publications\/naughton09fault.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/pdf.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Publication\"><\/a> <a href=\"publications\/naughton09fault.ppt.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/ppt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Presentation\"><\/a> <a href=\"?page_id=55#naughton09fault\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<\/ol>\n<h4>Peer-reviewed Conference Posters<\/h4>\n<ol>\n<li>Stephen L. Scott, Christian Engelmann, Geoffroy R. Vall&eacute;e, Thomas Naughton, Anand Tikotekar, George Ostrouchov, Chokchai (Box) Leangsuksun, Nichamon Naksinehaboon, Raja Nassar, Mihaela Paun, Frank Mueller, Chao Wang, Arun B. Nagarajan, and Jyothish Varma. <b>A Tunable Holistic Resiliency Approach for High-Performance Computing Systems<\/b>. Poster at the <a href=\"http:\/\/institute.lanl.gov\/resilience\/conferences\/2009\" target=\"institute.lanl.gov\/resilience\/conferences\/2009\" rel=\"noopener\">National HPC Workshop on Resilience 2009<\/a>, Arlington, VA, USA, August 12-14, 2009. <a href=\"javascript:showAbstract('In order to address anticipated high failure rates, resiliency characteristics have become an urgent priority for next-generation extreme-scale high-performance computing (HPC) systems. This poster describes our past and ongoing efforts in novel fault resilience technologies for HPC. Presented work includes proactive fault resilience techniques, system and application reliability models and analyses, failure prediction, transparent process- and virtual-machine-level migration, and trade-off models for combining preemptive migration with checkpoint\/restart. This poster summarizes our work and puts all individual technologies into context with a proposed holistic fault resilience framework.');\"><img decoding=\"async\" src=\"images\/txt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Abstract\"><\/a> <a href=\"publications\/scott09tunable2.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/pdf.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Publication\"><\/a> <a href=\"?page_id=55#scott09tunable2\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<li>Stephen L. Scott, Christian Engelmann, Geoffroy R. Vall&eacute;e, Thomas Naughton, Anand Tikotekar, George Ostrouchov, Chokchai (Box) Leangsuksun, Nichamon Naksinehaboon, Raja Nassar, Mihaela Paun, Frank Mueller, Chao Wang, Arun B. Nagarajan, and Jyothish Varma. <b>A Tunable Holistic Resiliency Approach for High-Performance Computing Systems<\/b>. Poster at the <a href=\"http:\/\/ppopp09.rice.edu\" target=\"ppopp09.rice.edu\" rel=\"noopener\">14th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming (PPoPP) 2009<\/a>, Raleigh, NC, USA, February 14-18, 2009. <a href=\"javascript:showAbstract('In order to address anticipated high failure rates, resiliency characteristics have become an urgent priority for next-generation extreme-scale high-performance computing (HPC) systems. This poster describes our past and ongoing efforts in novel fault resilience technologies for HPC. Presented work includes proactive fault resilience techniques, system and application reliability models and analyses, failure prediction, transparent process- and virtual-machine-level migration, and trade-off models for combining preemptive migration with checkpoint\/restart. This poster summarizes our work and puts all individual technologies into context with a proposed holistic fault resilience framework.');\"><img decoding=\"async\" src=\"images\/txt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Abstract\"><\/a> <a href=\"publications\/scott09tunable.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/pdf.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Publication\"><\/a> <a href=\"?page_id=55#scott09tunable\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<\/ol>\n<h4>White Papers<\/h4>\n<ol>\n<li>Christian Engelmann, Geoffroy R. Vall&eacute;e, Thomas Naughton, and Frank Mueller. <b>Dynamic Self-Aware Runtime Software for Exascale Systems<\/b>. <i>White paper for the U.S. Department of Energy&#39;s <a href=\"http:\/\/collab.cels.anl.gov\/display\/exaosr\/Position+Papers\" target=\"collab.cels.anl.gov\/display\/exaosr\/Position+Papers\" rel=\"noopener\">Exascale Operating Systems and Runtime Technical Council<\/a><\/i>, July 1, 2012. <a href=\"javascript:showAbstract('At exascale, the power consumption, resilience, and load balancing constraints, especially their dynamic nature and interdependence, and the scale of the system require a radical change in future high-performance computing (HPC) operating systems and runtimes (OS\/Rs). In contrast to the existing static OS\/R solutions, an exascale OS\/R is needed that is aware of the dynamically changing resources, constraints, and application needs, and that is able to autonomously coordinate (sometimes conflicting) responses to different changes in the system, simultaneously and at scale. To provide awareness and autonomic management, a novel, scalable and self-aware OS\/R is needed that becomes the brains of the entire X-stack. It dynamically analyzes past, current, and future system status and application needs. It optimizes system usage by scheduling, migrating, and restarting tasks within and across nodes as needed to deal with multi-dimensional constraints, such as power consumption, permanent and transient faults, resource degradation, heterogeneity, data locality, and load balance.');\"><img decoding=\"async\" src=\"images\/txt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Abstract\"><\/a> <a href=\"publications\/engelmann12dynamic.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/pdf.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Publication\"><\/a> <a href=\"publications\/engelmann12dynamic.ppt.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/ppt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Presentation\"><\/a> <a href=\"?page_id=55#engelmann12dynamic\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<li>Nathan DeBardeleben, James Laros, John T. Daly, Stephen L. Scott, Christian Engelmann, and Bill Harrod. <b>High-End Computing Resilience: Analysis of Issues Facing the HEC Community and Path-Forward for Research and Development<\/b>. <i>White paper for the U.S. National Science Foundation&#39;s High-end Computing Program<\/i>, December 1, 2009. <a href=\"publications\/debardeleben09high-end.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/pdf.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Publication\"><\/a> <a href=\"?page_id=55#debardeleben09high-end\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<\/ol>\n<h4>Technical Reports<\/h4>\n<ol>\n<li>Chao Wang, Frank Mueller, Christian Engelmann, and Stephen L. Scott. <b>Hybrid Full\/Incremental Checkpoint\/Restart for MPI Jobs in HPC Environments<\/b>. Technical Report, ORNL\/TM-2010\/162, Oak Ridge National Laboratory, Oak Ridge, TN, USA, August 1, 2010. <a href=\"javascript:showAbstract('As the number of cores in high-performance computing environments keeps increasing, faults are becoming common place. Checkpointing addresses such faults but captures full process images even though only a subset of the process image changes between checkpoints. We have designed a high-performance hybrid disk-based full\/incremental checkpointing technique for MPI tasks to capture only data changed since the last checkpoint. Our implementation integrates new BLCR and LAM\/MPI features that complement traditional full checkpoints. This results in significantly reduced checkpoint sizes and overheads with only moderate increases in restart overhead. After accounting for cost and savings, benefits due to incremental checkpoints significantly outweigh the loss on restart operations. Experiments in a cluster with the NAS Parallel Benchmark suite and mpiBLAST indicate that savings due to replacing full checkpoints with incremental ones average 16.64 seconds while restore overhead amounts to just 1.17 seconds. These savings increase with the frequency of incremental checkpoints. Overall, our novel hybrid full\/incremental checkpointing is superior to prior non-hybrid techniques.');\"><img decoding=\"async\" src=\"images\/txt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Abstract\"><\/a> <a href=\"publications\/wang10hybrid.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/pdf.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Publication\"><\/a> <a href=\"?page_id=55#wang10hybrid\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<li>Chao Wang, Frank Mueller, Christian Engelmann, and Stephen L. Scott. <b>Proactive Process-Level Live Migration and Back Migration in HPC Environments<\/b>. Technical Report, ORNL\/TM-2010\/161, Oak Ridge National Laboratory, Oak Ridge, TN, USA, August 1, 2010. <a href=\"javascript:showAbstract('As the number of nodes in high-performance computing environments keeps increasing, faults are becoming common place. Reactive fault tolerance (FT) often does not scale due to massive I\/O requirements and relies on manual job resubmission. This work complements reactive with proactive FT at the process level. Through health monitoring, a subset of node failures can be anticipated when one&amp;#39;s health deteriorates. A novel process-level live migration mechanism suppor ts continued execution of applications during much of processes migration. This scheme is integrated into an MPI execution environment to transparently sustain health-inflicted node failures, which eradicates the need to restart and requeue MPI jobs. Experiments indicate that 1-6.5 seconds of prior warning are required to successfully trigger live process migration while similar operating system virtualization mechanisms require 13-24 seconds. This self-healing approach complements reactive FT by nearly cutting the number of checkpoints in half when 70% of the faults are handled proactively. The work also provides a novel back migration approach to eliminate load imbalance or bottlenecks caused by migrated tasks. Experiments indicate the larger the amount of outstanding execution, the higher the benefit due to back migration will be.');\"><img decoding=\"async\" src=\"images\/txt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Abstract\"><\/a> <a href=\"publications\/wang10proactive.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/pdf.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Publication\"><\/a> <a href=\"?page_id=55#wang10proactive\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<\/ol>\n<h4>Talks and Lectures<\/h4>\n<ol>\n<li>Christian Engelmann. <b>Resilience Challenges and Solutions for Extreme-Scale Supercomputing<\/b>. Invited talk at the <a href=\"http:\/\/www.usna.edu\" target=\"www.usna.edu\" rel=\"noopener\">United  States Naval Academy<\/a>, Annapolis, MD, USA, February 18, 2016. <a href=\"javascript:showAbstract('The path to exascale computing poses several research challenges related to power, performance, resilience, productivity, programmability, data movement, and data management. Resilience, i.e., providing efficiency and correctness in the presence of faults, is one of the most important exascale computer science challenges as systems scale up in component count (100,000-1,000,000 nodes with 1,000-10,000 cores per node by 2022) and component reliability decreases (7 nm technology with near-threshold voltage operation by 2022). This talk provides an overview of recent and ongoing resilience research and development activities at Oak Ridge National Laboratory in advanced checkpoint storage architectures, process-level incremental checkpoint\/restart, proactive fault tolerance using prediction-triggered process or virtual machine migration, MPI process-level software redundancy, and soft-error injection tools to study the vulnerability of science applications.');\"><img decoding=\"async\" src=\"images\/txt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Abstract\"><\/a> <a href=\"publications\/engelmann16resilience2.ppt.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/ppt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Presentation\"><\/a> <a href=\"?page_id=55#engelmann16resilience2\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<li>Christian Engelmann. <b>Resilience Challenges and Solutions for Extreme-Scale Supercomputing<\/b>. Invited talk at the  19th Workshop on Distributed Supercomputing (SOS)   2015, Park City, UT, USA, March 2-5, 2015. <a href=\"javascript:showAbstract('The path to exascale computing poses several research challenges related to power, performance, resilience, productivity, programmability, data movement, and data management. Resilience, i.e., providing efficiency and correctness in the presence of faults, is one of the most important exascale computer science challenges as systems scale up in component count (100,000-1,000,000 nodes with 1,000-10,000 cores per node by 2022) and component reliability decreases (7 nm technology with near-threshold voltage operation by 2022). This talk provides an overview of recent and ongoing resilience research and development activities at Oak Ridge National Laboratory in advanced checkpoint storage architectures, process-level incremental checkpoint\/restart, proactive fault tolerance using prediction-triggered process or virtual machine migration, MPI process-level software redundancy, and soft-error injection tools to study the vulnerability of science applications.');\"><img decoding=\"async\" src=\"images\/txt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Abstract\"><\/a> <a href=\"publications\/engelmann15resilience.ppt.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/ppt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Presentation\"><\/a> <a href=\"?page_id=55#engelmann15resilience\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<li>Christian Engelmann. <b>Resilience Challenges and Solutions for Extreme-Scale Supercomputing<\/b>. Invited talk at the Technical University of Dresden,  Dresden, Germany, September 3, 2013. <a href=\"javascript:showAbstract('With the recent deployment of the 18 PFlop\/s Titan supercomputer and the exascale roadmap targeting 100, 300, and eventually 1,000 PFlop\/s by 2022, Oak Ridge National Laboratory is at the forefront of scientific capability computing. The path to exascale computing poses several research challenges related to power, performance, resilience, productivity, programmability, data movement, and data management. Resilience, i.e., providing efficiency and correctness in the presence of faults, is one of the most important exascale computer science challenges as systems scale up in component count (100,000-1,000,000 nodes with 1,000-10,000 cores per node by 2022) and component reliability decreases (7 nm technology with near-threshold voltage operation by 2022). This talk provides an overview of recent and ongoing resilience research and development activities at Oak Ridge National Laboratory in advanced checkpoint storage architectures, process-level incremental checkpoint\/restart, proactive fault tolerance using prediction-triggered process or virtual machine migration, MPI process-level software redundancy, and soft-error injection tools to study the vulnerability of science applications and of CMOS logic in processors and memory.');\"><img decoding=\"async\" src=\"images\/txt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Abstract\"><\/a> <a href=\"publications\/engelmann13resilience.ppt.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/ppt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Presentation\"><\/a> <a href=\"?page_id=55#engelmann13resilience\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<li>Christian Engelmann. <b>High-End Computing Resilience: Analysis of Issues Facing the HEC Community and Path Forward for Research and Development<\/b>. Invited talk at the Argonne National Laboratory (ANL)  Institute of Computing in Science (ICiS)  <a href=\"http:\/\/www.icis.anl.gov\/programs\/summer2012-4b\" target=\"www.icis.anl.gov\/programs\/summer2012-4b\" rel=\"noopener\">Summer Workshop Week on Addressing Failures in Exascale   Computing<\/a>, Park City, UT, USA, August 4-11, 2012. <a href=\"javascript:showAbstract('The path to exascale computing poses several research challenges related to power, performance, resilience, productivity, programmability, data movement, and data management. Resilience, i.e., providing efficiency and correctness in the presence of faults, is one of the most important exascale computer science challenges as systems scale up in component count (100,000-1,000,000 nodes with 1,000-10,000 cores per node by 2020) and component reliability decreases (7 nm technology with near-threshold voltage operation by 2020). To provide input for a discussion of future needs in resilience research, development, and standards work, this talk gives a brief summary of the outcomes from the National HPC Workshop on Resilience, held in Arlington, VA, USA on August 12-14, 2009.');\"><img decoding=\"async\" src=\"images\/txt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Abstract\"><\/a> <a href=\"publications\/engelmann12high-end.ppt.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/ppt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Presentation\"><\/a> <a href=\"?page_id=55#engelmann12high-end\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<li>Christian Engelmann. <b>Resilient Software for ExaScale Computing<\/b>. Invited talk at the Birds of a Feather Session on Resilient Software for ExaScale Computing at the <a href=\"http:\/\/sc11.supercomputing.org\" target=\"sc11.supercomputing.org\" rel=\"noopener\">24th IEEE\/ACM International Conference on High Performance  Computing, Networking, Storage and Analysis (SC) 2011<\/a>, Seattle, WA, USA, November 17, 2011. <a href=\"javascript:showAbstract('ExaScale computing systems will likely consist of millions of cores executing applications with billions of threads, based on 14nm or less CMOS technology, according to the ITRS roadmap. Processing elements built on this technology, coupled with dynamic power management will exhibit high variability in performance, between cores and across different runs. Even worse, preliminary figures indicates that on average about every couple of minutes - at least - something in the system will break. Traditional checkpointing strategies are unlikely to work, given the time it will take to save the huge quantities of data combined with the fact that they will need to be restored frequently. This BoF wants to investigate resilient software: software that is able to survive failing hardware and continue to run, without minimal performance impact. Furthermore, we may also discuss tradeoffs between rerunning the application and the cost of instrumentation to deal with resilience.');\"><img decoding=\"async\" src=\"images\/txt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Abstract\"><\/a> <a href=\"publications\/engelmann11resilient.ppt.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/ppt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Presentation\"><\/a> <a href=\"?page_id=55#engelmann11resilient\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<li>Christian Engelmann. <b>Resilience and Hardware\/Software Co-design for Extreme-Scale Supercomputing<\/b>. Seminar at the <a href=\"http:\/\/www.bsc.es\" target=\"www.bsc.es\" rel=\"noopener\">Barcelona Supercomputing Center<\/a>, Barcelona, Spain, July 27, 2011. <a href=\"javascript:showAbstract('Oak Ridge National Laboratory (ORNL) provides the most powerful high-performance computing (HPC) resources in the world for open scientific research. Jaguar, a 224,162-core Cray XT5 with a LINPACK performance of 1.759 PFlop\/s, for example, is the world&amp;#39;s 3rd fastest supercomputer. 80% of its resources are allocated through a reviewed process to address the most challenging scientific problems in climate modeling, renewable energy, materials science, fusion and other areas. ORNL's Computer Science and Mathematics Division performs computer science and mathematics research to increase supercomputer efficiency and application scientist productivity while accelerating time to solution for scientific breakthroughs. This talk details recent research advancements at ORNL in two areas: (1) resilience and (2) hardware\/software co-design for extreme-scale supercomputing. Both are essential on the road toward exa-scale HPC systems with millions-to-billions of cores. Due to the expected drastic increase in scale, the corresponding decrease in system mean-time to interrupt warrants a rethinking of the traditional checkpoint\/restart approach for HPC resilience. New concepts discussed in this talk range from preventative measures, such as task migration based on fault prediction, to more aggressive fault masking, such as various levels of redundancy. Further, the expected drastic increase in task parallelism requires redesigning algorithms to avoid the consequences of Amdahl's law at extreme scale. As million-way task parallel systems don't exist yet, this talk discusses a lightweight system simulation approach for performance estimation of algorithms at scale.');\"><img decoding=\"async\" src=\"images\/txt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Abstract\"><\/a> <a href=\"publications\/engelmann11resilience.ppt.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/ppt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Presentation\"><\/a> <a href=\"?page_id=55#engelmann11resilience\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<li>Christian Engelmann. <b>Scalable HPC System Monitoring<\/b>. Invited talk at the 3rd HPC Resiliency Summit: Workshop on Resiliency for Petascale HPC 2010, in conjunction with the <a href=\"http:\/\/www.lanl.gov\/conferences\/lacss\/2010\" target=\"www.lanl.gov\/conferences\/lacss\/2010\" rel=\"noopener\">3rd Los Alamos Computer Science Symposium (LACSS) 2010<\/a>, Santa Fe, NM, USA, October 13, 2010. <a href=\"javascript:showAbstract('We present a monitoring system for large-scale parallel and distributed computing environments that allows to trade-off accuracy in a tunable fashion to gain scalability without compromising fidelity. The approach relies on classifying each gathered monitoring metric based on individual needs and on aggregating messages containing classes of individual monitoring metrics using a tree-based overlay network. The MRNet-based prototype is able to significantly reduce the amount of gathered and stored monitoring data, e.g., by a factor of  56 in comparison to the Ganglia distributed monitoring system. A simple scaling study reveals, however, that further efforts are needed in reducing the amount of data to monitor future-generation extreme-scale systems with up to 1,000,000 nodes. The implemented solution did not had a measurable performance impact as the 32-node test system did not produce enough monitoring data to interfere with running applications.');\"><img decoding=\"async\" src=\"images\/txt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Abstract\"><\/a> <a href=\"publications\/engelmann10scalable.ppt.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/ppt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Presentation\"><\/a> <a href=\"?page_id=55#engelmann10scalable\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<li>Christian Engelmann. <b>Beyond Application-Level Checkpoint\/Restart &#8211; Advanced Software Approaches for Fault Resilience<\/b>. Talk at the <a href=\"http:\/\/www.speedup.ch\/workshops\/w39_2010.html\" target=\"www.speedup.ch\/workshops\/w39_2010.html\" rel=\"noopener\">39th SPEEDUP Workshop on High Performance Computing<\/a>, Zurich, Switzerland, September 6, 2010. <a href=\"publications\/engelmann10beyond.ppt.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/ppt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Presentation\"><\/a> <a href=\"?page_id=55#engelmann10beyond\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<li>Christian Engelmann and Stephen L. Scott. <b>Reliability, Availability, and Serviceability (RAS) for Petascale High-End Computing and Beyond<\/b>. Talk at the <a href=\"http:\/\/www.usenix.org\/events\/fastos10\" target=\"www.usenix.org\/events\/fastos10\" rel=\"noopener\">Forum to Address Scalable Technology for Runtime and Operating Systems (FAST-OS) Workshop<\/a>, in conjunction with the <a href=\"http:\/\/www.usenix.org\/events\/confweek10\" target=\"www.usenix.org\/events\/confweek10\" rel=\"noopener\">USENIX Federated Conferences Week (USENIX) 2010<\/a>, Boston MA, USA, June 22, 2010. <a href=\"javascript:showAbstract('This project aims at scalable technologies for providing high-level RAS for next-generation petascale scientific high-performance computing (HPC) resources and beyond as outlined by the U.S. Department of Energy (DOE) Forum to Address Scalable Technology for Runtime and Operating Systems (FAST-OS) and the U.S. National Coordination Office for Networking and Information Technology Research and Development (NCO\/NITRD) High-End Computing Revitalization Task Force (HECRTF) activities. Based on virtualized adaptation, reconfiguration, and preemptive measures, the ultimate goal is to provide for non-stop scientific computing on a 24x7 basis without interruption. The taken technical approach leverages system-level virtualization technology to enable transparent proactive and reactive fault tolerance mechanisms on extreme scale HPC systems. This effort targets: (1) reliability analysis for identifying pre-fault indicators, predicting failures, and modeling and monitoring component and system reliability, (2) proactive fault tolerance technology based on preemptive migration away from components that are about to fail, (3) reactive fault tolerance enhancements, such as checkpoint interval and placement adaptation to actual and predicted system health threats, and (4) holistic fault tolerance through combination of adaptive proactive and reactive fault tolerance.');\"><img decoding=\"async\" src=\"images\/txt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Abstract\"><\/a> <a href=\"publications\/engelmann10reliability.ppt.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/ppt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Presentation\"><\/a> <a href=\"?page_id=55#engelmann10reliability\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<li>Christian Engelmann. <b>Resilience Challenges at the Exascale<\/b>. Talk at the <a href=\"http:\/\/www.csm.ornl.gov\/workshops\/SOS14\" target=\"www.csm.ornl.gov\/workshops\/SOS14\" rel=\"noopener\">14th Workshop on Distributed Supercomputing (SOS) 2010<\/a>, Savannah, GA, USA, March 8-11, 2010. <a href=\"javascript:showAbstract('The path to exascale computing poses several research challenges related to power, performance, resilience, productivity, programmability, data movement, and data management. Resilience, i.e., providing efficiency and correctness in the presence of faults, is one of the most important exascale computer science challenges as systems scale up in component count and component reliability decreases. This talk discusses the future needs in resilience research, development, and standards work based on the outcomes from the National HPC Workshop on Resilience, held in Arlington, VA, USA on August 12-14, 2009.');\"><img decoding=\"async\" src=\"images\/txt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Abstract\"><\/a> <a href=\"publications\/engelmann10resilience.ppt.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/ppt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Presentation\"><\/a> <a href=\"?page_id=55#engelmann10resilience\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<li>Christian Engelmann and Stephen L. Scott. <b>HPC System Software Research at Oak Ridge National Laboratory<\/b>. Seminar at the <a href=\"http:\/\/www.lrz-muenchen.de\" target=\"www.lrz-muenchen.de\" rel=\"noopener\">Leibniz  Rechenzentrum (LRZ)<\/a>, Garching, Germany, February 22, 2010. <a href=\"javascript:showAbstract('Oak Ridge National Laboratory (ORNL) is the largest energy laboratory in the United States. Its National Center for Computational Sciences (NCCS) provides the most powerful computing resources in the world for open scientific research. Jaguar, a Cray XT5 system at NCCS, is the fastest supercomputer in the world. It recently ranked #1 in the Top 500 List of Supercomputer Sites with a maximal LINPACK benchmark performance of 1.759 PFlop\/s and a theoretical peak performance of 2.331 PFlop\/s, where 1 PFlop\/s is 10^15 Floating Point Operations Per Second. Annually, 80 percent of Jaguar&amp;#39;s resources are allocated through the U.S Department of Energy's Innovative and Novel Computational Impact on Theory and Experiment (INCITE) program, a competitively selected, peer reviewed process open to researchers from universities, industry, government and non-profit organizations. These allocations address some of the most challenging scientific problems in areas such as climate modeling, renewable energy, materials science, fusion and combustion. In conjunction with NCCS, the Computer Science and Mathematics Division at ORNL performs basic and applied research in HPC, mathematics, and intelligent systems. This talk gives a summary of the HPC research and development in system software performed at ORNL, including resilience at extreme scale and virtualization technologies in HPC. Specifically, this talk will focus on advanced resilience technologies, such as migration of computation away from components that are about to fail and on management and customization of virtualized environments.');\"><img decoding=\"async\" src=\"images\/txt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Abstract\"><\/a> <a href=\"publications\/engelmann10hpc.ppt.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/ppt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Presentation\"><\/a> <a href=\"?page_id=55#engelmann10hpc\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<li>Christian Engelmann. <b>High-Performance Computing Research Internship and Appointment Opportunities at Oak Ridge National Laboratory<\/b>. Seminar at the <a href=\"http:\/\/www.cs.reading.ac.uk\" target=\"www.cs.reading.ac.uk\" rel=\"noopener\">Department of Computer Science<\/a>, <a href=\"http:\/\/www.reading.ac.uk\" target=\"www.reading.ac.uk\" rel=\"noopener\">University of Reading<\/a>, Reading, United Kingdom, December 14, 2009. <a href=\"javascript:showAbstract('Oak Ridge National Laboratory (ORNL) is the largest energy laboratory in the United States. Its National Center for Computational Sciences (NCCS) provides the most powerful computing resources in the world for open scientific research. Jaguar, a Cray XT5 system at NCCS, is the fastest supercomputer in the world. It recently ranked #1 in the Top 500 List of Supercomputer Sites with a maximal LINPACK benchmark performance of 1.759 PFlop\/s and a theoretical peak performance of 2.331 PFlop\/s, where 1 PFlop\/s is 10^15 Floating Point Operations Per Second. Annually, 80 percent of Jaguar&amp;#39;s resources are allocated through the U.S Department of Energy's Innovative and Novel Computational Impact on  Theory and Experiment (INCITE) program, a competitively selected, peer reviewed process open to researchers from universities, industry, government and non-profit organizations. These allocations address some of the most challenging scientific problems in areas such as climate modeling, renewable energy, materials science, fusion and combustion. In conjunction with NCCS, the Computer Science and Mathematics Division at ORNL performs basic and applied research in HPC, mathematics, and intelligent systems. This talk gives a summary of the HPC research performed at ORNL. It provides details about the Jaguar peta-scale computing resource, an overview of the computational science research carried out using ORNL's computing resources, and a description of various computer science efforts targeting solutions for next-generation HPC systems. This talk also provides information about internship opportunities for MSc students and research appointment opportunities for recent graduates.');\"><img decoding=\"async\" src=\"images\/txt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Abstract\"><\/a> <a href=\"publications\/engelmann09high2.ppt.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/ppt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Presentation\"><\/a> <a href=\"?page_id=55#engelmann09high2\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<li>Christian Engelmann. <b>Modeling Techniques Towards Resilience<\/b>. Invited talk at the <a href=\"http:\/\/institute.lanl.gov\/resilience\/conferences\/2009\" target=\"institute.lanl.gov\/resilience\/conferences\/2009\" rel=\"noopener\">National HPC Workshop on Resilience 2009<\/a>, Arlington, VA, USA, August 12-14, 2009. <a href=\"publications\/engelmann09modeling.ppt.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/ppt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Presentation\"><\/a> <a href=\"?page_id=55#engelmann09modeling\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<li>Christian Engelmann. <b>System Resilience Research at ORNL in the Context of HPC<\/b>. Invited talk at the <a href=\"http:\/\/www.inria.fr\/inria\/organigramme\/fiche_ur-ren.fr.html\" target=\"www.inria.fr\/inria\/organigramme\/fiche_ur-ren.fr.html\" rel=\"noopener\">Institut National de Recherche en Informatique et en Automatique (INRIA)<\/a>, Rennes, France, May 15, 2009. <a href=\"javascript:showAbstract('The continuing growth in high performance computing (HPC) system scale poses a challenge for system software and scientific applications with respect to reliability, availability and serviceability (RAS). With only very few exceptions, the availability of recently installed systems has been lower in comparison to the same deployment phase of their predecessors. As a result, sites lower allowable job run times in order to force applications to store intermediate results (checkpoints) as insurance against lost computation time. However, checkpoints themselves waste valuable computation time and resources. In contrast to the experienced loss of availability, the demand for continuous availability has risen dramatically with the trend towards capability computing, which drives the race for scientific discovery by running applications on the fastest machines available while desiring significant amounts of time (weeks and months) without interruption. These machines must be able to run in the event of frequent interrupts in such a manner that the capability is not severely degraded. Thus, research and development of scalable RAS technologies is paramount to the success of future extreme-scale systems. This talk summarizes our accomplishments in the area of high-level RAS for HPC, such as developed concepts and implemented proof-of-concept prototypes.');\"><img decoding=\"async\" src=\"images\/txt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Abstract\"><\/a> <a href=\"publications\/engelmann09system.ppt.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/ppt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Presentation\"><\/a> <a href=\"?page_id=55#engelmann09system\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<li>Christian Engelmann. <b>High-Performance Computing Research and MSc Internship Opportunities at Oak Ridge National Laboratory<\/b>. Seminar at the <a href=\"http:\/\/www.cs.reading.ac.uk\" target=\"www.cs.reading.ac.uk\" rel=\"noopener\">Department of Computer Science<\/a>, <a href=\"http:\/\/www.reading.ac.uk\" target=\"www.reading.ac.uk\" rel=\"noopener\">University of Reading<\/a>, Reading, United Kingdom, May 11, 2009. <a href=\"javascript:showAbstract('Oak Ridge National Laboratory (ORNL) is the largest energy laboratory in the United States. Its National Center for Computational Sciences (NCCS) provides the most powerful computing resources in the world for open scientific research. Jaguar, a Cray XT5 system at NCCS, is the second HPC system to exceed 1 PFlop\/s (10^15 Floating Point Operations Per Second), and the fastest open science supercomputer in the world. It recently ranked #2 in the Top 500 List of Supercomputer Sites with a maximal LINPACK benchmark performance of 1.059 PFlop\/s and a theoretical peak performance of 1.3814 PFlop\/s. Annually, 80 percent of Jaguar&amp;#39;s resources are allocated through the U.S Department of Energy's Innovative and Novel Computational Impact on Theory and Experiment (INCITE) program, a competitively selected, peer reviewed process open to researchers from universities, industry, government and non-profit organizations. These allocations address some of the most challenging scientific problems in areas such as climate modeling, renewable energy, materials science, fusion and combustion. In conjunction with NCCS, the Computer Science and Mathematics Division at ORNL performs basic and applied research in HPC, mathematics, and intelligent systems. This talk gives a summary of the HPC research performed at ORNL. It provides details about the Jaguar peta-scale computing resource, an overview of the computational science research carried out using ORNL's computing resources, and a description of various computer science efforts targeting solutions for next-generation HPC systems. This talk also provides information about internship opportunities for MSc students.');\"><img decoding=\"async\" src=\"images\/txt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Abstract\"><\/a> <a href=\"publications\/engelmann09high.ppt.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/ppt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Presentation\"><\/a> <a href=\"?page_id=55#engelmann09high\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<li>Christian Engelmann. <b>Proactive Fault Tolerance Using Preemptive Migration<\/b>. Invited talk at the <a href=\"http:\/\/acet.rdg.ac.uk\/events\/details\/cancun.php\" target=\"acet.rdg.ac.uk\/events\/details\/cancun.php\" rel=\"noopener\">3rd Collaborative and Grid Computing Technologies Workshop (CGCTW) 2009<\/a>, Cancun, Mexico, April 22-24, 2009. <a href=\"javascript:showAbstract('The continuing growth in high-performance computing (HPC) system scale poses a challenge for system software and scientific applications with respect to reliability, availability and serviceability (RAS). In order to address anticipated high failure rates, resiliency characteristics have become an urgent priority for next-generation HPC systems. The concept of proactive fault tolerance prevents compute node failures from impacting running parallel applications by preemptively migrating application parts away from nodes that are about to fail. This talk presents our past and ongoing efforts in proactive fault resilience for HPC. Presented work includes proactive fault resilience techniques, transparent process- and virtual-machine-level migration, system and application reliability models and analyses, failure prediction, and trade-off models for combining preemptive migration with checkpoint\/restart. All these individual technologies are put into context with a proposed holistic HPC fault resilience framework.');\"><img decoding=\"async\" src=\"images\/txt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Abstract\"><\/a> <a href=\"publications\/engelmann09proactive2.ppt.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/ppt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Presentation\"><\/a> <a href=\"?page_id=55#engelmann09proactive2\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<li>Christian Engelmann. <b>Resiliency<\/b>. Panel at the <a href=\"http:\/\/www.cs.sandia.gov\/Conferences\/SOS13\" target=\"www.cs.sandia.gov\/Conferences\/SOS13\" rel=\"noopener\">13th Workshop on Distributed Supercomputing (SOS) 2009<\/a>, Hilton Head, SC, USA, March 9-12, 2009. <a href=\"?page_id=55#engelmann09resiliency\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<li>Christian Engelmann. <b>High-Performance Computing Research at Oak Ridge National Laboratory<\/b>. Invited talk at the Reading Annual Computational Science  Workshop, Reading, United Kingdom, December 8, 2008. <a href=\"javascript:showAbstract('Oak Ridge National Laboratory (ORNL) is the largest energy laboratory in the United States. Its National Center for Computational Sciences (NCCS) provides the most powerful computing resources in the world for open scientific research. Jaguar, a Cray XT5 system at NCCS, is the second HPC system to exceed 1 PFlop\/s (10^15 Floating Point Operations Per Second), and the fastest open science supercomputer in the world. It recently ranked #2 in the Top 500 List of Supercomputer Sites with a maximal LINPACK benchmark performance of 1.059 PFlop\/s and a theoretical peak performance of 1.3814 PFlop\/s. Annually, 80 percent of Jaguar\u2019s resources are allocated through the U.S Department of Energy\u2019s Innovative and Novel Computational Impact on Theory and Experiment (INCITE) program, a competitively selected, peer reviewed process open to researchers from universities, industry, government and non-profit organizations. These allocations address some of the most challenging scientific problems in areas such as climate modeling, renewable energy, materials science, fusion and combustion. In conjunction with NCCS, the Computer Science and Mathematics Division at ORNL performs basic and applied research in HPC, mathematics, and intelligent systems. This talk gives a summary of the HPC research performed at ORNL. It provides details about the Jaguar peta-scale computing resource, an overview of the computational science research carried out using ORNL\u2019s computing resources, and a description of various computer science efforts targeting solutions for next-generation HPC systems.');\"><img decoding=\"async\" src=\"images\/txt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Abstract\"><\/a> <a href=\"publications\/engelmann08high.ppt.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/ppt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Presentation\"><\/a> <a href=\"?page_id=55#engelmann08high\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<li>Christian Engelmann. <b>Modular Redundancy in HPC Systems: Why, Where, When and How?<\/b>. Invited talk at the 1st HPC Resiliency Summit: Workshop on Resiliency for Petascale HPC 2008, in conjunction with the <a href=\"http:\/\/www.lanl.gov\/conferences\/lacss\/2008\" target=\"www.lanl.gov\/conferences\/lacss\/2008\" rel=\"noopener\">1st Los Alamos Computer Science Symposium (LACSS) 2008<\/a>, Santa Fe, NM, USA, October 15, 2008. <a href=\"javascript:showAbstract('The continuing growth in high-performance computing (HPC) system scale poses a challenge for system software and scientific applications with respect to reliability, availability and serviceability (RAS). With only very few exceptions, the availability of recently installed systems has been lower in comparison to the same deployment phase of their predecessors. As a result, sites lower allowable job run times in order to force applications to store intermediate results (checkpoints) as insurance against lost computation time. However, checkpoints themselves waste valuable computation time and resources. In contrast to the experienced loss of availability, the demand for continuous availability has risen dramatically with the trend towards capability computing, which drives the race for scientific discovery by running applications on the fastest machines available while desiring significant amounts of time (weeks and months) without interruption. These machines must be able to run in the event of frequent interrupts in such a manner that the capability is not severely degraded. Thus, research and development of scalable RAS technologies is paramount to the success of future extreme-scale systems. This talk summarizes our past accomplishments, ongoing work, and future plans in the area of high-level RAS for HPC.');\"><img decoding=\"async\" src=\"images\/txt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Abstract\"><\/a> <a href=\"publications\/engelmann08modular.ppt.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/ppt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Presentation\"><\/a> <a href=\"?page_id=55#engelmann08modular\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<li>Christian Engelmann. <b>Resiliency for High-Performance Computing<\/b>. Invited talk at the <a href=\"http:\/\/acet.rdg.ac.uk\/events\/details\/cancun.php\" target=\"acet.rdg.ac.uk\/events\/details\/cancun.php\" rel=\"noopener\">2nd Collaborative and Grid Computing Technologies Workshop (CGCTW) 2008<\/a>, Cancun, Mexico, April 10-12, 2008. <a href=\"javascript:showAbstract('In order to address anticipated high failure rates, resiliency characteristics have become an urgent priority for next-generation high-performance computing (HPC) systems. One major source of concern are non-recoverable soft errors, i.e., bit flips in memory, cache, registers, and logic. The probability of such errors not only grows with system size, but also with increasing architectural vulnerability caused by employing accelerators and by shrinking nanometer technology. Reactive fault tolerance technologies, such as checkpoint\/restart, are unable to handle high failure rates due to associated overheads, while proactive resiliency technologies, such as preemptive migration, simply fail as random soft errors can&amp;#39;t be predicted. This talk proposes a new, bold direction in resiliency for HPC as it targets resiliency for next-generation extreme-scale HPC systems at the system software level through computational redundancy strategies, i.e., dual- and triple-modular redundancy.');\"><img decoding=\"async\" src=\"images\/txt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Abstract\"><\/a> <a href=\"publications\/engelmann08resiliency.ppt.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/ppt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Presentation\"><\/a> <a href=\"?page_id=55#engelmann08resiliency\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<li>Christian Engelmann. <b>Advanced Fault Tolerance Solutions for High Performance Computing<\/b>. Seminar at the <a href=\"http:\/\/www.laas.fr\" target=\"www.laas.fr\" rel=\"noopener\">Laboratoire d&#39;Analyse et d&#8217;Architecture des Syst&eacute;mes<\/a>, <a href=\"http:\/\/www.cnrs.fr\" target=\"www.cnrs.fr\" rel=\"noopener\">Centre National de la Recherche Scientifique<\/a>, Toulouse, France, February 11, 2008. <a href=\"javascript:showAbstract('The continuing growth in high performance computing (HPC) system scale poses a challenge for system software and scientific applications with respect to reliability, availability and serviceability (RAS). With only very few exceptions, the availability of recently installed systems has been lower in comparison to the same deployment phase of their predecessors. As a result, sites lower allowable job run times in order to force applications to store intermediate results (checkpoints) as insurance against lost computation time. However, checkpoints themselves waste valuable computation time and resources. In contrast to the experienced loss of availability, the demand for continuous availability has risen dramatically with the trend towards capability computing, which drives the race for scientific discovery by running applications on the fastest machines available while desiring significant amounts of time (weeks and months) without interruption. These machines must be able to run in the event of frequent interrupts in such a manner that the capability is not severely degraded. Thus, research and development of scalable RAS technologies is paramount to the success of future extreme-scale systems. This talk summarizes our accomplishments in the area of high-level RAS for HPC, such as developed concepts and implemented proof-of-concept prototypes, and describes existing limitations, such as performance issues, which need to be dealt with for production-type deployment.');\"><img decoding=\"async\" src=\"images\/txt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Abstract\"><\/a> <a href=\"publications\/engelmann08advanced.ppt.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/ppt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Presentation\"><\/a> <a href=\"?page_id=55#engelmann08advanced\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<\/ol>\n<h4>Co-advised Theses<\/h4>\n<ol>\n<li>Swen B&ouml;hm. <b>Development of a RAS Framework for HPC Environments: Realtime Data Reduction of Monitoring Data<\/b>. Master&#8217;s thesis, <a href=\"http:\/\/www.cs.reading.ac.uk\" target=\"www.cs.reading.ac.uk\" rel=\"noopener\">Department of Computer Science<\/a>, <a href=\"http:\/\/www.reading.ac.uk\" target=\"www.reading.ac.uk\" rel=\"noopener\">University of Reading<\/a>, UK, March 12, 2010. Thesis research performed at Oak Ridge National Laboratory. Advisors: Prof. Vassil N. Alexandrov (University of Reading); Christian Engelmann (Oak Ridge National Laboratory); George Bosilca (University of Tennessee, Knoxville). <a href=\"javascript:showAbstract('The advancements of high-performance computing (HPC) systems in the last decades lead to more and more complex systems containing thousands or tens-of-thousands computing systems that are working together. While the computational performance of these systems increased dramaticaly in the last years the I\/O subsystems have not gained such a significant improvement. With increasing nummbers of hardware components in the next generation HPC systems maintaining the relaiability of such systems becomes more and more difficult since the probability of hardware failures is increasing with the number of components. The capacities of traditional reactive fault tolerance technologies are exceeded by the development of next generation systems and alternatives have to be found. This paper discusses a monitoring system that is using data reduction techniques to decrease the amount of the collected data. The system is part of a proactive fault tolerance system that may challenge the reliability problems of exascale HPC systems.');\"><img decoding=\"async\" src=\"images\/txt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Abstract\"><\/a> <a href=\"publications\/boehm10development.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/pdf.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Publication\"><\/a> <a href=\"publications\/boehm10development.ppt.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/ppt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Presentation\"><\/a> <a href=\"?page_id=55#boehm10development\" ><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<li>Antonina Litvinova. <b>RAS Framework Engine Prototype<\/b>. Master&#8217;s thesis, <a href=\"http:\/\/www.cs.reading.ac.uk\" target=\"www.cs.reading.ac.uk\" rel=\"noopener\">Department of Computer Science<\/a>, <a href=\"http:\/\/www.reading.ac.uk\" target=\"www.reading.ac.uk\" rel=\"noopener\">University of Reading<\/a>, UK, September 22, 2009. Thesis research performed at Oak Ridge National Laboratory. Advisors: Prof. Vassil N. Alexandrov (University of Reading); Christian Engelmann (Oak Ridge National Laboratory); George Bosilca (University of Tennessee, Knoxville). <a href=\"javascript:showAbstract('Extreme high performance computing (HPC) systems constantly increase in scale from a few thousands of processors cores to thousands of thousands of processors cores and beyond. However their system mean-time to interrupt decreases according. The current approach of fault tolerance in HPC is checkpoint\/restart, i.e. a method based on recovery from experienced failures. However checkpoint\/restart cannot deal with errors in the same efficient way anymore, because of HPC systems modification. For example, increasing error rates, increasing aggregate memory, and not proportionally increasing input\/output capabilities. The recently introduced concept is proactive fault tolerance which avoids experiencing failures through preventative measures. Proactive fault tolerance uses migration which is an emerging technology that prevents failures on HPC systems by migrating applications or application parts away from a node that is deteriorating to a spare node. This thesis discusses work conducted at ORNL to develop a Proactive Fault Tolerance Framework Engine Prototype for HPC systems with high reliability, availability and serviceability. The prototype performs environmental system monitoring, system event logging, parallel job monitoring and system resource monitoring in order to analyse HPC system reliability and to perform fault avoidance through a migration.');\"><img decoding=\"async\" src=\"images\/txt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Abstract\"><\/a> <a href=\"publications\/litvinova09ras.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/pdf.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Publication\"><\/a> <a href=\"publications\/litvinova09ras.ppt.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/ppt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Presentation\"><\/a> <a href=\"?page_id=55#litvinova09ras\" ><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<\/ol>\n<p><em><small>Symbols: <img decoding=\"async\" style=\"border-style: none;\" src=\"images\/txt.gif\" border=\"0\" alt=\"Abstract\" height=\"10pt\"> Abstract, <img decoding=\"async\" style=\"border-style: none;\" src=\"images\/pdf.gif\" border=\"0\" alt=\"Publication\" height=\"10pt\"> Publication, <img decoding=\"async\" style=\"border-style: none;\" src=\"images\/ppt.gif\" border=\"0\" alt=\"Presentation\" height=\"10pt\"> Presentation, <img decoding=\"async\" style=\"border-style: none;\" src=\"images\/bib.gif\" border=\"0\" alt=\"BibTeX Citation\" height=\"10pt\"> BibTeX Citation<\/small><\/em><\/p>\n<p><script language=\"JavaScript\">\nfunction showAbstract (text) {\n  var width  = 400;\n  var height = 400;\n  var left   = (screen.width  - width ) \/ 2;\n  var top    = (screen.height - height) \/ 2;\n  var win    = window.open('',\n                           'Abstract',\n                           'width='  + width  + ', ' + \n                           'height=' + height + ', ' +\n                           'left='   + left   + ', ' +\n                           'top='    + top    + ', ' +\n                           'toolbar=no, '     +\n                           'location=no, '    +\n                           'directories=no, ' +\n                           'status=no, '      +\n                           'menubar=no, '     +\n                           'copyhistory=no, ' +\n                           'scrollbars=yes, ' +\n                           'resizable=yes')\n  win.document.write(text);\n  win.document.close();\n}\n<\/script><\/p>\n","protected":false},"excerpt":{"rendered":"<p>This project aims at scalable technologies for providing high-level RAS for next-generation petascale scientific high-end computing (HEC) resources and beyond as outlined by the U.S. Department of Energy (DOE) Forum to Address Scalable Technology for Runtime and Operating Systems (FAST-OS) and the U.S. National Coordination Office for Networking and Information Technology Research and Development (NCO\/NITRD)&hellip;&nbsp;<\/p>\n","protected":false},"author":1,"featured_media":0,"parent":145,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"neve_meta_sidebar":"","neve_meta_container":"","neve_meta_enable_content_width":"off","neve_meta_content_width":100,"neve_meta_title_alignment":"","neve_meta_author_avatar":"","neve_post_elements_order":"","neve_meta_disable_header":"","neve_meta_disable_footer":"","neve_meta_disable_title":"","footnotes":""},"class_list":["post-309","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/www.christian-engelmann.info\/index.php?rest_route=\/wp\/v2\/pages\/309","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.christian-engelmann.info\/index.php?rest_route=\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/www.christian-engelmann.info\/index.php?rest_route=\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/www.christian-engelmann.info\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.christian-engelmann.info\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=309"}],"version-history":[{"count":9,"href":"https:\/\/www.christian-engelmann.info\/index.php?rest_route=\/wp\/v2\/pages\/309\/revisions"}],"predecessor-version":[{"id":548,"href":"https:\/\/www.christian-engelmann.info\/index.php?rest_route=\/wp\/v2\/pages\/309\/revisions\/548"}],"up":[{"embeddable":true,"href":"https:\/\/www.christian-engelmann.info\/index.php?rest_route=\/wp\/v2\/pages\/145"}],"wp:attachment":[{"href":"https:\/\/www.christian-engelmann.info\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=309"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}