{"id":1120,"date":"2026-09-06T08:00:06","date_gmt":"2026-09-06T08:00:06","guid":{"rendered":"https:\/\/www.christian-engelmann.info\/?page_id=1120"},"modified":"2026-09-07T00:45:49","modified_gmt":"2026-09-07T00:45:49","slug":"2024-a-resilient-federated-ecosystem-for-self-driving-laboratories","status":"publish","type":"page","link":"https:\/\/www.christian-engelmann.info\/?page_id=1120","title":{"rendered":"2024-&#8230;: A Resilient Federated Ecosystem for Self-Driving Laboratories"},"content":{"rendered":"<p>Failure resilience in federated ecosystems for instrument science is a critical challenge. Failures disrupt experiments and make them potentially useless, wasting valuable instrument, network and computing allocations and creating setbacks for scientists. Oak Ridge National Laboratory&#8217;s (ORNL&#8217;s) <a href=\"https:\/\/www.ornl.gov\/intersect\" target=\"www.ornl.gov_intersect\">Interconnected Science Ecosystem (INTERSECT) Initiative<\/a> offers a federated ecosystem for instrument science, enabling autonomous experiments, self-driving laboratories, smart manufacturing, and artificial intelligence (AI) driven design, discovery, and evaluation. While it currently does not offer failure resilience, existing INTERSECT experiments require it, such as to reliably steer an <a href=\"https:\/\/intersect-architecture.readthedocs.io\/en\/latest\/examples\/aam\" target=\"intersect-architecture.readthedocs.io_en_latest_examples_aam\">autonomous additive manufacturing (AAM)<\/a> process using real-time data streamed to a simulation in the feedback loop.<\/p>\n<p align=\"center\"><iframe loading=\"lazy\" style=\"display:block; margin:auto;\" width=\"720\" height=\"405\" src=\"https:\/\/www.youtube.com\/embed\/UGTADFR1O2U\" title=\"YouTube video player\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" allowfullscreen=\"1\"><\/iframe>Figure 1: The INTERSECT autonomous additive manufacturing experiment uses a thermomechanical simulation in a live feedback loop to control the residual stress in a printed part <\/p>\n<p>This project creates a <a href=\"https:\/\/intersect-architecture.readthedocs.io\" target=\"intersect-architecture.readthedocs.io\">resilient INTERSECT ecosystem architecture<\/a> using resilience design patterns, a resilient system of systems (SoS) architecture, and a resilient microservices architecture. Its proof-of concept prototype implements a resilient federated ecosystem using the <a href=\"https:\/\/github.com\/INTERSECT-SDK\" target=\"github.com_INTERSECT_SDK\">INTERSECT software development kit (SDK)<\/a> and demonstrates resilience capabilities for the INTERSECT AAM cross-facility experiment between the <a href=\"https:\/\/www.ornl.gov\/facility\/mdf\" target=\"www.ornl.gov_facility_mdf\">Manufacturing Demonstration Facility (MDF)<\/a> and the <a href=\"https:\/\/www.olcf.ornl.gov\" target=\"www.olcf.ornl.gov\">Oak Ridge Leadership Computing Facility (OLCF)<\/a> Advanced Computing Ecosystem (ACE) testbed. The outcome of this project facilitates the proper development and deployment of resilience for federated ecosystems. It creates, implements and demonstrates a consistent design methodology that allows scientists to pick and choose the right solution for the resilience problem at hand and deploy it with ease. It enables a resilient federated ecosystem that facilitates the US Department of Energy&#8217;s (DOE&#8217;s) Integrated Research Infrastructure (IRI) vision.<\/p>\n<h4>Prominent Solutions<\/h4>\n<ul>\n<li><a href=\"?page_id=570\">The INTERSECT Federated Architecture for the Laboratory of the Future<\/a><\/li>\n<\/ul>\n<h4>Funding Sources<\/h4>\n<ul>\n<li>\n<a href=\"http:\/\/www.ornl.gov\/intersect\" target=\"www.ornl.gov_intersect\">Interconnected Science Ecosystem (INTERSECT) Initiative<\/a>, Laboratory Directed Research and Development, <a href=\"http:\/\/www.ornl.gov\" target=\"www.ornl.gov\">Oak Ridge National Laboratory<\/a>\n<\/li>\n<\/ul>\n<h4>Participants<\/h4>\n<ul>\n<li>Christian Engelmann (PI), Andrew Ayres, Michael Brim, Stephen DeWitt, Swen Boehm, Addi Malviya, Marshall McDonnell, and Ryan Prout &#8212; <a href=\"http:\/\/www.ornl.gov\" target=\"www.ornl.gov\">Oak Ridge National Laboratory<\/a>\n<\/li>\n<\/ul>\n<h4>Peer-reviewed Workshop Publications<\/h4>\n<ol>\n<li>Christian Engelmann, Andrew Ayres, Stephen DeWitt, Michael J. Brim, and Brett Eiffert. <b>Building Resilient Self-Driving Laboratories with the INTERSECT Federated Ecosystem<\/b>. In <i>Proceedings of the <a href=\"http:\/\/sc26.supercomputing.org\" target=\"sc26.supercomputing.org\">39th International Conference on High Performance Computing, Networking, Storage and Analysis (SC) Workshops 2026<\/a>: <a href=\"http:\/\/wordpress.cels.anl.gov\/xloop-2026\/\" target=\"wordpress.cels.anl.gov\/xloop-2026\/\">8th Annual Workshop on Extreme-Scale Experiment-in-the-Loop Computing (XLOOP) 2026<\/a><\/i>, Chicago, IL, USA, November 15, 2026. <a href=\"http:\/\/www.computer.org\" target=\"www.computer.org\">IEEE Computer Society, Los Alamitos, CA, USA<\/a>. To appear. <a href=\"javascript:showAbstract('Failure resilience in federated ecosystems for instrument science presents a critical challenge. Failures disrupt experiments and make them potentially useless, wasting valuable resources and creating setbacks. Oak Ridge National Laboratory&amp;#39;s Self-driven Experiments for Science \/ Interconnected Science Ecosystem (INTERSECT) offers a federated ecosystem for instrument science, enabling autonomous experiments, self-driving laboratories, smart manufacturing, and AI-driven design, discovery, and evaluation. This paper documents the recent advances in creating a resilient INTERSECT ecosystem. The proposed solution includes a resilient architecture with resilience design patterns, a resilient system of systems (SoS) architecture, and a resilient microservices architecture; and a resilient software development kit with reliable service communication and asynchronous and synchronous failure detection and notification. The resilience capabilities are demonstrated for an autonomous additive manufacturing process with a real-time feedback loop.');\"><img decoding=\"async\" src=\"images\/txt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Abstract\"><\/a> <a href=\"?page_id=55#engelmann26building\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<\/ol>\n<h4>White Papers<\/h4>\n<ol>\n<li>Ryan Adamson and Christian Engelmann. <b>Cybersecurity and Privacy for Instrument-to-Edge-to-Center Scientific Computing Ecosystems<\/b>. <i>White paper accepted at the U.S. Department of Energy&#39;s <a href=\"http:\/\/www.orau.gov\/2021ascr-cybersecurity\" target=\"www.orau.gov\/2021ascr-cybersecurity\">ASCR Workshop on Cybersecurity and Privacy for Scientific  Computing Ecosystems<\/a><\/i>, November 3-5, 2021. <a href=\"javascript:showAbstract('The DOE&amp;#39;s Artificial Intelligence (AI) for Science report outlines the need for intelligent systems, instruments, and facilities to enable science breakthroughs with autonomous experiments, 'self-driving' laboratories, smart manufacturing, and AI-driven design, discovery and evaluation. The DOE's Computational Facilities Research Workshop report identifies intelligent systems\/facilities as a challenge with enabling automation and eliminating human-in-the-loop needs as a cross-cutting theme. Autonomous experiments, 'self-driving' laboratories and smart manufacturing employ machine-in-the-loop intelligence for decision-making. Human-in-the-loop needs are reduced by an autonomous online control that collects experiment data, analyzes it, and takes appropriate operational actions in real time to steer an ongoing or plan the next experiment. DOE laboratories are currently in the process of developing and deploying federated hardware\/software architectures for connecting instruments with edge and center computing resources to autonomously collect, transfer, store, process, curate, and archive scientific data. These new instrument-to-edge-to-center scientific ecosystems face several cybersecurity and privacy challenges.');\"><img decoding=\"async\" src=\"images\/txt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Abstract\"><\/a> <a href=\"publications\/adamson21cybersecurity.pdf\" target=\"publication\"><img decoding=\"async\" src=\"images\/pdf.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Publication\"><\/a> <a href=\"?page_id=55#adamson21cybersecurity\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<li>Hal Finkel, Pete Beckman, Christian Engelmann, Shantenu Jha, and Jack Lange. <b>Research Opportunities in Operating Systems for Scientific Edge Computing<\/b>. <i>White paper by the U.S. Department of Energy&#39;s <a href=\"http:\/\/www.orau.gov\/OSRoundtable2021\" target=\"www.orau.gov\/OSRoundtable2021\">ASCR Roundtable Discussions on Operating-Systems Research 2021<\/a><\/i>, January 25, 2021. <a href=\"javascript:showAbstract('As scientific experiments generate ever-increasing amounts of data, and grow in operational complexity, modern experimental science demands unprecedented computational capabilities at the edge - physically proximate to each experiment. While some requirements on these computational capabilities are shared with high-performance-computing (HPC) systems, scientific edge computing has a number of unique challenges. In the following, we survey current trends in system software and edge systems for scientific computing, associated research challenges and open questions, infrastructure requirements for operating-systems research, communities who should be involved in that research, and the anticipated benefits of success.');\"><img decoding=\"async\" src=\"images\/txt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Abstract\"><\/a> <a href=\"publications\/finkel21research2.pdf\" target=\"publication\"><img decoding=\"async\" src=\"images\/pdf.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Publication\"><\/a> <a href=\"?page_id=55#finkel21research2\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<li>Hal Finkel, Pete Beckman, Ron Brightwell, Rudi Eigenmann, Christian Engelmann, Roberto Gioiosa, Kamil Iskra, Shantenu Jha, Jack Lange, Tapasya Patki, and Kevin Pedretti. <b>Research Opportunities in Operating Systems for High-Performance Scientific Computing<\/b>. <i>White paper by the U.S. Department of Energy&#39;s <a href=\"http:\/\/www.orau.gov\/OSRoundtable2021\" target=\"www.orau.gov\/OSRoundtable2021\">ASCR Roundtable Discussions on Operating-Systems Research 2021<\/a><\/i>, January 25, 2021. <a href=\"javascript:showAbstract('As high-performance-computing (HPC) systems continue to evolve, with increasingly diverse and heterogeneous hardware, increasingly-complex requirements for security and multi-tenancy, and increasingly-demanding requirements for resiliency and monitoring, research in operating systems must continue to seed innovation to meet future needs. In the following, we survey current trends in system software and HPC systems for scientific computing, associated research challenges and open questions, infrastructure requirements for operating-systems research, communities who should be involved in that research, and the anticipated benefits of success.');\"><img decoding=\"async\" src=\"images\/txt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Abstract\"><\/a> <a href=\"publications\/finkel21research.pdf\" target=\"publication\"><img decoding=\"async\" src=\"images\/pdf.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Publication\"><\/a> <a href=\"?page_id=55#finkel21research\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<\/ol>\n<h4>Technical Reports<\/h4>\n<ol>\n<li>Brian Etz, Oral, Sarp, Rafael Ferreira Da Silva, Ryan Adamson, Anees Alnajjar, Tom Beck, Ashley Barker, Michael Brim, Paul Bryant, Christian Engelmann, Anjus George, Samuel Herts, Gustav Jansen, Rajesh Kalyanam, Ahmad Maroof Karimi, Jack Lange, Kellen Leland, Ketan Maheshwari, Marshall McDonnell, Bronson Messer II, Ross Miller, Daniel S. Pelfrey, Suzanne Prentice, Bran Radovanovic, David Rogers, Daniel Rosendo, A.J. Ruckman, Mallikarjun (Arjun) Shankar, Amir Shehata, Tyler Skluzacek, Renan Santos Souza, Veronica Melesse Vergar, Feiyi Wang, Jordan Webb, Patrick Widener, and Christopher Zimmer. <b>OLCF&#39;s Advanced Computing Ecosystem (ACE): FY25 Update for Ongoing Efforts<\/b>. Technical Report, ORNL\/TM-2025\/4050, Oak Ridge National Laboratory, November 30, 2025. DOI <a href=\"http:\/\/dx.doi.org\/10.2172\/3006499\" target=\"publication\">10.2172\/3006499<\/a>. <a href=\"publications\/etz25olcf.pdf\" target=\"publication\"><img decoding=\"async\" src=\"images\/pdf.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Publication\"><\/a> <a href=\"?page_id=55#etz25olcf\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<li>Rafael Ferreira da Silva, Robert Moore, Benjamin Mintz, Rigoberto Advincula, Anees Alnajjar, Luke Baldwin, Craig Bridges, Ryan Coffee, Ewa Deelman, Christian Engelmann, Brian Etz, Millie Firestone, Ian Foster, Panchapakesan Ganesh, Leslie Hamilton, Dale Huber, Ilia Ivanov, Shantenu Jha, Ying Li, Yongtao Liu, Jay Lofstead, Anirban Mandal, Hector Martin, Theresa Mayer, Marshall McDonnell, Vijayakumar Murugesan, Sal Nimer, Nageswara Rao, Martin Seifrid, Mitra Taheri, Michela Taufer, and Konstantinos Vogiatzis. <b>Shaping the Future of Self-Driving Autonomous Laboratories Workshop<\/b>. Technical Report, ORNL\/TM-2024\/3714, Oak Ridge National Laboratory, January 2, 2024. DOI <a href=\"http:\/\/dx.doi.org\/10.5281\/zenodo.14430232\" target=\"publication\">10.5281\/zenodo.14430232<\/a>. <a href=\"javascript:showAbstract('The Shaping the Future of Self-Driving Autonomous Laboratories workshop, held in Denver on November 7-8, 2024, brought together leading experts from materials science and computing to address the growing need to revolutionize scientific research through AI-driven autonomous laboratories. The workshop identified critical challenges, including the integration of heterogeneous data, development of AI systems that understand fundamental physical principles, and comprehensive safety protocols. Key recommendations emerged around developing universal laboratory equipment interfaces, implementing automated metadata collection systems, and creating hybrid AI approaches that combine data-driven learning with scientific principles. The workshop emphasized maintaining human oversight while leveraging automation, transforming scientific education to prepare the next generation of researchers, and establishing a national consortium leveraging DOE facilities as anchors for broader collaboration with academia and industry. Participants stressed the urgency of addressing the growing disconnect between human decision-making timescales and modern instrumentation capabilities, highlighting the need for strategic automation while preserving essential human insight and oversight in the research process.');\"><img decoding=\"async\" src=\"images\/txt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Abstract\"><\/a> <a href=\"publications\/dasilva24shaping.pdf\" target=\"publication\"><img decoding=\"async\" src=\"images\/pdf.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Publication\"><\/a> <a href=\"?page_id=55#dasilva24shaping\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<\/ol>\n<h4>Talks and Lectures<\/h4>\n<ol>\n<li>Christian Engelmann. <b>The Federated Computing Environment for Autonomous Smart Laboratories<\/b>. Invited talk at the <a href=\"http:\/\/sos27.cscs.ch\" target=\"sos27.cscs.ch\">27th Workshop on Distributed Supercomputing (SOS) 2025<\/a>, Engelberg, Switzerland, March 20, 2025. <a href=\"javascript:showAbstract('The open Interconnected Science Ecosystem (INTERSECT) architecture connects scientific instruments and robot-controlled laboratories with computing and data resources at the edge, the Cloud or the high-performance computing center to enable autonomous experiments, self-driving laboratories, smart manufacturing, and artificial intelligence driven design, discovery and evaluation. Its a novel approach consists of science use case design patterns, a system of systems architecture, and a microservice architecture. Failure resilience in federated ecosystems for instrument science is a critical challenge. Failures disrupt experiments and make them potentially useless, wasting valuable instrument, network and computing allocations and creating setbacks for scientists. A diverse, yet resilient, federated high-performance computing ecosystem is needed with traditional and accelerated capacity and capability computing resources and proper network and data storage resources, in part with on-demand and real-time features. This talk presents an overview of the resilient INTERSECT architecture, illustrates a resilient autonomous additive manufacturing use case, and discusses the future needs for incorporating such computational workloads into high-performance computing systems and facilities.');\"><img decoding=\"async\" src=\"images\/txt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Abstract\"><\/a> <a href=\"publications\/engelmann25federated.ppt.pdf\" target=\"publication\"><img decoding=\"async\" src=\"images\/ppt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Presentation\"><\/a> <a href=\"?page_id=55#engelmann25federated\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<\/ol>\n<p><em><small>Symbols: <img decoding=\"async\" style=\"border-style: none;\" src=\"images\/txt.gif\" border=\"0\" alt=\"Abstract\" height=\"10pt\"> Abstract, <img decoding=\"async\" style=\"border-style: none;\" src=\"images\/pdf.gif\" border=\"0\" alt=\"Publication\" height=\"10pt\"> Publication, <img decoding=\"async\" style=\"border-style: none;\" src=\"images\/ppt.gif\" border=\"0\" alt=\"Presentation\" height=\"10pt\"> Presentation, <img decoding=\"async\" style=\"border-style: none;\" src=\"images\/bib.gif\" border=\"0\" alt=\"BibTeX Citation\" height=\"10pt\"> BibTeX Citation<\/small><\/em><\/p>\n<p><script language=\"JavaScript\">\nfunction showAbstract (text) {\n  var width  = 400;\n  var height = 400;\n  var left   = (screen.width  - width ) \/ 2;\n  var top    = (screen.height - height) \/ 2;\n  var win    = window.open('',\n                           'Abstract',\n                           'width='  + width  + ', ' + \n                           'height=' + height + ', ' +\n                           'left='   + left   + ', ' +\n                           'top='    + top    + ', ' +\n                           'toolbar=no, '     +\n                           'location=no, '    +\n                           'directories=no, ' +\n                           'status=no, '      +\n                           'menubar=no, '     +\n                           'copyhistory=no, ' +\n                           'scrollbars=yes, ' +\n                           'resizable=yes')\n  win.document.write(text);\n  win.document.close();\n}\n<\/script><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Failure resilience in federated ecosystems for instrument science is a critical challenge. Failures disrupt experiments and make them potentially useless, wasting valuable instrument, network and computing allocations and creating setbacks for scientists. Oak Ridge National Laboratory&#8217;s (ORNL&#8217;s) Interconnected Science Ecosystem (INTERSECT) Initiative offers a federated ecosystem for instrument science, enabling autonomous experiments, self-driving laboratories, smart&hellip;&nbsp;<\/p>\n","protected":false},"author":1,"featured_media":0,"parent":83,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"neve_meta_sidebar":"","neve_meta_container":"","neve_meta_enable_content_width":"off","neve_meta_content_width":100,"neve_meta_title_alignment":"","neve_meta_author_avatar":"","neve_post_elements_order":"","neve_meta_disable_header":"","neve_meta_disable_footer":"","neve_meta_disable_title":"","footnotes":""},"class_list":["post-1120","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/www.christian-engelmann.info\/index.php?rest_route=\/wp\/v2\/pages\/1120","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.christian-engelmann.info\/index.php?rest_route=\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/www.christian-engelmann.info\/index.php?rest_route=\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/www.christian-engelmann.info\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.christian-engelmann.info\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1120"}],"version-history":[{"count":29,"href":"https:\/\/www.christian-engelmann.info\/index.php?rest_route=\/wp\/v2\/pages\/1120\/revisions"}],"predecessor-version":[{"id":1452,"href":"https:\/\/www.christian-engelmann.info\/index.php?rest_route=\/wp\/v2\/pages\/1120\/revisions\/1452"}],"up":[{"embeddable":true,"href":"https:\/\/www.christian-engelmann.info\/index.php?rest_route=\/wp\/v2\/pages\/83"}],"wp:attachment":[{"href":"https:\/\/www.christian-engelmann.info\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1120"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}