The INTERSECT Federated Architecture for the Laboratory of the Future

Summary: The open Interconnected Science Ecosystem (INTERSECT) architecture connects scientific instruments and robot-controlled laboratories with computing and data resources at the edge, the Cloud or the high-performance computing center to enable autonomous experiments, self-driving laboratories, smart manufacturing, and artificial intelligence driven design, discovery and evaluation. Its a novel approach consists of science use case design patterns, a system of systems architecture, and a microservice architecture.

Connecting scientific instruments and robot-controlled laboratories with computing and data resources at the edge, the Cloud or the high-performance computing (HPC) center enables autonomous experiments, self-driving laboratories, smart manufacturing, and artificial intelligence (AI) driven design, discovery and evaluation. The goal is to autonomously collect, transfer, store, process, curate, and archive scientific data and reduce human-in-the-loop needs for controlling, steering and even designing experiments.

A federated instrument-to-edge-to-center hardware/software architecture needs to provide (a) common interfaces that leverage community and custom software; (b) pluggability that permits adaptable solutions, reuse of existing solutions, and digital twins for testing and evaluation; and (c) an open standard to enable adoption by science facilities world-wide. The Interconnected Science Ecosystem (INTERSECT) architecture enables science breakthroughs using intelligent networked systems, instruments and facilities with an open federated hardware/software architecture for the laboratory of the future. It relies on a novel approach, consisting of (1) science use case design patterns, (2) a system of systems (SoS) architecture, and (3) a microservice architecture.

Figure 1: The elements of the INTERSECT architecture in the context of the ecosystem for interconnected smart laboratories

Science use cases are described as design patterns that identify and abstract the involved hardware/software components and their interactions in terms of control, work, and data flow. The basic template for a science use case design pattern is defined in a loop control problem paradigm. There are two classes of science use case design patterns (Figure 1): strategy patterns and architectural patterns. Strategy patterns define high-level solution methods using experiment control architecture features at a very coarse granularity. Architectural patterns define more specific solution methods using hardware and software architecture features at a finer granularity. While the architectural patterns do inherit the features of certain parent strategy patterns, they also address additional problems that are not exposed at the high abstraction level of the strategy patterns. A specific solution may require the composition of patterns, such as the Experiment Steering and Design of Experiments strategy patterns and correspondingly the Local Experiment Steering and Distributed Design of Experiments architectural patterns (Figure 2).

Figure 2: Classification of the science use case design patterns

Figure 3: Components and control/data flow of Local Experiment Steering and Distributed Design of Experiments architectural pattern composition

The SoS architecture decomposes the federated hardware/software ecosystem into smaller and less complex systems and components within these systems (Figure 3). It permits the development of individual systems and components with clearly defined interfaces, data formats and communication protocols. This not only separates concerns and functionality for reusability, but also promotes pluggability and extensibility with uniform protocols and system/component life cycles. Instead of developing individual monolithic solutions for each science use case, the SoS architecture provides one solution that can be easily adapted to different use cases using different compositions of systems. It offers operational and managerial independence of systems and of components within systems, geographical distribution with a physically distributed and federated ecosystem, emergent behavior based on the interplay between systems and components, and evolutionary development through pluggability and extensibility.

Figure 4: Infrastructure systems contain multiple logical systems and logical systems span multiple infrastructure systems

Figure 5: Example of an autonomous electron microscopy use case

The microservice architecture provides a catalog of infrastructure and experiment-specific microservices. The microservices are defined to facilitate composition within the federated SoS architecture (Figure 4). INTERSECT infrastructure microservices represent common service functionality and capabilities, such as data management, computing, messaging, and workflow orchestration that are likely to be generally useful across many science ecosystems without the need for customization. Experiment-specific microservices, on the other hand, represent services whose implementation may require detailed application knowledge, such as experiment planning or steering services that require knowledge of experiment-specific control parameters and their associated constraints.

Latest INTERSECT architecture specification:

• Michael Brim and Christian Engelmann. INTERSECT Architecture Specification: Microservice Architecture (Version 0.9). Technical Report, ORNL/TM-2023/3171, Oak Ridge National Laboratory, Oak Ridge, TN, USA, September 1, 2023. DOI 10.2172/2333815.
• Christian Engelmann and Suhas Somnath. INTERSECT Architecture Specification: Use Case Design Patterns (Version 0.9). Technical Report, ORNL/TM-2023/3133, Oak Ridge National Laboratory, Oak Ridge, TN, USA, September 1, 2023. DOI 10.2172/2229218.
• Olga A. Kuchar, Swen Boehm, Thomas Naughton, Suhas Somnath, Ben Mintz, Jack Lange, Scott Atchley, Rohit Srivastava, and Patrick Widener. INTERSECT Architecture Specification: System-of-systems Architecture (Version 0.5). Technical Report, ORNL/TM-2022/2717, Oak Ridge National Laboratory, Oak Ridge, TN, USA, September 1, 2022. DOI 10.2172/1968700.

Research Projects

• 2021-…: An Open Federated Architecture for the Laboratory of the Future

Funding Sources

• Interconnected Science Ecosystem (INTERSECT) Initiative, Laboratory Directed Research and Development, Oak Ridge National Laboratory

Participating Institutions

• Oak Ridge National Laboratory

Peer-reviewed Conference Publications

1. Christian Engelmann and Suhas Somnath. Science Use Case Design Patterns for Autonomous Experiments. In Proceedings of the 28th European Conference on Pattern Languages of Programs (EuroPLoP) 2023, pages 1-14, Kloster Irsee, Germany, July 5-9, 2023. ACM Press, New York, NY, USA. ISBN 979-8-4007-0040-8. DOI 10.1145/3628034.3628060.
2. Christian Engelmann, Olga Kuchar, Swen Boehm, Michael J. Brim, Thomas Naughton, Suhas Somnath, Scott Atchley, Jack Lange, Ben Mintz, and Elke Arenholz. The INTERSECT Open Federated Architecture for the Laboratory of the Future. In Communications in Computer and Information Science (CCIS): Accelerating Science and Engineering Discoveries Through Integrated Research Infrastructure for Experiment, Big Data, Modeling and Simulation. 18th Smoky Mountains Computational Sciences & Engineering Conference (SMC) 2022, pages 173-190, August 24-25, 2022. Springer, Cham. ISBN 978-3-031-23605-1. DOI 10.1007/978-3-031-23606-8_11. Acceptance rate 32.4% (24/74).

Peer-reviewed Conference Posters

1. Christian Engelmann, Swen Boehm, Michael Brim, Jack Lange, Thomas Naughton, Patrick Widener, Ben Mintz, and Rohit Srivastava. INTERSECT: The Open Federated Architecture for the Laboratory of the Future. Poster at the 52nd International Conference on Parallel Processing (ICPP) 2023, Salt Lake City, UT, USA, August 7-10, 2023.

White Papers

1. Ryan Adamson and Christian Engelmann. Cybersecurity and Privacy for Instrument-to-Edge-to-Center Scientific Computing Ecosystems. White paper accepted at the U.S. Department of Energy's ASCR Workshop on Cybersecurity and Privacy for Scientific Computing Ecosystems, November 3-5, 2021.
2. Hal Finkel, Pete Beckman, Christian Engelmann, Shantenu Jha, and Jack Lange. Research Opportunities in Operating Systems for Scientific Edge Computing. White paper by the U.S. Department of Energy's ASCR Roundtable Discussions on Operating-Systems Research 2021, January 25, 2021.
3. Hal Finkel, Pete Beckman, Ron Brightwell, Rudi Eigenmann, Christian Engelmann, Roberto Gioiosa, Kamil Iskra, Shantenu Jha, Jack Lange, Tapasya Patki, and Kevin Pedretti. Research Opportunities in Operating Systems for High-Performance Scientific Computing. White paper by the U.S. Department of Energy's ASCR Roundtable Discussions on Operating-Systems Research 2021, January 25, 2021.

Technical Reports

1. Michael Brim and Christian Engelmann. INTERSECT Architecture Specification: Microservice Architecture (Version 0.9). Technical Report, ORNL/TM-2023/3171, Oak Ridge National Laboratory, Oak Ridge, TN, USA, September 1, 2023. DOI 10.2172/2333815.
2. Christian Engelmann and Suhas Somnath. INTERSECT Architecture Specification: Use Case Design Patterns (Version 0.9). Technical Report, ORNL/TM-2023/3133, Oak Ridge National Laboratory, Oak Ridge, TN, USA, September 1, 2023. DOI 10.2172/2229218.
3. Michael Brim and Christian Engelmann. INTERSECT Architecture Specification: Microservice Architecture (Version 0.5). Technical Report, ORNL/TM-2022/2715, Oak Ridge National Laboratory, Oak Ridge, TN, USA, September 1, 2022. DOI 10.2172/1902805.
4. Christian Engelmann and Suhas Somnath. INTERSECT Architecture Specification: Use Case Design Patterns (Version 0.5). Technical Report, ORNL/TM-2022/2681, Oak Ridge National Laboratory, Oak Ridge, TN, USA, September 1, 2022. DOI 10.2172/1896984.
5. Olga A. Kuchar, Swen Boehm, Thomas Naughton, Suhas Somnath, Ben Mintz, Jack Lange, Scott Atchley, Rohit Srivastava, and Patrick Widener. INTERSECT Architecture Specification: System-of-systems Architecture (Version 0.5). Technical Report, ORNL/TM-2022/2717, Oak Ridge National Laboratory, Oak Ridge, TN, USA, September 1, 2022. DOI 10.2172/1968700.

Talks and Lectures

1. Christian Engelmann. The Interconnected Science Ecosystem (INTERSECT). Invited talk at the Hartree Centre, Science and Technology Facilities Council, Daresbury, UK, October 4, 2023.
2. Christian Engelmann. The Interconnected Science Ecosystem (INTERSECT) Architecture. Invited talk at the 20th Smoky Mountains Computational Sciences & Engineering Conference (SMC), Knoxville, TN, USA, August 21-23, 2023.
3. Christian Engelmann. The Interconnected Science Ecosystem (INTERSECT) Architecture. Seminar at the Leibniz Rechenzentrum (LRZ), Garching, Germany, July 10, 2023.
4. Christian Engelmann. The Interconnected Science Ecosystem (INTERSECT) Architecture. Invited talk at the 1st Ecosystems for Smart Autonomous Interconnected Labs (E-SAIL) Workshop, held in conjunction with the 38th ISC High Performance (ISC) 2023, Hamburg, Germany, May 25, 2023.
5. Ben Mintz, Christian Engelmann, Elke Arenholz, and Ryan Coffee. Enabling Self-Driven Experiments for Science through an Interconnected Science Ecosystem (INTERSECT). Panel at the 17th Smoky Mountains Computational Sciences & Engineering Conference (SMC), October 20, 2021.

Symbols: Abstract, Publication, Presentation, BibTeX Citation