{"id":348,"date":"2023-02-14T08:00:59","date_gmt":"2023-02-14T08:00:59","guid":{"rendered":"https:\/\/christian-engelmann.de\/?page_id=348"},"modified":"2023-02-15T20:14:17","modified_gmt":"2023-02-15T20:14:17","slug":"2006-09-harness-workbench-unified-and-adaptive-access-to-diverse-hpc-platforms","status":"publish","type":"page","link":"https:\/\/www.christian-engelmann.info\/?page_id=348","title":{"rendered":"2006-09: Harness Workbench: Unified and Adaptive Access to Diverse HPC Platforms"},"content":{"rendered":"<p>The goal of this project is to enhance the overall productivity of applications science on diverse high performance computing (HPC) platforms by conducting research in two innovative software environments. The first is a virtualized command toolkit (VCT) for application building and execution that provides a common view across diverse HPC systems. The VCT consists of a software backplane architecture that presents a uniform but extensible interface for preparatory and pre-execution stages of application execution, which interfaces to instance-specific software via customizable plug-in modules.<\/p>\n<p>The second is a next generation runtime environment (RTE) that similarly provides a flexible, adaptive framework for plugging in modules optimized for a specific HPC system and allows dynamic interfacing to a variety of user environments. Both these environments employ platform specific pluggable modules disseminating target-specific knowledge and expertise immediately to all end-users who can continue to interface to a familiar environment.<\/p>\n<h4>Funding Sources<\/h4>\n<ul>\n<li>\n<a href=\"http:\/\/science.energy.gov\/ascr\" target=\"science.energy.gov_ascr\" rel=\"noopener\">Office of Advanced Scientific Computing Research<\/a>, Office of Science, U.S. Department of Energy<\/li>\n<\/ul>\n<h4>Participating Institutions<\/h4>\n<ul>\n<li><a href=\"http:\/\/www.ornl.gov\" target=\"www.ornl.gov\" rel=\"noopener\">Oak Ridge National Laboratory<\/a><\/li>\n<li><a href=\"http:\/\/www.utk.edu\" target=\"www.utk.edu\" rel=\"noopener\">University of Tennessee, Knoxville<\/a><\/li>\n<li><a href=\"http:\/\/www.emory.edu\" target=\"www.emory.edu\" rel=\"noopener\">Emory University<\/a><\/li>\n<\/ul>\n<h4>Peer-reviewed Conference Publications<\/h4>\n<ol>\n<li>Bj&ouml;rn K&ouml;nning, Christian Engelmann, Stephen L. Scott, and George A. (Al) Geist. <b>Virtualized Environments for the Harness High Performance Computing Workbench<\/b>. In <i>Proceedings of the <a href=\"http:\/\/www.pdp2008.org\" target=\"www.pdp2008.org\" rel=\"noopener\">16th Euromicro International Conference on Parallel, Distributed, and network-based Processing (PDP) 2008<\/a><\/i>, pages 133-140, Toulouse, France, February 13-15, 2008. <a href=\"http:\/\/www.computer.org\" target=\"www.computer.org\" rel=\"noopener\">IEEE Computer Society, Los Alamitos, CA, USA<\/a>. ISBN 978-0-7695-3089-5. DOI <a href=\"http:\/\/dx.doi.org\/10.1109\/PDP.2008.14\" target=\"publication\" rel=\"noopener\">10.1109\/PDP.2008.14<\/a>. Acceptance rate 40% (83\/207). <a href=\"javascript:showAbstract('This paper describes recent accomplishments in providing a virtualized environment concept and prototype for scientific application development and deployment as part of the Harness High Performance Computing (HPC) Workbench research effort. The presented work focuses on tools and mechanisms that simplify scientific application development and deployment tasks, such that only minimal adaptation is needed when moving from one HPC system to another or after HPC system upgrades. The overall technical approach focuses on the concept of adapting the HPC system environment to the actual needs of individual scientific applications instead of the traditional scheme of adapting scientific applications to individual HPC system environment properties. The presented prototype implementation is based on the mature and lightweight chroot virtualization approach for Unix-type systems with a focus on virtualized file system structure and virtualized shell environment variables utilizing virtualized environment configuration descriptions in Extensible Markup Language (XML) format. The presented work can be easily extended to other virtualization technologies, such as system-level virtualization solutions using hypervisors.');\"><img decoding=\"async\" src=\"images\/txt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Abstract\"><\/a> <a href=\"publications\/koenning08virtualized.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/pdf.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Publication\"><\/a> <a href=\"publications\/koenning08virtualized.ppt.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/ppt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Presentation\"><\/a> <a href=\"?page_id=55#koenning08virtualized\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<\/ol>\n<h4>Peer-reviewed Conference Posters<\/h4>\n<ol>\n<li>George A. (Al) Geist, Christian Engelmann, Jack J. Dongarra, George Bosilca, Magdalena M. S&#322;awi&#324;ska, and Jaros&#322;aw K. S&#322;awi&#324;ski. <b>The Harness Workbench: Unified and Adaptive Access to Diverse High-Performance Computing Platforms<\/b>. Poster at the <a href=\"http:\/\/www.hpcsw.org\" target=\"www.hpcsw.org\" rel=\"noopener\">1st High-Performance Computer Science Week (HPCSW) 2008<\/a>, Denver, CO, USA, March 30 &#8211; April 5, 2008. <a href=\"javascript:showAbstract('This poster summarizes our past and ongoing research and development efforts in novel software solutions for providing unified and adaptive access to diverse high-performance computing (HPC) platforms. The poster showcases developed proof-of-concept implementations of tools and mechanisms that simplify scientific application development and deployment tasks, such that only minimal adaptation is needed when moving from one HPC system to another or after HPC system upgrades.');\"><img decoding=\"async\" src=\"images\/txt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Abstract\"><\/a> <a href=\"publications\/geist08harness.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/pdf.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Publication\"><\/a> <a href=\"?page_id=55#geist08harness\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<\/ol>\n<h4>Talks and Lectures<\/h4>\n<ol>\n<li>Christian Engelmann. <b>High-Performance Computing Research at Oak Ridge National Laboratory<\/b>. Invited talk at the Reading Annual Computational Science  Workshop, Reading, United Kingdom, December 8, 2008. <a href=\"javascript:showAbstract('Oak Ridge National Laboratory (ORNL) is the largest energy laboratory in the United States. Its National Center for Computational Sciences (NCCS) provides the most powerful computing resources in the world for open scientific research. Jaguar, a Cray XT5 system at NCCS, is the second HPC system to exceed 1 PFlop\/s (10^15 Floating Point Operations Per Second), and the fastest open science supercomputer in the world. It recently ranked #2 in the Top 500 List of Supercomputer Sites with a maximal LINPACK benchmark performance of 1.059 PFlop\/s and a theoretical peak performance of 1.3814 PFlop\/s. Annually, 80 percent of Jaguar\u2019s resources are allocated through the U.S Department of Energy\u2019s Innovative and Novel Computational Impact on Theory and Experiment (INCITE) program, a competitively selected, peer reviewed process open to researchers from universities, industry, government and non-profit organizations. These allocations address some of the most challenging scientific problems in areas such as climate modeling, renewable energy, materials science, fusion and combustion. In conjunction with NCCS, the Computer Science and Mathematics Division at ORNL performs basic and applied research in HPC, mathematics, and intelligent systems. This talk gives a summary of the HPC research performed at ORNL. It provides details about the Jaguar peta-scale computing resource, an overview of the computational science research carried out using ORNL\u2019s computing resources, and a description of various computer science efforts targeting solutions for next-generation HPC systems.');\"><img decoding=\"async\" src=\"images\/txt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Abstract\"><\/a> <a href=\"publications\/engelmann08high.ppt.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/ppt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Presentation\"><\/a> <a href=\"?page_id=55#engelmann08high\"><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<\/ol>\n<h4>Co-advised Theses<\/h4>\n<ol>\n<li>Bj&ouml;rn K&ouml;nning. <b>Virtualized Environments for the Harness Workbench<\/b>. Master&#8217;s thesis, <a href=\"http:\/\/www.cs.reading.ac.uk\" target=\"www.cs.reading.ac.uk\" rel=\"noopener\">Department of Computer Science<\/a>, <a href=\"http:\/\/www.reading.ac.uk\" target=\"www.reading.ac.uk\" rel=\"noopener\">University of Reading<\/a>, UK, March 14, 2007. Thesis research performed at Oak Ridge National Laboratory. Advisors: Prof. Vassil N. Alexandrov (University of Reading); Christian Engelmann (Oak Ridge National Laboratory). <a href=\"javascript:showAbstract('The expanded use of computational sciences today leads to a significant need of high performance computing systems. High performance computing is currently undergoing vigorous revival, and multiple efforts are underway to develop much faster computing systems in the near future. New software tools are required for the efficient use of petascale computing systems. With the new Harness Workbench Project the Oak Ridge National Laboratory intends to develop an appropriate development and runtime environment for high performance computing platforms. This dissertation project is part of the Harness Workbench Project, and deals with the development of a concept for virtualised environments and various approaches to create and describe them. The developed virtualisation approach is based on the \\verb|chroot| mechanism and uses platform-independent environment descriptions. File structures and environment variables are emulated to provide the portability of computational software over diverse high performance computing platforms. Security measures and sandbox characteristic are integrable.');\"><img decoding=\"async\" src=\"images\/txt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Abstract\"><\/a> <a href=\"publications\/koenning07virtualized.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/pdf.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Publication\"><\/a> <a href=\"publications\/koenning07virtualized.ppt.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/ppt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Presentation\"><\/a> <a href=\"?page_id=55#koenning07virtualized\" ><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<li>Ronald Baumann. <b>Design and Development of Prototype Components for the Harness High-Performance Computing Workbench<\/b>. Master&#8217;s thesis, <a href=\"http:\/\/www.cs.reading.ac.uk\" target=\"www.cs.reading.ac.uk\" rel=\"noopener\">Department of Computer Science<\/a>, <a href=\"http:\/\/www.reading.ac.uk\" target=\"www.reading.ac.uk\" rel=\"noopener\">University of Reading<\/a>, UK, March 6, 2006. Thesis research performed at Oak Ridge National Laboratory. Double diploma in conjunction with the <a href=\"http:\/\/www.f1.fhtw-berlin.de\" target=\"www.f1.fhtw-berlin.de\" rel=\"noopener\">Department of Engineering I<\/a>, <a href=\"http:\/\/www.f1.fhtw-berlin.de\" target=\"www.f1.fhtw-berlin.de\" rel=\"noopener\">Technical College for Engineering and Economics (FHTW) Berlin<\/a>, Germany. Advisors: Prof. Vassil N. Alexandrov (University of Reading); George A. (Al) Geist and Christian  Engelmann (Oak Ridge National Laboratory). <a href=\"javascript:showAbstract('This master thesis examines plug-in technology, especially the new field of parallel plug-ins. Plug-ins are popular because they extend the capabilities of software packages such as browsers and Photoshop, and allow an individual user to add new functionality. Parallel plug-ins also provide the above capabilities to a distributed set of resources, i.e., a plug-in now becomes a set of coordinating plug-ins. Second, the set of plugins may be heterogeneous either in function or because the underlying resources are heterogeneous. This new dimension of complexity provides a rich research space which is explored in this thesis. Experiences are collected and presented as parallel plug-in paradigms and concepts. The Harness framework was used in this project, in particular the plugin manager and available communication capabilities. Plug-ins provide methods for users to extend Harness according to their requirements. The result of this thesis is a parallel plug-in paradigm and template for Harness. Users of the Harness environment will be able to design and implement their applications in the form of parallel plug-ins easier and faster by using the paradigm resulting from this project. Prototypes were implemented which handle different aspects of parallel plug-ins. Parallel plug-in configurations were tested on an appropriate number of Harness kernels, including available communication and error-handling capabilities. Furthermore, research was done in the area of fault tolerance while parallel plug-ins are (un)loaded, as well as while a task is performed.');\"><img decoding=\"async\" src=\"images\/txt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Abstract\"><\/a> <a href=\"publications\/baumann06design.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/pdf.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Publication\"><\/a> <a href=\"publications\/baumann06design.ppt.pdf\" target=\"publication\" rel=\"noopener\"><img decoding=\"async\" src=\"images\/ppt.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"Presentation\"><\/a> <a href=\"?page_id=55#baumann06design\" ><img decoding=\"async\" src=\"images\/bib.gif\" border=\"0\" style=\"border-style:none\" height=\"10pt\" alt=\"BibTeX Citation\"><\/a><\/li>\n<\/ol>\n<p><em><small>Symbols: <img decoding=\"async\" style=\"border-style: none;\" src=\"images\/txt.gif\" border=\"0\" alt=\"Abstract\" height=\"10pt\"> Abstract, <img decoding=\"async\" style=\"border-style: none;\" src=\"images\/pdf.gif\" border=\"0\" alt=\"Publication\" height=\"10pt\"> Publication, <img decoding=\"async\" style=\"border-style: none;\" src=\"images\/ppt.gif\" border=\"0\" alt=\"Presentation\" height=\"10pt\"> Presentation, <img decoding=\"async\" style=\"border-style: none;\" src=\"images\/bib.gif\" border=\"0\" alt=\"BibTeX Citation\" height=\"10pt\"> BibTeX Citation<\/small><\/em><\/p>\n<p><script language=\"JavaScript\">\nfunction showAbstract (text) {\n  var width  = 400;\n  var height = 400;\n  var left   = (screen.width  - width ) \/ 2;\n  var top    = (screen.height - height) \/ 2;\n  var win    = window.open('',\n                           'Abstract',\n                           'width='  + width  + ', ' + \n                           'height=' + height + ', ' +\n                           'left='   + left   + ', ' +\n                           'top='    + top    + ', ' +\n                           'toolbar=no, '     +\n                           'location=no, '    +\n                           'directories=no, ' +\n                           'status=no, '      +\n                           'menubar=no, '     +\n                           'copyhistory=no, ' +\n                           'scrollbars=yes, ' +\n                           'resizable=yes')\n  win.document.write(text);\n  win.document.close();\n}\n<\/script><\/p>\n","protected":false},"excerpt":{"rendered":"<p>The goal of this project is to enhance the overall productivity of applications science on diverse high performance computing (HPC) platforms by conducting research in two innovative software environments. The first is a virtualized command toolkit (VCT) for application building and execution that provides a common view across diverse HPC systems. The VCT consists of&hellip;&nbsp;<\/p>\n","protected":false},"author":1,"featured_media":0,"parent":145,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"neve_meta_sidebar":"","neve_meta_container":"","neve_meta_enable_content_width":"off","neve_meta_content_width":100,"neve_meta_title_alignment":"","neve_meta_author_avatar":"","neve_post_elements_order":"","neve_meta_disable_header":"","neve_meta_disable_footer":"","neve_meta_disable_title":"","footnotes":""},"class_list":["post-348","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/www.christian-engelmann.info\/index.php?rest_route=\/wp\/v2\/pages\/348","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.christian-engelmann.info\/index.php?rest_route=\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/www.christian-engelmann.info\/index.php?rest_route=\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/www.christian-engelmann.info\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.christian-engelmann.info\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=348"}],"version-history":[{"count":4,"href":"https:\/\/www.christian-engelmann.info\/index.php?rest_route=\/wp\/v2\/pages\/348\/revisions"}],"predecessor-version":[{"id":403,"href":"https:\/\/www.christian-engelmann.info\/index.php?rest_route=\/wp\/v2\/pages\/348\/revisions\/403"}],"up":[{"embeddable":true,"href":"https:\/\/www.christian-engelmann.info\/index.php?rest_route=\/wp\/v2\/pages\/145"}],"wp:attachment":[{"href":"https:\/\/www.christian-engelmann.info\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=348"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}