Pegasus (workflow management)

Pegasus is an open-source workflow management system.[1][2][3] It provides the necessary abstractions for scientists to create scientific workflows[4] and allows for transparent execution of these workflows on a range of computing platforms including high performance computing clusters, clouds, and national cyberinfrastructure.[5][6] In Pegasus, workflows are described abstractly as directer acyclic graphs (DAGs) using a provided API for Jupyter Notebooks, Python, R, or Java.[7] During execution, Pegasus translates the constructed abstract workflow into an executable workflow[8][9] which is executed and managed by HTCondor.[10][11]

Pegasus
Developer(s)University of Southern California, Information Sciences Institute, University of Wisconsin-Madison
Stable release
5.0 Beta1 / July 27, 2020 (2020-07-27)
Written in Java, Python, C
Operating systemmacOS, Linux
Available in Java, Python, C
TypeWorkflow management system
LicenseApache License 2.0
Websitepegasus.isi.edu

Pegasus is being used in a number of different disciplines including astronomy, gravitational-wave physics, bioinformatics, earthquake engineering, and helioseismology.[12] Notably, the LIGO Scientific Collaboration has used it to directly detect a gravitational wave for the first time.[8][13][14]

Area of applications

Application examples:[5][6]

  • Gravitational-Wave Physics
  • Earthquake Science
  • Bioinformatics
  • Workflows for Volcanic Mass Flows
  • Diffusion Image Processing and Analysis
  • Spallation Neutron Source (SNS)

History

The development of Pegasus started in 2001.

See also

References
  1. E. Deelman, K. Vahi, G. Juve, M. Rynge, S. Callaghan, P. J. Maechling, R. Mayani, W. Chen, R. Ferreira da Silva, M. Livny, and K. Wenger, "Future Generation Computer Systems", Elsevier; 46, pp. 17-35 (2015)
  2. E.A. Huerta, R. Haas, E. Fajardo, D.S. Katz, S. Anderson, P. Couvares ,J. Willis, T. Bouvet, J. Enos, W.T.C. Kramer, H.W. Leong, and D. Wheeler, "BOSS-LDG: A Novel Computational Framework That Brings Together Blue Waters, Open Science Grid, Shifter and the LIGO Data Grid to Accelerate Gravitational Wave Discovery", 2017 IEEE 13th International Conference on e-Science (e-Science); pp. 335-344 (2017)
  3. B. Riedel, B. Bauermeister, L. Bryant, J. Conrad, P. de Perio, R. W. Gardner ,L. Grandi, F. Lombardi, A. Rizzo, G. Sartorelli, M. Selvi, E. Shockley, J. Stephen, S. Thapa, and C. Tunnell "Distributed Data and Job Management for the XENON1T Experiment", PEARC '18: Proceedings of the Practice and Experience on Advanced Research Computing;9, pp. 1-8 (2018)
  4. G. Amalarethinam, T. Lucia, A. Beena, “Scheduling Framework for Regular Scientific Workflows in Cloud”, International Journal of Applied Engineering Research; 10, no. 82 (2015)
  5. E. Deelman, G. Singh, M. Su, J. Blythe, Y. Gil, C. Kesselman, G. Mehta, K. Vahi, B. G. Berriman, J. Good, A. Laity, J. C. Jacob, and D. S. Katz, “Pegasus: a Framework for Mapping Complex Scientific Workflows onto Distributed Systems”, Scientific Programming; 13, pp. 19 (2005)
  6. The Scientific Workflow Integrity with Pegasus (SWIP), by Center for Applied Cybersecurity Research; published 16 September 2016; retrieved 1 May 2020
  7. D. Weitzel, B. Bockelman, D. Brown, P. Couvares, F. Würthwein, and E.F. Hernandez, “Data Access for LIGO on the OSG”, Proceedings of the Practice and Experience in Advanced Research Computing 2017 on Sustainability, Success and Impact - PEARC17; 24, no. 1-6 (2017)
  8. "Testing LIGO's Sensitivity". Research.gov. September 1, 2007. Retrieved April 30, 2020.
  9. Duncan Brown and Ewa Deelman, "Looking for gravitational waves: A computing perspective", at Science Node; published June 8, 2011; retrieved April 30, 2020
  10. $1M NSF award goes to IU-led data integrity project, by Indiana University; published 16 September 2016; retrieved 1 May 2020
  11. Brian Mattmiller, "High Throughput Computing helps LIGO confirm Einstein’s last unproven theory", at Morgridge Institute for Research; published March 7, 2016; retrieved May 1, 2020
  12. Sanden Totten, "Caltech Wasn’t the Only SoCal School Helping Discover Gravitational Waves", at KPCC; published 11 February 2016; retrieved May 1, 2020
  13. D.A. Brown, P.R. Brady, A. Dietz, J. Cao, B. Johnson, J. McNabb, “A Case Study on the Use of Workflow Technologies for Scientific Analysis: Gravitational Wave Data Analysis. In: I.J Taylor, E. Deelman, D.B. Gannon, M. Shields (eds) Workflows for e-Science”, Springer, London; 13, pp. 39-59 (2007)
  14. D. Davis, T. Massinger, A. Lundgren, J.C. Driggers, A.L. Urban, and L. Nuttall, “Improving the sensitivity of Advanced LIGO using noise subtraction”, Classical and Quantum Gravity; 36, no. 5 (2019)
This article is issued from Wikipedia. The text is licensed under Creative Commons - Attribution - Sharealike. Additional terms may apply for the media files.