CaltechAUTHORS
Published 2007 | Version Published
Journal Article Open

Optimizing Workflow Data Footprint

Creators

  • 1. ROR icon University of Southern California
  • 2. ROR icon University of Manchester
  • 3. ROR icon California Institute of Technology
  • 4. ROR icon University of Wisconsin–Milwaukee
  • 5. ROR icon Infrared Processing and Analysis Center
  • 6. ROR icon Louisiana State University

Abstract

In this paper we examine the issue of optimizing disk usage and scheduling large-scale scientific workflows onto distributed resources where the workflows are data-intensive, requiring large amounts of data storage, and the resources have limited storage resources. Our approach is two-fold: we minimize the amount of space a workflow requires during execution by removing data files at runtime when they are no longer needed and we demonstrate that workflows may have to be restructured to reduce the overall data footprint of the workflow. We show the results of our data management and workflow restructuring solutions using a Laser Interferometer Gravitational-Wave Observatory (LIGO) application and an astronomy application, Montage, running on a large-scale production grid-the Open Science Grid. We show that although reducing the data footprint of Montage by 48% can be achieved with dynamic data cleanup techniques, LIGO Scientific Collaboration workflows require additional restructuring to achieve a 56% reduction in data space usage. We also examine the cost of the workflow restructuring in terms of the application's runtime.

Additional Information

© 2007 Hindawi Publishing Corporation. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. This work was supported by the National Science Foundation under the grant CNS 0615412. R. Sakellariou and H. Zhao would like to acknowledge partial support from the EU-funded CoreGrid Network of Excellence (grant FP6-004265) and the UK EPSRC grant GR/S67654/01. The authors also thank the Open Science Grid for resources used for the motivation of this work. K. Blackburn and D. Meyers were supported by the LIGO Laboratory and NSF grants PHY-0107417 and PHY-0326281. The work of D. Brown was supported by the LIGO Laboratory and NSF grant PHY-0601459. The work of S. Fairhurst was supported by the LIGO Laboratory and NSF grant PHY-0326281 and PHY-0200852. LIGO was constructed by the California Institute of Technology and the Massachusetts Institute of Technology and operates under cooperative agreement PHY-0107417. This paper has been assigned LIGO Document Number LIGO-P070017-00-Z. Montage was supported by the NASA Earth Sciences Technology Office Computing Technologies (ESTOCT) program under Cooperative Agreement Notice NCC 5-6261. This research was done using resources provided by the Open Science Grid, which is supported by the National Science Foundation and the U.S. Department of Energy's Office of Science.

Attached Files

Published - 701609.pdf

Files

701609.pdf

Files (1.8 MB)

Name Size
md5:59f9ab8e89bf5084e04a61e339ad7b5f
1.8 MB Preview Download

Additional details

Identifiers

Eprint ID
86108
Resolver ID
CaltechAUTHORS:20180427-155708572

Funding

NSF
CNS-0615412
CoreGrid Network of Excellence
FP6-004265
Engineering and Physical Sciences Research Council (EPSRC)
GR/S67654/01
LIGO Laboratory
NSF
PHY-0107417
NSF
PHY-0326281
NSF
PHY-0601459
NSF
PHY-0326281
NSF
PHY-0200852
NSF
PHY-0107417
NASA
NCC 5-6261
Department of Energy (DOE)

Dates

Created
2018-04-30
Created from EPrint's datestamp field
Updated
2021-11-15
Created from EPrint's last_modified field

Caltech Custom Metadata

Caltech groups
Infrared Processing and Analysis Center (IPAC) , LIGO
Other Numbering System Name
LIGO Document
Other Numbering System Identifier
P070017-00-Z