CaltechAUTHORS
Published January 2021 | Version Supplemental Material + Submitted
Journal Article Open

DeepCell Kiosk: scaling deep learning–enabled cellular image analysis with Kubernetes

Creators

  • 1. ROR icon California Institute of Technology
  • 2. ROR icon Stanford University
  • 3. ROR icon Cloud Posse (United States)

Abstract

Deep learning is transforming the analysis of biological images, but applying these models to large datasets remains challenging. Here we describe the DeepCell Kiosk, cloud-native software that dynamically scales deep learning workflows to accommodate large imaging datasets. To demonstrate the scalability and affordability of this software, we identified cell nuclei in 10⁶ 1-megapixel images in ~5.5 h for ~US$250, with a cost below US$100 achievable depending on cluster configuration. The DeepCell Kiosk can be downloaded at https://github.com/vanvalenlab/kiosk-console; a persistent deployment is available at https://deepcell.org/.

Additional Information

© The Author(s), under exclusive licence to Springer Nature America, Inc. 2021. Received 15 October 2019; Accepted 23 November 2020; Published 04 January 2021. We thank numerous colleagues including A. Anandkumar, M. Angelo, J. Bois, I. Brown, A. Butkovic, L. Cai, I. Camplisson, M. Covert, M. Elowitz, J. Freeman, C. Frick, L. Geontoro, A. Ho, K. Huang, K. C. Huang, G. Johnson, L. Keren, D. Litovitz, D. Macklin, U. Manor, S. Patel, A. Raj, N. Pelaez Restrepo, C. Pavelchek, S. Shah and M. Thomson for helpful discussions and contributing data. We gratefully acknowledge support from the Shurl and Kay Curci Foundation, the Rita Allen Foundation, the Paul Allen Family Foundation through the Allen Discovery Center at Stanford University, the Rosen Center for Bioengineering at Caltech, Google Research Cloud, Figure 8's AI For Everyone award, and a subaward from NIH U24-CA224309-01. Data availability: All data that were used to generate the figures in this paper are available at https://deepcell.org/data and at https://github.com/vanvalenlab/deepcell-tf under the deepcell.datasets module. Code availability: We used Kubernetes and TensorFlow, along with the scientific computing stack for Python. A persistent deployment of the software described can be accessed at https://deepcell.org/. All source code, including version requirements and explicit usage, is under a modified Apache license and is available at https://github.com/vanvalenlab. Detailed instructions are available at https://deepcell-kiosk.readthedocs.io. Author Contributions: D.B., W.G. and D.V.V. conceived the project; D.B., W.G., E.O. and D.V.V. designed the software architecture; D.B., E.O. and W.G. wrote the core components of the software; D.B., E.M., M.S., E.B., V.V., B.C., E.O., W.G. and D.V.V. contributed to the code base; T.K. and E.P. collected data for annotation; E.M., M.S., N.G., D.B., W.G. and D.V.V. wrote documentation; D.B., E.M., W.G. and D.V.V. wrote the paper; D.V.V. supervised the project. Competing interests: The authors have filed a provisional patent for the described work; the software described here is available under a modified Apache license and is free for non-commercial uses. Peer review information: Nature Methods thanks Ola Spjuth and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. Rita Strack was the primary editor on this article and managed its editorial process and peer review in collaboration with the rest of the editorial team.

Errata

Bannon, D., Moen, E., Schwartz, M. et al. Publisher Correction: DeepCell Kiosk: scaling deep learning–enabled cellular image analysis with Kubernetes. Nat Methods (2021). https://doi.org/10.1038/s41592-021-01059-w

Attached Files

Submitted - 505032v4.full.pdf

Supplemental Material - 41592_2020_1023_MOESM1_ESM.pdf

Supplemental Material - 41592_2020_1023_MOESM2_ESM.pdf

Files

41592_2020_1023_MOESM1_ESM.pdf

Files (7.8 MB)

Name Size
md5:1dd0c540411e40bf31feeeec84ab1112
2.4 MB Preview Download
md5:270c7c107af1c2c242faaa5f80a5612c
2.7 MB Preview Download
md5:f547ef3d80a01917f3f6430bc9b5cfd8
2.7 MB Preview Download

Additional details

Additional titles

Alternative title
Dynamic allocation of computational resources for deep learning-enabled cellular image analysis with Kubernetes

Identifiers

Eprint ID
98655
Resolver ID
CaltechAUTHORS:20190916-101510993

Describes
10.1038/s41592-021-01059-w (DOI)
10.1101/505032 (DOI)
https://github.com/vanvalenlab/kiosk-console (URL)
https://deepcell.org/data (URL)
https://github.com/vanvalenlab/deepcell-tf (URL)
https://deepcell-kiosk.readthedocs.io (URL)

Funding

Shurl and Kay Curci Foundation
Rita Allen Foundation
Paul Allen Family Foundation
Donna and Benjamin M. Rosen Bioengineering Center
Google Research Cloud
Figure 8's AI for Everyone
NIH
U24-CA224309-01

Dates

Created
2019-09-16
Created from EPrint's datestamp field
Updated
2023-06-01
Created from EPrint's last_modified field

Caltech Custom Metadata

Caltech groups
Rosen Bioengineering Center , Division of Biology and Biological Engineering (BBE)