Dataset Placement and Data Loading Optimizations for Cloud-Native Deep Learning Workloads
Author
Abstract
The primary challenge facing cloud-based deep learning systems is the need for efficient orchestration of large-scale datasets with diverse data formats and provisioning of high-performance data loading capabilities. To that end, we present DLCache, a cloud-native dataset management and runtime-aware data-loading solution for deep learning training jobs. DLCache supports the low-latency and high-throughput I/O requirements of DL training jobs using cloud buckets as persistent data storage and a dedicated computation cluster for training.
Year of Publication
2023
Conference Name
IEEE International Symposium on Real-time Computing (ISORC)
Date Published
May
Publisher
IEEE
Conference Location
Nashville, TN
ISBN Number
979-8-3503-3902-4
Accession Number
23517989
URL
https://ieeexplore.ieee.org/document/10196902
DOI
10.1109/ISORC58943.2023.00023
Google Scholar | BibTeX | XML | DOI