Dataset Placement and Data Loading Optimizations for Cloud-Native Deep Learning Workloads | |
---|---|
Author | |
Abstract |
The primary challenge facing cloud-based deep learning systems is the need for efficient orchestration of large-scale datasets with diverse data formats and provisioning of high-performance data loading capabilities. To that end, we present DLCache, a cloud-native dataset management and runtime-aware data-loading solution for deep learning training jobs. DLCache supports the low-latency and high-throughput I/O requirements of DL training jobs using cloud buckets as persistent data storage and a dedicated computation cluster for training.
|
Year of Publication |
2023
|
Conference Name |
IEEE International Symposium on Real-time Computing (ISORC)
|
Date Published |
May
|
Publisher |
IEEE
|
Conference Location |
Nashville, TN
|
ISBN Number |
979-8-3503-3902-4
|
Accession Number |
23517989
|
URL |
https://ieeexplore.ieee.org/document/10196902
|
DOI |
10.1109/ISORC58943.2023.00023
|
Google Scholar | BibTeX | XML | DOI |