Big-data applications—from the analysis of information in industries ranging from health care to entertainment to transportation—have large volumes of data. This data must be maintained, managed, and operated on disk-based computer-storage systems via input/output (I/O) systems, which read, process, and ultimately deliver the information. An ongoing problem in working with big data is that the performance improvement of storage systems has been much slower than that of advancements made in memory speed and capacity, thereby creating an I/O performance gap, or bottleneck.
The research group of Xian-He Sun, Distinguished Professor of Computer Science, has received a grant from the National Science Foundation (NSF) for the project “Framework: Software: NSCI: Collaborative Research: Hermes: Extending the HDF Library to Support Intelligent I/O Buffering for Deep Memory and Storage Hierarchy Systems.” The award is expected to total $3 million over four years.
“Our main objective is to develop an I/O buffering system that will utilize a deep memory and storage hierarchy design that will significantly accelerate I/O performance,” says Sun, a leading researcher in this field, whose team has received NSF grants over the past 10 years for related work. Sun is collaborating with The HDF Group, the developer of the widely used HDF5 high-performance I/O library, to extend the HDF5 library with the Hermes I/O buffering system. “To reduce this gap, storage subsystems are undergoing extensive changes, adopting new technologies and adding more software management functionalities into the memory/storage hierarchy,” adds Sun.
The collaborators aim to transform their methodologies into HDF5 software libraries to benefit the general big data and high-performance computing communities.