Enabling Efficient and Reliable Transition from Replication to Erasure Coding for Clustered File Systems
Refereed conference paper presented and published in conference proceedings


Times Cited
Altmetrics Information
.

Other information
AbstractTo balance performance and storage efficiency, modern clustered file systems (CFSes) often first store data with random replication (i.e., distributing replicas across randomly selected nodes), followed by encoding the replicated data with erasure coding. We argue that random replication, while being commonly used, does not take into account erasure coding and hence will raise both performance and availability issues to the subsequent encoding operation. We propose encoding-aware replication, which carefully places the replicas so as to (i) avoid cross-rack downloads of data blocks during encoding, (ii) preserve availability without data relocation after encoding, and (iii) maintain load balancing as in random replication. We implement encoding-aware replication on HDFS, and show via tested experiments that it achieves significant encoding throughput gains over random replication. We also show via discrete-event simulations that encoding-aware replication remains effective under various parameter choices in a large-scale setting. We further show that encoding-aware replication evenly distributes replicas as in random replication.
All Author(s) ListLi R., Hu Y., Lee P.P.C.
Name of Conference45th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2015
Start Date of Conference22/06/2015
End Date of Conference25/06/2015
Place of ConferenceRio de Janeiro
Country/Region of ConferenceBrazil
Detailed descriptionorganized by IEEE/IFIP,
Year2015
Month9
Day14
Volume Number2015-September
Pages148 - 159
ISBN9781479986293
LanguagesEnglish-United Kingdom

Last updated on 2020-06-08 at 01:56