Week 0 : Internship Begins

Hi everyone!

I would be writing this blog series to document my internship at Code and Design Lab, IISc.

The internship came out to me as a surprise as Dr.P.V.Kumar replied to my mail which was over a month old and I had to have a interview on the day my semester exams ended. Luckily I could do well at the unprepared interview, taken by Myna Vajha et al., and well got selected for the internship.

For the next two months I would be working to implement the theoretical framework of Clay Codes, which are a high rate MSR (maximum regenerating codes) erasure codes for distributed storage, in HDFS in Hadoop 3.0.

Erasure codes in storage for distributed systems perform very well over the popular 3x replication and are known to reduce storage overheads significantly to less than 50% compared to that of 200% former. But they also come with computation and network bandwidth considerations. MSR codes are vector codes which try to optimise the repair bandwidth, which is the amount of data required to recover lost data nodes (storage units). Erasure codes were introduced to Hadoop in its 3.0 release.

These initial days at my internship were spent setting up my PC and getting it ready for installation of Hadoop. Also spent the major part of the week in understaning Erasure coding, through a lecture series by Vinayak and getting settled with the team. Hope to learn more about Clay Codes and their framework. Will try to write a post compiling the details about the same.

That’s all folks for this week.