Monday, September 26, 2011

The Google File System

This paper shows a good example of how a good engineering becomes research. The authors explains their scalable distributed file system, Google File System, keeping the following assumptions in mind:

  • Cost effectiveness is their first priority. They use attached disk drives on inexpensive commodity servers. Their approach is clearly distinguished from the practice of dedicated storage area network at that time.
  • They tried to optimize their fille system to Google's applications and workloads. Most disk access patterns are sequential, most files are big (at least multi GBs), throughput is way more important than latency, and most updates to the file system are appended to the end of file.
So their decision making is somewhat extraordinary from a traditional point of view. Chunk size is unusually big (64MB) since large, sequential access is dominant in their workloads. 

Another point is that GFS uses a single master, which is sometimes regarded as the axis of evil. I think this is a good example of the "centralize if you can, distribute if you must" principle. Many decision makings including large chunk size, metadata caching on client, lease, and master replication make this centralized master affordable, enabling their implementation simple enough to manage.

No comments:

Post a Comment