sys·ad·min·ol·o·gy [sis-ad-mih-nol-uh-jee]

noun

  1. The scientific study of system administration and related phenomena.

Sunday, 20 January 2013

Amazon Cloud: Ephemeral Drives

I've been working with the Amazon Cloud computing platform for sometime now, and thought it best to document some of my experiences, and knowledge. This first post will deal with Ephemeral drives!

What are they?

They are drives which come with any instance type, expect that of Micro instances. The amount of space, and number of drives you get is dependant on the instance type you select.

I was mainly working with M1.Large and M1.Extra Large instances, and so received two or four 420Gb ephemeral drives accordingly.

What's more, they are drives which are physically connected to the host your instance resides on. That means that in terms of speed, they are probably the quickest type of storage available to your instance(s), especially when compared to EBS volumes.

So why use EBS volumes if these ephemeral drives are quicker?

So here's the bad news! Ephemeral storage only persists for as long as the instance is running. Data will survive a reboot, but not a shut down or termination of an instance. Therefore, ephemeral storage should not be used as a reliable method of storing important data, unless its backed up to an EBS volume or something else with greater longevity.

So what can you use ephemeral storage for?

Anything that benefits from quick access, but is not important in terms of data retention. This includes:

  • Swap space - probably the most obvious benefactor of ephemeral drives
  • tmp folder - to make the read / write of tmp files quicker
  • Session files - This depends on your use of sessions, but can speed session clear down considerably.
  • Temporary Mysql databases / tables
  • Caching - In Squid for example you can define where cached data is stored. In any cache situation, you want the cached data to be accessed as quickly as possible.

How do I access the ephemeral drives?

They are attached when the instance is booted and should be visible in your /dev folder. You can get a complete list of drives on your instance (assuming its linux) by typing:

fdisk -l

In Ubuntu they are usually called /dev/xvdb & /dev/xvdc and so on...

They can be partitioned in the usual way using fdisk. 

Should I put drives created on ephemeral storage in the fstab?

It might seem like an obvious "yes" but its not so straight forward. Remember that data on ephemeral storage will survive a reboot, but not instance shut down or termination. Also remember that ephemeral storage is a physical disk (or disks) attached directly to the host your instance is running on, and hardware can and will fail (eventually)

Best practice is to assume that ephemeral storage only has persistence while the instance is running, therefore when your instance boots up, it should via scripts or whatever means, setup the ephemeral drive for use according to your own needs. This includes mounting.

This way, hardware failure aside, you can guarantee the ephemeral storage will be in a usable state by the time the instance is booted and ready to do its job.

How about you? Do you have any special uses for ephemeral storage? Use the comments section to share your thoughts!

No comments:

Post a Comment