In a recent blog post by Jeff Barr, Chief Evangelist at Amazon Web Services, he announced that Elastic Block Storage (EBS) now supports Elastic Volumes, which is a game changer in the world of block storage in the cloud.
About Elastic Volumes
At its most simplistic level, storage tiering in the cloud is now available for block storage in a similar way there were lifecycle policies on services like S3. Dynamic tiering and storage volume elasticity is not a new feature in the world of block storage. Your classic on-premises array vendors (for example, Dell EMC Fully Automated Storage Tiering (FAST)) and storage virtualization technologies (for example, VMware’s Storage vMotion & Storage DRS) have offered these features for a long time. The benefit of these on-premises offerings is to allow for the dynamic, live migration of workloads to the appropriate tiers as workload and application lifecycle dictates. The benefit of these technologies is to allow for the best price and performance of workloads as they evolve. However, this often requires buffer capacity to ensure that data could dynamically tier or be capable of migrating between storage classes through performance lifecycle policies. Traditionally, that meant requiring a minimum of a 10% buffer capacity that could never be consumed leaving stranded capacity and a loss of capital investments.
With this new release, Elastic Volumes aims to provide feature parity with the on-premises storage vendor and virtualization technology counterparts. Prior to this release, the ability to migrate workloads between storage tiers (General Purpose SSD, Provisioned IOPS, Magnetic Storage) was not a simple task. Expanding volumes required downtime (for example, powering down your EC2 instance or unmounting the volume from the EC2 instance, taking an EBS snapshot of your existing volume, then creating a new volume with a large size or different EBS volume type and reconnecting the volume to the EC2 instance) or required significant overhead to copy the data from one EBS-backed volume to another. This was wildly inefficient, especially for simple use cases like moving data from General Purpose SSD to Magnetic Storage, handling bursty workload through automated performance allocation, or simply expanding a volume by a certain capacity.
With AWS Elastic Volumes for EBS, there are three key use cases that are solved with this new offering:
1. Changing Workloads – Information lifecycle policies are not new and the same goes for application workloads. Over the course of your application’s life, you may want to move from Provisioned IOPS to General Purpose to Throughput Optimized as your application evolves. Here is a link to more information on EBS volume types and use cases.
My Thoughts – For storage admins, this is very much like selective storage tiering that most are used to with on-premises workloads via internal LUN migration utilities or Storage vMotion capabilities. Being able to dynamically make changes based on storage performance requirements or pricing needs, it solves a critical need to do this online as a background process. This will help ensure that as the application grows, scales, evolves, and is decommissioned, the EBS volume will live on the appropriate performance and pricing tier.
2. Demand Spikes – For workloads that have predictable or unpredictable performance spikes, you can now manually or through automated services like CloudWatch Events and AWS Lambda, dynamically scale up performance (ex. Scale Provisioned IOPS volumes up and down as workload dictates).
My Thoughts – This new offering essentially provides basic IOPS QoS capabilities where through monitoring and other automation tools, you can help keep performance optimal and right size for cost as the workload demands grow and scales. This dynamic allocation is very similar to the provisioned capacity units in DynamoDB today for those familiar with that AWS service.
3. Increasing Storage – Perhaps the biggest improvement with Elastic Volumes is being able to start with a smaller root/non-root volumes and expand them dynamically, online. There are a few caveats listed in the technical documentation listed linked above, but this will offer dramatically improved price optimizations by right-sizing and growing volumes as needed versus pre-allocating significant capacity in advance.
My Thoughts – In my opinion, this is the biggest cost optimization and native management capability enhancement that came out of this announcement. This will give AWS customers the flexibility to provision EBS capacity that is needed Day 1 with automatic capabilities to grow through integrations with CloudWatch Events, AWS Lambda, for Day X. This was always a unique feature with most recent operating systems and VMware or Hyper-V being able to dynamically expand VMs. By being able to hot-add capacity to Amazon AMIs, this is key to helping dynamically optimize capacity and cost as the application grows and scales.
Elastic Volumes solves a large number of technical and operational challenges by addressing capacity optimization, dynamic workload tiering, and automating storage performance as demand dictates. Enhancing this new offering with AWS CloudWatch, AWS Lamdba, and other tools will further automate the lifecycle of EBS-backed volumes by dynamically adjusting performance and cost as requirements dictate.
Benefits of the Enhancement
The primary benefit that I can see with Elastic Volumes addresses the Cost Optimization Pillar of Amazon’s Well-Architected Framework white paper. The AWS Well-Architected Framework white paper is a set of recommended principles around how to build applications and services in a public cloud environment. With the Cost Optimization Pillar, there are four key areas businesses need to address in their build and now with Elastic Volumes for EBS that can be achieved in more fully automated ways:
- Cost-effective resources – Start with one EBS volume type with the necessary performance and capacity characteristics required day 1. Gone are the days of needing to plan for day 2 through day X.
- Matching supply with demand – Handle both planned and unplanned demand spikes with your Amazon EBS workloads by dynamically adding performance (Provisioned IOPS EBS volumes) or live-migrating a volume to a higher/lower performance tier.
- Expenditure awareness – No longer is there a need to provision 1TB of capacity when 40GB will suffice. Expand the volume as needs arise without costly downtime or painful volume migrations. Modern operating systems will detect the new capacity and expand volumes online. Leveraging tools like CloudWatch and Lambda, automated expansion at the AWS EBS and EC2 AMI Operating System layers can be automatically triggered and expanded.
- Optimizing over time – This is all part of storage lifecycle management policies. As applications age over time and demand grows or fades, Elastic Volumes can be programmatically enabled to optimize where the EBS volume resides to ensure the lowest cost of ownership via CloudWatch, AWS Lambda, and other tools. This ensures that you maintain the optimum costs for your EBS volumes throughout the application lifecycle.
Building this automation requires some investment on the front-end, but will lead to significant cost savings over time. In an upcoming post, we’ll show how to build some of this automation programmatically and how to build this intelligence into your AWS EBS deployments.
Does Elastic Volumes solve all use cases for EBS-backed volumes? No, it does not. For example, workloads that have frequent performance spikes and decreases (less than every 6 hours) are not ideal candidates for this new feature as there are limits to the frequency of changes with Elastic Volumes. Shrinking volumes is also not possible, but that is an issue with most operating systems and not EBS. Alternative AWS solutions or different architecture approaches might be required to solve those particular challenges and that’s where AHEAD can help discuss further requirements with you.
Amazon continues to impress with how quickly it evolves its service offerings. The new EBS Elastic Volumes offering is generally available as of today. If you’d like to discuss how to further optimize your AWS environment or explore alternative AWS solutions, contact us today!