Deep Dive: Maximizing Amazon EC2 and Amazon Elastic Block Store Performance

  • Published on

  • View

  • Download

Embed Size (px)


<ol><li> 1. 2015, Amazon Web Services, Inc. or its affiliates. All rights reserved Amazon Elastic Block Store Deep Dive Dougal Ballantyne, HPC Solutions Architect </li><li> 2. What well cover Amazon EBS overview Volumes Snapshots Performance Encryption Q&amp;A </li><li> 3. EBS overview </li><li> 4. For most builders AWS is get in and go! Source: </li><li> 5. A normal hard drive </li><li> 6. EBS = </li><li> 7. What is EBS? Network block storage Designed for five nines of availability Attaches to Amazon EC2 within the same Availability Zone Provides point-in-time snapshots to Amazon S3 </li><li> 8. More about EBS Its a service! Its independent of EC2 It has regional and AZ availability goals All EBS volumes are designed for 99.999% availability Over 1.5 million volumes are created per day </li><li> 9. A few definitions IOPS: Input/output operations per second (#) Throughput: Read/write rate to storage (MB/s) Latency: Delay between request and completion (ms) Capacity: Volume of data that can be stored (GB) Block size: Size of each I/O (KB) </li><li> 10. EBS volume types General Purpose (SSD) Provisioned IOPS (SSD) Magnetic When performance matters, use SSD-backed volumes </li><li> 11. EBS SSD volumes Applies to both General Purpose and Provisioned IOPS IOPS measured up to 256 KB Single-digit ms latency Designed for 99.999% availability </li><li> 12. EBS General Purpose volumes (SSD) New default volume type for EBS Every volume can burst up to 3,000 IOPS Larger volumes can burst for longer periods 3 IOPS per GB baseline performance, maximum of 10,000 IOPS 99% performance consistency Up to 160 MB/s throughput </li><li> 13. General Purpose (SSD) Burst &amp; baseline 16 KB I/O size </li><li> 14. (2) Max I/O credit per bucket is 5.4M (1) Always accumulating 3 IOPS per GB per second (3) You can spend up to 3000 IOPS per second Understanding General Purpose (SSD) bursting Baseline performance = 3 IOPS per GB </li><li> 15. Minutes to empty a full I/O credit bucket for various volume sizes The larger the volume, the longer it takes to empty the I/O credit bucket 1 TB or larger volume will never exhaust its I/O credit bucket </li><li> 16. Minutes to empty a full I/O credit bucket for various sizes The larger the volume, the longer it takes to empty the I/O credit bucket 1 TB or larger volume will never exhaust its I/O credit bucket </li><li> 17. General Purpose (SSD) volumes example Microsoft Windows 30 GB boot volume: Gets initial I/O credit of 5.4M Could burst for up to 30 mins @ 3000 IOPS Always accumulating 90 I/O credits per second </li><li> 18. Improved instance boot time m3.medium Volume type Boot time Access time OS GP2 3:31 4:33 Windows Server 2012 Magnetic 4:30 7:16 Windows Server 2012 GP2 0:36 0:45 CentOS6 Magnetic 0:57 1:16 CentOS6 40% Reduction in boot times by using General Purpose SSD </li><li> 19. Database volume 1 TB PIOPS volume with 4K IOPS = $526.40 per month per volume GP2 1 TB volume with 3000 IOPS = $102.40 GP2 2 x 500 GB volume at 3K, Burst to 6K = $102.40 80% cost savings, 50% more peak I/O with General Purpose SSD </li><li> 20. Guidelines for sizing General Purpose (SSD) volumes Generic boot, developer, test/dev, and web apps: Provision GB required for your applications Database apps: 1. Calculate the IOPS required in steady state 2. Perform this calculation: (steady state IOPS) 3 = GB to provision Note: I/O bursts will support: Database load or table scan operations Spike in I/O workload 20 </li><li> 21. EBS PIOPS (SSD) volumes Best for I/O intensive databases that require highest consistency Throughput up to 320 MB/sec Provision up to 20,000 IOPS per volume (supports IOPS:GB ratio of 30) Designed for 99.9% performance consistency </li><li> 22. EBS Magnetic volumes Best for cold workloads (rarely accessed data that needs always-on access) IOPS: ~100 IOPS steady-state, with best-effort bursts Throughput: variable by workload, best effort to 10s of MBs Latency: Varies, reads typically ~20-40 ms, writes typically ~2-10 ms </li><li> 23. EBS volume types - summary General Purpose (SSD) Provisioned IOPS (SSD) Magnetic Recommend use cases Boot volumes Small to med DBs Dev and test I/O-intensive workloads Large DBs Cold storage Storage media SSD-backed SSD-backed Magnetic-backed Volume size 1 GB - 16 TB 4 GB - 16 TB 1 GB - 1 TB Max IOPS per volume 10,000 IOPS 20,000 IOPS ~100 IOPS Burst &lt; 1 TB to 3000 IOPS baseline baseline Read and write peak throughput 160 MB/s 320 MB/s ~50-90 MBps Max IOPS per node (16k) 48,000 48,000 48,000 Peak throughput node 800 MB/s 800 MB/s 800 MB/s Latency (random read) 1-2 ms 1-2 ms 20-40 ms API Name gp2 io1 standard Price* $.10/GB-month $.125/GB-month $.065/provisioned IOPS $.05/GB-month $.05/ 1M I/O </li><li> 24. Why is General Purpose SSD the default? High baseline level of performance Burst to higher level of IOPS Single, capacity-based pricing dimension Makes forecasting very easy Eliminates sizing complexity Attractive price/gigabyte/price/IOPS density </li><li> 25. Always use General Purpose (SSD) for boot volumes </li><li> 26. Migrating to General Purpose (SSD) volumes Change volume type during launch Use EBS snapshots You may be able to resize the file system Use General Purpose (SSD) sizing guide </li><li> 27. Benefits of using EBS snapshots More durable than an EBS volume Stored in Amazon S3 Differential (space-efficient) First snapshot is a clone Pay only for what you use Availability Zone-independent Clone into any AZ Can be copied efficiently across regions </li><li> 28. Tagging snapshots Use tags to add metadata to snapshots: Type (daily, weekly) Version Instance ID Volume ID Application stack </li><li> 29. Performance </li><li> 30. Queuing theory Littles Law Littles Law is the foundation for performance tuning theory Mathematically proven by John Little in 1961 L = A * W L = Queue length = average number of requests waiting A = Arrival rate = the rate of requests arriving W = Wait time = average wait time EBS performance is related to this law </li><li> 31. Performance optimization is measured by: IOPS: Read/write I/O rate (IOPS) Latency: Time between I/O submission and completion (ms) Throughput: Read/write transfer rate (MB/s); throughput = IOPS X I/O size </li><li> 32. Four key components of performance optimization 1. EC2 instance 2. I/O 4. EBS 3. Network link </li><li> 33. Tools available for performance tuning: 1. EC2 instance: Network bandwidth (Mbps) 2. EBS-optimized instance: EC2 instance option (On/Off) 3. Workload: Block size, read/write ratio, serialization 4. Queue depth: The number of outstanding I/Os 5. RAID: Stripe volumes to maximize performance 6. Pre-warming: Eliminate first-touch penalty </li><li> 34. 1. EC2 instance Compute-optimized C3,C4 Memory-optimized R3 General-Purpose M3 EBS EC2 Select the EC2 instance that has the right network, RAM, and CPU resources for your applications </li><li> 35. 2. EBS-optimized instance Most instance families support the EBS-optimized flag EBS-optimized instances now support up to 4 GB/s Drive 32,000 16K IOPS or 500 MB/s EC2 *.8xlarge instances support 10 Gb/s network Max IOPS per node supported is ~48,000 IOPS @ 16K I/O </li><li> 36. Use EBS-optimized instances for consistent EBS performance </li><li> 37. 3. Workload I/O size: 4 KB to 64 MB I/O pattern: Sequential and random I/O type: Read and write I/O concurrency: Number of concurrent I/O EBS SSD-backed volumes measure I/O size up to 256 KB EBS SSD-backed volumes deliver same performance for read and write </li><li> 38. EBS IOPS and throughput limits 20,000 IOPS PIOPS volume 20,000 IOPS 320 MB/s throughput You can achieve 20,000 IOPS when driving smaller I/O operations You can achieve up to 320 MB/s when driving larger I/O operations </li><li> 39. EBS IOPS and throughput limits 8,000 IOPS PIOPS volume 8,000 IOPS 320 MB/s throughput 8,000 x 64 KB=512 MB/s 1,250 x 256 KB = 320 MB/s 8,000 X 8 KB = 64 MB/s 8,000 X 16 KB = 128 MB/s 16,000 x 8 KB = 128 MB/s 8,000 x 32 KB = 256 MB/s </li><li> 40. Block (I/O) size determines whether your application is IOPS bound or throughput bound </li><li> 41. 4. Queue depth An I/O operation EBS After its gone, its gone EC2 Queue depth is the pending I/O for a volume </li><li> 42. Monitoring EBS volumes Important Amazon CloudWatch metrics: IOPS and bandwidth Latency Queue depth </li><li> 43. I/O latency Elapsed time between I/O submission and its completion time Performance requirements may be driven by IOPS or latency or both There is an interdependency among IOPS, queue depth, and latency </li><li> 44. Latency: What does it look like? </li><li> 45. Latency: Introducing the boxplot </li><li> 46. Latency: Baseline </li><li> 47. Latency: General Purpose SSD volume </li><li> 48. Latency: Latest generation instance </li><li> 49. EC2: Instance comparison m2.4xlarge CPU: Intel Xeon vCPU: 8 Memory: 68.4 GiB Price: $0.98/hour r3.2xlarge CPU: Intel Xeon E5-2670 v2 vCPU: 8 Memory: 61 GiB Enhanced Networking Price: $0.70/hour * All pricing from us-east-1 </li><li> 50. Random read latency 0.075 35.1 0 5 10 15 20 25 30 35 40 1 4 8 12 16 20 24 28 32 LatencyTP90(ms) Queue depth Random read latency across various queue depths Latency (TP90) Read latency linearly increases with increase in queue depth </li><li> 51. Random read latency 0.075 35.1 2.09 1,865 4,152 3,851 - 500 1,000 1,500 2,000 2,500 3,000 3,500 4,000 4,500 0 5 10 15 20 25 30 35 1 4 8 12 16 20 24 28 32 LatencyTP90(ms) Queue depth 16 KB random read IOPS, latency across various queue depths Latency (TP90) Avg Read IOPS IOPS Queue depth of 1 has the lowest latency, but also has the lowest IOPS </li><li> 52. Random read latency 0.075 35.1 2.09 1,865 4,152 3,851 - 500 1,000 1,500 2,000 2,500 3,000 3,500 4,000 4,500 0 5 10 15 20 25 30 35 1 4 8 12 16 20 24 28 32 LatencyTP90(ms) Queue depth 16 KB random read IOPS, latency across various queue depths Latency (TP90) Avg Read IOPS IOPS Queue depth between 4 and 8 has the optimal IOPS and latency performance </li><li> 53. Random read latency 0.075 35.1 2.09 1,865 4,152 3,851 - 500 1,000 1,500 2,000 2,500 3,000 3,500 4,000 4,500 0 5 10 15 20 25 30 35 1 4 8 12 16 20 24 28 32 LatencyTP90(ms) Queue depth 16 KB random read IOPS, latency across various queue depths Latency (TP90) Avg Read IOPS IOPS Higher queue depths have negative impact on IOPS and latency </li><li> 54. Random write latency 0.08 7.71 845 4,152 0 500 1,000 1,500 2,000 2,500 3,000 3,500 4,000 4,500 0 1 2 3 4 5 6 7 8 9 10 1 4 8 12 16 20 24 28 32 LatencyTP90(ms) Queue depth 16 KB random write IOPS, latency across various queue depths Latency (TP90) AvgIOPS IOPS Write latency queue depth and IOPS interaction is similar to that of read latency </li><li> 55. Optimal queue depth to achieve lower latency and highest IOPS is typically between 4-8; ~1 queue depth per 500 IOPS EBS-optimized instances provide consistent latency experience Use SSD volumes with latest-generation EC2 instances </li><li> 56. 5. RAID Increases performance, or capacity, or both Over 320 MB/sec or 20K IOPS, striping needed Dont mix volume types Typically RAID 0 or LVM stripe Avoid RAID for redundancyEBS EC2 </li><li> 57. Maximum performance per instance How should you think about taking snapshots on a striped volume? Quiesce file systems and take snapshot Unmount file system and take snapshot Use OS-specific tools 12400 GB PIOPS, pre-warmed, RAID 0 LVM, stripe size 128 KB, attached to CR1 instance </li><li> 58. Use stripe size of 128 KB or 256 KB </li><li> 59. 6. Pre-warming Eliminates first-access penalty Typically 5%, extreme worst case of 50% performance reduction in IOPS and latency when volumes are used without pre-warming: Performance is as provisioned when all the chunks are accessed Recommendations before benchmarking: For new volumes: Linux: DD write Windows: NTFS full format Takes roughly an hour to pre-warm 1 TB PIOPS/General Purpose (SSD) volumes Always check latest documentation </li><li> 60. Use large block size to speed up your pre-warming Example: sudo dd if=/dev/xvdf of=/dev/xvdf conv=notrunc bs=1M </li><li> 61. Final tips Try to use Ext4 or XFS Alignment can matter; check tools use 4k </li><li> 62. Workload/ software Typical block size Random/ Seq? Max EBS @ 500 MB/s instances Max EBS @ 1 GB/s instances Max EBS @ 10 GB/s instances Oracle DB Configurable:2 KB 16 KB Default 8 KB random ~7,800 IOPS ~15,600 IOPS ~48,000 IOPS Microsoft SQL Server 8 KB w/ 64 KB extents random ~7,800 IOPS ~15,600 IOPS ~48,000 IOPS MySQL 16 KB random ~4,000 IOPS ~7,800 IOPS ~48,000 IOPS PostgreSQL 8 KB random ~7,800 IOPS ~15,600 IOPS ~48,000 IOPS MongoDB 4 KB serialized ~15,600 IOPS ~31,000 IOPS ~48,000 IOPS Apache Cassandra 4 KB random ~15,600 IOPS ~31,000 IOPS ~48,000 IOPS GlusterFS 128 KB sequential ~500 IOPS ~1,000 IOPS ~6,000 IOPS Cheat sheet sample: Storage workloads on AWS </li><li> 63. EBS-optimized instance Four key components: balanced (Oh, YEAH!!) EC2 A boatload of I/O Right-sized EBS </li><li> 64. Tools available for tuning EC2 Instance: Network transfer rate (Mbps) EBS Optimized: EC2 instance option (On/Off) Workload: Block size, read/write ratio, serialization Queue Depth: The number of outstanding I/Os (#) RAID: Stripe volumes to maximize performance Pre-warm: Eliminates first touch penalty </li><li> 65. Encryption </li><li> 66. Why encrypt data volumes? Security: Protects against someone who might gain unauthorized physical access to the volume Can help with internal or external compliance efforts: Chief Information Security Officer wants encryption to protect sensitive corporate data 3rd-party auditors want to see evidence that sensitive customer data is encrypted Ease of use and operating cost reduction: Unlike open-source or 3rd-party solutions, such as Trend Micro SecureCloud, SafeNet ProtectV, etc., EBS encryption offers: Checkbox encryption at no extra cost Automated, secure key management </li><li> 67. AWS KMS A service that simplifies encrypting data and managing keys Allows customers to create, use, and manage encryption keys from within their own applications and supported AWS services (Amazon S3, EBS, Amazon Redshift) Key management functions include: Create, enable, disable, rotate, and define usage policy on master keys Generate a data key that can be exported from the service after its encrypted by a master key Audit use of master keys in AWS CloudTrail Available in 9 commercial regions </li><li> 68. Use Create Volume in the EC2 console </li><li> 69. How AWS services integrate with AWS KMS 2-tiered key hierarchy using envelope encryption Unique data key encrypts customer data AWS KMS master keys encrypt data keys Benefits of envelope encryption: Limits risk of a compromised data key Better performance for encrypting large data Easier to manage a small number of master keys than millions of data keys Master key(s) Data key 1 S3 object EBS volume Amazon Redshift cluster Data key 2 Data key 3 Data key 4 Custom application KMS </li><li> 70. Summary Use encryption if you need it Take snapshotsSelect the right instance for your workload Select the right volume for your workload </li><li> 71. Q&amp;A </li><li> 72. NEW YORK </li></ol>