{"id":25053,"date":"2015-12-12T16:56:23","date_gmt":"2015-12-12T14:56:23","guid":{"rendered":"https:\/\/mamchenkov.net\/wordpress\/?p=25053"},"modified":"2015-12-12T16:56:23","modified_gmt":"2015-12-12T14:56:23","slug":"cpu-steal-time-now-on-amazon-ec2","status":"publish","type":"post","link":"https:\/\/mamchenkov.net\/wordpress\/2015\/12\/12\/cpu-steal-time-now-on-amazon-ec2\/","title":{"rendered":"CPU Steal Time. Now on Amazon EC2"},"content":{"rendered":"\n

Yesterday I wrote the blog post, trying to figure out what is the CPU steal time<\/a> and why it occurs. \u00a0The problem with that post was that I didn’t go deep enough.<\/p>\n

I was looking at this issue from the point of view of a generic virtual machine. \u00a0The case that I had to deal with wasn’t exactly like that. \u00a0I saw the CPU steal time on the Amazon EC2 instance. \u00a0Assuming that these were just my neighbors acting up or Amazon having a temporary hardware issue was a wrong conclusion.<\/p>\n

That’s because I didn’t know enough about Amazon EC2. \u00a0Well, I’ve learned a bunch since then, so here’s what I found.<\/p>\n

<\/p>\n

It all started with this ServerFault answer<\/a>, saying:<\/p>\n

Depending on the EC2 instance type and the underlying hardware, you may not be paying for access to all of the underlying CPU cycles. Amazon is not going to give you access to 100% of a modern, fast CPU if you have asked for an m1.small which is promised to be equivalent to an old, slow CPU.<\/p>\n

On EC2, steal doesn’t depend on the activity of other virtual machine neighbors. It is simply a matter of EC2 making sure you are not getting more CPU cycles than you are paying for.<\/p><\/blockquote>\n

Wait, what? I need to know more about this. \u00a0So I dig deeper, and I find this PDF document<\/a> describing “The Top 5 AWS EC2 Performance Problems”. \u00a0Just for the record, they are:<\/p>\n

\n
    \n
  1. Unpredictable EBS Disk I\/O<\/li>\n
  2. EC2 Instance ECU Mismatch and Stolen CPU<\/li>\n
  3. Running out of EC2 Instance Memory<\/li>\n
  4. ELB Loading Balancing Traffic Latency<\/li>\n
  5. AWS Maintenance and Service Interruptions<\/li>\n<\/ol>\n<\/blockquote>\n

    Hmm. \u00a0It’s good that I haven’t hit all five (yet?). \u00a0But the issue was having is clearly on the list. \u00a0So, scrolling down to page 11 of that document, I find a very helpful explanation, of what is an ECU:<\/p>\n

    While AWS has a large number of physical servers under management it does not rent them per se, rather only access to these servers in the form of virtual machines is available. The types of virtual machines are limited to a small list so as to make choosing an instance relatively easy. Somehow a virtual machine type, e.g. m1.large, can run on very different underlying hardware platforms and yet yield roughly the same performance in terms of compute.<\/p>\n

    To standardize compute, AWS has created a logical computation unit known as an Elastic Compute Unit (ECU). ECUs equate to a certain amount of computing cycles in a way that is independent of the actual hardware \u2013 1 ECU is defined as the compute power of a 1.0-1.2Ghz of a 2007 server CPU.<\/p><\/blockquote>\n

    … and further on down on what is “stolen” CPU:<\/p>\n

    Stolen CPU is a metric that\u2019s often looked at but can be hard to understand. It implies some malevolent intent from your virtual neighbors. In reality it is a relative measure of the cycles a CPU should have been able to run but could not due to the hypervisor diverting cycles away from the instance. From the point of view of your application, stolen CPU cycles are cycles that your application could have used.<\/p>\n

    Some of these diverted cycles stem from the hypervisor enforcing a quota based on the ECU you have purchased. In other cases, such as the one shown below, the amount of diverted or stolen CPU cycles varies over time, presumably due to other instances on the same physical hardware also requesting CPU cycles from the underlying hardware.<\/p><\/blockquote>\n

    … and as to why it occurs:<\/p>\n

    Since the price per instance type is the same in a given region, AWS, through the use of hypervisors, ensures that all virtual machines get a fair share of access to the underlying hardware to actually run.<\/p><\/blockquote>\n

    Recommended problem avoidance and resolution:<\/p>\n

    \n