{"id":28571,"date":"2018-05-09T17:04:23","date_gmt":"2018-05-09T15:04:23","guid":{"rendered":"https:\/\/mamchenkov.net\/wordpress\/?p=28571"},"modified":"2018-05-09T17:04:23","modified_gmt":"2018-05-09T15:04:23","slug":"creating-a-1-3-million-vcpu-grid-on-aws-using-ec2-spot-instances-and-tibco-gridserver","status":"publish","type":"post","link":"https:\/\/mamchenkov.net\/wordpress\/2018\/05\/09\/creating-a-1-3-million-vcpu-grid-on-aws-using-ec2-spot-instances-and-tibco-gridserver\/","title":{"rendered":"Creating a 1.3 Million vCPU Grid on AWS using EC2 Spot Instances and TIBCO GridServer"},"content":{"rendered":"\n

This Amazon AWS blog post<\/a> provides a great insight into the benefits of the cloud computing in general and Amazon AWS in particular.\u00a0 The whole thing is well worth the read, but here are a few of my favorite bits.<\/p>\n

The scale:<\/p>\n

The grid grew to 61,299 Spot Instances (1.3 million vCPUs drawn from 34 instance types spanning 3 generations of EC2 hardware) as planned, with just 1,937 instances reclaimed and automatically replaced during the run, and cost $30,000 per hour to run, at an average hourly cost of $0.078 per vCPU. If the same instances had been used in On-Demand form, the hourly cost to run the grid would have been approximately $93,000.<\/p><\/blockquote>\n

The size of the Amazon AWS customers:<\/p>\n

1.3 million vCPUs (5x the size of the largest on-premises grid)<\/p><\/blockquote>\n

The evolution of computing power over the last few years:<\/p>\n

To give you a sense of the compute power, we computed that this grid would have taken the #1 position on the\u00a0TOP 500<\/a>\u00a0supercomputer list in November 2007 by a considerable margin, and the #2 position in June 2008. Today, it would occupy position #360 on the list.<\/p><\/blockquote>\n

Now, just for fun, exercise the idea of building something like this in house…<\/p>\n\n","protected":false},"excerpt":{"rendered":"\n

This Amazon AWS blog post provides a great insight into the benefits of the cloud computing in general and Amazon AWS in particular.\u00a0 The whole thing is well worth the read, but here are a few of my favorite bits. The scale: The grid grew to 61,299 Spot Instances (1.3 million vCPUs drawn from 34 … Continue reading Creating a 1.3 Million vCPU Grid on AWS using EC2 Spot Instances and TIBCO GridServer<\/span><\/a><\/p>\n\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"Creating a 1.3 Million vCPU Grid on AWS using EC2 Spot Instances and TIBCO GridServer #AWS #cloud #hosting","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"_links_to":"","_links_to_target":""},"categories":[1,133,62,1334],"tags":[3270,2366,2289],"keyring_services":[],"class_list":["post-28571","post","type-post","status-publish","format-standard","hentry","category-general","category-sysadmin","category-technology","category-web-work","tag-amazon-aws","tag-cloud-computing","tag-web-hosting"],"aioseo_notices":[],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack-related-posts":[{"id":28177,"url":"https:\/\/mamchenkov.net\/wordpress\/2017\/11\/06\/persisting-state-between-aws-ec2-spot-instances\/","url_meta":{"origin":28571,"position":0},"title":"Persisting state between AWS EC2 spot instances","author":"Leonid Mamchenkov","date":"November 6, 2017","format":false,"excerpt":"\"Persisting state between AWS EC2 spot instances\" is a handy guide into using Amazon EC2 spot instances instead of on-demand or reserved instances and preserving the state of the instance between terminations.\u00a0 This is not something that I've personally tried yet, but with the ever-growing number of instances I managed\u2026","rel":"","context":"In "All"","block_context":{"text":"All","link":"https:\/\/mamchenkov.net\/wordpress\/category\/general\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/mamchenkov.net\/wordpress\/wp-content\/uploads\/2017\/11\/aws-console-create-instance-500x332.png?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":25053,"url":"https:\/\/mamchenkov.net\/wordpress\/2015\/12\/12\/cpu-steal-time-now-on-amazon-ec2\/","url_meta":{"origin":28571,"position":1},"title":"CPU Steal Time. Now on Amazon EC2","author":"Leonid Mamchenkov","date":"December 12, 2015","format":false,"excerpt":"Yesterday I wrote the blog post, trying to figure out what is the CPU steal time and why it occurs. \u00a0The problem with that post was that I didn't go deep enough. I was looking at this issue from the point of view of a generic virtual machine. \u00a0The case\u2026","rel":"","context":"In "All"","block_context":{"text":"All","link":"https:\/\/mamchenkov.net\/wordpress\/category\/general\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":25061,"url":"https:\/\/mamchenkov.net\/wordpress\/2015\/12\/17\/amazon-ec2-t2-nano-instances\/","url_meta":{"origin":28571,"position":2},"title":"Amazon EC2 t2.nano instances","author":"Leonid Mamchenkov","date":"December 17, 2015","format":false,"excerpt":"If you thought t2.micro was a tiny machine, I have news for you - Amazon announced t2.nano instance type. \u00a0It features 512 MB of RAM, 1 vCPU, and up to two Elastic network interfaces. \u00a0Price for on-demand instance - $0.0065 per hour. This instance type is perfect for small websites,\u2026","rel":"","context":"In "All"","block_context":{"text":"All","link":"https:\/\/mamchenkov.net\/wordpress\/category\/general\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":34995,"url":"https:\/\/mamchenkov.net\/wordpress\/2019\/02\/18\/build-load-balanced-servers-in-aws-ec2-using-cloudformation\/","url_meta":{"origin":28571,"position":3},"title":"Build load-balanced servers in AWS EC2 using CloudFormation","author":"Leonid Mamchenkov","date":"February 18, 2019","format":false,"excerpt":"\"Build load-balanced servers in AWS EC2 using CloudFormation\" is an excellent guide on deploying load balancer servers with EC2 instances to Amazon AWS cloud with CloudFormation infrastructure management tool. The guide covers a variety of topics from the actual deployment to security and monitoring. There are many different approaches for\u2026","rel":"","context":"In "All"","block_context":{"text":"All","link":"https:\/\/mamchenkov.net\/wordpress\/category\/general\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/mamchenkov.net\/wordpress\/wp-content\/uploads\/2019\/02\/aws.jpg?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/mamchenkov.net\/wordpress\/wp-content\/uploads\/2019\/02\/aws.jpg?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/mamchenkov.net\/wordpress\/wp-content\/uploads\/2019\/02\/aws.jpg?resize=525%2C300&ssl=1 1.5x"},"classes":[]},{"id":27312,"url":"https:\/\/mamchenkov.net\/wordpress\/2017\/02\/09\/amazon-aws-mtu-for-ec2\/","url_meta":{"origin":28571,"position":4},"title":"Amazon AWS : MTU for EC2","author":"Leonid Mamchenkov","date":"February 9, 2017","format":false,"excerpt":"I came across this handy Amazon AWS manual for the maximum transfer unit (MTU) configuration for EC2 instances. \u00a0This is not something one needs every day, but, I'm sure, when I need it, I'll otherwise be spending hours trying to find it. The maximum transmission unit (MTU) of a network\u2026","rel":"","context":"In "All"","block_context":{"text":"All","link":"https:\/\/mamchenkov.net\/wordpress\/category\/general\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/mamchenkov.net\/wordpress\/wp-content\/uploads\/2017\/02\/packet-headers-500x337.png?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":36094,"url":"https:\/\/mamchenkov.net\/wordpress\/2019\/02\/25\/how-to-build-a-serverless-ci-cd-pipeline-on-aws\/","url_meta":{"origin":28571,"position":5},"title":"How To Build a Serverless CI\/CD Pipeline On AWS","author":"Leonid Mamchenkov","date":"February 25, 2019","format":false,"excerpt":"\"How To Build a Serverless CI\/CD Pipeline On AWS\" is a nice guide to some of the newer Amazon AWS services, targeted at developers and DevOps. It shows how to tie together the following: Amazon EC2 (server instances)Docker (containers)Amazon ECR (Elastic Container Registry)Amazon S3 (storage)Amazon IAM (Identity and Access Management)Amazon\u2026","rel":"","context":"In "All"","block_context":{"text":"All","link":"https:\/\/mamchenkov.net\/wordpress\/category\/general\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/mamchenkov.net\/wordpress\/wp-content\/uploads\/2019\/02\/aws.png?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/mamchenkov.net\/wordpress\/wp-content\/uploads\/2019\/02\/aws.png?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/mamchenkov.net\/wordpress\/wp-content\/uploads\/2019\/02\/aws.png?resize=525%2C300&ssl=1 1.5x, https:\/\/i0.wp.com\/mamchenkov.net\/wordpress\/wp-content\/uploads\/2019\/02\/aws.png?resize=700%2C400&ssl=1 2x, https:\/\/i0.wp.com\/mamchenkov.net\/wordpress\/wp-content\/uploads\/2019\/02\/aws.png?resize=1050%2C600&ssl=1 3x, https:\/\/i0.wp.com\/mamchenkov.net\/wordpress\/wp-content\/uploads\/2019\/02\/aws.png?resize=1400%2C800&ssl=1 4x"},"classes":[]}],"jetpack_sharing_enabled":true,"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/posts\/28571","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/comments?post=28571"}],"version-history":[{"count":0,"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/posts\/28571\/revisions"}],"wp:attachment":[{"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/media?parent=28571"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/categories?post=28571"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/tags?post=28571"},{"taxonomy":"keyring_services","embeddable":true,"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/keyring_services?post=28571"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}