Archive for the ‘Amazon EC2’ Topic

We're always looking for ways to make AWS an even better value for our customers. If you've been reading this blog for an extended period of time you know that we reduce prices on our services from time to time.

Effective September 1, 2010, we've reduced the On-Demand and Reserved Instance prices on the m2.2xlarge (High-Memory Double Extra Large) and the m2.4xlarge (High-Memory Quadruple Extra Large) by up to 19%.  If you have existing Reserved Instances your hourly usage rate will automatically be lowered to the new usage rate and your estimated bill will reflect these changes later this month.  As an example, the hourly cost for an m2.4xlarge instance running Linux/Unix in the us-east Region from $2.40 to $2.00. This price reduction means you can now run database, memcached, and other memory-intensive workloads at substantial savings. Here's the full EC2 price list.

As a reminder, there are many different ways to optimize your costs. When compared to On-Demand instances, Reserved Instances enable you to reduce your overall instance costs by up to 56%.  You pay a low, one-time fee to reserve an instance for a one or three year period. You can then run that instance whenever you want, at a greatly reduced hourly rate.

For background processing and other jobs where you have flexibility in when they run, you can also use Spot Instances by placing a bid for unused capacity. You job will run as long as your bid is higher than the current spot price.

– Jeff;

I almost missed a really important anniversary! Yesterday marked Amazon EC2’s fourth birthday. Here are some of the ways that EC2 has grown and changed in the last four years:

We’ve done quite a bit, but we’re not resting, not for a minute. We have a lot of open positions on the AWS team, including a really interesting developer position within the EC2 team. This developer will focus on EC2’s dynamic market pricing features. In addition to experience with Ruby, Perl, Java, C, or C++, candidates should have some experience building large-scale distributed systems and an interest in operational scheduling, optimization, and constraint satisfaction. You can read more here and you can send your resume directly to amazon-ec2-spot-jobs@amazon.com.

While I am on the subject of anniversaries, eight years ago this month I abandoned my full-time consulting practice to take a development position with the Amazon Associates Team, with the agreement that I could spend some of my time helping out with the effort to create and market the E-Commerce Service (which has since become the Product Advertising API). A few months in, I was asked if I would mind speaking at a conference. I guess I did ok, because they asked me to do another one, and before too long they invited me to apply for the position of Web Services Evangelist. I took on that title in the spring of 2003 and have been spreading the word about our web service efforts ever since. All things considered, this is a really awesome place to work. Day after day, week after week, things get more and more exciting around here. The pace is quick and I do my best to keep up. We do our best to understand and to meet the needs of our customers with regard to features, reliability, scale, business models, and price. I get to work with and to learn from a huge number of world-class intellects. If this sounds like the kind of place for you, check out our list of open jobs and apply today!

– Jeff;

You can now use the Linux kernel of your choice when you boot up an Amazon EC2 instance. 

We have created a set of AKIs (Amazon Kernel Images) which contain the PV-Grub loader. This loader simply chain-boots the kernel provided in the associated AMI (Amazon Machine Image). Net-net, your instance ends up running the kernel in the AMI instead of the kernel specified in the boot process.

You need to install an "EC2 compatible" kernel and create an initrd (initial RAM disk) as part of your AMI. You also need to create a menu (/boot/grub/menu.lst) for the Grub boot loader. Once you've done this you can create the AMI and then launch instances by using one of the PV-Grub "kernels" as described above. You may find this document to be helpful if you want to learn more about the Linux boot process.

To be compatible with EC2, a Linux kernel must support Xen's pv_ops (paravirtual ops) infrastructure with XSAVE disabled or the Xen 3.0.2 interface. The following kernels have been tested and/or have vendor support:

• Fedora 8-12 Xen kernels
• SLES/openSUSE 10x, 11.0, and 11.1 Xen kernels
• SLES/openSUSE 11.x EC2 Variant
• Ubuntu EC2 Variant
• RHEL 5.x
• CentOS 5.x

Other kernels may not start reliably within EC2. We're working with the providers of popular AMIs to make sure that they will start to use PV-Grub in the near future.

You can read more about this in our "Enabling User Provided Kernels in Amazon EC2" document.

– Jeff;

PS – You could (if you are sufficiently adept) use this facility to launch an operating system that we don't support directly (e.g. FreeBSD). If you manage to do this, please feel free to let me know.

We've added two new features to the Amazon Virtual Private Cloud (VPC) to make it more powerful and easier to use. Here's the scoop:

• IP Address Control - You can now assign the IP address of your choice to each of the EC2 instances that you launch in your Virtual Private Cloud. The address must be within the range of addresses that you designated for the VPC, it must be available for use within the instance's network subnet, and it must not conflict with any of the addresses that are reserved for internal use by AWS. You can specify the desired address as an optional parameter to the RunInstances function. This will allow you to have additional control of your network configuration, and has been eagerly anticipated by many of our customers. Two use cases that we've heard about already are running DNS servers and Active Directory® Domain Controllers.
• Config File Generation – VPC can now generate configuration files (example at right) for several different types of devices including the Cisco ISR and a number of Juniper products including the J-Series Service Router, the SSG (Secure Services Gateway), and the ISG (Integrated Security Gateway). The files can be generated from the command line or from within ElasticFox. Generating the config files in this way lets you avoid common configuration issues and allows you to be up and running in minutes.

If you want to connect a Linux-based VPN gateway to your Virtual Private Cloud, take a look at Amazon VPC With Linux. This article will show you how to set up IPSec and BGP routing and includes detailed configuration information.

If you are running OpenSolaris, take a look at the OpenSolaris VPC Gateway Tool.

– Jeff;

A number of AWS users have been using Amazon EC2 to solve a variety of computationally intensive problems. Here's a sampling:

• Atbrox and Lingit use Elastic MapReduce to build data sets that help individuals with dyslexia to improve their reading and writing skills.
• Systems integrator Cycle Computing helps Varian to run compute-intensive Monte Carlo simulations.
• Harvard Medical School's Laboratory for Personalized Medicine creates innovative genetic testing models.
• Pathwork Diagnostics runs tens of thousands of models to help oncologists to diagnose hard-to-identify cancer tumors.
• Razorfish processes huge datasets on a very compressed timescale.
• The Server Labs helps the European Space Agency to build the operations infrastructure for the Gaia project.

Some of these problems are examples of what are called
“embarrassingly parallel” computing.  Others leverage the Hadoop framework for data-intensive computing, spreading the workload across a large number of EC2 instances.

Our customers have also asked us about the ability to run even larger and more computationally complex workloads in the cloud.

It is clear that people are now figuring out that they can do HPC (High-Performance Computing) in the cloud. We want to make it even easier and more efficient for them to do so!

Our new Cluster Compute Instances will fit the bill. With Cluster Compute Instances, you can now run many types of large-scale network-intensive jobs without
losing the core advantages of EC2: a pay-as-you-go pricing model and the ability to
scale up and down to meet your needs.

Each Cluster Compute Instance consists of a pair of quad-core Intel "Nehalem" X5570 processors with a total of 33.5 ECU (EC2 Compute Units), 23 GB of RAM, and 1690 GB of local instance storage, all for $1.60 per hour.

Because many HPC applications and other network-bound applications make heavy use of network communication, Cluster Compute Instances are connected using a 10 Gbps network. Within this network you can create one or more placement groups of type "cluster" and then launch Cluster Compute Instances within each group. Instances within each placement group of this type benefit from non-blocking bandwidth and low latency node to node communication.

The EC2 API's, the command-line tools, and the AWS Management Console have all been updated to support the creation and use of placement groups. For example, the following pair of commands creates a placement group called biocluster and then launches 8 Cluster Compute Instances inside of the group:

$ ec2-create-placement-group biocluster -s cluster

$ ec2-run-instances ami-2de43f55 –type cc1.4xlarge –placement-group biocluster -n 8

The new instance type is now available for Linux/UNIX use in a single Availability Zone in the US East (Northern Virginia) region. We'll support it in additional zones and regions in the future. You can purchase individual Reserved Instances for a one or a three year term, but you can't buy them within specific cluster placement groups just yet. There is a default usage limit for this instance type of 8 instances (providing 64 cores). If you wish to run more than 8 instances, you can request a higher limit using the Amazon EC2 instance request form.

The Cluster Compute Instances use hardware-assisted (HVM) virtualization instead of the paravirtualization used by the other instance types and requires booting from EBS, so you will need to create a new AMI in order to use them. We suggest that you use our Centos-based AMI as a base for your own AMIs for optimal performance. See the EC2 User Guide or the EC2 Developer Guide for more information.

The only way to know if this is a genuine HPC setup is to benchmark it, and we've just finished doing so. We ran the gold-standard High Performance Linpack benchmark on 880 Cluster Compute instances (7040 cores) and measured the overall performance at 41.82 TeraFLOPS using Intel's MPI (Message Passing Interface) and MKL (Math Kernel Library) libraries, along with their compiler suite. This result places us at position 146 on the Top500 list of supercomputers. The input file for the benchmark is here and the output file is here.

Putting this all together, I think that we have put together a true fire-breathing dragon of an offering. You can now get world-class compute and network performance on an economical, pay-as-you-go basis.  The individual instances perform really well, and you can tie a bunch of them together using a fast network to attack large-scale problems. I'm fairly certain that you can't get this much compute power so fast or so economically anywhere else.

I'm looking forward to writing up and sharing some of the success stories from the customers who've been helping us to test the Cluster Compute instances during our private beta test. Feel free to share your own success stories with me once you've had a chance to give them a try.

Update – Here's some additional info:

• Werner Vogels: Expanding the Cloud – Cluster Compute Instances for Amazon EC2.
• TechCunchIT: AWS Launches Cluster Compute EC2 Instances For High Performance Applications.
• ZDNet: Amazon Web Services tackles high performance computing instances.
• And our own HPC Applications page.

– Jeff;