Fork me on GitHub

n. Slang a rough lawless young Kuali developer.
[perhaps variant of Houlihan, Irish surname]
kualiganism n

Blog of an rSmart Java Developer. Full of code examples, solutions, best practices, et al.

Friday, July 13, 2012

Utilizing EC2 Instances to Speed Up Agile Release Cycles


One great advantage of cloud virtualization is the disposable nature of virtual machines. This lends itself very well to Agile software development because before and after testing/releasing, environments can be disposed to ensure sanity with each run.

EC2 in particular is great for this because of their flexibility and abundance of tools to help manage your virtualization configuration. EC2 can arguably be said is too flexible because of the related caveats. In this post, I want to speak to some of those caveats as well as share some code and examples on how to overcome them. In turn, it is expected that anyone reading this will be able to improve the quality and automation behind their testing/release cycles.

1 Jenkins EC2 Plugin

More can be read about the plugin from the Jenkins Wiki. Basically, you can point Jenkins to an AMI and your AWS account information to allow it to automatically raise and destroy EC2 slave instances. Another advantage is that you can label instances. This allows you to group EC2 slaves. You can have EC2 slaves exclusively for testing and others exclusively for sandboxes.

2 Managing Images

The caveat of having Jenkins manage your EC2 images though is that it doesn't manage them at all. It will erect new instances as it needs and terminate them as it needs. By default, EC2 instances that use EBS volumes do not destroy the volumes on terminate. This is a troublesome feature. Some cases you do want volumes to stick around. That is why the default. For Jenkins though, you want your volumes to get cleaned up or else they can get pretty costly.

3 Cleaning up

To get around this, I add a tag to my AMI. This way I know which instances were created by Jenkins. Otherwise, I do not know which volumes to cleanup. I get all my Jenkins volumes and I am left with 2 options that are equally good in my opinion. One is delete it if it's unattached (unused). The other is to set it to automatically delete itself.

(00:06:06) [63] ec2-describe-instance-attribute -v -b i-3a179642 | grep deleteOnTermination

This shows it is off. To turn it on, simply use something like this:

(00:06:06) [63] instanceid=i-3a179642; ec2-modify-instance-attribute -b /dev/sda1=$(ec2-describe-instance-attribute -b $instanceid| grep ^BLOCK | cut -f 3 -d ' '):true $instanceid

I usually choose to delete it though. If by chance the instance is already terminated, then delete on terminate will do you no good at all. Going around after-the-fact only helps if you're deleting.

Here's the code I used to accomplish this:

You can see there how I am using a tag on the image to determine which volumes to cleanup.

If you care to see more of my scripts on this process, checkout my ec2 scripts project.

4 Kuali Scripts in SVN

Actually, way after I had written those scripts, I'd found that Jonathan Keller from UC Davis had actually written a plethora of very useful scripts that can be found at Much of the scripts are useful includes for starting/stopping or even identifying objects on AWS which I would have found very helpful when writing my scripts had I know of them.

Further, I am hosting my scripts on github because I do not have rights to commit to that repo.

No comments:

Post a Comment