Fork me on GitHub

n. Slang a rough lawless young Kuali developer.
[perhaps variant of Houlihan, Irish surname]
kualiganism n

Blog of an rSmart Java Developer. Full of code examples, solutions, best practices, et al.

Friday, July 27, 2012

Kuali Implementation Development Pre and Post Go Live


I named this such because I think it effects all Kuali software implementations. This post is about the concept that pre and post-golive are actually 2 separate implementation projects. Many of you are probably saying, "Of course! You have customization mode and support mode." That may be true, but that is not what I want to point out. I am speaking specifically to development process and strategy rather than the administrative impacts of the application.

I have only had the opportunity to witness a few implementations go live, so I cannot speak to every implementation out there. From what I have witnessed though, I will say that in each case development continues on almost as if nothing has happened. This may in part be because prior to go live, implementations are already preparing by mimicking production.

After some time, there is a kind of evolution that occurs as development teams mature and become more familiar with their software and processes behind maintaining their software. For example, the University of Arizona is starting an initiative where they are employing a release process paradigm called Git flow. One of the primary issues before and after go live is the matter of controlling fixes and features as they flow from environment to environment. Git flow is a paradigm that is tempered to the point where support software has been written to support it and enforce its tenets. I wish them the best of luck with their initiative and look forward from learning from their experience.

I want to share an idea that I think many have already thought of or at least had it in the backs of their minds. I can tell because of how Continuous Integration has embedded itself into our release cycles and environment management. There's been talk of Continuous Deployment, but still on a release cycle. It's like we're at the cusp of realizing something.


What if the world of development before and after go live are actually so far from each other, that they become entirely different processes? I recently came across this article about a month ago and at first I really didn't think anything of it. I didn't even think it applied to me in any way because I don't plan to ever work for github. I then realized that github is a SaaS just like KFS, KC, and just about anything built with Rice. As soon as that occurred to me, I hurried back to read about Github Flow.

Scott Chacon is discussing why they don't use Git flow at github. Here's what really struck me from what he stated:

"We don’t really have 'releases' because we deploy to production every day - often several times a day."

By now you are spraying whatever you were drinking earlier out of your nose, right? I know because that's what I did. They deploy to production every day. Some implementations do that. Several times a day is what we all wish we could do. Can you imagine the software agility? To be able to drop in a fix with little to now administrative overhead?

"If you deploy every few hours, it’s almost impossible to introduce large numbers of big bugs."

It's true. Once, I had found a tremendous bug on github. I thought I was so cool I posted an issue. Before I could blog about it, the bug was fixed. I had to retract my blog. Big bugs don't exist. When you can deploy fixes with such agility, your only problem is prioritizing which ones first.

"We try to make the process of testing and shipping as simple as possible so that every employee feels comfortable doing it."

I said it before, but they made it their mission to defeat administrative overhead in order to provide this capability. Every employee feels comfortable not just because they do it regularly, but because they made it simple and easy.


Now that we know github's process, what if we wanted to follow this? Do we have to use git? I don't think so. Github's process isn't anything specific to git. Of course, they have their own tools like hubot and of course their UI. Each of those is exclusive from git which means SVN would be just as applicable. I think probably the biggest obstacle would be the application startup time. The application has to go down for deployment, so what then of restarting it? In production, I expect that institutions will probably have failover configured. One node can cover for another while it is updating and back again. Since optimal startup requires failover and the only environment that will probably have it is production, this implies there's only one environment after go live. This probably makes you a little nervous. It may even make you shudder. Imagine though that what Scott explained is true and that major bugs are wiped out by this process and that everyone can follow it (even business analysts, graphic designers, or web developers).


Prior to go live, institutions are concerned with pruning the system, conversion (data migration), environment setup, quality assurance, managing vendor sources, and configuring their software. Preparing for go live is incremental and each increment is treated like a release. These releases are scheduled. Sometimes close together and sometimes far apart. Institutions will have some environments they deploy to several times a day, but those are generally not releases. Those are development modifications. These are not changes that have been QA'd and part of a release. There will be eventually be a release.

Post go live, the need for releases doesn't exist anymore. Most of the work is support issues or new feature requests. This is pretty much like how github operates. I think Scott is right.

Thursday, July 26, 2012

How I Setup My KFS Instance Built from Archetype

This is related to KFS Maven Overlays Made Easy. I didn't give details on how I configured my environment because I felt that's really a post in itself. I'm going to show/describe what configuration files I use and how I went about encrypting my passwords.

Encrypting Passwords

Ok. I'm going to break this down into the following:
  • Creating a PEM public key.
  • Adding the private/public keypair to $HOME/kuali/main/dev/rice.keystore
  • Encrypting the password with the public key

1 Create a PEM public key

I don't need to generate a private/public keypair because I'm using my SSH keys for this. Of course, you could generate your own public/private keypairs with openssl. That's fine, but I'm assuming you're like me and you already have an SSH keypair. Now you just need to create a PEM public key because your SSH RSA public key just isn't going to cut it.

The above will create you a nice little id_rsa.pem file. That's your PEM public key.

2 Import your RSA Private/Public Keys

For the files necessary refer to my post Decryption/ReEncryption With the DemonstrationGradeEncryptionServiceImpl. Don't let the name fool you. There's a in there. Even if you don't use it to replace the DemonstrationGradeEncryptionServiceImpl, you can use it to import your Private/Public keys into your rice.keystore.

You can show your keys like this:

3 Now we want to encrypt our password

The Whole Process went like this for me

My Config

This is what my config ended up looking like. Since I'm working on tem, I have a $HOME/kuali/main/dev/kfs-tem-config.xml


Hope this helps you to setup your config that you can now store in your institution's VCS.

Wednesday, July 25, 2012

KFS Maven Overlays Made Easy

That's right. Yet another one of these. Before I start, I want to announce that the archetype is now at version 1.0.4 and the plugin is at version 1.0.8. We've passed version 1.0 and the plugin is ready for wide use.

I have added screencasts on using the KFS Maven Archetypes.

It's here. I've been talking about a maven archetype for KFS creating for a long long while, but I never shared it. It just wasn't ready for wide spread use. It even had its own usability problems. I have since completely rewritten it so people besides myself may use it. You can find it on github here KFS Maven Plugin. That's right, it's mixed in with a kfs maven plugin. That's how I solved my usability issues. There's a custom plugin and it will be used for much more than just the archetype. I had planned for a use case of migrating a kfs distribution, but I am rethinking it. I want to really really promote the overlay concept and I think a KFS maven distribution would kind of defeat that at this point. I'd rather give people the opportunity to take an existing war that they can make pretty quickly and just build on it.

In this video, I show that I'm using Configuration from VCS with on the fly encryption/decryption aaaand!!! Low-carb kfs configuration. It'll save you the trouble of setting that up for yourselves. The downside is there is still some setup. You will be required to add a private/public keypair to your rice.keystore. Good news!!! I already tell you how to do that here Encryption/decryption. Don't worry. There will be a video on that soon as well.

I am working on one for Rice (thin clients and native) and KC as well.


Wednesday, July 18, 2012

Using the Convert KFS Maven Archetype

I've been talking a lot about my archetype plugin that let's me convert a KFS distribution to a maven project. You can have a look at my kfs-maven-plugin project on GitHub. Not quite there yet, but it is coming.

I did make significant progress on the kfs-archetype which is actually part of the kfs-maven-plugin project. While I am still working on the plugin, you can try the archetype now. Here's how.

This of course, implies that the archetype is in maven central. It should be there in a couple days. You can check by looking up my groupId. If you are impatient and do not want to wait for the archetype to arrive at maven central, you can try it by checking out the project and building it like this:

More updates to come. I'll let you as I make progress.

Friday, July 13, 2012

Utilizing EC2 Instances to Speed Up Agile Release Cycles


One great advantage of cloud virtualization is the disposable nature of virtual machines. This lends itself very well to Agile software development because before and after testing/releasing, environments can be disposed to ensure sanity with each run.

EC2 in particular is great for this because of their flexibility and abundance of tools to help manage your virtualization configuration. EC2 can arguably be said is too flexible because of the related caveats. In this post, I want to speak to some of those caveats as well as share some code and examples on how to overcome them. In turn, it is expected that anyone reading this will be able to improve the quality and automation behind their testing/release cycles.

1 Jenkins EC2 Plugin

More can be read about the plugin from the Jenkins Wiki. Basically, you can point Jenkins to an AMI and your AWS account information to allow it to automatically raise and destroy EC2 slave instances. Another advantage is that you can label instances. This allows you to group EC2 slaves. You can have EC2 slaves exclusively for testing and others exclusively for sandboxes.

2 Managing Images

The caveat of having Jenkins manage your EC2 images though is that it doesn't manage them at all. It will erect new instances as it needs and terminate them as it needs. By default, EC2 instances that use EBS volumes do not destroy the volumes on terminate. This is a troublesome feature. Some cases you do want volumes to stick around. That is why the default. For Jenkins though, you want your volumes to get cleaned up or else they can get pretty costly.

3 Cleaning up

To get around this, I add a tag to my AMI. This way I know which instances were created by Jenkins. Otherwise, I do not know which volumes to cleanup. I get all my Jenkins volumes and I am left with 2 options that are equally good in my opinion. One is delete it if it's unattached (unused). The other is to set it to automatically delete itself.

(00:06:06) [63] ec2-describe-instance-attribute -v -b i-3a179642 | grep deleteOnTermination

This shows it is off. To turn it on, simply use something like this:

(00:06:06) [63] instanceid=i-3a179642; ec2-modify-instance-attribute -b /dev/sda1=$(ec2-describe-instance-attribute -b $instanceid| grep ^BLOCK | cut -f 3 -d ' '):true $instanceid

I usually choose to delete it though. If by chance the instance is already terminated, then delete on terminate will do you no good at all. Going around after-the-fact only helps if you're deleting.

Here's the code I used to accomplish this:

You can see there how I am using a tag on the image to determine which volumes to cleanup.

If you care to see more of my scripts on this process, checkout my ec2 scripts project.

4 Kuali Scripts in SVN

Actually, way after I had written those scripts, I'd found that Jonathan Keller from UC Davis had actually written a plethora of very useful scripts that can be found at Much of the scripts are useful includes for starting/stopping or even identifying objects on AWS which I would have found very helpful when writing my scripts had I know of them.

Further, I am hosting my scripts on github because I do not have rights to commit to that repo.

Saturday, July 7, 2012

Configuration from VCS with on-the-fly Password Decryption


To follow up Low-carb KFS Configuration and Decryption/ReEncryption With the DemonstrationGradeEncryptionServiceImpl, I want to now give an example of how public/private key encryption can be used to encrypt passwords used in KFS for the purpose of safely storing these configurations in the VCS.


Let's get started.

1 Encrypt Your Password

First thing we need to do is get our password encrypted. I'm using my SSH key for encryption. You can do this as well.

1.1 Make Sure Your Public Key is PEM Format

Run the following to convert your ssh public key:

1.2 Encrypt Your Password

Using the datasource.password as an example, I'm going to encrypt it and append it back to the file. I am appending it back with the name datasource.encrypted. The reason for this, is I want to distinguish encrypted data from unencrypted data by convention. Any property that ends with .encrypted in the name must be encrypted. This is important because I want to add function, not replace it. Unencrypted passwords still have their purpose. Developers, for example, would find it inconvenient to have to encrypt everything for a number of reasons.

1.3 Now we are encrypted

Let's check it out.

2 PropertyLoadingFactoryBean

Now that the password is encrypted, we need to handle decryption. Since properties are loaded through the PropertyLoadingFactoryBean, we can do it there as we are reading in the properties. The best strategy is probably to do the following:
  1. Load all properties
  2. Iterate over properties searching for the ones ending with .encrypted
  3. Decrypt the property values
  4. Replace property values with the unencrypted ones and change the key names to end with .password by convention

Sounds pretty simple.

The above shows that the getObject is modified to decrypt properties at the end. here is the decryptProps method.

Notice it uses the keystore.filename and keystore.password properties. This means the keystore.password cannot be encrypted. Since it cannot be encrypted, this is probably the only password that cannot be added to the VCS. Likewise, the rice.keystore is a file you want to keep out of VCS since it contains the private and public keys.

Which brings me to the point about the private and public keys being stored in the rice.keystore. This post assumes you are familiar with Decryption/ReEncryption With the DemonstrationGradeEncryptionServiceImpl where I went over adding the SSH private/public keys to the rice keystore.


Now you have enough to encrypt your passwords and decrypt them as properties on-the-fly during KFS startup.

Decryption/ReEncryption With the DemonstrationGradeEncryptionServiceImpl


Some institutions have a requirement to access the sensitive information in Kuali systems externally. I want to have a look at how to decrypt data in Kuali externally given the use case of Business Intelligence Reports.

This all came from a case when I tried to decrypt values in KFS using the default encryption key, but could not. I tried all manner of debugging to prove I was using the correct key and algorithm, but it did not matter. I continued to encounter the following error:

I knew something had to be different, so finally, I decided to attack this at the byte level and examine bytes directly. When I used the UTF-8 encoded string for the default encryption key, I was always using the following bytes:

-20 -128 -70 -29 14 -92 -80 -75 36

while KFS actually was hoping for

-20 -128 -70 -29 14 -92 0 0 0

The length is the same, but there is additional padding at the end. The padding is significant. Whenever I tried to apply the default encryption key, the padding did not match, so I was forced to reproduce with a byte array. No matter what, it looked as though I was going to have to make some modifications to this service to do what I want. I decided to go the extra mile and break it down into some changes and best practice improvements.

1 Setting Up a New Encryption Key

One of the first improvements I'd like to make is not using a String-based key. Keys are natively non-String formats. In order for them to be embedded in text files and human-readable, we Base64 encode them. Though this is a workable solution, it is not preferred. Also, as we have seen, it does not always produce reliable results. Rather, let us utilize the rice.keystore and embed our key within it using the appropriate tools.

That is the existing rice private key. I want to add another.

I have now created a new DES key with the same password as the keystore.

To use this new service, I had to update several spring files in rice and KFS. For example, the spring-kfs-rice-overrides.xml:

Notice, I'm passing the location of the keystore and the passkey into the constructor.

The reason for that is best explained by showing the modified service I created:

I modified it to not be dependent upon external libraries (you will see why in a moment). I also modified it to read the key from the keystore instead of from a string. Another thing I did was added a main method. This way, the service can be run from the command line. It acts as a utility as shown below:

Now I have the option of using the string as a key or the java code representing the actual key. This java code will be useful later.

1.1 Caveats

Once you start using your system with encrypted data (this can actually be on submitting your first document since document data is encrypted), changing your key becomes very difficult. You need to be able to decrypt and then re-encrypt. Unfortunately, such a method is not available in Kuali software right now. It is possible to do though.

First, you need to determine which fields are encrypted. Below is an incomplete list between KFS an Rice:
  • KRNS_LOOKUP_RSLT_T.SERIALZD_RSLTS (Probably not important)

The above tables may have entire columns encrypted. One interesting caveat is that the KRIM_ENTITY_EXT_ID_T.EXT_ID column may not be entirely encrypted. Some values may be encrypted and some may not. The only way to know when a field is encrypted is when KRIM_EXT_ID_TYP_T.KRIM_EXT_ID_TYP_T is 'Y'.

The following SQL will reveal the fields that are encrypted.

2 Decrypting via Oracle

There are several methods for decrypting via Oracle, but if I want to use the exact same method as I'm using in Kuali, it's just easier to update Oracle directly. Below is the following SQL for adding the modified decryption class to Oracle.

2.1 Modified Decryption Class

2.2 Calling Java from Oracle

Now I can call decrypt fields in oracle like this:

You may have noticed that the field is passed in directly without any key. The reason for this is that I have embedded the key into the class. Normally, this is a pretty bad programming practice, but it is the best way the key is used correctly from Oracle.

3 Changing Your Encryption Key

That's just crazy talk! If you need to though, the EncryptionService has an encrypt method that will allow you to encrypt strings.

4 Alternative Encryption Algorithms

4.1 SSH RSA Key

My favorite use case is to use your system's SSH key for encrypting keys. This way your key is specific to an environment and uses SSH more ubiquitously. This is probably not ideal for everyone, but I personally favor it.

4.1.1 Convert SSH Public Key to PEM Format

4.1.2 Import RSA Key into the keystore

Just like with the DES key, we will add our RSA key to the keystore. I created another custom encryption service class with a main method that handles this:

To execute:

That's right. It uses the bouncycastle api. I include the jar in the classpath during execution. What happens is my RSA private key and its X.509 PEM counterpart are imported into the rice.keystore. To verify, try this:

You can see that now my RSA private key and SHA1 signed certificate are in the rice.keystore.

4.1.3 Using the RsaEncryptionServiceImpl

You use it pretty much the same as my modified DemonstrationGradeEncryptionServiceImpl. Just add the spring bean and change the class to use the RsaEncryptionServiceImpl class.

4.2 PGP

PGP (Pretty Good Privacy) is another private/public key encryption algorithm similar to RSA. It is a little bit of a diversion from RSA in java because we cannot use the keytool. Rather, we will use the PGP Keyring instead. For this specific reason, it follows after the RSA encryption method. The preference is to always use the java keystore whenever possible.

4.2.1 Setup PGP Keyring

To get started, we need a PGP Keyring.

Now I have two sets of keys:

In PGP, the Private Key is called a Secret and the Public Key is a ... Public Key.

Here is the encryption service

To use it, just make the following modifications to the spring-kfs-rice-overrides.xml

That's it

I plan to do another post similar to this on decrypting your database password from the PropertyLoadingFactoryBean. This will allow you to store any configuration in VCS because passwords would be encrypted.