Surviving Integration Hell -or- How *not* to Handle Projects Outsourced to Vendors

Integration--the process of porting ones build artifacts from a development to end production environment--in my mind is the biggest unknown for a software project. Production environments are usually locked down, log files are hard to get to and requests for provisions or server resets -- long to wait for. What's even worse? When your integration process involves not only the end goal of "hosting" your application, but also incorporates build and deploy processes and policies that directly affect your code base.

For instance, a client I worked for needed a site. They provided a nice set of requirements and wireframes, in fact they provided all visual aspects for the site mocked as ASP pages (including the AJAX elements). What more could I ask for? It seemed like the perfect storm, a developer I hired would swoop in and add the dynamic elements to the site and we would be done in no time.

We weren't limited in the frameworks we chose in fact early on it was assumed the client only cared that the end result ran fine under IBM WAS. No problem (I thought), WAS is a servlet container and that's the beauty of Java web development, there is actually a spec describing the functionality of application servers. This means that the application being produced should operate in a bubble of sorts, it should know nothing (well that's not always possible) of where it's living, it's just a war and it needs a place to live.

Then reality kicked in. About a few weeks into the project the client presented a "Build and Deployment process for the <insert internal acronym> environment" document. As far as the build document was concerned Ant was the only real requirement. Their continuous integration (CI) tool however did support Maven which was great because Maven is what I used. So, with too much trust in the reasonability of clients we continued on our development path.

Development went relatively smoothly, we deployed the code to a local Tomcat instance so our QA people could do some basic smoke tests. When we were confident the code was ready for UAT, we asked the client if they could deploy the code to their servers. This required a few things:

1. Import the code into their SVN repository
2. Update some configuration files to match some of their environment settings (i.e. DB host names, file system folder paths, etc...)
3. Have their CI server build the code




Steps 1 and 2 were a breeze but then I needed to get the client involved in step 3. I was not familiar with their CI server and I didn't have time to learn it's ins and outs. I asked the client to setup the CI server to compile and deploy the code by providing them with the Maven goals which needed to be executed. They didn't like this. Even though their CI server was perfectly capable of building code via Maven, their corporate standards did not have a section on Maven and therefore Maven did not exist to them. ANT was the only way to build, ANT was the only way they were going to get our code compiled and deployed. The catch here was that they didn't want to create an ANT script for us, they wanted us to do it.

Maven to the rescue - Ant Script Generation

With one command: mvn ant:ant I was able to generate an ant script based off the maven pom.xml file I had already created for the project. I then ran mvn dependencies:directory to have maven place all the dependencies for the project into a directory under target such that I could have the ant script reference the directory instead of having it download the dependencies, something the end build server would not allow.

With this I thought I was in the clear, I had a working Ant script and life was good. Then the client said "but this doesn't look like the one that was there already". "The one that was there already?" I thought? What do you mean? This is a new project. Apparently there was a "static" site already created for the project that did not include the dynamic part we were creating. Well, I shouldn't say that, it did have a few small Struts 1.2x apps attached to it, but that was it. I tried to assure them that the static site could remain in place, and our app would simply live at a context under the main site. All they would have to do is build and deploy both sites. Their answer was (can you guess?) No. They wanted one ant script which built everything. And they didn't want to have to build and deploy several war files, that is, they didn't want to build the struts apps and our app, they just wanted one big war file.

Not to mention, they wanted the code to have a specific (and very antiquated) directory structure. That's right, they wanted me to completely and utterly gut my Maven src/main/java type directory structure and replace it with their own. No... freaking... way...

Git to the rescue

During the entire project I had been using Git for my local source control. This allowed me to keep version copies of my source code as the project progressed and only when I felt the code was ready to be put onto the client's SVN server would I do a git svn dcommit. This proved to be very good foresight on my part as it allowed me to keep two separate versions of the code base. One that I kept in git, and one that I pushed to the client, but it wasn't as cut and dry as that. Read on...

Maven Assemblies to the rescue

Now that it was determined that that directory structure of my project was to conform to the client's custom structure I was in a quandry. I like Maven, it's great and I really couldn't see myself going back to Ant. So what I did was I created a Maven Assembly which takes the code and artifacts from my local Maven project and spits out a project who's directories conform to the client's wishes. I then rsync the assembled directory over to a place on my hard drive that contains the SVN checkout of the client's code.

Using these techniques I can now keep my existing maven layout, commit my changes to git whenever I wish. When I'm ready to give the client a build, I follow these steps:

• mvn assembly:directory
• cd target/project-assembly.dir
• rsync -a ./ ~/src/client-svn
• cd ~/src/client-svn
• svn commit

Other General Problems You as a Client should Avoid

If you are going to outsource projects to outside vendors here are some tips:

• Let vendors have access to log files and web server restarts in DEV
  • I had access to neither. It slowed down integration at least 10 fold. I had to request both log files and server restarts and it could take up to 12 hours to get a response. I actually resorted to intentionally throwing Exceptions with my log messages in them just to get by. If I could have tail -F'd a log file or two, things would have been so much better.
• I get build standards, really I do. But when dealing with outside vendors you either have to be upfront about them from day 1, or you have to have people on staff who can help migrate a project to the standard. I still feel to this day that it was not my responsibility to convert my codebase to Ant and to their folder structures, my job was to create a deliverable. In the Java world that means a) source code and b) a means to build it to a WAR file
• VPN - for the love of god people allow your freaking VPN to resolve domain names at the very least. The VPN I was stuck with would not resolve external or internal domain names. Everything was done by IP address. I kept pinching myself to see if I was in a nightmare or something... Also, no external (outside the VPN) connections were allowed. This means I couldn't even freaking google something without disconnecting, nor could I send email, nothing. It was like a bad joke or something
  • Honestly, the best VPNs I find are the ones that say "oh, google is not one of our IPs, I'll ignore that and let life outside the VPN connection handle it"

That's all for now.

</rant>

P.S. I ended up re-writing those Struts 1.2x apps in Wicket because they absolutely refused to build them separately.