Catarang in 2016 – Building a CI Server

Catarang is a Continuous Integration (CI) server that I’ve been working on for the past two years in my spare time and have made a ton of progress on in 2016. CI servers, for those who don’t know, do repetitive tasks like compiling code every time a change is checked in or automating pushes to production servers. The most popular one out there is Jenkins, but there are a ton of other options listed out on this excellent Wikipedia page: Comparison of continuous integration software. In this post, I’ve written up the progress I’ve made on Catarang in the past year and what I’m looking to do with it in the future.

Why build a new CI Server?

A question I often get when I talk about Catarang to people is why build a new one if there are a bunch of options out there already? The two main reasons are that none of the CI Servers out there fulfill all my needs and I wanted a project to learn Go on. I’d heard good things about the language and had tried out the little playground they had, but I learn best when I actively use the language on a project. Throughout the two years I’ve been working on Catarang, I’ve fallen in love with Go and find it an utter joy to program in. Once I got a hang of the language (which was very quick), I’ve been able to get a surprising amount done with it with a limited amount of head scratching issues coming up.

I’ve also learned a ton about CI servers and unique issues surrounding how they work. These things are incredibly complicated and take a lot of fiddling to get right! Lastly, it’s been fun to build a large scale tool by myself and be able to rework all of the code at the drop of a hat if I don’t like how it’s structured. I try to keep that to a minimum and make constant forward progress, but at times it’s beneficial to take a step backward and to the side so that it’s easier to go forward in the future.

Progress in 2016

Catarang Contributions for 2016 The amount of energy I had to work on Catarang fluctuated quite a bit this year due to work, illnesses, life issues, and politics. There were several weeks where I worked on it non-stop in my free time and then there were several months where I didn’t have any energy to even think about it.

This makes sense considering I’m only working on my own time and not being paid for it, but there were certainly days when I felt guilty about not programming on it when I had wanted to. Catarang as a whole has only 105 commits, so with 81 of them coming in the 2016, it’s easy to see that I did a significant chunk of work on the project this year.

Feature Set

Catarang is still in its infancy, so I wouldn’t suggest anyone start using it yet, but I am proud of how the features are coming together and it’s starting to feel like a real project. Here are all of the features I worked on this year:

Plugin System

I started off building all of the functionality straight into Catarang for git, running arbitrary commands, and saving off artifacts, but realized it’d be better to have a defined interface to make writing plugins easier. I’ve gone through several iterations and am at a middle point with this, but am happy with where I am right now. Each plugin is completely segregated from the rest of the code and I eventually plan on moving them outside of the project itself so that they can be updated independent from Catarang itself. I briefly looked at Hashicorp’s go-plugin and even got a prototype working, but stopped going down that route as it felt like I was going a bit too deep into that system and ignoring other necessary features. Right now plugins are as easy to add as filling out a basic interface and adding a single line to the plugin list to register it.

Job Template and Instantiation

I created a job template so that you can create a job via a single file that you keep in your depot. Catarang will pull in the appropriate plugins from the plugin system and run the commands that you specify in the template. Catarang can already handle when you check in a change to the config file and will rebuild the job based off of the new template, which took a bit of work to allow. This is the number one feature that I had originally wanted in a self-hosted CI server that wasn’t available in the one I was using at work, so it’s nice to have made really good progress on it. The template itself is pretty simple and this is what Catarang’s currently looks like:

It’s a bit verbose since it’s in JSON and that will likely be changed at a later point in time, but it’s a great start.

Job Instances

Each time a job runs, it creates a new instance of itself and saves off all of the log files associated with running that job. Each instance is separate because if you change a job’s configuration template, you’ll get very different output and want to keep that information around. It’s also useful to go back and see why certain runs of the job failed and why some succeeded. This feature has changed quite a bit over the past year due to creating the job template and the plugin system, but it’s at a pretty good place right now.

Unique Logging of Commands

I’m pretty proud of how Catarang logs all of the commands it runs. For every command it keeps

1. A list of the arguments used to run the command
2. A high level description of what’s being run and the plugin that ran the command
3. The output from the command segmented into the Standard Out and Standard Error sections

Because of that last part, I can highlight on the CI server (and in emails) which parts actually failed so you don’t have to go digging for it when it does! I haven’t seen any other CI server do this, so it felt good to get in. It was a bit difficult to get in as I learned there are some deep, dark secrets of StdOut and StdErr that I didn’t know before. Did you know you can get interwoven lines of output in your console that look like this:

Yikes! It gets even worse if you have sub-commands being run in parallel and only a single output window.

A Web Server

One of the awesome parts of Go is that it’s suuuuuper easy to set up a web server and serve content. The most basic setup is a single line of code, and a more complex example isn’t far from that. I know next to nothing about web development, so I have a very simple web interface for Catarang that allows you to add new jobs, run them, delete them, clean them, and see all of the output from each job instance. This is the largest thing that will have to be worked on before I release it to the world, but it’s also the least important until I get a solid feature set up and running. I used to have fancy websockets working so the site would live-update, but that broke at some point and I didn’t care enough to go back and fix it since so much of the architecture of the program was changing.

What’s Next?

In 2017 I’m going to work much more on making Catarang stable and usable for the general public. It’d be neat to be able to release a very alpha version and start getting feedback on it, but that’s a bit of a stretch goal.

As for the next features I’m going to be implementing, a lot of it will focus on expanding out the Job’s capabilities. I’d like to add the ability to chain multiple jobs together in a pipeline using if/or/and blocks. Getting this right is the most important thing for a CI to be useful to users, so it will likely be quite a bit of iteration before I’m happy with it.

I’d also like to start expanding the plugins to other things people might find useful like Slack notifications and emails. There’s a long way to go, but if I implement some of the basics that most people are going to want, then I’ll get some traction in order for people to want to create their own to flesh out the parts they want but I haven’t made yet.

Tests are something I have a small amount of, but not nearly enough. I haven’t bothered with them since I’m more focused on building out a prototype, but now that it’s getting a bit larger they’re becoming much more useful. I’m going to be aiming for a fairly high test coverage percentage and we’ll see where I get with that.

Lastly, I’d like to start fleshing out the UI on the server into something that’s not a programmer’s prototype. I’m not very good at web development mostly due to lack of experience, so this either entails me finding someone that wants to work on this project with me (for free) or spending a lot of time learning how to build a highly interactive website from scratch. Both seem fairly difficult to do, so we’ll see how this goes.

I’m still super excited to be working on Catarang two years after I first came up with the idea, which is great. I’d love to be able to release it and have people other than myself use it on a day to day basis so I keep building it with that in mind. I know that if I got some dedicated time to work on it I could really build it up into something special, so I’ll have to slowly work in that direction since I can only use spare time right now. If you want to keep up to date on my progress you can follow me on Twitter or follow Catarang on GitHub.