I’ve been moving my development towards micro-services, containers, and Kubernetes. One of my objectives with the new version of the Software Factory (SF) was to make the generation engine work in a container.
The good news is: That I was able, for the first time, to successfully run the SF engine in a container yesterday.
It may sound trivial these days to run an application in a container, but it was nothing but in this case. There were significant changes that needed to be incorporated just to support containerization. This thing had deep-reaching tentacles into my personal development machine…
In addition, this was an entirely new version with 12 new micro-service with new models, 5 new client applications, and close to 60% rewrite of the SF engine, which is relatively complicated and extensive(486 files, 19k LOC)…
The first challenge was to define how to trigger builds. The previous version was an MVC application. I would just navigate to the UI, click a button to initiate a build, and access a web page to see the results. Easy, but quite limiting.
Because I want to use the SF in an automated way, I wanted the SF engine to be independent. I didn’t want to interact with the SF engine directly either, so I set up a system where requests could be sent to another service, and the SF would monitor it to figure out when to run a new build.
The SF is a hub of information; it takes in large amounts of data of different types from multiple services. It processes it and generates a large amount of result information and thousands of output files. It does so every time I run a build. And I can run builds hundreds of times a day…
The results were stored directly in a local database when running on my computer, and the output files were injected directly into my workspace. The output files were reasonably easy to manage because my workspace is stored in a git repository.
For the results, they would accumulate. When the volume of information became too great (>200M records) and performance was starting to be impacted, I would just delete the database, and the system would create a new one automatically. But that meant I was losing information.
When running in a container, I needed a more robust solution.
So I created a new SF result service. This way, I could access and manage my results without affecting the SF engine.
But, I also need to find a mechanism to manage the volume. So I added the notion of a lifetime. Lifetimes are managed in a separate micro-service and can be attached to results.
A third service can then be used to apply lifetime to results. Delete results that are no longer required and keep those that are important.
Dealing with output files
To access the generated files, I needed an easily accessible solution. I thought that using my git repository to store every build as a separate commit would be an interesting solution.
I created a module to perform basic git commands in the service (checkout, pull, commit, push). Then create tasks to initialize the repository at the beginning of the build and commit and push at the end of a successful build.
Again, because of decoupling, I setup up a service containing all my git repositories, and, in the build, I get to pick which repository to store the output files.
Then I had to install and configure git on the container to access the git repositories.
And voila! All I have to do to access the files is set up a workspace on the same repository. Every time I do a pull, I get the latest results from the SF engine.
Are we there yet?
Not exactly, but we’re a lot closer than we were… There are a lot of issues left, and I don’t expect to be able to resolve them all in this version.
As I like to say: A software factory is not complete until it can generate itself…
Or at least as much as the previous version did.