Dovetail 4.17.0: Under the hood
Everything changed
At 2021, I started with a new team working on the integration platform Dovetail. The main assignments we got from CEO Erik Greefhorst were:
- Make Integration Easy (Again).
- Give integration developers more insights.
- Modernize the technology stack.
In previous blogs I already spoke about the first two, but in this blog I will focus on the latter.
Changing the basic building blocks
When working on Dovetail flows, you may have noticed a “Copyright” notice with the dates 2012–2024.
Is Dovetail that old? Yes, it is. It came into existence at the consulting firm Kabisa. In 2012, they were integrating their customers using Apache Camel. In Camel, as a Java integration framework, you have to write most things unsurprisingly in Java. This was a problem for Kabisa as they wanted to provide their customers with ways to modify and manage integrations themselves.
Initially these were XML and configuration files, but this was far from convenient, so the developers created a web application with the following basic building blocks:
- BackBone.js (Frontend Framework)
- Ruby on Rails (API’s, also referred as the backend of the frontend)
- Apache Karaf + Apache Camel (Backend)
When Dovetail spin off, these building blocks have been updated several times. But the technologies themselves didn’t change that much. As the world of IT changed, this became increasingly problematic.
Headaches
BackBone.js, for example, was a vibrant project with multiple releases each year when it was selected in 2012. Starting in 2016, development slowed down and many other frameworks like Angular, React, and Vue took its place. In 2023, we switched from BackBone.js to React, which greatly improved our front-end capabilities.
The next victim of our modernization was the backend. We faced a similar problem as on the frontend. Our integration engine, Camel, was running on Apache Karaf. A runtime for Java software. Karaf itself was based on the OSGi standard. We have always been very happy with this, because OSGi enables dynamic and modular Java services.
Apache Camel always provided good support for OSGi. Other Java libraries and frameworks however not so so much, and this became more and more of a headache. OSGi lost support because it required more effort from Java developers than pure Java. Over the years, many library maintainers concluded that it wasn’t worth it.
OSGi became less and less supported, which created challenges for us when updating our custom components. A lot of class loading problems, aka headaches, occurred. Eventually even Apache Camel gave in and OSGi became best effort for their framework. It was time to move on.
The search for a new runtime
So we were kind of stuck on the backend. Not only could we not easily update the Java libraries, but the Camel framework was stuck at version 2 (the last version we could support was from September 2019). Fortunately, Apache Camel has become runtime agnostic over the years, supporting various other options such as WildFly, Spring Boot, Quarkus, and Kubernetes. I have blogged about the available runtimes before.
It was time to choose a new runtime for our backend, and with this change, one of the core building blocks. Our basic requirements for the new runtime:
- Don’t tie the “integration framework” (Camel) to the “runtime framework”.
- The runtime must be API driven (REST endpoints).
- The runtime must support Dovetail concepts such as flows and versions.
- The runtime must be able to run current configurations (Camel 2) as well as new configurations (Camel 3 and 4).
- Support for the latest versions of Java.
We couldn’t immediately find a runtime that met our needs. The integration runtime, as the place where data is processed, is so core to Dovetail that we decided to create our own. “Isn’t that a lot of work?” you might ask. The answer is yes.
Assimbly
It took us two years to create a runtime that met all of our needs. The new runtime is based on the Assimbly open source project. It provides modules on top of open source integration frameworks like Apache Camel and Apache ActiveMQ.
Assimbly is fully API-driven, which means its able to receive requests from the new frontend, like installing a flow, getting the latest status of flows and their stats. There is also a compatibility layer that can take an Apache Camel 2 configuration and convert it to a newer version.
The upgrade was done in two steps. First, we upgraded to Camel 3, version 3.20.x from December 2022. This was used in Dovetail 4.15 and 4.16. From the new 4.17 version we will use Camel 4.8.0 that came out in September, 2024. Yes, that’s really up-to-date. But Camel was not the only thing that we updated.
What technologies were updated? All!
The backend is something that no end user will see. Sure, they know immediately if data is not being processed correctly, but it’s not something that sells. This sometimes leads to an “as long as it works” mentality, where the backend ends up being neglected. A situation best illustrated by Homer Simpson:
The Dovetail development team was happy to see that a significant amount of resources was invested in migrating and upgrading the backend technologies. We have already talked about the migration from Karaf to Assimbly, but many other technologies have been upgraded as well.
So what has changed? Everything. Because once you start upgrading one technology, you touch other components that need to be upgraded as well. It’s often not easy to just upgrade to the latest, but upgrading is like a game of chess, you really have to think ahead. Sometimes you have to move pieces forward, and sometimes you have to move a piece back.
To give an idea about the upgrades, here is a list of software that was upgraded:
This list is just a few of the highlights. Many more libraries and frameworks have been updated, such as logging frameworks, database drivers, and test frameworks. 198 dependencies in total. One of the biggest things still on our roadmap is the migration of Ruby on Rails to Spring Boot.
A new policy
In today’s fast-paced technology world, keeping your software up to date seems like a no-brainer. Almost every IT manager and even C-level executives agree. In practice, however, priorities are set elsewhere, and the effort required is usually much harder than expected. Take Java, for example: many companies are still running Java 8, a version that came out in 2014. That’s ten years ago.
Upgrading one thing, however, never stands alone. There are other libraries that are being adapted. At Dovetail, we used Java EE (Enterprise Edition) a lot, where the libraries were moved from the Javax namespace to the Jakarta namespace due to legal battles with Oracle. However, the move to Jakarta also caused a lot of other libraries to be upgraded, some that didn’t even support Jakarta yet. As I said, sometimes upgrading is like a game of chess.
One library that wasn’t running with Java 21, was even from 2010. The problem was however, that the functionality provided by the library was used by a lot of users. That’s why the library was completely rewritten by my colleague Bruno Vieira Gonçalves with the modern library Jackson. There were many of these challenges we faced, until we came to the level we are now.
So it’s not easy to change a whole framework or do major upgrades all at once, so we’ve changed our policy of selecting new technologies and keeping them up to date. The main points are:
- We prefer technologies that are open source, supported by large organizations, and have a large community. This makes us a bit of a member of the boring technology club. Boring maybe, but it provides proven technologies with regular releases. This allows us to focus on innovating on the platform itself. On solutions that matter to our customers.
- The second policy is that we have automatic monitors in place when updates occur. We schedule security and patch updates for every patch release, and major updates for every major or minor release. We prefer LTS (long-term support) releases. This doesn’t apply to every technology, but to major ones like Ubuntu, Java and Camel.
With this Herculean task, which we’ve been working on for more than 2 years, and a better update policy, we think we’ll have a good foundation for a stable platform for years to come.
