Assimbly in the cloud
3.0 is just released, and we are now working on some new experimental features for 3.1. These are:
Support for all Camel components
This feature let users select all available Camel components. Those components are loaded dynamically in the background.
Modularization
Breaks up the Assimbly connector into separate modules. In this way developers can decide what parts of Assimbly they include as dependency and what not.
Microservices: Running on Kubernetes
Allow running Assimbly on Kubernetes. This lets Assimbly scale horizontally. Assimbly is split in a gateway (frontend) and a microservice (backend).
All these functions have in common that it will run Assimbly in a more modular, flexible and scalable way. This is especially (but not only) targeted at the cloud.
If you only wanted the management summary of what we are working on than you know enough.
If not? We will continue this blog for all techies with details on the ideas and technologies on these 3 new features.
Functionality vs Performance
Sometimes it’s a fine line between performance and functionality. Too much functionality and it becomes unmaintainable and heavy. Too less and it is not useful anymore.
Both frameworks used in Assimbly, Jhipster and Camel, take these into account. Jhipster for example asks what sort of components you want to include when generating a new application. Afterwards you can enhance the data model or add extra modules.
Camel (especially in version 3) is highly modularized so you can choose only what you need. If you for example only need to support one protocol then you just include that component and a few core components.
Compile time and runtime
Like most things, it’s not that easy… The modules of Jhipster and the components Camel are mainly targeted at development (compile time). What if you want to support multiple use cases in runtime?
Most of such uses cases are covered with configuration. In Assimbly configuration files are used to configure the web application and at Camel level we use a database to configure flows.
But not all things can be solved by configuration. Sometimes you need a more modular approach. For this we take some inspiration from Linux. The OS is highly modular so all kinds of distributions arose. Still even when users are running their distribution they can change this with a package manager. They can add software, change the desktop environment and so on.
With the help of students of the University of Applied Sciences Utrecht similar approaches for Assimbly are researched and implemented in our latest development branch.
Modular functionality
One of the powers of Camel is that it supports a lot of components. These can be older technologies like file, FTP and SQL as well as newer ones like Kubernetes, Git and Slack.
Currently, there are:
- 25 of Core Components (in 22 jar artifacts)
- 316 of Non-Core Components (In 246 JAR artifacts)
Those components aren’t supported out-of-the-box. You need to add the component as a dependency. By default, dependency management tools like Maven and Gradle will add these in compile time. This means that the jar artifacts will be added to the main jar of Assimbly.
When writing a small microservice one can take just enough core and non-core components to run a Camel integration. This leads to fast startup and small memory footprint.
But what if you need to support another protocol? Normally you can create another service or write some more code and add the extra component as a dependency. Thereafter, you recompile and redeploy the application. But what if you don’t know what kind of extra functionality a user wants? And what if there is no time to add these extra dependencies?
There are some runtimes that can share resources, like Karaf (based on OSGi modularization) and Wildfly. There you can add modules that shares dependencies. In this case no compile-time dependencies are needed, but it’s assumed that the dependencies are provided. But even in these cases the dependencies somehow need to be available by another module.
But what if you don’t know what runtime will be used? What if you don’t have these big setups provided? In this case you want something like Linux. You need something more dynamically.
New Features
Support all Camel components
Until version 3.0 Assimbly was not very dynamic. At one of the first versions we only added a few components. That were components used for data integration like JMS and FTP. Assimbly was around 100 MB in size. In the following releases we added more and more components. And now we are supporting 30 components with a size of 300 MB. Still, this is only a small portion of what Camel is capable of.
As users need a specific component we could add a new component manually. This has some disadvantages:
- We never know beforehand which components will be used? What use cases Assimbly will be used for (and we don’t want to know).
- We need to add the dependency manually and make a new release.
- Every added dependency will make the download of Assimbly larger. For example adding the AWS components lead to more than 100 MB size of the Assimbly jar file.
What we’ve done in the 3.1 branch is let user choose out of all available Camel components from the GUI. On the background three things are done:
- Search for the Maven dependency in the Camel Catalog.
- Resolve the dependencies programmatically by the Java dependencies manager Jeka.
- Load the jar dependencies dynamically in runtime.
This solution may not always be perfect. Firstly it requires an internet connection. Also, some components require extra code or dependencies to make them useful. Still, this opens up using Assimbly for a lot more use cases.
Modularization
Besides loading components in runtime, we also want to modularize the project in compile time. This leads to faster compilation time and let other projects decide what modules they need or not.
Current situation in 3.0 is:
- Assimbly Connector contains all the “Integration logic” which consists of an API (Connector and Broker), Config, Camel implementation, ActiveMQ implementations for Classic and Artemis.
- Assimbly Gateway makes use of the Assimbly connector and Jhipster libraries.
In 3.1 the connector will be separated in a
- Config API
- Connector API
- Broker API
- Main module
- Connector implementation modules
- Broker implementation modules
When for example I only need ActiveMQ Artemis I don’t need others. This leads to smaller jar sizes and less class loading.
Microservices: running on Kubernetes
The choice that Assimbly would be a monolith, instead of a microservice, started at the very beginning of the project.
When starting tinkering with what would be the best web application framework Jhipster was chosen. Jhipster let you generate the web application. There you can for example choose if you want Angular or React as frontend. It also has the choice between a monolith and gateway/microservice application.
When Jhipster talks about a gateway, this doesn’t mean a message gateway (like Assimbly does), but it’s a web frontend that handles the web traffic.
Firstly both options, monolith and gateway/microservice, were tried. For the Jhipster gateway this lead to a small problem, because the word ‘gateway’ was reserved. At the end both applications were running fine.
So both architectures were supported and both worked well, but they can not work both at the same time. At the end it was decided to stick with the monolith, because it was easier to develop, and we didn’t need two maintain code bases (for only 1 developer at the time…).
Because of more and more cloud set-ups, modularization and scaling the idea of microservices came back to the Assimbly project.
Supporting both architectures
As we don’t want to many code bases. We decided for 3.1 that Assimbly message gateway will be enhanced with Jhipster gateway functionality. This means that Assimbly gateway can still be used as standalone (monolith) application, but when detecting a Kubernetes environment it will act as a frontend only. The Jhipster architecture on Kubernetes is as follows:
In the above overview all parts run in a Docker container. A user logs into the Jhipster gateway. When it creates a new Assimbly flow the configuration is saved into the database. Also, a microservice is started on the background that will run the flow. This microservice is detected through the Jhipster registry. The user can start the flow without any delay.
The graphics below translate the Jhipster architecture to Assimbly:
To sum up:
- Assimbly Gateway can act as a standalone application which contains an embedded database and the Assimbly connector. When it detects a Kubernetes cluster it will act as a front-end only.
- Assimbly ConnectorService is microservice that runs one flow (A configured Camel Route). It can run as one or multiple instances.
We hope to release an alpha version of 3.1 in few weeks.
