Data Integration Language

Background: Low-code to low-level

In 2018, I started my open source project Assimbly. A project based on Apache Camel to create platform-independent connectors and services.

The project is divided into two main parts:

Since 2021 I work as a free contractor for a low-code integration platform Dovetail. They like to use Assimbly runtime to run their integrations.

Lots of formats

Dovetail uses JSON as a format. This file is generated with a visual designer and then stored in MongoDB. When installing a flow, the JSON file is converted to Camel routes (Blueprint.xml). This XML file runs on Apache Karaf.

The format of Dovetail is pretty nice with lots of abstractions on top of Camel, because of its low-code nature. The used terminology however got a little mixed up with Camel. Is it a Camel component or a Dovetail component? As the company wants to change the runtime to Assimbly runtime, I had to figure out how these various formats fit together.

Both Dovetail and Assimbly are a couple of years old, but in the meantime another effort was created CamelK. This project not only pioneered running Apache Camel in the cloud, but also has its own format that abstracted away from Camel: Kamelets. Originally, Kamelets were written YAML, but now also other formats are supported (though not JSON). Some of these Kamelets would also be interesting for Dovetail and Assimbly. Thus, yet another format to consider.

We now have four formats:

Let’s show some examples to see how the various formats look like:

Camel XML DSL (blueprint.xml)

<camelContext id="ID_627a596f38c74a0374000321" xmlns="http://camel.apache.org/schema/blueprint" useMDCLogging="true" streamCache="true">
<onException> <exception>java.lang.Exception</exception>
<redeliveryPolicy maximumRedeliveries="0" redeliveryDelay="5000"/>
<setExchangePattern pattern="InOut"/>
</onException>
<route id="915da660-d069-11ec-83f5-3747809ef661">
<from uri="activemq:ID_627a596f38c74a0374000321_test_84601220-d05c-11ec-83f5-3747809ef661"/>
<split streaming="false" parallelProcessing="false">
<xpath saxon="true" threadSafety="true">/split</xpath>
<setHeader headerName="CamelSplitIndex">
<simple>${exchangeProperty.CamelSplitIndex}</simple>
</setHeader>
<setHeader headerName="CamelSplitSize">
<simple>${exchangeProperty.CamelSplitSize}</simple>
</setHeader>
<setHeader headerName="CamelSplitComplete">
<simple>${exchangeProperty.CamelSplitComplete.toString().trim()}</simple>
</setHeader>
<to uri="activemq:ID_627a596f38c74a0374000321_test_915da660-d069-11ec-83f5-3747809ef661_BottomCenter_split?timeToLive=86400000&amp;exchangePattern=InOut"/>
</split>
<to uri="activemq:ID_627a596f38c74a0374000321_test_915da660-d069-11ec-83f5-3747809ef661_split?timeToLive=86400000&amp;exchangePattern=InOut"/>
</route>
</camelContext>

Camel Java DSL

from("direct:a")
.choice()
.when(simple("${header.foo} == 'bar'"))
.to("direct:b")
.when(simple("${header.foo} == 'cheese'"))
.to("direct:c")
.otherwise()
.split(body().tokenize("\n"))
.to("direct:d");

Note that the above Java DSL is without the routeBuilder class and its configure method where the Java DSL is usually placed.

Kamelet

apiVersion: camel.apache.org/v1alpha1
kind: KameletBinding
metadata:
name: delay-action-binding
spec:
source:
ref:
kind: Kamelet
apiVersion: camel.apache.org/v1alpha1
name: timer-source
properties:
message: Hello
steps:
- ref:
kind: Kamelet
apiVersion: camel.apache.org/v1alpha1
name: delay-action
properties:
milliseconds: "1000"
sink:
ref:
kind: Channel
apiVersion: messaging.knative.dev/v1
name: mychannel

Dovetail

{
"_id": "627a6b7338c74a00130007f9",
"components": [
{
"_id": "04a6c550-d067-11ec-83f5-3747809ef661",
"_type": "InboundHttpComponent",
"endpoint": "#{self.flow_name}",
"exchangePattern": "requestReply",
"matchPrefix": false,
"note": null,
"preserveHttpHeaders": false,
"previousEndpoint": "",
"previousNode": "",
"protocol": "https",
"tenantPart": "id",
"x": 280,
"y": 196
},
{
"_id": "04a6ec60-d067-11ec-83f5-3747809ef661",
"_type": "VelocityComponent",
"note": null,
"previousEndpoint": "RightMiddle",
"previousNode": "75be5f00-d0f8-11ec-83f5-3747809ef661",
"template": "Message Body:\n\n${bodyAs(String)}",
"x": 784,
"y": 294
},
{
"_id": "75be5f00-d0f8-11ec-83f5-3747809ef661",
"_type": "AggregateCurrentComponent",
"aggregateFileType": "xml",
"completionCount": "3",
"completionCountTimeout": "-1",
"completionInterval": "",
"note": null,
"previousEndpoint": "BottomCenter",
"previousNode": "797f5ea0-d0f8-11ec-83f5-3747809ef661",
"x": 602,
"y": 294
},
{
"_id": "797f5ea0-d0f8-11ec-83f5-3747809ef661",
"_type": "SplitCurrentComponent",
"aggregateFileType": "none",
"exchangePattern": "requestReply",
"expression": "/names/name",
"expressionType": "xpath",
"namespace": "",
"note": null,
"nsprefix": "",
"parallelProcessing": false,
"previousEndpoint": "RightMiddle",
"previousNode": "04a6c550-d067-11ec-83f5-3747809ef661",
"streaming": false,
"x": 434,
"y": 196
}
],
"created_at": "2022-05-10T13:41:07.581Z",
"environment_variables": [],
"error_components": [
{
"_id": "627a6b7338c74a00130007fa",
"_type": "FailedExchangeComponent",
"note": null,
"previousEndpoint": null,
"previousNode": null,
"redeliveryAttempts": 0,
"redeliveryInterval": 5000,
"x": 400,
"y": 300
}
],
"flow_group_id": "627a4aa838c74a000e0005cc",
"icon": null,
"icon_name": null,
"isFlowComponent": false,
"lock_user_id": "61a0030f8249dde2022c7616",
"name": "Aggregate",
"request_timeout": 20000,
"tracing_ttl": null,
"transport": "activemq",
"trashed": false,
"updated_at": "2022-07-30T22:56:36.690Z",
"bundle_id": "ID_627a6b7338c74a00130007f9",
"lockUserName": "Super Admin",
"lockCurrentUserCanEdit": null
}

Assimbly

<?xml version="1.0" encoding="UTF-8"?>
<integrations>
<integration>
<id>1</id>
<name>default</name>
<type>ADAPTER</type>
<environmentName>Dev1</environmentName>
<stage>DEVELOPMENT</stage>
<defaultFromEndpointType>FILE</defaultFromEndpointType>
<defaultToEndpointType>FILE</defaultToEndpointType>
<defaultErrorEndpointType>FILE</defaultErrorEndpointType>
<offloading/>
<flows>
<flow>
<id>2</id>
<name>FILE2FILE</name>
<autostart>false</autostart>
<offloading>false</offloading>
<maximumRedeliveries>0</maximumRedeliveries>
<redeliveryDelay>3000</redeliveryDelay>
<logLevel>OFF</logLevel>
<endpoint>
<id>2</id>
<type>from</type>
<uri>file://C:\test1</uri>
</endpoint>
<endpoint>
<id>2</id>
<type>to</type>
<uri>file://C:\test2</uri>
<options>
<directoryMustExist>true</directoryMustExist>
</options>
</endpoint>
<endpoint>
<id>2</id>
<type>error</type>
<uri>file://C:\test3</uri>
</endpoint>
</flow>
<services/>
<headers/>
<environmentVariables/>
</integration>
</integrations>

Universal language

Based on a comparison between the various formats, I noticed many similarities, but also some specific functionality. Similar were for example that both Kamelets, Dovetail and Assimbly are on a higher-level than routes. They target specific use cases, instead of being technical components with many options.

There were also differences:

The different formats show the multiplicity of writing configurations. I decided to gather the various formats and put them in an universal language, called DIL. This gives a common way to speak about integrations. Most of the terminology is to my belief intuitive for common developers. For example, a flow and a step:

I also took some ideas from the various formats:

In the next chapter we discuss the concept of levels further by discussion low-level programming and low-coding.

Low-level to Low-code

Programmers like to automate their own work. It’s thus no surprise that low-code platforms, RAD tools, code generators sprung from the minds of developers. I really love those platforms, languages and frameworks that work on a higher-abstraction level. Using components that let me work and think more intuitively and build solutions faster.

I always try to work on a low-code level when possible. But I keep getting back to low-level stuff, because all of these low-code solutions have a couple of downsides. Some of them are:

DIL is, because of above reasons not low-code, but like low-code, tries to run on a higher-abstraction level. These higher-levels of DIL can be easily used by low-code platforms.

Low-code and High-code

Developers hate low-code, because they find low-code limited and inflexible, while business developer hate coding because it’s too complex and development is too slow. Low-level means here real programming while low-code something drag-and-drop like.

Is it always low-level vs low-code?

That low-code and low-level don’t get along, is at least what you mostly read in the media. In general however, people don’t care that much. They just want to get to a solution. It’s true that (high-)code and low-code use different approaches to get to a solution, but they are much closer as often being outlined.

Consider for example the low-code development platform Mendix. It actually started as a no-code platform. The original slogan was “No code, just glory”. However, from the start they already used XPath to retrieve values. Also visual models like ERD-diagrams were used (and the code was generated from there). And it uses microflows to ‘program’ the business logic. And it even became possible to call Java from such flow. So the change from no-code to low-code was for sure more accurate.

A Mendix Microflow

Modeling and flows in general are actually pretty close to programming. It’s just not textual, but visual programming. On the other hand coders used those models for decades (only the generating tools weren’t that advanced, so they wrote the code themselves). Additionally programmers have always been pushing the boundaries of programming languages to become more high-level.

DIL acknowledges different ways of programming. Not generational, after each other, but on certain levels next to each other.

In the picture below are the various levels of the DIL language. Where the core components are closer to programming in third generation languages and the higher level components are closer to the business and architecture of a specific organization.

People work, based on their role, from low-level to low-code. Thus programmers create, in Java or another JVM language, various core level integration components. This is the place where integration specialists and programmers meet.

DIL’s aim is thus not to choose between high-code and low-code, but to bring them together and let developers go through various levels, just like it using the elevator in the company building. For low-level to low-code!

Final note

In my career, I have filled all four roles in the above building. I sometimes literally need to go up and down in the building to speak with someone with another role. The building however is more to make a point than to take the metaphor too literally. It’s about that based on where you are and what you want to achieve, you be on a certain level. That you need the right tool for the right job. And that there is no silver-bullet. And more of those wisdoms…

More reading:

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store