Sep 23, 2022

8 min read

Data Integration Language

Background: Low-code to low-level

In 2018, I started my open source project Assimbly. A project based on Apache Camel to create platform-independent connectors and services.

The project is divided into two main parts:

Assimbly runtime: Runs integrations with Camel.
Assimbly gateway: A web application to configure and manage integrations.

Since 2021 I work as a free contractor for a low-code integration platform Dovetail. They like to use Assimbly runtime to run their integrations.

Lots of formats

Dovetail uses JSON as a format. This file is generated with a visual designer and then stored in MongoDB. When installing a flow, the JSON file is converted to Camel routes (Blueprint.xml). This XML file runs on Apache Karaf.

The format of Dovetail is pretty nice with lots of abstractions on top of Camel, because of its low-code nature. The used terminology however got a little mixed up with Camel. Is it a Camel component or a Dovetail component? As the company wants to change the runtime to Assimbly runtime, I had to figure out how these various formats fit together.

Both Dovetail and Assimbly are a couple of years old, but in the meantime another effort was created CamelK. This project not only pioneered running Apache Camel in the cloud, but also has its own format that abstracted away from Camel: Kamelets. Originally, Kamelets were written YAML, but now also other formats are supported (though not JSON). Some of these Kamelets would also be interesting for Dovetail and Assimbly. Thus, yet another format to consider.

We now have four formats:

Camel DSL
Kamelets
Dovetail
Assimbly

Let’s show some examples to see how the various formats look like:

Camel XML DSL (blueprint.xml)

<camelContext id="ID_627a596f38c74a0374000321" xmlns="http://camel.apache.org/schema/blueprint" useMDCLogging="true" streamCache="true">
  <onException>      <exception>java.lang.Exception</exception>
      <redeliveryPolicy maximumRedeliveries="0" redeliveryDelay="5000"/>
      <setExchangePattern pattern="InOut"/>
</onException>
  <route id="915da660-d069-11ec-83f5-3747809ef661">
    <from uri="activemq:ID_627a596f38c74a0374000321_test_84601220-d05c-11ec-83f5-3747809ef661"/>
    <split streaming="false" parallelProcessing="false">
      <xpath saxon="true" threadSafety="true">/split</xpath>
      <setHeader headerName="CamelSplitIndex">
        <simple>${exchangeProperty.CamelSplitIndex}</simple>
      </setHeader>
      <setHeader headerName="CamelSplitSize">
        <simple>${exchangeProperty.CamelSplitSize}</simple>
      </setHeader>
      <setHeader headerName="CamelSplitComplete">
        <simple>${exchangeProperty.CamelSplitComplete.toString().trim()}</simple>
      </setHeader>
      <to uri="activemq:ID_627a596f38c74a0374000321_test_915da660-d069-11ec-83f5-3747809ef661_BottomCenter_split?timeToLive=86400000&amp;exchangePattern=InOut"/>
    </split>
    <to uri="activemq:ID_627a596f38c74a0374000321_test_915da660-d069-11ec-83f5-3747809ef661_split?timeToLive=86400000&amp;exchangePattern=InOut"/>
  </route>  
</camelContext>

Camel Java DSL

from("direct:a")
    .choice()
        .when(simple("${header.foo} == 'bar'"))
            .to("direct:b")
        .when(simple("${header.foo} == 'cheese'"))
            .to("direct:c")
        .otherwise()
            .split(body().tokenize("\n"))
            .to("direct:d");

Note that the above Java DSL is without the routeBuilder class and its configure method where the Java DSL is usually placed.

Kamelet

apiVersion: camel.apache.org/v1alpha1
kind: KameletBinding
metadata:
  name: delay-action-binding
spec:
  source:
    ref:
      kind: Kamelet
      apiVersion: camel.apache.org/v1alpha1
      name: timer-source
      properties:
        message: Hello
  steps:
    - ref:
        kind: Kamelet
        apiVersion: camel.apache.org/v1alpha1
        name: delay-action
      properties:
        milliseconds: "1000"
  sink:
    ref:
      kind: Channel
      apiVersion: messaging.knative.dev/v1
      name: mychannel

Dovetail

{
 "_id": "627a6b7338c74a00130007f9",
 "components": [
  {
   "_id": "04a6c550-d067-11ec-83f5-3747809ef661",
   "_type": "InboundHttpComponent",
   "endpoint": "#{self.flow_name}",
   "exchangePattern": "requestReply",
   "matchPrefix": false,
   "note": null,
   "preserveHttpHeaders": false,
   "previousEndpoint": "",
   "previousNode": "",
   "protocol": "https",
   "tenantPart": "id",
   "x": 280,
   "y": 196
  },
  {
   "_id": "04a6ec60-d067-11ec-83f5-3747809ef661",
   "_type": "VelocityComponent",
   "note": null,
   "previousEndpoint": "RightMiddle",
   "previousNode": "75be5f00-d0f8-11ec-83f5-3747809ef661",
   "template": "Message Body:\n\n${bodyAs(String)}",
   "x": 784,
   "y": 294
  },
  {
   "_id": "75be5f00-d0f8-11ec-83f5-3747809ef661",
   "_type": "AggregateCurrentComponent",
   "aggregateFileType": "xml",
   "completionCount": "3",
   "completionCountTimeout": "-1",
   "completionInterval": "",
   "note": null,
   "previousEndpoint": "BottomCenter",
   "previousNode": "797f5ea0-d0f8-11ec-83f5-3747809ef661",
   "x": 602,
   "y": 294
  },
  {
   "_id": "797f5ea0-d0f8-11ec-83f5-3747809ef661",
   "_type": "SplitCurrentComponent",
   "aggregateFileType": "none",
   "exchangePattern": "requestReply",
   "expression": "/names/name",
   "expressionType": "xpath",
   "namespace": "",
   "note": null,
   "nsprefix": "",
   "parallelProcessing": false,
   "previousEndpoint": "RightMiddle",
   "previousNode": "04a6c550-d067-11ec-83f5-3747809ef661",
   "streaming": false,
   "x": 434,
   "y": 196
  }
 ],
 "created_at": "2022-05-10T13:41:07.581Z",
 "environment_variables": [],
 "error_components": [
  {
   "_id": "627a6b7338c74a00130007fa",
   "_type": "FailedExchangeComponent",
   "note": null,
   "previousEndpoint": null,
   "previousNode": null,
   "redeliveryAttempts": 0,
   "redeliveryInterval": 5000,
   "x": 400,
   "y": 300
  }
 ],
 "flow_group_id": "627a4aa838c74a000e0005cc",
 "icon": null,
 "icon_name": null,
 "isFlowComponent": false,
 "lock_user_id": "61a0030f8249dde2022c7616",
 "name": "Aggregate",
 "request_timeout": 20000,
 "tracing_ttl": null,
 "transport": "activemq",
 "trashed": false,
 "updated_at": "2022-07-30T22:56:36.690Z",
 "bundle_id": "ID_627a6b7338c74a00130007f9",
 "lockUserName": "Super Admin",
 "lockCurrentUserCanEdit": null
}

Assimbly

<?xml version="1.0" encoding="UTF-8"?>
<integrations>
   <integration>
      <id>1</id>
      <name>default</name>
      <type>ADAPTER</type>
      <environmentName>Dev1</environmentName>
      <stage>DEVELOPMENT</stage>
      <defaultFromEndpointType>FILE</defaultFromEndpointType>
      <defaultToEndpointType>FILE</defaultToEndpointType>
      <defaultErrorEndpointType>FILE</defaultErrorEndpointType>
      <offloading/>
      <flows>
         <flow>
            <id>2</id>
            <name>FILE2FILE</name>
            <autostart>false</autostart>
            <offloading>false</offloading>
            <maximumRedeliveries>0</maximumRedeliveries>
            <redeliveryDelay>3000</redeliveryDelay>
            <logLevel>OFF</logLevel>
            <endpoint>
               <id>2</id>
               <type>from</type>
               <uri>file://C:\test1</uri>
            </endpoint>
            <endpoint>
               <id>2</id>
               <type>to</type>
               <uri>file://C:\test2</uri>
               <options>
                  <directoryMustExist>true</directoryMustExist>
               </options>
            </endpoint>
            <endpoint>
               <id>2</id>
               <type>error</type>
               <uri>file://C:\test3</uri>
            </endpoint>
         </flow>
      <services/>
      <headers/>
      <environmentVariables/>
   </integration>
</integrations>

Universal language

Based on a comparison between the various formats, I noticed many similarities, but also some specific functionality. Similar were for example that both Kamelets, Dovetail and Assimbly are on a higher-level than routes. They target specific use cases, instead of being technical components with many options.

There were also differences:

Different default formats (XML, JSON or YAML)
Different structure
Different terminology
Added functionality

The different formats show the multiplicity of writing configurations. I decided to gather the various formats and put them in an universal language, called DIL. This gives a common way to speak about integrations. Most of the terminology is to my belief intuitive for common developers. For example, a flow and a step:

I also took some ideas from the various formats:

Assimbly → Levels
Dovetail → Links
CamelK: → Blocks (Templates/Kamelets)

In the next chapter we discuss the concept of levels further by discussion low-level programming and low-coding.

Low-level to Low-code

Programmers like to automate their own work. It’s thus no surprise that low-code platforms, RAD tools, code generators sprung from the minds of developers. I really love those platforms, languages and frameworks that work on a higher-abstraction level. Using components that let me work and think more intuitively and build solutions faster.

I always try to work on a low-code level when possible. But I keep getting back to low-level stuff, because all of these low-code solutions have a couple of downsides. Some of them are:

Low-code software isn’t built with the help of a programming language, but with a platform. And most platforms are proprietary causing closed resources and vendor lockins.
You are mostly limited by the platform and there is often no easy way to define your own blocks.
It’s often unclear or even not possible to scale low-code solutions.
There is the “no technical background needed” myth. One still need to think like a developer and platforms still require developer expertise beyond simple use cases.
No portability. When you create an integration flow in a low-code platform, you cannot use it in another. A transformation written in XSLT on the other hand works on almost every platform and is thus portable.
Lack of flexibility and customization.
Incomplete pipelines (versioning, testing, deploying, security, debugging) for a good lifecycle of a program.

DIL is, because of above reasons not low-code, but like low-code, tries to run on a higher-abstraction level. These higher-levels of DIL can be easily used by low-code platforms.

Low-code and High-code

Developers hate low-code, because they find low-code limited and inflexible, while business developer hate coding because it’s too complex and development is too slow. Low-level means here real programming while low-code something drag-and-drop like.

Is it always low-level vs low-code?

That low-code and low-level don’t get along, is at least what you mostly read in the media. In general however, people don’t care that much. They just want to get to a solution. It’s true that (high-)code and low-code use different approaches to get to a solution, but they are much closer as often being outlined.

Consider for example the low-code development platform Mendix. It actually started as a no-code platform. The original slogan was “No code, just glory”. However, from the start they already used XPath to retrieve values. Also visual models like ERD-diagrams were used (and the code was generated from there). And it uses microflows to ‘program’ the business logic. And it even became possible to call Java from such flow. So the change from no-code to low-code was for sure more accurate.

Modeling and flows in general are actually pretty close to programming. It’s just not textual, but visual programming. On the other hand coders used those models for decades (only the generating tools weren’t that advanced, so they wrote the code themselves). Additionally programmers have always been pushing the boundaries of programming languages to become more high-level.

DIL acknowledges different ways of programming. Not generational, after each other, but on certain levels next to each other.

In the picture below are the various levels of the DIL language. Where the core components are closer to programming in third generation languages and the higher level components are closer to the business and architecture of a specific organization.

People work, based on their role, from low-level to low-code. Thus programmers create, in Java or another JVM language, various core level integration components. This is the place where integration specialists and programmers meet.

DIL’s aim is thus not to choose between high-code and low-code, but to bring them together and let developers go through various levels, just like it using the elevator in the company building. For low-level to low-code!

Final note

In my career, I have filled all four roles in the above building. I sometimes literally need to go up and down in the building to speak with someone with another role. The building however is more to make a point than to take the metaphor too literally. It’s about that based on where you are and what you want to achieve, you be on a certain level. That you need the right tool for the right job. And that there is no silver-bullet. And more of those wisdoms…