Data Integration Language

Part 2: Syntax and examples

Syntax

A component or a link are defined by the following attributes:

• Type
• Function
• Parameters

The syntax of the attributes of a component is:

type:function?parameters

or in XML:

<component>
     <type></type>
     <function></function>
     <parameters>
        <parameter></parameter>
     <parameters> 
</component>

Where

1. Type defines the pattern of the component and thereby the number of possible links.
2. Function: A self-contained module that accomplishes a specific task (written in the form of a URI).
3. Parameter: A collection of key/value pairs, that are the input parameters for a function.

It doesn’t matter on which level a component or link is, the syntax can be applied to each level.

Syntax examples

The syntax of DIL can be represented in text or visually. Textual represented in a tree-like fashion (like in XML or in a Fluent API) and visually in a flow graph.

Say we have as a source an endpoint that gets XML messages on an HTTP Endpoint, then it splits the message, replace some text, aggregates it and at the end send it to a database.

XML:

<flow>
 <name>sample</name>
 <type>inbound</type>
 <options>
  <transport>synchronous</transport>
 </options>
 <steps>
  <step>
   <type>source</type>
   <uri>https://sample</uri>
   <options>
    <preserveHttpHeader>true</preserveHttpHeader>
   </options>
  </step>
  <step>
   <type>action</type>
   <uri>split:xpath</uri>
   <options>
    <expression>/names/name</expression>
   </options>
  </step>
  <step>
   <type>action</type>
   <uri>replace:myregex</uri>
  </step>
  <step>
   <type>action</type>
   <uri>aggregate:xml</uri>
  </step>
  <step>
   <type>sink</type>
   <uri>sql:myquery</uri>
   <options>
    <database>mysql</database>
    <username>example</username>
    <host>xx</host>
   </options>
  </step>
 </steps>
</flow>

Fluent

flow
  .inbound(”sample?transport=synchronous")
    .source(“https:sample?preserveHttpHeader=true”)
    .action(“split:xpath?expression=/names/name&out=split”)
    .action(“replace:myregex?in=split”)
    .action(“aggregate:xml?completionCount=3”)
    .sink(“sql:query?database=mysql&username=example&host=xx”)

Visually:

In the visual diagram, clicking on the step, the parameters are shown. All the above are just examples, in the next chapters we will provide real implemented examples.

Component Libraries

Components that are implemented can together form a library. Libraries consist of related components. A collection of libraries can form a catalog that can be used by third-parties.

The Transpiler

In DIL, we convert our code from DIL code to Camel code. It’s a bit like Typescript that produces JavaScript as output. This is also called source-to-source compiling, but a clearer term is transpiler.

The basic process of transpilation in DIL is very simple:

Like most compilers, the process of getting to the result code is not a one step-conversion. This is because there are multiple layers of abstraction that needs to be translated.

We have a source file in DIL format and give it to the transpiler and the transpiler produces the camel code. Transpiling goes in the following steps:

From a user perspective, we can now define the whole process:

1. Source code

The user first defines an integration:

a. Visually from a user interface like Dovetail, Assimbly Gateway or maybe in the future with other tools like Karavan or Kaoto.

or

b. Textually as JSON, XML or YAML in a text editor like VIM, Notepad++ or Visual Studio Code.

2. The source code gets transpiled.

3. The target Camel code get loaded and run in Assimbly runtime.

Hello world!

In most general-purpose programming languages, one starts with “Hello world!”. In domain-specific languages like XSLT and SQL, this doesn’t make a lot of sense. The same counts to some extent for DIL. The language is to process and transport data, not to print text. However, it’s perfectly possible to print a message in DIL.

As an example, we use the same “basic” example as Camel does. Here the data is triggered by a timer which runs by default every second. After the timer, we place a print step that prints “Hello world!”.

Example:

<flow>
   <name>HelloWorld</name>
   <steps>
      <step>
         <type>source</type>
         <uri>timer:foo</uri>
       </step>
      <step>
         <type>sink</type>
         <uri>print:Hello World!</uri>
      </step>
   </steps>
</flow>

The output:

2022-09-02 15:54:32.675  INFO 1848 --- [2 - timer://foo] 1-2                                      : Hello World

Let us short analyze this program:

1. The program is named “Hello World”.
2. It contains two levels: flow and step.
3. There is one source step triggers the program and the sink step (the last step) prints “Hello World!”.

The example is quite minimal and implicit. In some cases, it’s better to be very explicit about what the transpiler needs to do. For example, when we want to have more control, or when we call the runtime API. The last is most likely the case when using low-code tools to generate the DIL code.

The same example, but now explicit:

<dil>
   <integrations>
      <integration>
         <id>1</id>
         <name>default</name>
         <options>
            <environmentName>PRODUCTION</environmentName>
            <stage>PRODUCTION</stage>
         </options>
         <flows>
            <flow>
               <id>12345</id>
               <name>HelloWorld2</name>
               <steps>
                  <step>
                     <id>1</id>
                     <type>source</type>
                     <uri>component:timer:foo</uri>
                     <links>
                       <link>
                         <id>link1to2</id>
                         <type>sync</type>                                   
                        <bound>out</bound>
                       </link>
                     </links>
                  </step>
                  <step>
                     <id>2</id>
                     <type>sink</type>
                     <uri>block:print:Hello World!</uri>
                     <links>
                       <link>
                         <id>link1to2</id>
                         <type>sync</type>                                   
                        <bound>in</bound>
                       </link>
                     </links>
                  </step>
               </steps>
            </flow>
         </flows>
      </integration>
   </integrations>
</dil>

Though this gives exactly the same output as the first example, it explicitly defines:

1. Levels: Make the flow part of integration. An integration can have one or multiple (linked) flows.
2. ID’s: Many databases use ID’s to store objects. It’s with an ID easier to manage an integration.
3. Links: The links define the flow of control. If no links are defined, it just processes the steps from first to last. If they are defined, the order of steps doesn’t matter.

Try it yourself!

The “hello-world” example and other DIL configuration can be tried out with Assimbly Gateway. An open source integration tool.

Download & Run

• Download 3.8.0 beta
• Run

java -jar gateway-[version].jar

There are three ways to try out the examples:

Folder

1. Create a file in a text editor
2. Save the file in the deploy folder:

/{user.home}/.assimbly/deploy

The file (and changes in the file) are automatically detected. Delete the file to stop the flow.

GUI

1. Login to the gateway (admin/admin by default):

http://localhost:8080

2. On the main page, go to:

Actions → Create flow

3. Create and save a flow

4. On the main page, run it.

API

1. After the gateway started, go to the following URL:

http://localhost:8088/#/admin/docs

2. On the (Swagger) API documentation page use the following endpoint:

/api/integration/{integrationId}/flow/test/{flowId}

Note that integrationid and flowId should match the IDs in the DIL configuration. As ‘integrationid’ you can set 1 as default.

Conclusion

I hope the language reference and examples gave an idea what DIL is about. The basic idea behind DIL was to create something that is both powerful and easy. To achieve this, a lot of layers were added.

Some people say with every layer of abstraction, an onion is created. When you are on an outer layer, it’s not easy to understand the inner layers. They say when you cut the layers of computing open, you really have to cry.

Hopefully DIL didn’t make you cry too much, but let developers explore every layer easily. A new integration recipe.

More reading:

Data Integration Language

More examples in DIL format:

camel-examples/dil at main · assimbly/camel-examples