Data Integration Language

Building blocks, Kamelets and Route Templates

7 min readSep 23, 2022

This blog contains some ideas on building blocks. Building blocks in DIL are so to say, Camel route templates on steroids. Well, Kamelets are the same. So let’s start with Kamelets.

The structure of a Kamelet:

Comparison of a Kamelet with a Step

To some extent we could say that a whole Kamelet is a step in DIL (thus has nothing to do with steps as defined in a Kamelet) and a template is comparable to a block. The route that use one or more Kamelets can be compared to a flow.

But a route, even a route that calls Kamelets still is relatively bounded to Java. It’s not really clear that it encapsulates other routes through the Kamelets. Besides, templates are really just parameterized routes, not really reusable building blocks for creating steps.

In DIL a route, but also a (route)template, (route)configuration, a message, a connection and so on, are all core components. These core components can be referenced by a block that multiple of them together form a step. The idea is not only that core components can be reused by multiple blocks, but also that the parameters guide what kind is deployed and run.

Blocks

The current route template (which is a subset of the functionality of Kamelets) could be further developed into blocks. I already created a Jira ticket earlier with some ideas to enhance route templates:

RouteTemplates: A flexible builder

I sometimes use a RouteTemplate: https://camel.apache.org/manual/route-template.html Then I create a route as follows…

issues.apache.org

In my opinion, this would bring more powerful functionality to the Camel framework as any other project. In the following list there are a few ideas to move templates to blocks, and Kamelets to steps.

Some ideas to improve templates into bulding blocks:

Kamelets

Kamelets are partly available in the broader Camel ecosystem through route templates and through the Kamelet component. The first are parameterized routes. What a route template misses from a Kamelet are metadata and other specs (types, dependencies, documentation properties).

A route template could be part of a block. Other blocks are for example connections. Multiple related blocks form a step. The metadata and specs defined in a separate block or in other words a step configuration could be used to create a step. Defining this block (step congifuration) could also be in a separate construct (thus optional). A step configuration is thus similar to a route configuration.

2. Libraries/Catalog/Marketplace

When a block is on a Kamelet level (thus has both metadata/spec and the template) then related blocks/Kamelets can be packed as a jar and published to the Kamelet catalog. Here we add third-party libraries to the catalog. Thus Kamelets catalog is more used as marketplace. By just adding the dependency this becomes available in a Camel application.

3. Autodiscovery

Developers now need to explicitly add the route template:

context.addRoutes(new myRouteTemplates());

Thus they cannot be autodiscovered like components (that have the discovery file in META-INF. It would be nice that RouteTemplates can register themselves (especially when you have a lot of them, they now need to be listed). The class that contains the route templates could get for example an annotation like:

@BindToRegistry

or with a new annotation:

@routetemplate

And then it can scan and load the classes. When this available there should also be methods to retrieve (like a catalog) what templates are available and what routes and parameters can be used. Registering is at best runtime independent (thus works not only in Spring for example, but also in Karaf, Jbang etc).

4. Auto update

Currently, when creating (an already defined/added route template) a new route from a template then to load it. If it is already loaded then you need to update it. Thus, you need to check before loading if it already loaded or not. Better and easier is thus give and Camel takes care to load or update the routes.

5. Route parts/snippets

A template must always be a valid route. Thus one cannot take a part of route and use it within another route

.setHeader("myKey","myValue")
.to("log:something")
.to("http:myurl?parameters")

It’s thus not possible to leave the from away and use only part of the route. In Kamelets through kamelet:source and kamelet:sink there seem some possibilities to do this, but currently not in route templates.

6. Parameterize EIP through conditionals

Now only values/parameters within an EIP can be parameterized

routeTemplate("myTemplate")          
            .templateOptionalParameter("name")
            .from("timer:name?period=1000")
                .log("Hello {{name}}");

However what is missing is to add or remove an entire EIP through parameters. For example:

routeTemplate("myTemplate")          
            .templateOptionalParameter("name")
            .from("timer:name?period=1000")
               .add().whenParameter("name").exist()
                .log("Hello {{name}}");
               .endAdd()

Because this is currently possible, developers need to copy the template a lot. It’s not possible to strictly follow the DRY-principle.

7. Add route templates

Currently, routes programmaticaly can be added like this

context.addRouteFromTemplate(templateId, routeId, map);

It would be nice if they can directly be add

context.addRouteFromTemplate(resource);

or (string template as xml / yaml)

context.addRouteFromTemplate(stringtemplate);

There is a way to use the routeLoader:

extendedCamelContext = context.adapt(ExtendedCamelContext.class); 
loader = extendedCamelContext.getRoutesLoader();
loader.loadRoutes(resource);

This is however not well known.

8. Documentation

I could be better documented how to load / unload and run routes programmatically. In the documentation, it only focuses on defining the template and then run the route from the template.

9. Support JSON

XML is traditionally the format to create routes. Today we see YAML as configuration format becoming popular. However, a lot of webapplications use JSON. Thus to work with webapplications it make sense to support JSON as a format.

10. Route Loader

It’s possible to load routes with routeLoader like this:

extendedCamelContext = context.adapt(ExtendedCamelContext.class); 
loader = extendedCamelContext.getRoutesLoader();
loader.loadRoutes(resource);

However it’s not possible through the ‘updatesRoutes’ method

extendedCamelContext = context.adapt(ExtendedCamelContext.class); 
loader = extendedCamelContext.getRoutesLoader();
loader.updatesRoutes(resource);

With regular routes this is no problem when the route already exists, but with routeTemplates and routeConfigurations this is not possible.

11. Route Loader: Error feedback

When loading the routetemplate with routeLoader like this:

extendedCamelContext = context.adapt(ExtendedCamelContext.class); 
loader = extendedCamelContext.getRoutesLoader();
loader.loadRoutes(resource);

Then it doesn’t always give a warn/error when a required parameter is missing.

12. Load single routeConfiguration / routeTemplate as XML

It’s possible to load routes in XML format likes:

<routes xmlns="http://camel.apache.org/schema/spring">
 <route>
  <from uri="timer:xml?period=5s"/>
  <log message="I am XML"/>
 </route>
</routes>

However it’s also possible to load it as a singel route

<route>
  <from uri="timer:xml?period=5s"/>
  <log message="I am XML"/>
 </route>

See for example the routeloader example.

This is however not possible for routeConfigurations or routeTemplates. Thus you cannot do this:

<templatedRoute routeTemplateRef="generic-action" routeId="100-2">
  <parameter name="in" value="direct:2"/><parameter name="out" value="direct:3"/>
  <parameter name="uri" value="log:MyLog"/>
 </templatedRoute>

But must do this:

<templatedRoutes xmlns="http://camel.apache.org/schema/spring">
 <templatedRoute routeTemplateRef="generic-action" routeId="100-2">
  <parameter name="in" value="direct:2"/><parameter name="out" value="direct:3"/>
  <parameter name="uri" value="log:MyLog"/>
 </templatedRoute>
</templatedRoutes>

12. Custom RouteID

RouteID must be set on the route level:

<templatedRoutes xmlns="http://camel.apache.org/schema/spring">
 <templatedRoute routeTemplateRef="generic-action" routeId="100-2">
  <parameter name="in" value="direct:2"/><parameter name="out" value="direct:3"/>
  <parameter name="uri" value="log:MyLog"/>
 </templatedRoute>
</templatedRoutes>

It’s not possible to set it manually in the route itself and then call it as parameter

<templatedRoutes xmlns="http://camel.apache.org/schema/spring">
 <templatedRoute routeTemplateRef="generic-action" >
  <parameter name="in" value="direct:2"/><parameter name="out" value="direct:3"/>
  <parameter name="uri" value="log:MyLog"/>
  <parameter name="routeId" value="100-2"/></templatedRoute>
</templatedRoutes>

13. getTemplates?

There is a method to addRouteFromTemplate as follows

context.addRouteFromTemplate("templateid","routeid",map);

and

context.removeRouteTemplates(pattern);

However there is no

context.getRouteTemplates();

context.getTemplateNames();

14. autostartup

By default the routes startup. There is no way to specifically tell a route start or not.

15. ExchangePattern

In Kamelets one can set the ExchangePattern (InOnly, InOut, InOptionalOut). This is not a setting on a routetemplate.

16. Loops

Currently it’s possible to set a specific value:

"routeTemplate("myTemplate")          
            .templateOptionalParameter("name")
            .from("timer:name?period=1000")
                .setHeader("name","{{name}}")
                .log("Hello ${header.name}");"

However what if the name of the header is set by the user

routeTemplate("myTemplate")          
            .templateOptionalParameter("headername")
            .templateOptionalParameter("name")
            .from("timer:name?period=1000")
                .setHeader("{{headername}}","{{name}}")
                .log("Hello ${header.headername}");

One specified value can be set, but if you want more you need to call the template several times. It’s not possible to have for example a map as entry (a map of maps) and then loop over the so that multiple values can be set:

routeTemplate("myTemplate")          
            .templateOptionalParameter("headers")
            .templateOptionalParameter("name")
            .from("timer:name?period=1000")
                .forEach(Map header: header)
                      .setHeader("{{header.name}}","{{header.value}}")
                .endForEach(
                .log("Hello ${header.headername}");

In the above not a new template needs to be initiated, but can be done from one template. Now only workaround is to create specific processors/components for functionality that is actually already in the Camel EIP.

17. Global parameters

Global parameters are parameters that are set on creation of a route from a template. They are not parameters of a specific template, but they can be set ‘globally’ on any template. Currently, there is only one global parameter:

routeid

Unfortunately, you can not specify the global parameters as a normal parameter in the route (see point 10). There could be other global parameters that were practical like:

RouteConfigurationID (ID of routeConfiguration)
ErrorHandler
Autostartup
ExchangePattern
MessageHistory
No… (Delayer,MessageHistory,Tracing, Autostartup, StreamCaching)
Description
RouteDescription
Threads

One would normally set these directly after the “from” endpoint. Here you set them directly on route creation from a template.

RouteID is used as (global) path parameter. It would make more sense when RouteID and other global parameters would used as Query Parameters.

18. Class types

Currently it’s only possible to use String as template parameter. Sometimes it would be useful to have other types. For example:

routeTemplate("myTemplate")          
            .templateOptionalParameter("headername")
            .templateParameter("path")
            .templateParameter("pattern",Exchange.class)            
            .templateParameter("timeout",Long.class)
            .from("timer:name?period=1000")
            .setExchangePattern("{{myExchangePattern}}")                    
            .pollEnrich().simple("{{path}}").timeout("{{myTimeout}}");

The alternative is that every configuration in the Java DSL can be set as String. See also:

https://lists.apache.org/list?users@camel.apache.org:2022-9

19. Call as template as a component

It’s already possible to call a Kamelet as component. Templates still require “templatedRoute” to create one. Would be better to call it as a normal component:

template:mytemplate?parameter1=value1&parameter2=value2

In this way one can create base templates that can be called other templates or routes. Together nesting will be possible.

20. EIP as a to statement

Because route template one can write most options as strings, it will be possible to write a lot of EIP’s as “to” endpoints (or when using them as a DIL step with links, everything can be a “to” endpoint). For example:

routeTemplate("xmltojson-action")
.templateParameter("in")
.templateParameter("delayTimeout","5000")
.templateParameter("out")
.from("{{in}}")
   .to("delay:{delayTimeout}}")?asyncDelayed=true&callerRunsWhenRejected=false)
   .to("{{out}}");

The broader ecosystem

The previous list is just to share a lot of ideas on creating building blocks (which are basically enhanced Kamelets). The number of ideas just show how powerful the Kamelet concept is (however building blocks and steps is much easier to understand in my opinion). It has lots of potential when it’s developed and made available to the broader ecosystem. These have so much potential and power that it can cumulate in a Camel not as framework, but as a (general-purpose) programming language on its own.