Camel variables: new ways to manipulate data pipelines
In programming, variables are where you store and manipulate information in your program. In Apache Camel, this works a little differently. There you have a message exchange (the information) and routes (the program). A route defines the flow of data, and the message exchange is the unit of data.
The Exchange
A message exchange goes through the route (see image above). It lives in memory and goes through each step of a route. Within an exchange, there are several places where you can store data:
The payload is typically stored in the body. Think of XML or JSON, but it could also be a string, binary, list, or unstructured text.
Metadata in an exchange is stored in headers or properties. Both are key-value pairs. The difference is that headers are part of the message and properties are not. This means that when data is sent to an external system, only the message (headers and body) is sent. For example, when sending data to an HTTP endpoint, the message headers are automatically converted to HTTP headers. Properties remain on the exchange and are only used internally.
Message-Oriented vs. Object-orientated
The fact that properties stay on the exchange is a handy way to store temporary data (such as holding the payload during data enrichment). It still means that this data stays on the exchange, so sometimes it can be good to remove properties or headers for a lighter exchange.
The way Camel handles data is called message-oriented middleware. Camel’s message-oriented paradigm is different from what most programmers are used to, such as an object-oriented paradigm where everything lives in objects with methods and variables.
From this second perspective, the following questions are often asked when using Apache Camel:
- Can I store data independently of the exchange?
- Can I store data independently of the route?
- Can I store data independently of the Camel context?
Until now, you had to store values either in properties or in an external system like a database. You have to be careful to remove the data from the exchange or database when it’s no longer needed. So the answer to most questions is yes, but using routes and exchanges aren’t always ideal for storing and manipulating data values.
Camel variables
In version 4.4.0, Camel introduces variables, which are closer to what programmers are used to. A Camel variable is conceptually somewhere between an exchange property and a Java variable. It’s a key/value pair for storing and manipulating values.
Besides the fact that there is no ceremony to prepare the message body and headers and clean up afterward, variables have some characteristics that make them useful and powerful within a Camel program:
1. Scopes
The scope of a variable is the area in which a variable is accessible. You cannot use a variable beyond its scope. By default, Camel variables are private, this means you can use the variable only within an exchange.
Private variable (exchange scope):
from("timer:foo")
.setVariable("myVar", constant("world"))
.log("Hello ${variable.myVar}!");You, can however also use a private variable in another route. Even when the routes are not connected.
Private variable (route scope):
from("scheduler:foo?repeatCount=1").routeId("first")
.setVariable("route:second:myVar").constant("world");
from("scheduler:foo2?repeatCount=1&initialDelay=3000").routeId("second")
.setBody().variable("route:second:myVar")
.log("Hello ${body}!");The last possible scope is to use a global variable in any route throughout the Camel context.
Global variable (context scope):
from("scheduler:foo?repeatCount=1").routeId("first")
.setVariable("global:myVar").constant("world");
from("scheduler:foo2?repeatCount=1&initialDelay=3000").routeId("second")
.setBody().variable("global:myVar")
.log("Hello ${body}!");
Note that in the examples of the route and global scope, no transport endpoints like direct or seda are used to connect the two routes. Still, we were able to pass values easily. How cool is that!
2. Patterns
Variables in Camel aren’t just getters and setters, but much more. They come with the full power of the simple expression language and enterprise integration patterns.
Get a variable in simple language:
${variable.foo}Use variable in a choice:
from("direct:a")
.choice()
.when(simple("${variable.myVar} == 'hello'"))
.log("Hello ${variable.myVar}!");
.otherwise()
.log("Goodbey ${variable.myVar}!");Use variableReceive on incoming message
from("scheduler:foo?repeatCount=1").routeId("first")
.setBody().constant("world")
.to("direct:start");
fromV("direct:start", "myVar").routeId("second")
.log("This body is empty: ${body}")
.setBody().constant("Hello")
.log("${body} ${variable.myVar}!");Notice that in the second route, we use fromV instead of a regular from. This stores the incoming data directly in a variable, while we can use the route body after it for other data.
Stores
By default, both private and global variables are held in memory by the Camel context. However, using “variableSend” and “variableReceive” it is possible to store the variable values completely outside the Camel context, such as in an external server or database. This makes it possible to
- Use and share the data with other Camel Contexts.
- Use and share the data with external systems.
- Use less memory, especially for very big data.
Here is an example that gets data from:
http://ip.jsontest.com/
This endpoint returns your IP address, that is set as the variable myVar. After each step, there is a log to show what the variables do with the message.
<route id="sendAndReceiveVariables">
<from uri="scheduler:foo?repeatCount=1"/>
<setBody>
<constant>1</constant>
</setBody>
<log message="1. Body=${body}"/>
<enrich variableReceive="myVar">
<constant>http://ip.jsontest.com/</constant>
</enrich>
<log message="2. Body=${body}"/>
<setBody>
<constant>2</constant>
</setBody>
<log message="3. Body=${body}"/>
<to uri="stream:out" variableSend="myVar"/>
<log message="4. Body=${body}"/>
</route>Log output:
1. Body=1
2. Body=1
3. Body=2
{"ip": "8.8.8.8"}
4. Body={"ip": "8.8.8.8"}In this example an HTTP endpoint is used, but send and receive variable also do fine with stores like Redis, PostgreSQL or MongoDB.
Conclusion
Camel variables make it possible to use Camel beyond the message-orientated paradigm. This make Apache Camel more powerful and flexible for your design choices when messages flow through your data pipelines.
More about variables:
XML DSL Examples
Most examples were given for the Java DSL, here are the same examples for the XML DSL.
Private variable (exchange scope)
<route>
<from uri="timer:foo"/>
<setVariable name="myVar">
<constant>world</constant>
</setVariable>
<log message="Hello ${variable.myVar}!"/>
</route>Private variable (route scope)
<routes id="camel" xmlns="http://camel.apache.org/schema/spring">
<route id="first">
<from uri="scheduler:foo?repeatCount=1"/>
<setVariable name="route:second:myVar">
<constant>world</constant>
</setVariable>
</route>
<route id="second">
<from uri="scheduler:foo2?repeatCount=1&initialDelay=3000"/>
<setBody>
<variable>route:second:myVar</variable>
</setBody>
<log message="Hello ${body}!"/>
</route>
</routes>Global variable (context scope):
<routes id="camel" xmlns="http://camel.apache.org/schema/spring">
<route id="first">
<from uri="scheduler:foo?repeatCount=1"/>
<setVariable name="global:myVar">
<constant>world</constant>
</setVariable>
</route>
<route id="second">
<from uri="scheduler:foo2?repeatCount=1&initialDelay=3000"/>
<setBody>
<variable>global:myVar</variable>
</setBody>
<log message="Hello ${body}!"/>
</route>
</routes>Receive variable
<routes id="camel" xmlns="http://camel.apache.org/schema/spring">
<route id="first">
<from uri="scheduler:foo?repeatCount=1"/>
<setBody>
<constant>world</constant>
</setBody>
<to uri="direct:start"/>
</route>
<route id="second">
<from uri="scheduler:foo2?repeatCount=1&initialDelay=3000"/>
<setBody>
<variable>global:myVar</variable>
</setBody>
<log message="Hello ${body}!"/>
</route>
</routes>