17. Can artificial intelligence build integrations?
Innovation in data integration tools
The Bakery Group is pretty much suffering from staff shortages. And those who can’t find staff have to be smart. The first innovation is robotization in the warehouse. But couldn’t other business processes be smarter too? Can’t we just click data integrations together with low-code tools or use ChatGPT to solve integration issues?
The Evolution of Integration Tools
Most tools available to develop software are focused on web or mobile applications. Data integrations often do not even have a graphical interface or a database. So these need their own tools.
Initially, integration tools were little different than standard tools for writing software. Soon, integration specialists found that specialized tools added a lot of value to their workflow. In a sense, ESB platforms were among the very first low-code platforms. They offered dedicated tools and flow editors. Examples of ESB platforms include Tibco, Fusion and Sonic. Common tools used by these platforms are:
Flow editors: editors that create a flow with the processing steps of a message:
Mapping editors: mapping between different formats:
API designer: Designing and configuring APIs:
Online data integration
In traditional data integration workflows, the build phase is usually done in Eclipse IDE. In doing so, this build phase is separate from the runtime. You build on your laptop and then deploy it to the servers. More modern platforms like Apache NiFi or Dell Boomi bring building and running integrations closer together. They do this online with models and flows. You create integrations in the browser and run them instantly.
Of course, the integrations themselves do not run in the browser, but somewhere else (usually somewhere in the cloud).
Examples of Low-Code integration tools:
The process of integrating
When you think of tools than you quickly think of building integrations. The process of integrating from idea to production has many more aspects.
By definition, an integration involves integrating two or more systems. You will need to analyze these systems:
- What kind of systems are they, what kind of protocols and data formats do they use?
- What are the differences between the systems and how do we bridge them?
Finally when the integrations are running we need to know what happens, where and why. There is a lot of software out there that helps to analyze and observe the software. What is still missing in many integration tools is an integrated environment that goes through all aspects. Could AI help us here?
AI Chatbots in data integration
Some tools from traditional ESB and the new iPaaS vendors are quite sophisticated. Yet they cannot write integrations themselves. In many ways, they are useful only to integration specialists. You really need to have the knowledge of protocols, data formats and integration patterns. Can we do the same thing without an expert knowledge?
What would an AI data integration solution look like? The first thing that will be different is to start not with a functional and technical design, but with the intent. For example, we have an online shop and a warehouse system. Our intent is to integrate them with each other. We ask the AI, “Create an order integration from the webshop to the warehouse system.”
The AI can also ask questions back. Traditionally, the workflow would be as follows:
Especially in the analysis phase, different parties come together to discuss the specifications, the business analyst makes a design based on these specifications. Based on this design, the developer then creates the integration again. This is tested and put into production by the administrator.
An AI may already be trained based on best practices in data integration and code from existing integrations. Instead of the business analyst, an AI chatbot conducts the conversation. Code can already be generated during the conversation.
No developer is needed. The tests can see if the generated code is good, thus creating a feedback loop between the AI and the results of the tests on the integration platform. Finally, if all possible scenarios get a green light, the software is automatically deployed to production. End of conversation.
A functional bridge
Of course, during the conversation, the AI chatbot must know where these systems are and how to access them. It can request this from the conversational partner. Once there is access, the AI explores the available endpoints. Usually these are REST endpoints with an OpenAPI (Swagger) specification. Then the AI knows what valid endpoints there are on one side.
Next, the AI system must rely on the input language models and parameters. For example, that the field “EmployeeID” in the source system is related to another system. Algorithms map the mapping between these systems. The result is a functional mapping.
This functional bridge can be turned into a code. Suppose we use Apache Camel as an integration framework. There we can feed the AI with code on the Internet and also with our current repository. At the end, we let it test itself until a successful interface is the result.
Beyond tools
Low-code and chatbots are tools that can help us in the process of integration. Can we also imagine a future where these tools are no longer needed? In this case, integration would no longer take place at the level of data, but at the level of systems.
Each system has a module with information about the system. Once the system is running within an application landscape, this module can self-report to an AI system. The AI system can then research what it can best integrate with and can then create the integrations completely autonomously. It is like a company’s brain automatically making connections.
Conclusion
Right now, AI is not yet at the level where it can independently write full data integrations or AI can make connections autonomously. We are still responsible to oversight ourselves, but we can certainly already use low-code tools and AI chatbots to support the process of integration. It is the time to experiment, and we may be surprised how far we can already get.
