The IKEA concept
Part 2: Architectural principles of the Assimbly project
“The process of technological development is like building a cathedral. Over the course of several hundred years new people come along and each lays down a block on top of the old foundations, each saying, ‘I build a cathedral’. Next month another block is placed atop the previous one. Then comes along an historian who asks, ‘Well, who built the cathedral’ Peter added some stones here, and Paul added a few more. If your are not careful, you can con yourself into believing you did the most important part. But the reality is that each contribution has to follow onto previous work. Everything is tied to everything else.”
Paul Baran, one of the founders of the internet.
Since the beginning of programming, whether programming mainframes, database or applications, there’s been a lack of good software engineers. As these engineers can’t clone themselves, they are looking for ways to automate themselves.
Software is not made of stone and can be optimized and copied easily. But engineers often think the tools they are using are cumbersome. They also think that it takes a lot of time to create solutions. So, they create their own tools and try to automate their work.
Nevertheless, working more and more on software tools, brings them further from creating solutions for real-world problems. Often, managers that have deadlines and business targets don’t like to see this happen.
Traditionally, a separation was created between software vendors and the IT department at the customer site. The customer uses the vendor software for their specific business purposes. The idea is, the software vendor focus on the building materials, the IT department on building solutions with these materials.
Software factories
The software vendors try to make their software very generic, so they can be sold to many customers as possible. Often business software, especially backend software like data integration platforms, needs many adjustments to make the software really productive for an organization.
The customers thus want to build specific products, but the software vendor mostly only provide very basic building blocks. Again, the software engineers at the IT-department try building the products faster. With the basic blocks, they set up a (software) factory. The input of this factory are business problems, the output business solutions.
For integration, the business often wants to integrate their applications to support their business processes. The input is a specific business problem, while the output are integration solutions.
Integrations however don’t spin in the void. You need to build integrations, but also a platform where they are running on. So there are actually two factories:
The business wants integrations as much as possible, but often a lot of time is spent on setting up the platform and the integration factories.
Who is building these factories?
Any factory before it produces products needs to be set up first. For a car factory, for example, there must be a place where the factory will be located. And the factory needs a foundation, a building, machines, offices, storage space, logistics and so on. But who is building integration factories? Is it the software vendor or the IT department? The answer is neither.
Often the software vendors don’t have enough resources to actually set up the factories. The customers also don’t have the time or knowledge. Therefore, vendors often work with local partners like consultancy firms and implementation partners. These companies need to bridge the gap between the software vendors and the business.
Translating this to integration software:
But once again, this approach takes a lot of software engineers. There are software engineers at different places in the delivery process. Whether these are the vendors themselves, IT partners or IT departments.
This full-service approach is costly, needs a lot of specialists and takes a lot of time, before the factories finally stand (and often the factories are rebuilt and renovated multiple times to catch up with the latest demand).
Do-it-yourself
The full-service approach is like a furniture store, where people can watch all kind of home furnishings, make plans with architects and interior designers and let contractors build the designs. But again, like good architects, designers and contractors, the IT architects, programmers and integration specialists are sparse and expensive.
While styling their home, people that don’t want to spend their time managing all the parties (or just don’t have enough money) go to a do-it-yourself hardware store. There they have totally freedom to create any solution themselves with the most basic building blocks and tools.
In software, IT departments or startups, can use similar basic building blocks with the help of open-source software. But instead of going to Ace Hardware or Hornbach, they go to the Apache or the Eclipse foundation. There they download whatever framework or open source tool they need.
With these tools and software, they can build the factory themselves. But often these open source software take a lot of knowledge and time to set up. And often they don’t directly do what their organization wants.
So once again, these software engineers start improving the open source tools and frameworks. This has one major advantage over the closed-source approach of the software vendors. Multiple organization can work together on the tools and frameworks. They can profit from each other work, add their specific wishes and assure that their interest are looked after.
But again and again (and this seems somewhat of a circle) the organization needs good software engineers and often needs to improve low-level frameworks and tools to get real benefits. This diverts from their purpose of building solutions. And again the managers get nervous.
The hard road
The solution is once again software. Only not the software that makes the building and the machines in the software factory, but software that creates software pipelines itself.
There are many software frameworks and tools developed last decade that contribute to set up a factory. There is GIT for version control, build tools (like Gradle), CI/CD for automating build, release and deployment (like Jenkins). And of course there is the whole world of containerization, with technologies like Docker, Kubernetes and Istio.
With the help of all these open source frameworks, libraries and tools one could set up the best factories that are reliable and scalable.
But once again the problem is that these tools are very complex, and it takes good engineers to bring all the parts together. Do-it-you-self is a hard road.
iPaas
A solution that offer both service and flexibility is to be found in Software-as-a-Service (SaaS). Several vendors created software or use open software projects and build the factory (as-a-service) in the cloud. Often accompanied by user interfaces and low-code modeling tools.
These SaaS services save customers a lot of time in setting up the factory. Several traditional software companies like SAP, and also in the space of data integration, like Tibco, Dell and Mule offer an all-in-one cloud services platform. In integration, this is known as integration Platform as-a service or in the short form as iPaas.
But this approach is much a continuation of the full service approach that lacks some attributes of the do-it-yourself/open source approach. It isn’t ‘the right-tool for the right job’ approach. It isn’t an open approach. And often iPaas falls technologically behind various open source tools with big communities and big tech supporters. Also, they aren’t flexible and modular enough.
The IKEA concept: Cloud stacks
Cloud vendors, like Amazon and Microsoft, saw an opportunity here. Instead of offering an all-in-one solution, they offer (mostly based on open source software) various small factories that all focus on one functionality. In data integration this can be data ingesting, data transport or data transformation.
This approach is much like IKEA. At IKEA, customers can walk around and try things in the showroom, they can buy several products and build it easily at home (without difficult tools and without prior knowledge). The cloud vendors, instead of offering furniture, offer cloud services. The customer, like IKEA, only need to have a basic knowledge to build the solution they need.
Each cloud service is like a small factory that is already set up. And like factories together form an industrial park, various cloud services together form a stack. Microsoft and AWS both have an integration stack.
Below is a table of integration stacks with matching products/services. The last column contains open source software from Apache that has the same functionality:
These cloud integration stacks will probably become more popular in the coming years.
Hybrid multi-cloud
The big issue with these stacks is that they enlarge the vendor-lockin like we have never seen before. The open source software movement got us out of the dependency on software vendors. Cloud-vendors caught us back in.
We need a new movement that creates cloud-vendor independent ways of creating cloud stacks. And also to keep the possibility to try and use the software on premise. This approach is called hybrid multi-cloud.
To compete with the cloud integration stacks of Amazon and Microsoft, other integration tools and cloud services need to be user-friendly and have a low-code and modular approach. One of these places where you can find such software is OperatorHub.io which also has a “Integration & Delivery” stack:
In this stack you can for example find a WS02 as API manager, but also CamelK for integration functions and AtlasMap for transformations.
OperatorHub incorporates the IKEA-Style of creating software. It’s however compared to the stacks of Amazon and Microsoft still relatively low-level, incomplete and technological orientated.
The Assimbly project tries to contribute with creating modules that have easier interfaces and target at specific functions. Read more in the next part.
Or start at the beginning:
