13. The IKEA concept
The changing role of the integration platform
Aleksandra had big plans for the new office. I should have a cafeteria with a recreation room. However, the cost of the whole construction is a bit disappointing. She cuts the knot:
“Well, then let’s go to IKEA and put it together ourselves. Then we’ll have a nice Swedish design, and it’s still within budget. Roll up your sleeves!”
Meanwhile, the cost of data integrations is also going up quite nicely. The whole setup requires quite a bit of specialism. “Can’t we just do it the IKEA way too?”, Aleksandra wonders aloud. But what is the IKEA concept? And how might you apply it to data integration?
Economics
You may remember it from economics: Porter’s three strategies. These are the different ways the companies can be competitive in the marketplace. This can be on cost, differentiation (service/quality) or focus.
IKEA’s concept is hard to place under any of these three. The company wants to offer competitive prices, but at the same time it guarantees design, quality and service. IKEA may not have the very highest quality, but with their high turnovers, it simply cannot afford that, for example, all kitchen cabinets fail quickly.
IKEA itself speaks of “offering a wide range of well-designed, functional home furnishing products, at prices so low that as many people as possible can afford them.” But as on all corporate websites are such texts uninformative. You will get more from the following Dutch article “These 7 neuromarketing tricks IKEA uses successfully”:
The article states that at IKEA, it’s not just about cost control and service, but mostly about the psychology of the customer and how to give them a total experience.
Looking at this experience, it appears that everything and really everything has been thought through. From the soft ice cream to the IKEA bag (nice and big) and from the Småland childcare (“Liam wants to be picked up”) to the Swedish meatballs. IKEA keeps honing its concept, because of course customers are not crazy.
Different concepts
One example of the unique aspects of the IKEA concept is the showrooms where entire living rooms or bedrooms are set up. Customers can sit there for a while, open and close kitchen cabinets and rummage among the pots and pans. All without an annoying salesman breathing down their necks. At the end of the tour, they can take the items they want home right away and assemble them at home. And assembling is really easy.
IKEA, of course, is not alone in the market, competing with more traditional concepts. These often align better with Porter’s business strategies. For example, a furniture store that delivers high-quality furniture, designs the furnishings together with the customer, and delivers and installs them at home. Think of Ashley Homestore. They assume a differentiation strategy where customers receive high service and quality.
At the other end of the spectrum are the DIY stores. Hartville Hardware, for example, wants to carry competitive prices and lets the customer build everything themselves, exactly the way they want it. The customer, if he or she has the skills, can customize everything the way he or she envisions and ends up being the cheapest.
So what on earth does this economics lesson on furniture concepts have to do with data integration? Quite a lot, actually. In fact, in this market you also see that traditionally the same Porter strategies are being followed, but new IKEA-like concepts are also emerging.
The traditional integration market
The very first integration products in the 1990s were particularly in the differentiation corner. They were developed to differentiate from application products with their limited integration capabilities. At the center came middleware.
Middleware sales were still very traditional. The vendor would first come to the customer to give a demonstration. If the customer was interested, the next step was to discuss how the customer wanted to use the integration software. Sales engineers then worked out an implementation plan for the setup.
The goal of a new setup was to separate the application layer from the integration layer. This in turn had the goal of running data barrier-free and reliably through the organization. Such integrations run on the integration platform. So the integration layer is a collection of integrations on top of this platform.
In order for an integration platform to be deployed by an organization, software is required, of course. Most integration packages emerged in the late 1990s. They bundled multiple integration functions into one suite.
For example, an integration function can be “transport” through a broker or “integration patterns” through an ESB. When an integration function is implemented in an integration platform, integration capabilities are also referred to at the organizational level.
Until this new software hit the market, most of the links were point-to-point using scripts, adapters or other software. Work was mostly done directly from code. Basically just as many custom applications are still being developed today.
A traditional platform
As more and more applications and integrations arose, a faster and more uniform way to connect was sought. First, ESBs (Enterprise Services Bus) emerged. In the first decade of this century, they became immensely popular. In an ESB, one or more services process the data and together form the integration. This can be a mapping, routing or filtering service.
On an ESB, services are reusable and configurable as much as possible. It also became possible to write your own services in code in most software. Almost all ESBs, such as Tibco, Webmethods and Oracle Fusion, were Java-based. BizTalk (.Net) and InterSystems (C language) are some of the exceptions that do not use Java.
The ESB software was often extended with a broker acting as a transport layer. Later, SOAP services, APIs, Connectors, etc. were also supported by the packages. Thus, these integration suites got more and more modules (capabilities).
A traditional integration platform is based on a so-called integration suite (sometimes supplemented here and there with customization and tools). Such a suite bundles several products. As these integration suites gained more and more capabilities, these suites became increasingly heavy, expensive and complex.
Enterprise organizations have nevertheless seen reasonably successful implementations, but other organizations avoided these packages because of their high cost and complexity.
In summary, software vendors built code-based (usually Java) integration modules that make up an integration suite. Vendor partners and customers turned this into an integration platform. On this, customers built and ran the integrations.
Open source integration software
The complexity, knowledge and cost of traditional integration suites have become an increasing issue over the years. Another disadvantage of integration suites is that software vendors sometimes have difficulty keeping up with technological developments. So perhaps the message broker and ESB capabilities are highly developed, but the API and streaming broker capabilities are not (or just the other way around).
Therefore, many tech giants have not chosen integration suites, but have developed their own tooling and frameworks. For example, the following brokers:
- Kafka (originally developed at LinkedIn)
- Pulsar (originally developed at Yahoo)
- RocketMQ (originally developed at Ali Baba)
- TubeMQ (originally developed at Tencent)
These software tools and frameworks are often more modern and sophisticated than the modules in integration suites.
Many of these initiatives have transitioned to open source projects. For example, all of the brokers mentioned above now fall under the Apache Foundation. This has made these open source tools and framework widely accessible.
Open source foundations such as the Apache Foundation, the Eclipse Foundation and the Linux Foundation have become the hardware stores of the digital world. So the same reservations about do-it-yourself projects apply here. Those who make the choices, put together the integration platform and develop the integrations must really know what they are doing.
Open source integration platform
At the software layer, the integration suite has been replaced by open source integration tools and frameworks.
When done right, this open source software offers many opportunities. Companies are free to choose their own tools and frameworks instead of buying an entire suite from one vendor. If something doesn’t work, they can submit a patch either completely independently or with the open source project.
Several companies such as Red Hat have chosen to bundle several Apache Projects and make them commercially available. For example, the integration software Red Hat Fuse consists of:
- Apache Camel (ESB Framework)
- Apache ActiveMQ (Broker)
- Apache CXF (Web Services)
Red Hat guarantees that the versions work well with each other and offers regular patches. On top of that, they offer documentation, support and training.
Because a different open source tool or framework is used for each integration capability, the integration layer is easier to evaluate. For example, if the broker is obsolete you swap it out for another open source broker, but leave other capabilities intact.
Modern integration teams
Modern integration teams, which traditionally both deliver the integrations and set up the platform, today have to deal with:
- Building the integration platform based on multiple frameworks and tools (rather than one suite).
- The business that wants to integrate applications faster.
The former is primarily low-level, while the latter requires low-code. It is an enormous challenge to bring both closer together. So both to set up a good platform and to deliver many integrations.
Traditionally, integration specialists already had to spend a relatively large amount of time setting up and maintaining the platform. With a lot of open source tools, frameworks and cloud services being used today, this has become even more difficult.
Thus, while the open source projects offer very powerful and flexible software, it is more difficult to turn this into a single platform. In addition, the management tools offered for management are often very limited and highly technical.
So, on the other hand, the business expects that integrations can be delivered faster, and that other teams can also do more themselves on the integration platform. This creates a split for many integration teams who on the one hand want to deliver a stable and reliable platform and on the other hand want to implement new integrations quickly and flexibly.
In short, a modern integration platform is expected to provide the service and quality of an Ashley Homestore, but still have do-it-yourself options.
Hybrid Integration platform
Open source thus provides an excellent basis for a good integration platform. However, it is difficult to create a hybrid integration platform that meets the different integration capabilities.
Therefore, there is not only a movement to bundle open source tools and frameworks, as Red Hat is doing, but also to make the integration platform easier to put together and make integrations on this more easily.
The showroom of this concept is the cloud
Instead of talking to vendors and consultants or putting the software together yourself, you can try out different trials of integration features in the cloud. For example, you can compare different brokers and try which one suits your organization. The idea is that you pay for exactly what you need (pay per use).
Thus, the various integration capabilities are not sold as one suite, but chosen as separate modules. These may differ from each other both technologically and from vendor to vendor. All are directly usable by the end user himself. A concept very similar to that of IKEA!
An example integration platform à la IKEA:
Unfortunately, it must be said that cloud providers are almost as big a labyrinth as IKEA itself. The route and additional costs are serious pitfalls. Like IKEA, the danger is that you walk out of the showroom with many more and very different items than you originally intended. In addition, the hybrid integration platform is not just about the cloud, but rather about connecting the cloud and on premise (see also chapter on Cloud Integration).
An additional layer
Finally, a trend has emerged for developers and other IT professionals to build their own integrations with these tools. AmazonMQ, which is based on the open source broker ActiveMQ, offers not only the advantage of being hosted by Amazon, but also resources for configuration and security of the broker.
In some cases, an entire low-code integration platform is built on top of open source frameworks, usually marketed as iPaaS (Integration Platform as a Service). Examples of iPaaS products include Dell Boomi, Red Hat Fuse Online and Dovetail. The latter two are based on Apache Camel, Apache ActiveMQ and other open source technologies.
These so-called iPaaS focus primarily on ESB functionality, while, for example, Azure Service Bus, Amazon SQS or AmazonMQ focus on broker functionality. For all these cloud services, they are based on open source frameworks and tools.
The new workflow
The new IKEA software allows less-technical employees to put together integrations themselves. Just as you don’t have to work with all kinds of different tools at IKEA, you no longer need all kinds of developer tools. The workflow is very direct.
The old and the new developer workflow:
In the traditional workflow, the developer installed the “Developer Version” of an integration suite on his laptop. This often consisted of an Eclipse IDE with a plugin for developing the integrations for that suite. The integrations were then developed and tested on the laptop. Then the code or configuration was pushed to a version control system (e.g., Git) and built by a build server (e.g., Jenkins). Finally, they were deployed to the integration server.
In the new workflow, developers (called citizen integrators) log in directly through the browser. There they create the integration and roll it out at the touch of a button. A flow similar to the traditional one is often still at work in the background, but it is abstracted from the developers.
The exit
The new way of putting together the integration platform has led to a new concept. An additional layer has been added, providing hosting and management:
A true hybrid integration platform combines these layers so that all capabilities are available and easily accessible.
In summary, open source, cloud and low-code represent a new way for organizations to build their integration layer in an IKEA-like way.
