11. Lessons learned!

What do you run into in integration practice?

“In a few months you will no longer need an integration specialist,” says Aleksandra’s IT manager. By then, all the data will be “tied together” with all kinds of integrations. All flows are automated and they use all kinds of EDI standards. Over time, the manager asks if he can’t hire another integration specialist after all?

No work for integration specialists? If only that was true! The opposite is more likely, with each newly introduced protocol, standard or technology, integration specialists get a little busier. Even if the solution is so advanced, the reality is usually a bit more unruly. So despite all the standardization, protocols, data formats and tools, many integrations remain a challenge.

Many of these challenges have been chronicled by Igor Tkac, director of IGT Consulting, in the #IntegrationStories series some of his observations.

The Bermuda Triangle

Where does a lot of complexity come from? In integration, to begin with, you are always dealing with at least two systems. Often not only do they differ in technology, but they were created by different people and are managed by different departments. On top of that, an organization wants to support its own business processes.

This leads to a kind of Bermuda Triangle:

Each corner has its own purposes and interests. Data can just disappear between them.

Mainly alignment

Suppose system A is an HR application and system B is an app for employee recruitment. The first system’s function is proper personnel administration. System B in turn takes care of everything around the “recruitment & selection” function.

Thus, the organization uses both systems for different purposes. For example, someone recruited by the organization (System B) should automatically enter the HRM system (System A). Practice shows that bridging the differences between these two systems requires a lot of deliberation. After all, a candidate is not the same as an employee.

It is jokingly held that the following is true:

90% is communication, the rest is analyzing, building, testing and deploying integrations. Now this is not science, but the feeling many who are working with integrations have. Where does that feeling come from?

Let us continue our example:

“Okay everybody, there is going to be a project that is going to replace the Finance system. The current system is 26 years old and no longer meets modern needs. We need to move to the cloud.”

Managers are enthusiastic about the business benefits, architects are ambitious about technical innovation. It is often underestimated that a lot has happened in those 26 years. Customized components, scripts and tools have been built around the integrations. All kinds of processes are in place and years of fine-tuning are behind us.

It also underestimates how much integration work is actually needed, if it is already budgeted very specifically. All in all, there is only a vague idea of what management wants in the project, but they already need an estimate from the project for the integration part.

The integration team:

➢How many interfaces are needed?
Project: “That’s not clear yet.”

➢ How many systems need to be integrated?
Project: “That’s not certain yet, but probably between 3 and 8.”

➢ What architecture do we want to use?
Project: “We are still discussing that. We are still working out the vision”

… and so on.

In addition, the architecture ambition (Event-Driven, API-Driven, Cloud-First, Microservices etc.) brings in many new elements that are sometimes disproportionate to the project.

For example, the idea is to move to a microservices architecture. After two weeks, the engineers figure out that you can do 90 percent of the integrations in this new way. But what about that other 10 percent?

The team also finds out that the new technology still has many teething problems and you need extra time to “program around” it. In addition, the new microservices architecture offers much more complexity.

Slowly, the development team falls into the pitfalls of distributed systems. And not only does the new technology bring additional complexity, there is often plenty of legacy left behind with which to integrate. Is there enough visibility into the legacy that shines in the reflection of the new applications?

Within a new architecture running in the cloud, for example, integration is often relatively easy. Legacy systems are also easily connected on premise. However, connecting both, cloud services with on premise applications, is hell.

Lessons

1. Lesson learned: a balance between old and new

If you want to move forward then there are risks. The more new elements, the more risk you will have to factor in. Agile work presupposes finding a good mix between proven technology and innovation. In doing so, you need both experienced people who point out the obstacles on the road and innovators who embrace the opportunities of new trends.

2. Lesson learned: an interface between old and new

It may not yet be entirely clear exactly how many links there will be, but it is useful to create a single interface to which both old and new applications can be connected. Often this is a broker or a gateway from which the cloud and on premise can be connected. Both sides will need network-technical access to this.

The degree of complexity

Okay, the project is starting to get off to a good start. There is now a clear picture of the number of integrations. The enterprise architect has created a architecture overview with each connection between middleware and applications forming one line. The solution architects elaborate on this based on message types and protocols.

Then the analysts come into action, trying to represent the different steps of each “line” in a diagram. Meanwhile, the integration has already become quite a long chain. Finally, the developers get to work, it turns out there are still a skeletons in the closet and some technical bumps to overcome.

The original picture:

Final integration:

The man is a message from A to B

But this is only about the integration steps involved. In modern integrations, there are very different challenges, namely:

1. infrastructure
2. security
3. networking

An integration specialist nowadays seems to spend more and more time on the prerequisites of integration. Secure and stable connections are indeed necessary, but it does not actually add business value from the integration perspective.

The process to request for access, network configuration, firewall ports, certificates, etc. takes a lot of the integration team’s time. IT is quite a broad concept and many users (sometimes even programmers) do not realize two important facts:

• Through how many (physical and virtual) layers their data must pass, from application A to reaching application B
• In how many places “something can go wrong” during this trip.

The picture below is already simplified:

So with middleware, there is a lot of work not only at the application layer (where the value is delivered), but also at the system layers below it. In addition, once the integrations are running, they are very dependent on these layers.

Igor Tkac notes:

One thing that makes me uneasy is how much the integration team depends on a stable infrastructure.

Maybe I’m just a naive “consumer of AWS/Azure ads,” but I really like an idea to focus on our work without fears of:

➢ Full disk space
➢ Blackouts due to ‘unscheduled maintenance’
➢ ‘Transparent’ backend upgrades (where we lose access rights to our platform)
➢ ‘Aha moments’ such as wanting your cluster on 2 different machines (because it was all added on one machine)
➢ You need a new DB? — give us 3 weeks

I understand that “owning” infrastructure certainly has its advantages:

➢ less chance of “spying”
➢ legal reasons
➢ money has already been invested in “our” data centers — so why spend it again?

Companies that provide integration tools and platforms are now releasing new versions that are more “portable” — so I guess I am not the only one with this need.

Lessons

Lesson learned 3: an eye for infrastructure

Implementation specifically requires a strategy for the infrastructure and network layers. How can issues be tackled quickly? Is it often necessary to have dedicated resources from both the integration team and infrastructure in a large project? Now to get management to do that….

Lesson learned 4: simplicity

KISS (Keep-It-Simple-Stupid) will have to be one of the guiding principles of the project. You achieve this by standardizing regularly, by refactoring and sometimes by saying “no” at the integration layer. Leonardo da Vinci already knew, “simplicity is the ultimate form of refinement.”

The team

The budget is finally agreed upon, it’s time to assemble the integration team. But how do you do this? It starts with a mix of roles (coordinating roles, technical roles and testing/management roles) and a mix of work (the “actual” work, training and fun).

But how do you find (young) people who are going to love data integration? There is no direct training for it, and often IT professionals have only had indirect dealings with integration. Job descriptions then often sprinkle all kinds of abbreviations of protocols and frameworks, but just as important is:

➢ broad knowledge of IT (after all, you deal with all kinds of technologies)
➢ flexible in learning new things
➢ good communication skills
➢ willing to specialize in a very specific area
➢ 50/50 balance between coding and communication!

Igor Tkac wonders in integration stories:

But why would anyone love a job where:

➣ the more “invisible” you are, the better you are.
➣ if something does go wrong then you will be the first to be blamed.

However, he goes on to indicate that there is also another side to this story, which is how you treat people. An integration specialist is someone who:

➣ enjoys listening to others
➣ derives satisfaction from helping others

What is really important is the passion to be part of the “bigger picture” where you never stop learning new things and trends.

The biggest challenge

The project takes off and then you have to deal with a chunk of communication. Um, communication?

Yes, communication: the GREATEST integration challenge.

Igor likes to describe integration work as “gluing business and technology together” or “building bridges between technologies”. Yet he adds, “but sometimes it feels like trying to glue water and fire together.” In practice, you’re not only dealing with many different technologies, but also with all kinds of parties.

The playing field of integration:

The problem is that integration suffers more when different parts are more in their own bubble without regard to the environment. Proper communication and willingness to listen from all sides is a key to success.

Lessons

Lesson learned 5: a balanced skillset

Often IT is divided into managers on one side and technicians on the other. Of course, you have all sorts of flavors in that. With an integration specialist, this is almost always 50/50, so soft and hard skills must be balanced.

Lesson learned 6: listening is gold

Often integration specialists have strong ideas about what makes a good integration. However, the key lies in listening to find a solution that meets the requirements and wishes of the stakeholders.

The inevitable P1

Finally the various integrations go to production. A standard thing with integration is that different data is always sent over production than during the test phase. So no matter how well you test, there are always instances that slip through. The important thing is to be able to detect such incidents quickly (monitoring), analyze them and implement the solution (which is mainly a CI/CD pipeline and good change management).

If something goes wrong shortly after going into production there is often plenty of support. After that, integration is an invisible IT component …until something really goes wrong. Being an integration specialist is sometimes a thankless role to have….

If you are responsible for front-end applications, you have feedback from users. The back-end is shrouded in mystery, but users know there is a place full of “computers” somewhere that processes everything they need. But middleware? Doesn’t data go through cables?

But even a small problem at the integration layer and 💥 … data doesn’t come through, applications show errors and the back-end crashes … Months of silence are over. The integration team comes out of the shadows. The better you do your job, the more invisible you are.

Even when we do our best, it happens from time to time — a P1 incident with a major production disruption. As the integration team knows, they will be the first line of defense (and sometimes blame). The following two sentences then help kill the mood nicely:

1. I checked on my end and everything looks OK!
2. No action needs to be taken on our end.

The reason why people say this is that they are afraid.
They do not want to be part of the problem, because very often when you want to help someone solve the problem it is presented as if he / she are responsible for that problem.

Results:

➢ No trust between teams.
➢ Slow responses/longer time needed for recovery.
➢ And most important: unknown root cause.

What happens if the data from system A does not reach system B?

Two choices:

a) verify that system A and B are working correctly
b) contact the integration team and tell them that the middleware is not working

Yep … ‘b’ is usually the choice. And this is where you need to prepare all your integration team members (especially juniors). The larger companies are, the weaker the communication is between different IT teams (who are sometimes literally on the other side of the world).

Integration as “glue and bridge” usually provides the best overview of what is happening within the company from an IT perspective. This is simultaneously “a blessing and a curse” leading to the aforementioned situation.

Yes, it puts integration in a bad light (which can be frustrating) but the ultimate goal is to solve the problem ASAP and this is the fastest approach where middleware should take the lead and initiate collaboration between teams.

Lessons

Lesson 7: visibility

Integration specialists: if you feel like you’re invisible, that’s okay — it means you’re doing a good job.

Lesson 8: recognition

The only way to avoid a situation where people hide, blur and cover their mistakes is to show them that mistakes can be made and that this should be seen as an area for improvement, not blame. Recognize people who helped with the solution as a “good example.

Lesson 9: working together

Working together and not pointing fingers is the right answer. With a little respect and appreciation for integration work, even ‘b’ can be okay for integration specialists. Let’s strive to find a solution, not just find “culprits.

Two people

Imagine a high-tech company with 10k+ employees.
They are one of the world leaders in their industry. Like most companies today, they fully understand that the only way to survive in a competitive market is to “think digital”.

So money is being invested in Apps, R&D and Cloud solutions. The complexity of the application landscape is increasing. There are more than 1,000 components in the landscape. And guess what, the entire integration layer is managed by only … TWO people.

These two people are true professionals, so the whole solution works fine on a limited budget.

What could go wrong now?

Well, the following:

1. Because of small capacity, they cannot assess all new IT initiatives and therefore application teams “sometimes” manage direct integrations (bypassing the integration team).
2. An unpredictable production environment disrupts development plans.
3. Some projects are put on hold.
4. The company cannot get the FULL potential out of their data.
5. Constant overload limits room for learning/innovation.

Lessons

Lesson learned 10: sufficient resources

Igor indicates “This is not a rare story. I have seen it in several companies and in two cases it even ended with burnout.”

An integration team will need to be aligned with the rest of IT. In addition, a good CIO (chief information officer) will need to understand the stakes of integration layer and ensure that the middleware team is a stable and reliable part of IT.

Conversely, integration will have to enable the organization to make its own integrations. That, though, is the most important lesson.