6. EDI

Standardization in integration

8 min readApr 8, 2024

Aleksandra regularly drove to the local wholesaler to get ingredients and other supplies. At one point, the number of trips grew rapidly. She called in Thomas. With his van, he brought the supplies from the wholesaler to the bakery.

But by now she has also outgrown this phase. There appears to be a huge demand for the light cakes from Eastern Europe. She has from than on started ordering directly from the suppliers. But they didn’t like that she ordered over the phone; rather they wanted her to use their API, with an exchange standard. So from her Salesforce application, she would have to start sending EDI (Electronic Data Interchange) messages, such as orders and invoices.

A classic problem

A classic problem in data integration is: how do we integrate two systems, each with a different data model? A data field in one system may not exist in the other, or may mean something slightly different.

This problem can be something very trivial, such as the article number field being capitalized in another system. It can also be very complex, where hundreds of fields differ from each other in name and structure.

There are three ways to deal with this:

Mapping

The systems both generate their own data format. Between the two formats is a mapping that relates and translates the two formats. So you need a transformation step here.

2. Canonical Data Model.

The systems each generate their own data format. This is converted to a generic format. This generic format is defined in a CDM (Corporate Data Model).

So although an additional translation step is required here, this can vary from one change to another. Suppose a new source system sends a message. In that case, you only need to modify the translation step to the CDM and not to all customers.

3. Standards

The systems use the same standard format, also called EDI (Electronic Data Interchange). Despite the fact that the data model differs for each system, they all use the same standard format for data exchange. No translation step is required in the integration.

This latter way is the topic of this chapter.

EDI

The term EDI (Electronic Data Interchange) has been around since the 1970s and originally came from the military who used it for logistics processes.

Back in 1971, Heathrow introduced the EDI format London Airport Cargo EDP Scheme (LACES) for exchanging cargo information with customs. Nowadays we would omit the word electronic because almost everything goes digital.

EDI was quickly adopted also in other industries where a lot of information needs to be exchanged between parties. However, EDI formats can also do well within an organization. It has since replaced paper and PDF in many organizations.

EDI in Practice

Standard formats are ideal for integration specialists because they take a lot of logic out of the integration layer. Yet EDI comes with its own problems. Indeed, the practice of standardized formats can make integration problems even more complicated.

First, you need a large organization to create and maintain the standard. In addition, there are often multiple versions of the standard. Finally, the systems must both support the format and the correct version, otherwise a translation step is still needed.

An example of such a problem situation is the GS1 standard for item information. From time to time is this standard being updated. During this update, article data may not be able to be sent worldwide for a week. First, everyone had to support the same version. Preparations then often run for months, sometimes years.

In addition, standards are vastly expanded. You often only need a fraction of them in your own organization. Suppose you want to send an article file with only the article number and product description. This is however not allowed by the GS1, as the article file should contain all required fields. To figure out which fields, the GS1 standard provides a guide of 500 pages with hundreds of fields divided into modules. Implementation of the standard therefore also requires a lot from a user organization.

The theory

Often, finding the right specification, for the right format (for example, in XML or JSON) in the right version is already difficult. In addition, reading the specifications is sometimes difficult. Below is a snippet from the GS1 specification:

A customer specific article (CSA) is broadly defined as any item where the supplier defines all possible manifestations of the article from which the customer may choose, and pre-allocation of article numbers at the lowest level is not feasible. CSAs are never made for stock, and hence are always made to order. However, made-to-order articles are not necessarily customer specific, but could be standard. A typical example of a CSA is a chair that is available in 300 different types of upholstery for the seat, back, and armrest. This list of available upholstery could also be used for other types of furniture the supplier offers. There are 27,000,000 ordering possibilities for this chair (300 x 300 x 300). Typically the supplier’s catalogue lists a generic style of chair as well as the 300 different upholstery options. The customer chooses the style of chair and selects upholstery for the seat, back, and armrest.

When you have to read through 500 pages of this….

Although standards are often very comprehensive, they are never complete and are often open for interpretation. An example is that EDI is handled differently in Germany than in the Netherlands. This often causes problems when importing goods from Germany. Also, software vendors often have many custom fields, which still require integration logic.

A sexy topic?

Despite everything, standardized formats have brought a lot. In particular, once a standard is established and has broad support, it greatly facilitates communication. You don’t need many sessions to align each other’s data models.

So creating, maintaining and using standardized data formats is not a very sexy topic, but it is an important one. It starts with knowing what standards are out there.

The most common standards in the market:

ACORD AL3 (Insurance)
AgroConnect (Agriculture)
ARTS (Retail)
EDINE (Energy)
HIPAA (Healthcare)
FIX (Financial)
GS1/EANCOM (Retail and healthcare)
HL7 (Healthcare)
IATA PADIS (Aviation)
NCPDP SCRIPT (Healthcare)
ODETTE (Auto industry)
O-DF (Internet of Things)
Opentrip (Logistics)
S@les (Construction)
SAP iDocs (SAP data format)
SWIFT (Banks)
TRADACOMS (British supermarkets)
UBL (commercial transactions)
UN/EDIFACT (commercial transactions).
X12 (Generic Data formats from ANSI)

Most data formats are maintained by independent standards organizations. The best known are GS1 (Global Standards One), ISO (International Organization for Standardization) and ANSI (American National Standards Institute). Most organizations operate as foundations and have been around for decades and work continuously to come up with new standards and improve existing ones.

GS1, for example, has been around since the 1970s. The organization (like most other organizations) is known to many only for some specific standards. Often when people think of GS1 they only think of barcodes, but as you can see below there are many more standards created by the GS1 than you think:

The previous list of standards organizations and the above list of standards from just one of those organizations gives a good indication of how many standard data formats exist worldwide.

Standards in the Dutch government

There are several standards and organizations within the Dutch government that deal with the exchange of data within and with the government. The basis for most organizations is NORA (Dutch Government Reference Architecture). This is an interoperability framework for the Dutch government and to this end translates legislation, policy and standards into architecture principles, descriptions and models. Today it is maintained online.

Based on the principles of NORA, various organizations are working to provide nodes. This is not arranged centrally, but usually by type of government agency, such as ministries, implementing agencies, water boards and municipalities.

Well-known organizations are:

Logius

Logius is a part of the Ministry of the Interior and Kingdom Relations. Logius offers various services, such as DigiD, Digipoort and Digikoppeling. The latter is a set of standards for exchanging EDI traffic within the government.

Digikoppeling is not about the content of the message, but about the packaging. In order to exchange data between ICT systems, agreements are made about its format(UBL or XML), the method of transport (Digital Network or Internet) and the packaging in which the data is sent (Digikoppeling).

BKWI

The Bureau Keteninformatisering Werk & Inkomen (BKWI) is a data hub within the SZW (Social Affairs and Employment) domain. It is an independent organizational unit within the UWV (Uitvoeringsinstituut Werknemersverzekeringen) that falls directly under the UWV management board and works on behalf of the Ministry of SZW. It forms the link between chain partners in the field of work and income, such as the UWV, the municipalities and the SVB.

In particular, the BKWI advises and manages chain products such as SUWInet. For the Work and Income Chain, BKWI offers so-called Suwinet Services. For the Debt and garnishment chain, it offers the vBVV the central facility for the simplification of the attachment-free foot.

Intelligence Agency Foundation

Stichting Inlichtingenbureau is an information hub for municipalities. The bureau was established in 2001 by the Ministry of Social Affairs and Employment (SZW) and the Association of Netherlands Municipalities (VNG). The purpose of its establishment was to simplify the exchange of (personal) data between municipalities and government agencies.

The bureau links digital files of government agencies. If this link issues an alert, the Intelligence Bureau passes it on to other government agencies. This linking is done using, among other things, the citizen service number.

WoGo4IT

WiGo4IT was founded in 2007 as a partnership at the initiative of the directors of the social services of the four major cities. They also arrange much of the data exchange for these large municipalities.

There are other standards and nodes for various other government entities, such as Geo-Standards for Geographic Data, GWSW (Water Boards), WDO (Customs) and StUF (government administrative data).

EDI Software

Nowadays, software to generate and process EDI is still expensive. In addition, most large companies work with lots of parties. Consider, for example, supermarkets that purchase from hundreds of suppliers. To avoid having to set up a connection with each supplier, intermediaries are used.

These intermediaries act as brokers and try to overcome many of the difficulties mentioned earlier. For example, they help member companies comply with the correct format. In the Netherlands, the best-known EDI companies are Descartes, Tie Kinetix, Coopernicus (Niklas), OneTrail and B2BE.

Basically, a message broker is a type of post office that routes messages based on address information to the right party. It is also referred to as VANs (value-added networks). A VAN has added value especially in situations with many formats, protocols or many parties.

Standardized data formats are both a blessing and a curse. As this chapter has made clear, there is not one standard EDI, but hundreds of standards, standardization organizations and data nodes.

Fortunately, there are plenty of technical solutions and knowledge organizations trying to make exchange easier and easier. As a result, many standards have increasingly become community assets.