Lessons from rolling out new data standards across 12 companies

Most of the hard work in a data standard happens after the specification is published. Someone has to read it, work out what it means for their own systems, and then persuade everyone they exchange data with to read it the same way. For several years at Transcoma Grupo Empresarial, a logistics group of 12 companies, that someone was me.

We had to adopt several iterations of EU customs messaging under the Union Customs Code (ICS, ECS, ICS2) and new e-invoicing rules (Facturae, SII, Veri*Factu) across a group running six different ERP systems, a central finance platform, and links to customs authorities and sector partners. By the end, our integration platform was moving 6,000 to 7,000 messages a day with an error rate under 0.2%. Getting there taught me more about people than about the technology.

1. Read the specification yourself

It is tempting to rely on a vendor's summary or a consultant's slide deck. Don't. Summaries flatten exactly the details that cause trouble later: which fields are optional in theory but required in practice, which code lists are open to interpretation, and what happens at the edges.

For example, in the case of e-invoicing, none of the standard consultant decks or summaries "for the layperson" covered any of the complex scenarios required for import/export fees and disbursements in the name of third parties. Getting those messages right requires an in-depth understanding of the businesses and legalities involved, as well as the processes in the companies that create those invoices.

2. Translate it into the business's language

Finance and operations teams don't need to know the message schema. They need to know what changes for them: which information they must now capture, at what point in the process, and what happens when it is missing. Every new mandatory field is a process change somewhere, and the specification never tells you where.

The new standard and its new processes need explaining many times in many different ways, to each of the teams involved. Each team needs to hear it in terms it understands, and most only want to know which parts apply to them. Managers need the end-to-end view of the processes they supervise. Senior management needs to know what to keep in mind strategically, and who to ask for more detail.

3. Twelve companies means twelve interpretations

Each company in the group had its own systems, habits and priorities. Left alone, each would have implemented the same rule slightly differently, and every difference would have surfaced as a rejected message. What worked was agreeing one shared interpretation early, writing it down, and treating it as the reference every time a question came up.

Even with a shared interpretation, exceptions still come up, and many of them start with words. Sometimes terminology overlaps, and has different meanings depending on who you are talking to. A disbursement can mean something different to an operations department than to a finance department. A summary declaration can mean different things to a freight forwarder, a customs agent, and a ship's agent. An understanding of these differences speeds up identifying and resolving issues when they come up, and each one resolved should be added to the shared reference.

4. Assign ownership of the data, not just the interface

In the case of Transcoma Grupo Empresarial, the companies were complementary, and vendors of each other. A freight forwarder would send information to a customs agency, and then to a ship's agent. Each of them needed information about a different part of the customs chain, and each wanted the others to do the hard work, especially talking to the end customer.

Creating data interfaces between the companies and systems is not enough. Each piece of data needs one, and preferably only one, owner: ideally the person responsible for collecting it. This is doubly important when there is more than one company involved in the chain, as the company collecting the data may not necessarily be the company responsible for declaring that data to the authorities. Each company needs its own validation checks, and a workflow that sends problem data back to whoever created it, quickly. Over time, those procedures should be reviewed, so that edge cases are handled correctly or avoided altogether.

5. Bring vendors and authorities into the room early

A standard only works when both ends of the exchange agree. We worked with our software vendors, messaging platforms and the authorities themselves to test against real cases, not just against the schema. The earlier we found a mismatch, the cheaper it was to fix.

We were also not afraid to talk to authorities and those implementing standards where those standards were ill-defined or inadequate. Several times we had to convince a public authority that their systems were not handling particular cases, sometimes some really common cases! In one case we had to convince an authority that their implementation directly contravened published law. Good relationships with every person involved are key to helping this process along.

6. Production is where the standard really gets tested

Go-live is the start, not the end. Real data finds the gaps no test case did. Watching error rates, sorting rejections by cause and fixing the source rather than the symptom is what eventually took us below 0.2%.

In particular, having good fallback procedures that the people involved were properly trained in kept the business going whilst the interdepartmental groups worked on analysing and fixing the edge cases we faced.

This also involved coping with legal obligations when a public authority's go-live failed. The operations departments needed to have an agile problem-solving flow with their counterparts in the authorities, which would then be documented and fed back to the implementation groups.

7. All these decisions affect people

Every new standard, process and data field can seem abstract in a meeting room, but it is people who live with the result.

Sometimes the nature of a new standard is to create more work, but that needs taking into account when implementing it, especially if it will require expanding or reassigning roles. A bad implementation can create much more work and much more chaos, which can affect people's ability and, more importantly, willingness to do their jobs effectively. Automation can remove some jobs entirely, and that affects the whole workforce, not just the people whose roles disappear.

Ultimately, some of the decisions taken when implementing a standard affect what people are doing physically on the ground. Ensuring that activity is not onerous or demeaning is part of what needs to be considered.

Why this matters beyond logistics

I now work with public transport data, on GTFS integrations and NeTEx, and the pattern is the same. Everybody has their own interpretation of the standard, and different needs from it. A standard is only as good as the shared understanding behind it. The technical work matters, but most of the effort goes into getting many organisations to read the same document the same way, and to keep doing so once real data starts to flow.

In public transport, the people who are most affected by these standards are those at the end of the chain: station and depot staff, drivers, maintenance workers, control room teams and, most importantly, passengers. A badly implemented standard reaches staff as extra work, confusing tools, and dissatisfaction with their jobs, and reaches passengers as frustration, trips being missed, or an unwillingness to use public transport.

Done well, a data standard is invisible and boring. That is, ultimately, the point. Making sure it's invisible and boring is what standard setting and implementation should be about.