Why .NET Integration Projects Fail: A Pattern Across Multiple Engagements

Integration projects fail on three patterns, and the first is detectable during scoping, before any code exists: nobody on the project can state precisely what the two systems disagree about.

The team can describe what each system does and what the end state should be. Asked what happens when a client exists in System A and not System B, or which side wins when both hold a value and the values differ, the answer becomes general. It will be handled during implementation. The fields get mapped in development. That is a business data question.

The vagueness is not a scoping oversight. It is the project's primary risk, deferred to the phase where resolving it is most expensive.

Pattern 1: The disagreement is the project

Moving data is the trivial component. The work is reconciling two systems that have modeled overlapping reality differently for years, each internally consistent and mutually incompatible.

Representative mismatches:

• System A models a client as one row. System B models a client as a hierarchy with a parent organization and locations.
• System A permits a record with no assigned owner. System B requires one.
• System A stores status as one of four values. System B has eleven, three of them deprecated and still present in data.
• System A treats a name as a single field. System B has five, with different validation on each.
• System A soft-deletes. System B removes.

Neither system is wrong. They were built for different purposes at different times by people solving different problems.

Every mismatch requires a decision, and nearly none of them are technical. Whether an unowned record is rejected, assigned a default owner, or synchronized with a null is a business rule. Choosing wrong is invisible in testing and expensive in production, because the consequence appears in reporting months later.

Projects that succeed perform this reconciliation during scoping, write the rules down, and obtain agreement from someone with authority to give it. Projects that fail defer it, and developers make the decisions implicitly. That is not arrogance. A developer facing an unspecified case has to write something. Fifty reasonable choices later, system behavior is the sum of decisions nobody made deliberately and nobody can now enumerate.

The deliverable that prevents this is a written reconciliation document: every field where the models diverge, the decision for each, and the name of the person who approved it. It takes one to two weeks and it is the cheapest week on the project.

Pattern 2: Handling it in the integration

The second pattern announces itself in the phrase used for it, and it is almost always the wrong choice.

It arises when profiling reveals a data quality problem in a source system: duplicate client records, inconsistent naming, a field that policy requires and that is empty in eight percent of rows. Fixing it upstream means a cleanup project with an owner, a budget, and a timeline. Handling it in the integration means normalization logic, and that can start tomorrow.

The choice is made because the cost lands later and on someone else. Two consequences follow.

The integration accumulates correction logic. Special cases for known-bad records, fuzzy matching for inconsistent naming, defaults for fields that should have been populated. This layer becomes the most complex part of the system, and its behavior cannot be predicted by anyone who has not read it.

The upstream problem becomes invisible. The bad data was producing friction, and the friction was generating pressure to fix it. The integration absorbs the friction, the pressure disappears, and the process that creates bad data continues unchanged, because nothing about it was changed.

The approach that holds: fix upstream what can be fixed, and where it cannot, have the integration reject bad records visibly rather than repairing them silently. Rejection is uncomfortable and it keeps the problem attached to the system that produces it.

Pattern 3: Error handling in one paragraph

Integration specifications describe the happy path across twenty pages of field mapping and dispose of everything else in a sentence about logging and retries.

Six months later this produces a recognizable state:

• The integration has failed intermittently for weeks, writing to a log file nobody reads, because no alert was ever defined
• Retries have been running and are not idempotent, so they created duplicate records
• The duplicates propagated to downstream systems
• The number of affected records is unknown, because failed items were never recorded in a queryable form
• Reconstructing what happened requires logs that reach back far enough, and they do not

Remediation costs more than the original integration, and the majority of that cost is the reconstruction rather than the fix.

What durable integrations include

Five items. They are the first things cut when scope has to meet a number, and each one is cheaper to build than to add later.

• A written reconciliation of how the systems disagree, with the decision for each case, agreed before development starts.
• Explicit behavior for non-matching records, surfaced in a report with a named owner. Never silently skipped.
• Idempotent operations, so a replay after failure duplicates nothing. This is a design property, not a feature. Retrofitting it means reworking the data model and the matching logic.
• Alerting on the absence of expected activity, not only on thrown exceptions. The dangerous failure is the job that runs, processes zero records, and reports success.
• A reconciliation check that can answer whether the two systems agree at this moment, on demand. Without one, the strongest available statement is that nobody has complained recently.