The Three Ways Offline Data Goes Wrong: Sync, Duplicates and Version Conflicts

Every offline data collection tool markets the same promise: collect without a connection, sync when you have one. The promise is real and largely solved. What is not solved, and what actually costs programmes their data quality, is what happens in the seconds after the device reconnects.

Three failure modes account for most of it. They are worth understanding as an M&E manager even if you never touch the technical settings, because the mitigations are partly procedural and those are yours to own.

Failure one: the sync that never completes

The naive design uploads submissions when the app is open and connectivity is present. In the field this fails for reasons that have nothing to do with signal: the enumerator closes the app, the device sleeps, the operating system kills the background process to save battery, the connection appears for forty seconds in a matatu and disappears.

What a robust design does instead:

• Queue locally and treat the device as authoritative. A submission is saved to local storage the moment it is completed and is never held in memory waiting for a network.
• Sync in a background worker, not only when the app is in the foreground, so the OS can run it opportunistically.
• Retry with exponential backoff. A failed upload retried immediately, repeatedly, drains the battery and hammers a server that is probably already struggling. Backing off — waiting progressively longer between attempts — is what makes retries survivable on both ends.
• Cap the retries and surface the failure. After a bounded number of attempts the submission should be flagged for human attention rather than retried forever in silence. A queue that retries invisibly for three weeks is worse than one that fails loudly on day one.

The procedural half, which is yours. Sync at a scheduled point each day when enumerators are somewhere with reliable connectivity, rather than relying on opportunistic sync in the field. And monitor the pending queue by enumerator, by day — a queue that is growing for one person is the single earliest signal you will get that something is wrong, and it usually appears days before the data gap does.

Failure two: duplicates

Duplicates are more damaging than missing data, because missing data announces itself and duplicates do not. A household counted twice inflates every indicator built on it, and reconciliation after the fact is slow, manual and never quite complete.

They arise from ordinary field behaviour: the enumerator taps submit twice because the first tap gave no feedback; the app is reopened and a sync fires again for records it already sent; a device is restored from backup and re-uploads its queue.

The only reliable defence is a client-generated identifier. The device assigns each submission a unique ID at the moment of creation, before any network is involved. The server enforces uniqueness on that ID, so a submission arriving twice is recognised as the same record rather than accepted as two.

Two details separate a correct implementation from a superficial one:

The uniqueness constraint must be enforced at the database level, not in application code. A check-then-insert in application logic has a race condition: two sync requests carrying the same ID can both pass the check before either writes. Under a flaky connection — exactly when re-sends happen — this is not a theoretical concern. The right behaviour is a database uniqueness constraint plus explicit handling of the collision, so that the request which loses the race returns the record that won rather than an error.

The constraint must be scoped to the organisation. Client IDs are generated on devices, and a system serving many organisations must not let one organisation’s identifier collide with another’s. Getting this wrong is not a duplicate problem; it is a data isolation problem, and a much more serious one.

Media attachments need the same treatment. A photo uploaded twice for the same question on the same submission should be recognised as one file, which means deduplicating on the pairing of submission identifier and field.

Failure three: version conflicts

The one that is least discussed and most often discovered at analysis.

Forms change during a round. A question is reworded after the pilot, a choice option is added, a section is dropped. Meanwhile there are twelve devices in the field carrying whichever version they last downloaded, and their queued submissions may have been collected on a version that no longer exists in the current form.

If submissions do not record which version they were collected against, you end up with one dataset containing answers to two different questions in the same column — and nothing marking where one ends and the other begins. That is not correctable after the fact.

The mitigation is straightforward: stamp every submission with the form version it was collected on. With the version recorded, you can segment your analysis by version, identify exactly which submissions predate a change, and make an informed decision about whether they are comparable.

What that does not do is decide for you. A stamp is a fact, not a judgement. Whether pre-change and post-change submissions can be pooled is an analytical call that depends on what changed, and no system should make it silently.

The useful middle ground is a system that detects the conflict and then stops, rather than resolving it on your behalf. When a device submits against a form version older than the one currently published, that submission should be marked as conflicted and routed to a person — who accepts it, rejects it, or records why the two versions are comparable. The decision stays human; what the system contributes is that the decision gets made at all, on the record, instead of the discrepancy passing through unnoticed.

The procedural mitigations matter more here than the technical one. Freeze the form before the round starts. If a change is genuinely unavoidable mid-round, make it deliberately: document what changed and when, push it to all devices at once rather than letting them update ad hoc, and note in the analysis plan how the two versions will be handled. A mid-round change made informally, device by device, produces a dataset whose version boundaries nobody can reconstruct.

The metadata that makes all three tractable

Whichever tool you use, check that a submission carries:

• Its client-generated identifier.
• The form version it was collected on.
• Timestamps for both collection and receipt — they differ under offline sync, and the difference is diagnostic.
• The enumerator identity.
• GPS, where the protocol calls for it.
• Its sync source, so mobile, web and offline-queued submissions are distinguishable.

Together these turn all three failure modes from mysteries into queries. Without them, an anomaly in the dataset is something you argue about.

The one-page checklist

1. Local queue, device authoritative, background sync worker.
2. Exponential backoff with a retry cap and a visible failure state.
3. Client-generated IDs, unique at the database level, scoped to the organisation.
4. Media attachments deduplicated by submission and field.
5. Form version stamped on every submission.
6. Form frozen before the round; mid-round changes documented and pushed at once.
7. Pending queue monitored per enumerator, per day.
8. Scheduled daily sync point, not opportunistic sync.

Items one to five are the tool’s job. Items six to eight are yours, and they are the ones that most often go undone.

---

Monival’s mobile app queues locally with a background sync worker, exponential backoff and a bounded retry count. Client IDs are enforced unique at the database level, scoped per organisation, with the race condition handled explicitly rather than surfacing as an error; media attachments deduplicate on submission and field. Every submission is stamped with its form version — and when that version is older than the one currently published, the submission is marked as conflicted and held in a review queue, where a reviewer accepts or rejects it with a note and the resolution is recorded against the submission. For form design and sync strategy, see the complete guide to offline data collection.