Over-the-air (OTA) updates are the only practical way to fix bugs and patch vulnerabilities on devices you cannot visit. They are also one of the riskiest operations you can run on a fleet, because a bad update can turn hardware into a brick. This article is a troubleshooting guide for when updates fail, followed by the design habits that make failure less likely.

Start by Classifying the Failure

When an update does not complete, the first question is where in the pipeline it stopped. Most failures fall into one of five stages.

1. Never received. The device did not learn an update existed.
2. Download failed. The device started but could not finish retrieving the image.
3. Verification failed. The image arrived but did not pass signature or integrity checks.
4. Install failed. The device could not write or apply the image.
5. Boot failed. The new firmware installed but did not run correctly.

Each stage points to a different set of causes, so resist the urge to retry blindly until you know which one you are in.

Stage-by-Stage Diagnosis

Never received. Check whether the device was online during the rollout window. Verify its group membership and targeting rules, and confirm it has the permission to read job notifications. Devices that sleep for long periods may only check in rarely.

Download failed. Common causes are unstable connectivity, timeouts that are too short for the image size, an expired download link, or insufficient storage for the incoming image. Look for patterns by region, network type or hardware revision.

Verification failed. Mismatched signing keys, a corrupted transfer, or an image built for a different hardware revision are typical. Confirm that the certificate or key the device trusts matches the one used to sign the release, and that your build pipeline did not alter the artifact after signing.

Install failed. Look at free space, flash write errors and interrupted power. Devices on batteries may refuse updates below a charge threshold, which is correct behaviour but looks like failure from the cloud side.

Boot failed. This is the most serious case. The firmware passed every check but crashes or hangs. A watchdog and a rollback mechanism are what stand between this and a dead device.

The Information You Need Before Anything Else

Troubleshooting is slow without data. Make sure devices report, at minimum:

• Current and target firmware versions
• Update state and the last error code
• Hardware revision and bootloader version
• Free storage and, where relevant, battery level
• Time since last successful connection

If your devices only report “failed”, you will spend your time guessing.

A Diagnosis Table

Symptom	Likely area	First thing to check
Update stays “pending” for many devices	Delivery or targeting	Group rules, device connectivity, job configuration
Failures cluster on one hardware revision	Image compatibility	Build target and revision checks
Failures cluster on one network or region	Download path	Timeouts, link expiry, intermediate proxies
Verification errors everywhere	Signing	Key rotation history and artifact integrity
Devices stop reporting after update	Boot failure	Rollback behaviour and watchdog configuration
Only low-battery devices fail	Install policy	Power thresholds in the update logic

Rollback Is a Feature, Not a Rescue

The safest update designs use two firmware slots. The device writes the new image to the inactive slot, boots into it on a trial basis, and only marks it as good after the firmware confirms it is healthy, for example after reconnecting to the cloud. If confirmation does not arrive before a timeout, the bootloader returns to the previous slot.

Key details to get right:

• Confirmation should check something meaningful, not merely that the code started.
• The bootloader itself should rarely change, and when it does, treat it as a higher-risk release.
• Rollback needs to cope with schema or configuration changes, so avoid releases that make stored data unreadable by the older version.

Roll Out in Stages

Even a perfect pipeline benefits from caution. Release to a small canary group first, wait long enough to observe real behaviour, then widen in steps. Define in advance what makes you pause: an error rate above a threshold you chose, devices dropping offline, or an unusual rise in reboots. Make pausing a single action that anyone on call can perform.

A Hypothetical Example

Imagine a fleet of outdoor gateways receives a new firmware release. A small canary group updates successfully, but when the rollout widens, a group of devices on one older hardware revision stays offline after rebooting. Because the team collects boot reasons, they see these units failed the health confirmation and rolled back automatically, then retried and failed again. Investigation shows the build included a driver change that assumes a newer radio module. The team pauses the rollout, adds a hardware-revision check to the job targeting, fixes the build and resumes. Because rollback worked, no device needed a site visit.

Preventing the Next Failure

• Test updates on every hardware revision you support, including old ones.
• Simulate interrupted power and network loss during the install step.
• Keep image sizes as small as practical, and consider delta updates if your platform supports them and your link is constrained.
• Sign every release and rotate keys on a plan you have rehearsed.
• Keep a recovery path, such as a minimal recovery image, for devices that lose both slots.

Frequently Asked Questions

Should I retry failed updates automatically? Yes, with backoff and a retry limit. Endless immediate retries waste bandwidth and battery and can hide a systematic fault.

How do I update devices that are rarely online? Design the job so it remains available until the device checks in, and make sure the image link or token stays valid for that period, or can be refreshed on demand.

Is it safe to update everything at once? Rarely. Staged rollouts limit the blast radius of a mistake.

What if a device is bricked despite rollback? Investigate whether both slots were affected or the bootloader failed. Physical recovery may be required, which is why prevention matters.

Do I need to sign firmware? Signing is a baseline security practice, since it prevents a device from accepting an image from an untrusted source. Follow your platform’s and your device vendor’s current guidance for implementation.

Next Step

Audit what your devices report during an update today. If any stage of the pipeline is invisible, add that reporting first, then run a deliberate failure drill on a test group by interrupting a download and a reboot to confirm rollback behaves as you expect.