Secure Remote Debugging for IoT Connected Devices

Secure Remote Debugging for IoT Connected Devices

One of the challenges confronted by embedded device developers is what happens when something goes wrong with an IoT device – meaning you can no longer connect to it. Bad news if this happens to a single device in the field, but commercially disastrous if the same issue affects even a small proportion of your fleet of IoT devices.

Imagine that a software bug escapes your otherwise comprehensive device test plan resulting in an interruption to the connectivity of 10% of your deployed devices; even in this case the cost of recovery would be huge.

Issues like this do unfortunately happen, and while you know that software bugs occur, and no doubt can be fixed, the risk-impact of issues like this is significant.

Separating the connectivity stack from the application code guarantees device reachability

Obviously, being able to guarantee connectivity to your remote edge devices is important – Twilio Microvisor solves this problem by disaggregating the on-chip application code and the connectivity software stack that runs on the device itself. This is achieved through the hardware based hypervisor approach of Microvisor, which leverages a physical separation between the application code (that may have just encountered an otherwise fatal software bug) and the connectivity software stack running on the edge device.

So even if the worst case scenario described above is of the application having died in some horrible unpredictable way (e.g. a flailing pointer on startup), the device itself remains connected and reachable – because the Microvisor component is connected and reachable.

Twilio Microvisor handles your device connectivity

Twilio Microvisor takes care of all of the connectivity from the edge device to the cloud. It is responsible for the IP stack on the device, as well as the Wi-Fi and/or cellular modem firmware and drivers – i.e. everything your device requires to remain connected. And with Twilio Microvisor running alongside your application and managing the connectivity software on your devices, even in the worst case scenario, you can still reach and recover your device.

Twilio Microvisor provides your device with industry-leading security

Twilio Microvisor also provides you with an industry-leading approach to IoT device security for the complete lifecycle of your device. It ensures constant ongoing improvement and protection for your devices without you having to invest any of your own engineering effort. This starts with a secure factory provisioning process built with Secure Boot at the heart of it, guaranteeing only your code runs on your devices, and goes all the way to regular in-the-field security patches and updates to the connectivity drivers and the IP stack itself for the lifetime of your device.

As a result of a simple API call to the Twilio Microvisor endpoint, you now have a way to roll back your firmware and application to the previously working version, and this in turn gets those devices back online at the push of a button. What a relief!

Twilio Microvisor now lets you securely debug individual devices

But on top of Microvisor providing the edge device with connectivity, security, and OTA firmware update capabilities, Microvisor now also provides the ability for you to securely remotely debug the application itself and work out what has just gone wrong with those devices that may have crashed.

Secure Remote Debugging takes this super power of separately managed secure connectivity to the next level when it comes to debugging the issue at hand. We all love being able to fix a bug in real time, single-stepping through code, changing variable values on the fly, inspecting the register contents, and having that Eureka moment of figuring out what the bug is that is making your application do something unexpected. But doing all that on cellular-connected devices deployed across the world? That has now become possible.

How Microvisor enables secure remote debugging

We do this by tunneling the GDB debug protocol from your local development PC or MAC all the way to the edge device itself wherever it is in the world. The Twilio CLI tool and the Microvisor plugin enable you to run a local GDB server on your development PC that you can hook up to your own toolchain / integrated debug experience, just as you might do with a device that you are debugging locally on your desk. This local server connects through the Twilio Microvisor service directly to the edge device that you want to debug. This connection is transparently routed over a cellular connection (powered by Twilio Super SIM) or via Wi-Fi, or Ethernet to your embedded device.

This enables you to access all of the capabilities of your debugger (command line or GUI-based) that you will need to figure out what went wrong with your application. The initial release of Microvisor’s remote debugging functionality supports the majority of the commands provided by GDB. Further commands are expected to be added as Microvisor progresses through its Beta phases. Find details here: https://www.twilio.com/docs/iot/microvisor/microvisor-remote-debugging#supported-gdb-functions

The secure remote debugging link is fully end-to-end encrypted and ensures only you have access to your devices. On top of this, having access to secure remote debugging means that the local debug ports can be physically disabled on the edge device itself, meaning that your device is even more secure in the field. This level of security is one extra way that you can protect your code running on your device.

Secure Remote Debug enables you as a device builder to respond much faster when the inevitable device side bugs do occur. It means that you can remotely connect to the actual device in the field that is experiencing the issue, step through the code, and deploy the fix without having to leave the comfort of your “work from home” desk or office.

The benefits of securely debugging IoT devices without local access

Being able to securely debug devices remotely comes with a number of business benefits:

1. Cost savings: The cost to find and fix a bug in the field is massively reduced. Now when one of your customers reports an issue that you can’t reproduce in the lab, you can access the actual device experiencing the issue in the field and see exactly what is going wrong. This is a potentially massive time and money saver for when the inevitable bugs do occur.
2. Feature development agility: This capability opens up the possibility of developing new features in the field. Perhaps you have a device that has specific behaviors that are hard to simulate back at base. Now instead of spending your time with a simulation of the real world in order to develop your product features, perhaps you can just deploy a development device in the real world instead and use this device to develop and test your next feature on.
3. Maintenance efficiencies: Developers can react to in-the-field product issues as they happen, rolling out fixes and upgrades as a regular part of their agile development process.

From a Twilio IoT team perspective, it’s not often we (as engineers) get to work on something that feels truly game changing. This capability is unique (as far as we can tell – let us know if that’s wrong! ); it is patent pending, and it enables secure remote debugging of MCU scale devices over a cellular or Wi-Fi connection in a way that has never been done before.

If you want to experience the power of secure remote debugging, consider applying to receive a free STM32U585 Microvisor Dev Board!

Jonathan Williams is a Product Manager for IoT and Wireless at Twilio. Having worked in both the semiconductor industry as well as in cloud video communications, he has a focus on building developer friendly platforms that solve real world problems.