Shopify Functions enables customizing the business logic of Shopify’s back end with server-side WebAssembly. There are many benefits to using a WebAssembly environment to host these customizations such as low cold-start latency and robust security compared to other alternatives. However, WebAssembly poses new challenges when debugging. These challenges aren’t unique to Shopify Functions and are common to other uses of WebAssembly including other server-side use cases. One of these challenges is a less refined developer experience when using step debuggers.
Step debuggers allow developers to pause execution on certain lines of code given certain conditions, execute lines of code one at a time, descend and ascend the current callstack, and see the values of in-scope variables at a particular point in execution. This flexibility can reduce the amount of time it takes to find a problem compared to debugging with print or log statements or studying the code. Some examples of step debuggers are GDB, LLDB, and the Visual Studio Debugger.
There are a few approaches that can be taken to step debug code that will be running in a WebAssembly environment:
- compile the code to a native architecture and use an appropriate step debugger for the particular language that code was written in
- use a debugger like LLDB or GDB to run the WebAssembly code in a WebAssembly environment like Wasmtime
- use a dedicated WebAssembly step debugger
- leverage browser developer tools’ step debuggers.
There are tradeoffs to each approach with none being optimal for all cases so choosing the appropriate strategy depends on the context of the issue you’re trying to debug. However, some approaches are likely to work better than others at the present time.
Compiling to a Native Architecture
Compiling code to a native architecture and using an appropriate debugger like LLDB or GDB to execute that code is likely to present the best experience. Debuggers for native code render a higher fidelity representation of the values of variables compared to debuggers for WebAssembly. As well, developers already likely have prior exposure or easier access to help on how to use these debuggers.
Compiling to a native architecture may require changes to the code structure to stub or replace calls to functions that are external to the WebAssembly code. This is accomplished with the use of preprocessor directives, build configurations, or whatever facility the language exposes for including different code depending on the target architecture or enabled features. Fortunately, this is optional if the only external calls are to WebAssembly System Interface (WASI) calls since those calls have native support.
Here’s a contrived example where an implementation that would normally invoke WASI’s
fd_read function is substituted with an implementation that iterates over a hard-coded string for input:
When compiling to WebAssembly, the standard input stream provided by WASI is used to provide input. When compiling a native architecture, a reader over a hardcoded string is used instead. This same approach can be used to change the implementation of any function that would normally call an imported WebAssembly function to use a different approach when compiling to a native architecture.
To debug Rust code with a native debugger, run
cargo build and don’t specify the
--target flag, then run
rust-lldb or a different debugger on the file written to
target/debug/your_app_name. For example,
rust-lldb target/debug/rust-app. Type
help for a list of LLDB commands. Alternatively, a VS Code extension called CodeLLDB can be used to run and control the debugger with VS Code.
In some cases, it doesn’t make sense to execute native code under a debugger to find a problem. One case is if the programming language only supports compiling to WebAssembly which is the case with AssemblyScript or Grain. Another case is if the unexpected behavior can’t be reproduced when running the code natively. Luckily there are some options available.
Using LLDB with Wasmtime
One approach is to run the WebAssembly file with Wasmtime under LLDB. Wasmtime can transform the debug symbols included in a WebAssembly file when it compiles the WebAssembly file to native code. LLDB can then use these new debug symbols to step through the source code for the WebAssembly file. This can be useful for diagnosing issues that only appear in a WebAssembly environment. It can also be useful if the WebAssembly code relies on WebAssembly features that are not supported by browsers or WebAssembly-specific debuggers. The tradeoffs are that the representations of local variables may have a lower fidelity compared to native debugging and some time may be needed to learn how to use LLDB.
For an example of the variable rendering problem, on native, rendering a string looks like:
While using LLDB running Wasmtime, strings take a bit more work to render as expected:
Rust has more issues with variables not being available:
With respect to Rust, executing native code with
rust-lldb is able to render vectors and strings in a more legible way, but
rust-lldb typically doesn’t work well with the debug symbols generated by Wasmtime, and LLDB on its own doesn’t render Rust vectors or strings properly.
The Debugging WebAssembly page on Wasmtime’s website goes into detail about how to start debugging with LLDB and Wasmtime. Shopify’s function-runner also has support for debugging with LLDB.
Using WebAssembly Specific Debuggers
Another approach is to use a WebAssembly specific debugger like Wasminspect. The advantages with this approach are that it’s a standalone tool, includes features that are useful like directly inspecting linear memory, and is able to see the WebAssembly instructions when disassembling as opposed to native instructions. The tradeoffs are that tools in this space don’t have the same feature set as mature debuggers and also don’t render non-numeric variables in a helpful way. Missing features include an inability to set a breakpoint at a particular line of source code and lack of support for the debugger adapter protocol for integrating into visual editors.
Using Browser-based Debugging
The Chrome extension works reasonably well. It does suffer from the same problem of other non-native approaches that the renderings of local variable values is of a lower fidelity than native debuggers. For the second approach, wasm-dwarf works reasonably for doing this conversion and emscripten also includes a tool that performs a similar conversion. Unfortunately, the approach of converting debug symbols to a source map appears to cause the local variables to become unavailable in the debugger.
For example, when using the Chrome extension, C strings render properly but wide strings don’t:
And when using wasm-dwarf or emscripten, the local variables are unavailable:
Chrome also has a memory inspector available so it’s possible to view the contents of linear memory at particular addresses:
Other approaches to step debugging server-side WebAssembly may be available or become available as this is an area that’s actively evolving.
Here’s a table with a very brief overview of the trade-offs of each approach:
LLDB and Wasmtime
WebAssembly specific debugger
There are a variety of tools and approaches for debugging code running on a WebAssembly runtime. At the moment, using a debugger to execute the code in a native environment presents the most effective way to debug potentially problematic code in a variety of common cases. If the behavior only occurs when run in a WebAssembly engine, then using one of the alternative approaches listed will be helpful. This guidance may change in the future as tools continue to improve rapidly in the WebAssembly space. Though there may be many on-going challenges with tooling around debugging server-side WebAssembly, we will hopefully continue to see advances that smooth the developer experience.
- Jeff Charles: Approaches for Debugging WebAssembly on YouTube
- How Shopify Uses WebAssembly Outside of the Browser
Jeff Charles is a Senior Developer on Shopify's Wasm Foundations team. You can connect with him on LinkedIn and Github.
If building systems from the ground up to solve real-world problems interests you, our Engineering blog has stories about other challenges we have encountered. Visit our Engineering career page to find out about our open positions. Join our remote team and work (almost) anywhere. Learn about how we’re hiring to design the future together—a future that is digital by design.