Shopify Functions enables customizing the business logic of Shopify’s back end with server-side WebAssembly. There are many benefits to using a WebAssembly environment to host these customizations such as low cold-start latency and robust security compared to other alternatives. However, WebAssembly poses new challenges when debugging. These challenges aren’t unique to Shopify Functions and are common to other uses of WebAssembly including other server-side use cases. One of these challenges is a less refined developer experience when using step debuggers.

Step debuggers allow developers to pause execution on certain lines of code given certain conditions, execute lines of code one at a time, descend and ascend the current callstack, and see the values of in-scope variables at a particular point in execution. This flexibility can reduce the amount of time it takes to find a problem compared to debugging with print or log statements or studying the code. Some examples of step debuggers are GDB, LLDB, and the Visual Studio Debugger.

There are a few approaches that can be taken to step debug code that will be running in a WebAssembly environment:

• compile the code to a native architecture and use an appropriate step debugger for the particular language that code was written in
• use a debugger like LLDB or GDB to run the WebAssembly code in a WebAssembly environment like Wasmtime
• use a dedicated WebAssembly step debugger
• leverage browser developer tools’ step debuggers.

There are tradeoffs to each approach with none being optimal for all cases so choosing the appropriate strategy depends on the context of the issue you’re trying to debug. However, some approaches are likely to work better than others at the present time.

Compiling to a Native Architecture

Compiling code to a native architecture and using an appropriate debugger like LLDB or GDB to execute that code is likely to present the best experience. Debuggers for native code render a higher fidelity representation of the values of variables compared to debuggers for WebAssembly. As well, developers already likely have prior exposure or easier access to help on how to use these debuggers.

Using a language-native debugger is also the only realistic approach when working with interpreted languages like JavaScript or Ruby. When they’re run in a WebAssembly environment, the code is executed by an interpreter that has been compiled to WebAssembly instructions as opposed to the interpreted code itself being compiled to WebAssembly instructions. So there’s no way to map what source-level instruction is executing without the active cooperation of the language interpreter.

Compiling to a native architecture may require changes to the code structure to stub or replace calls to functions that are external to the WebAssembly code. This is accomplished with the use of preprocessor directives, build configurations, or whatever facility the language exposes for including different code depending on the target architecture or enabled features. Fortunately, this is optional if the only external calls are to WebAssembly System Interface (WASI) calls since those calls have native support.

Here’s a contrived example where an implementation that would normally invoke WASI’s fd_read function is substituted with an implementation that iterates over a hard-coded string for input:

When compiling to WebAssembly, the standard input stream provided by WASI is used to provide input. When compiling a native architecture, a reader over a hardcoded string is used instead. This same approach can be used to change the implementation of any function that would normally call an imported WebAssembly function to use a different approach when compiling to a native architecture.

To debug Rust code with a native debugger, run cargo build and don’t specify the --target flag, then run rust-lldb or a different debugger on the file written to target/debug/your_app_name. For example, rust-lldb target/debug/rust-app. Type help for a list of LLDB commands. Alternatively, a VS Code extension called CodeLLDB can be used to run and control the debugger with VS Code.

In some cases, it doesn’t make sense to execute native code under a debugger to find a problem. One case is if the programming language only supports compiling to WebAssembly which is the case with AssemblyScript or Grain. Another case is if the unexpected behavior can’t be reproduced when running the code natively. Luckily there are some options available.

Using LLDB with Wasmtime

One approach is to run the WebAssembly file with Wasmtime under LLDB. Wasmtime can transform the debug symbols included in a WebAssembly file when it compiles the WebAssembly file to native code. LLDB can then use these new debug symbols to step through the source code for the WebAssembly file. This can be useful for diagnosing issues that only appear in a WebAssembly environment. It can also be useful if the WebAssembly code relies on WebAssembly features that are not supported by browsers or WebAssembly-specific debuggers. The tradeoffs are that the representations of local variables may have a lower fidelity compared to native debugging and some time may be needed to learn how to use LLDB. 

For an example of the variable rendering problem, on native, rendering a string looks like:

While using LLDB running Wasmtime, strings take a bit more work to render as expected:

Rust has more issues with variables not being available:

With respect to Rust, executing native code with rust-lldb is able to render vectors and strings in a more legible way, but rust-lldb typically doesn’t work well with the debug symbols generated by Wasmtime, and LLDB on its own doesn’t render Rust vectors or strings properly.

The Debugging WebAssembly page on Wasmtime’s website goes into detail about how to start debugging with LLDB and Wasmtime. Shopify’s function-runner also has support for debugging with LLDB.

Using WebAssembly Specific Debuggers

Another approach is to use a WebAssembly specific debugger like Wasminspect. The advantages with this approach are that it’s a standalone tool, includes features that are useful like directly inspecting linear memory, and is able to see the WebAssembly instructions when disassembling as opposed to native instructions. The tradeoffs are that tools in this space don’t have the same feature set as mature debuggers and also don’t render non-numeric variables in a helpful way. Missing features include an inability to set a breakpoint at a particular line of source code and lack of support for the debugger adapter protocol for integrating into visual editors.

Using Browser-based Debugging

A final approach to consider is using a browser-based debugger. There are two sub-approaches for how to go about this. One approach to consider is using the Chrome DevTools team’s beta C/C++ DevTools Support (DWARF) Chrome extension. And the other approach is to use a tool to convert the debug symbols included in a WebAssembly file into a sourcemap and modify the WebAssembly file to include a reference to that source map. Some HTML and JavaScript needs to be written to instantiate the WebAssembly module and invoke the appropriate WebAssembly function. As well, if the WebAssembly module uses WASI, those calls either need to be stubbed or a browser-compatible implementation of WASI needs to be made available to the module when instantiating it.

The Chrome extension works reasonably well. It does suffer from the same problem of other non-native approaches that the renderings of local variable values is of a lower fidelity than native debuggers. For the second approach, wasm-dwarf works reasonably for doing this conversion and emscripten also includes a tool that performs a similar conversion. Unfortunately, the approach of converting debug symbols to a source map appears to cause the local variables to become unavailable in the debugger.

For example, when using the Chrome extension, C strings render properly but wide strings don’t:

And when using wasm-dwarf or emscripten, the local variables are unavailable:

Chrome also has a memory inspector available so it’s possible to view the contents of linear memory at particular addresses:

Despite some rough edges in this area, browser-based debugging may present the most accessible debugging experience in the future given some work on tools for generating the HTML and JavaScript to power it and a local server to serve those files. The reason it may become the most accessible experience is the tooling to debug is less complicated to set up and use compared to other options.

Other approaches to step debugging server-side WebAssembly may be available or become available as this is an area that’s actively evolving.

Wrapping Up

Here’s a table with a very brief overview of the trade-offs of each approach:

	Pros	Cons
Native	Most useable rendering of variable values Familiar debugging experience Only option for interpreted languages	Cannot reproduce behaviors specific to WebAssembly May require stubbing external function calls
LLDB and Wasmtime	Reasonable feature support Most similar to production execution environment	Less useable rendering of variable values
WebAssembly specific debugger	Standalone tool	Missing features Less useable rendering of variable values
Browser debugger	Familiar debugging experience	Requires an HTML and JS harness May require stubbing external function calls Less useable rendering of variable values

 

There are a variety of tools and approaches for debugging code running on a WebAssembly runtime. At the moment, using a debugger to execute the code in a native environment presents the most effective way to debug potentially problematic code in a variety of common cases. If the behavior only occurs when run in a WebAssembly engine, then using one of the alternative approaches listed will be helpful. This guidance may change in the future as tools continue to improve rapidly in the WebAssembly space. Though there may be many on-going challenges with tooling around debugging server-side WebAssembly, we will hopefully continue to see advances that smooth the developer experience.

Additional Resources

• Jeff Charles: Approaches for Debugging WebAssembly on YouTube
• How Shopify Uses WebAssembly Outside of the Browser

If building systems from the ground up to solve real-world problems interests you, our Engineering blog has stories about other challenges we have encountered. Visit our Engineering career page to find out about our open positions. Join our remote team and work (almost) anywhere. Learn about how we’re hiring to design the future together—a future that is digital by design.