At Shopify we’re always looking for ways to increase the impact of our bug bounty program. As big believers in the importance of open source stewardship, we’ve been exploring ways to orient the impact of our bug bounty program towards the open source community. For example, by capitalizing on bug bounty reports to find opportunities to contribute to the open source projects we rely on.

In this post, I’ll share the details behind a recent contribution to the Rails library, inspired by a bug bounty report we received. I’ll go over the report and its root cause, how we fixed it in our system, and how we took it a step further to make Rails more secure by updating the default serializer for a few classes to use safe defaults. I’ll also share some tools we built that you can use in your own Ruby applications to spot similar issues. 

The Report 

We hosted a special event in 2021 for our private bug bounty program, Shopify Experiments. This event temporarily gave hackers access to source code for a subset of our projects. While we received a few interesting bug reports, one in particular directly led to open source contributions in Rails. At a high level, this report demonstrated the ability to escalate a leaked application secret for the application’s development environment into a Remote Code Execution (RCE) vulnerability. While limited in potential scope given that this is only relevant to development environments running locally on developer machines, leaking a development secret shouldn’t lead to RCE by default. We found that the root cause was a deserialization vulnerability. When deserialized, untrusted data has the potential to abuse application logic, deny service, or execute arbitrary code. 

Specifically, the report demonstrated the ability to use the leaked development secret to generate seemingly trusted data that would then be deserialized by the MessageEncryptor class provided by ActiveSupport in Rails. In versions up to 7.0.X, the MessageEncryptor class uses an unsafe serializer by default. This unsafe serialization strategy is provided by the Marshal library.

Above are the default values that the MessageEncryptor class uses to initialize itself in Rails versions 7.0.X and below, found here. Note that Marshal is the default serializer if none is provided.

The app in question was leveraging this default which meant that it was possible to provide seemingly valid, untrusted data to the MessageEncryptor to exploit the deserialization vulnerability. 

The Fix

After identifying the root cause, we quickly patched the issue in our app by providing JSON as the serializer to the MessageEncryptor instead of relying on the default. Next, we began investigating how we could make this better for all Rails apps, and how we could entirely deprecate the use of Marshal in our production environments. 

Making Rails Defaults More Secure

One of the great things about working at Shopify is that several top Rails contributors work here, making it easy to connect with an expert about potential contributions. After initial conversation with some of these experts, it became clear that there was no strong motivation to continue with Marshal as the default serialization strategy within Rails. We decided to write patches to change the default serializer to JSON for both the MessageEncryptor and MessageVerifier classes in Rails beginning in version 7.1.X. We also provided an upgrade path to allow existing apps to migrate the MessageEncryptor and MessageVerifier to JSON serialization. This provides a safer default given that JSON is a text-based serialization format as opposed to Marshal, which expects to convert collections of Ruby objects into a byte stream to be shared. You can find more details about the changes in the related pull requests on the Rails GitHub repository:

Switch ActiveSupport::MessageEncryptor Default Serializer to JSON

Switch ActiveSupport::MessageVerifier default serialization to JSON

Deprecating Marshal in Shopify Production Environments (And How You Can Do It Too)

After changing the default in Rails, we began exploring how to deprecate the use of Marshal in our production environments entirely. The patch below is similar to what we use in our own apps. It works by hooking the Marshal load and dump methods to verify that calls originating from gems to either of those methods are expected. If the call is unexpected, we raise an exception and instruct the recipient to use a safe alternative serializer or track the new call to Marshal.

This snippet provides a patch that can be applied in your own Ruby application. It is a mechanism to both actively track and deprecate the use of Marshal in your application.

Our patch runs in our local development environments and our continuous integration pipeline. The intent is to help prevent the introduction of new calls to Marshal while allowing us to maintain a list of gems still using it. We hope that by maintaining this list, we can encourage continued contributions to other open source libraries using Marshal as a default serializer. If you’re maintaining an application that is written in Ruby, we encourage you to adapt this patch to your project in an effort to help eliminate unnecessary Marshal serialization in open source software.

Making Open Source Safer for Everyone

If you’re a bug bounty program manager working with open source projects, consider keeping your eye out for untapped opportunities to make open source contributions based on the reports you receive. Not only is it a great way to increase the impact of your program and sharpen your development skills, it’s also a chance to give back to the open source programs we all benefit from.

If solving complex security and compliance challenges at scale sounds exciting to you, visit our Security Engineering careers page to find out about our open positions. Join our remote team and work (almost) anywhere. Learn about how we’re hiring to design the future together—a future that is digital by design