To build great mobile apps at scale, you need a great developer experience. It’s no secret that we love creating purpose-built internal tools that make complex tasks virtually effortless. Think back to the last mobile project you worked on. What was the longest, most mundane aspect of your development workflow? Chances are, you probably remember waiting for the app to compile every time you wanted to test out a change.

At Shopify, manually testing every GitHub pull request (PR) we ship as part of code review is part of our culture. We call this process “tophatting,” a term that originated in a time when GitHub did not yet support PR reviews. Instead, Shopify used emoji to convey the state of a PR review, posting a tophat emoji to indicate that the changes were not only reviewed, but also tophatted locally to validate that they work as expected. As you might imagine, needing to constantly stash changes, switch branches, and wait for Xcode or Android Studio to build a project adds up to countless hours of lost time.

What if we leveraged the mobile infrastructure we already have in place and make tophatting mobile changes as easy as a click of a button, without even pulling the branch?

We created tooling that provides a seamless and interactive tophatting experience to contributors. We built a desktop macOS application called “Tophat” that lives in the system menu bar.

Making Tophatting Fun

The Tophat tool took on different forms over the years. At Shopify, we use another internal tool called Dev, a powerful CLI interface for bootstrapping the codebases we work on. Initially, we integrated a mobile tophatting tool directly into Dev so that mobile developers can use commands like dev ios tophat to test changes. This worked well, but it was implemented outside of the mobile domain which made long-term maintenance challenging and required CLI knowledge to use. In 2020, we built the first iteration of the desktop Tophat app, which featured a simple interface and support for launching builds from PRs. In the latest version, we spared no effort in building an experience that would make developers want to use it.

There are three reasons why this investment in our tooling was justifiable:

1. We’re actively shipping many large changes at a rapid pace as part of our ongoing effort to migrate Shopify Mobile to React Native. As such, we needed to make sure that we had a tool that would enable us to ship changes with confidence—ensuring that we maintain a consistent level of quality as we migrate. This included being able to test on real devices more frequently which wasn’t supported by previous versions of Tophat.
2. Analytics suggested that usage of the previous iterations of the Tophat tool was relatively low, indicating that we either weren’t testing as frequently as we should have been, or we were still compiling locally and potentially missing opportunities to speed up our workflows.
3. Creating a unified interface to iOS and Android devices in a single app would make it easy to build other mobile development tooling in the future, making Tophat the best way to communicate with mobile devices at Shopify.

Under the hood, Tophat exposes a lightweight HTTP web server that we link to from GitHub PRs, since GitHub does not support custom URL schemes. In the Shopify Mobile repository, we use a simple bot that runs as part of our standard continuous integration (CI) build process. We store artifacts produced by our CI pipelines in Google Cloud Storage (GCS) buckets, making it easy to retrieve the built artifacts at a later time. This means that we reuse the iOS and Android application builds for tophatting, completely eliminating the local compilation process. Our GitHub bot posts links to the GCS artifacts, allowing PR reviewers to install the build to any iOS or Android device—simulator or real device—with a single click.

Tophat parallelizes installation tasks like downloading the app and booting the device, allowing builds to install faster. Every aspect of the Tophat user interface was designed to be a joy to use, from fun animations on the server’s landing page to the main menu which feels right at home in the menu bar. The new Tophat transforms what used to be a repetitive task into an enjoyable experience—which in turn means that we get more eyes on the PRs we ship to our outward-facing apps.

We use Tophat for more than just tophatting pull requests. The core component to a React Native development environment is a JavaScript bundler known as Metro that continuously observes JavaScript changes and then updates the running application. This means that we can attach any application build to the local bundler, however, we occasionally need to update the build after making changes to native modules or other Swift or Kotlin code. Because Tophat makes it so easy to grab the latest native code changes, we integrated a new feature directly into the Tophat interface called “Quick Launch.”

Each time a developer starts a new piece of work, they click the app in the Quick Launch palette to grab the latest build and can get started with JavaScript changes with confidence that all native modules are up to date.

Getting Real on iOS

Testing on virtual (emulated) devices is usually a good representation of how an app might work, but the only way to be certain that the app looks, works, and feels as intended is to test it on a real iPhone, iPad, or Android device.

The Android development toolchain provides a tool called adb which allows us to install and launch apps remotely on both emulated and real devices. Xcode provides a tool called simctl for controlling and managing simulators, but no equivalent first-party tool exists for interacting with real iOS devices like adb does for Android.

Another challenge arises when trying to install and launch an app on a real iOS device. In order to do so, the app must be signed with one of three certificates: development, enterprise, or distribution. This made it difficult to reuse our existing CI artifacts since they were only set up for unsigned installation in simulators in the past.

We needed to solve two problems:

1. Find a tool that allows us to install apps to real iOS devices.
2. Sign builds destined for real devices with a certificate suitable for development purposes, ensuring that we minimize the amount of effort required by contributors to get up and running.

Exploring Apple’s APIs

In order to keep iOS devices secure, Apple imposes restrictions on how apps can be installed to devices. Apps can be installed using the built-in App Store, for internal distribution with direct links to the app bundle using enterprise signing, or using Xcode during the development process. This is great for keeping users safe, but it makes development of automated tooling challenging since there are no publicly documented APIs for installing and launching apps on devices remotely.

macOS uses a private framework called MobileDevice.framework to communicate with iOS devices attached to the system. The framework provides an abstraction for the communication protocols that are used to control devices. After some research, we found a number of open-source projects that fully reimplement these protocols thereby eliminating the need to reverse-engineer MobileDevice.framework’s private headers, but we found this to be overkill (and likely too fragile) for our needs.

In the end, we settled on a community-maintained utility already used by the React Native command line tool called ios-deploy. It provides a command line interface (CLI) interface to MobileDevice.framework by using a set of reverse-engineered headers. We bundle ios-deploy directly in Tophat so developers don’t need to install extra dependencies on their systems.

Simplifying Signing

With app installation addressed, we needed to find a way to automatically handle signing Tophat artifacts so that they could be installed to real iOS devices. There are three common types of certificates used to sign iOS applications:

1. Development
2. Enterprise
3. Distribution

Development and Enterprise certificates (which contain the iPhone Software Development Signing extension) permit the installation of apps to devices outside of the App Store, but there are specific restrictions as to when they can be used.

Development certificates sign app bundles for a set period of time, with validity of the signature expiring after about a week. Development signing also requires that the app bundle contains an embedded provisioning profile that specifies which devices the app can be installed to. This means that the target device’s unique device identifier (UDID) must be registered in the Apple Developer Portal either manually by the developer or automatically by Xcode. Development signing makes debugging easy, but initial setup more difficult.

Enterprise signing is only available to Apple Developer Enterprise Program members like Shopify. Enterprise certificates sign app bundles for indefinite use and permit installation on an indefinite number of devices. Enterprise signing does not require any provisioning profiles to be included. However, signing an app bundle with an enterprise certificate disables the get-task-allow entitlement which is required for remotely launching and debugging, which means that we wouldn’t be able to use an enterprise-signed app for a few types of development tasks. Enterprise signing makes setup easy, but debugging more difficult.

Distribution certificates are meant for App Store submissions and can’t be used for anything else.

Xcode internally communicates with some internal Apple Developer Portal APIs to provision connected devices, manage signing certificates, and automatically sign application bundles. At first, we attempted to prototype a version of Tophat that communicated with these APIs in an effort to make Tophat just as seamless. Due to the lack of public documentation, however, we quickly discovered that this is an incredibly fragile process and weren’t able to implement a solution that produced consistent results, especially when it came to authentication with these services. Additionally, designing the best algorithm to manually re-sign an application bundle was a challenge as every application can have a completely different arrangement of Executables, Frameworks, and App Extensions.

Fortunately, there was another option we hadn’t explored. Across Shopify, we use Fastlane’s match utility to manage all signing certificates automatically. Shopify’s Mobile Tooling team had already set up a YAML-based API for automatically managing code signing, so all we needed to do was configure an Apple Development certificate and sign our application bundle with it in CI. This means that developers will need to provision their devices once manually, but the certificates and provisioning profiles will then be updated automatically. It may not be as seamless as allowing Tophat to handle the entire process, but it’s guaranteed to always work.

To help with troubleshooting, we integrated checks into Tophat that can notify developers if their device isn’t provisioned or if the app bundle is missing entitlements. With this extra guidance, developers know exactly what they need to do to resolve common issues. Tophat also checks if the app is signed using a certificate with the iPhone Software Development Signing extension object identifier (OID) by using a few APIs from the Apple Security framework:

With application installation implemented and code signing addressed, Tophat offers a physical device development experience much like Xcode’s or Android Studio’s that allows us to install builds to real iOS devices over USB (or even Wi-Fi) and launch them automatically with a single click. Just like magic.

Onboarding Projects

Tophat began its journey as a tool for Shopify Mobile contributors, but we wanted to make it the best way to install builds to devices for any mobile project at Shopify. We needed to design it in a way that required no manual setup tasks for developers that already have a functioning development environment.

One of the most useful features of the aforementioned Dev tool is a configurable command called dev up that automatically installs everything one might need to get to work on a project. So we leveraged up to automatically install Tophat for all mobile developers without any manual setup.

To make sure Tophat has a truly invisible setup process, we also needed a way to preconfigure its settings and integrate it with our other tooling. To do so, we built a command line utility called tophatctl that we can call from dev up steps in various projects. This utility can be used to populate the Quick Launch palette as well as launch apps.

We also built an onboarding flow that developers can use to make sure they have everything they need to get the most from Tophat, like setting up Xcode, Android Studio, the Google Cloud SDK used for downloading our CI artifacts, and the tophatctl command line helper.

In most cases, developers are greeted with all green check marks thanks to Dev which automatically configures all dependencies, but we included installation instructions in case any manual configuration is needed.

One of the key tools we use for managing the development and release of mobile apps at Shopify is an internal service called Shipit Mobile. Shipit Mobile is a web application that integrates with GitHub and Slack to facilitate automated mobile application releases. It does other things like creating internal builds from PRs and specific commits on demand. We call these builds “snapshot” builds. Snapshot builds can also be created on a nightly basis. We integrated Tophat links directly into Shipit Mobile so that all projects already using Shipit Mobile gain support for installing snapshot builds to virtual or real devices with a single click.

What is the best way to get powerful tools into the hands of developers that need them most? The answer is to make them available by default. By integrating Tophat into all the tools we already had and supporting a fully unsupervised installation and setup process, everyone at Shopify that works on a mobile project gets Tophat support for free without even lifting a finger.

Looking to the Future

We’ve been tracking daily usage statistics for Tophat by capturing analytics and visualizing them in a dashboard. Since we shipped these latest improvements, the number of times Tophat is used each day has doubled. Developers love the new experience and we’ve received overwhelmingly positive feedback. There are endless possibilities for further improving Tophat, with two in particular that can make it even more essential in any mobile development workflow.

1. Device Management: To fully eliminate the need for developers to open Xcode or Android Studio as part of their day-to-day work, we would love to integrate device management tools directly into Tophat—making Tophat the go-to native mobile development tool.
2. Bundled Dependencies: Tophat still relies on a fully functioning mobile development environment. Not everyone at Shopify is a developer, meaning that UX or Product teams still need to set up a development environment to use Tophat. Shopify’s Dev tool can assist in the installation of all the prerequisite toolchains, but we’re hoping to bundle more development tools directly into Tophat so the only step to getting started for non-developers is to drag Tophat to your Applications folder.

Tools like Tophat are key to building an efficient mobile development workflow at scale. By investing time into crafting a polished internal tool, we’re able to ship new features faster by significantly reducing the amount of time otherwise spent on recompiling the same codebase, resulting in significantly shorter feedback loops.

Since Tophat makes tophatting enjoyable, provides useful features for real-world testing, and requires virtually no steps to get up and running, it’s the ultimate multitool for shipping great mobile features and making commerce better for everyone.

---

We all get shit done, ship fast, and learn. We operate on low process and high trust, and trade on impact. You have to care deeply about what you’re doing, and commit to continuously developing your craft, to keep pace here. If you’re seeking hypergrowth, can solve complex problems, and can thrive on change (and a bit of chaos), you’ve found the right place. Visit our Engineering career page to find your role.