A core part of Shopify’s DNA is that we make sure to foster a QA culture. For our mobile apps, that culture depends on a small layer of end-to-end (E2E) tests at the top of the test pyramid: flows that drive the real app the way a merchant would, running as blocking CI on every pull request. The pyramid works really well, as long as the suite is trustworthy.
The Shopify mobile app (our largest) had gotten to the point where it was blocking more good PRs than bad ones. Tests were getting flaky because screens might take an extra second to load and it got so bad that we had to pull the E2E suite from our PR checks entirely.
This is the story of how we fixed it, hitting 98% test stability—up from 50% using the old API.
The problem with our old setup
Since 2023, the Shopify app’s E2E tests have run on Appium through WebdriverIO, using React Native Test IDs to find elements. That flexibility turned into a liability: Appium gave us low-level control, but nothing enforced good testing patterns. After tapping one element, tests could immediately try to tap the next before the new screen had rendered, which caused “element not found” failures.
The fix was to explicitly wait for elements to appear, but it was just as easy to drop in a pause(1000) that seemed to work locally and usually passed in CI (until a screen took just a bit long to load and it failed!). Over time, those shortcuts piled up into flaky tests. The app was fine, but the test suite kept failing.
Even when tests passed, they were often asserting the wrong thing: that a node existed in the component tree, not that a merchant could actually see or use it. We were testing implementation details instead of user experience.
The bottom inset is bad and the last cell is obscured. Our old API would have been able to click on this cell and incorrectly pass.
We’d seen the pattern before: flakiness grows and we then devote immense resources just to keep the suite green. The problem wasn’t that we were bad at bailing out the suite; it was that the framework itself kept creating the same failures. No amount of cleanup would solve that. We had to fix the underlying system.
The rebuild
We stopped trying to patch Appium and built an opinionated wrapper around it. It’s a two-parter: a strict, builder-style API that makes flaky tests hard to write, and computer vision that finds elements the way a user does, not by crawling the view hierarchy. Under the hood, Appium is still driving the device. Developers just don’t see it anymore, and they can’t reach past the wrapper to do the things that sank the old suite.
A builder-style test API
We write tests against a builder that only exposes actions we’re confident won’t flake:
A few things are deliberate here:
- Every step carries an assertion. You can’t tap, wait, or type without declaring what the screen should show afterward. If the app leaves the expected state, the test fails at the step where reality diverged, not four actions later when something downstream breaks.
-
Reusable slices.
logIntoAppis a named step sequence that any test in the app can pull in. -
Escape hatches are prefixed
UNSAFE_. Options exist that bypass the guardrails (like custom timeouts or script injection), but they’re named to discourage reaching for them.UNSAFE_timeoutInSecondsin a test is a signal for review. - Readable enough for AI agents to write. The surface area is small and the grammar is predictable, which means both humans and AI tools produce correct tests on the first try more often.
Computer vision instead of Test IDs
The bigger change sits one layer down. Every step takes a screenshot and finds its target visually, the way a merchant does: scan the screen for “Save” or a plus icon, then tap. PaddleOCR handles text; OpenCV matches screenshots against SVGs from our Polaris design system. Test IDs still work as a fallback for screens where generated content makes visual matching unreliable, but they’re opt-in through an UNSAFE_testID field. The naming itself discourages using them.
The real win is authoring speed. With Test IDs, adding a step meant opening an inspector, drilling into the component tree to find or add a testID, then wiring it up in the test. With computer vision, you look at the simulator, see “Save”, and write touch({ text: 'Save' }). And that’s the whole loop. AI agents get the same advantage: the grammar maps one-to-one with what’s on screen, so “write a test that creates a product” turns into correct code on the first try with no codebase knowledge required.
Every run produces an annotated video of what each step was looking for and where it looked. When a test fails, you see exactly why: which text OCR was searching for, what it found instead, where it tapped. Most failures diagnose themselves in a few seconds of video, with no rerun needed.
A CLI that runs the same tests everywhere
The runner is a single command that works the same way on a laptop, on a CI emulator, or when calling to a real device in a remote device farm:
This runs every test file matching logout on the iOS devices declared in the RemoteDeviceFarm config. Swap --runner remote-device-farm for --runner local and the same command runs on a simulator on your machine.
The migration, in numbers
A few weeks after promoting the new API into blocking CI on the Shopify app: 98% test stability, measured as individual test successes divided by total runs—up from 50% using the old API. Remaining test failures are largely from what we would expect: occasional network failures and simulators failing to boot properly.
We also built a pre-promotion flakiness gate. Before a new test is allowed into the blocking suite, a dedicated pipeline runs it multiple times and rejects it if it fails above a set threshold.
What’s next
We’ve validated this framework on our biggest app and are now exploring adopting it in our other apps.
For years we believed mobile E2E testing was inherently flaky, and that the best we could do was manage the flakiness. A lot of it turned out to live in the API, not in the tests themselves. When we replaced the API (using the principles of asserting at every step, finding elements the way a user does, and refusing the footguns), we found that a suite that couldn’t stay in blocking CI runs at 98% stability on two platforms.
As AI increases engineering velocity, frameworks like this become even more valuable. They allow teams to move faster without losing confidence in what they ship.
I want to build this myself
Maybe your CI also suffers from a flaky, hard-to-use, hard-to-interpret end-to-end testing framework. Here’s what we did to get to a better place:
- Limit the API to a small set of essential commands. Deeplink, swipe, type, touch, assert, and relaunch app.
- Use computer vision to interact with text and icons on screen. We evaluated many open-source OCR libraries, and PaddleOCR was the clear winner. For icon matching, we use OpenCV. We convert everything to grayscale, then match the icon (and its color-inverted variant) across multiple size variations until we find a match. For duplicate elements, we use adjacencies (i.e., “icon1 to the left of icon2”) to disambiguate them.
- Require every action to include an assertion or refutation. This prevents tests from progressing without verifying that anything actually happened. We also validate our assertions: an assertion must be false before the action, and true afterwards.
- Establish test stability before merging. A test is only allowed to pass if it has proven itself stable across multiple runs.
Now that the framework makes reliable tests the easiest ones to write, E2E testing does the job it was built for: catching bad changes without standing in the way of good ones.