Alex Page and Emma Orhun
The Polaris team creates tools, education and documentation that helps unite teams across Shopify to build better experiences. We created Polaris, our design system, and continue to maintain it. We are a multidisciplinary team with a range of experience. Some people have been at Shopify for over 6 years and others, like me, are a part of our Dev Degree program.
In 2020, myself, Alex, and the Polaris team were responsible for updating the design language across the Shopify product.
This was a huge undertaking that consisted of updating illustrations, colors, shadows, behaviour and shape throughout the Shopify product and its applications. A huge part of this work was updating the illustration style. Our illustrations were lighthearted and fun. However, our product needed to be direct and literal in it’s representation of information.
The previous design language illustration style on the left and the updated illustration on the right
As Shopify scaled, our usage of illustrations became fragmented across over 6000 repositories. Developers and illustrators faced roadblocks while updating this many illustrations at scale. We built, @shopify/get-repo-images, an open source solution that enables our illustration team to easily find illustrations and their references in code across multiple repositories. It gives them the ability to sort by file size or total references, and filter by specific repositories.
The Problem With Illustrations at Scale
Our illustration team started this update by going through the Shopify admin, documenting the illustrations and URLs in a large spreadsheet. This would then be handed over to developers to replace the images in our product. Our illustrators did the work, however there were gaps.
It wasn’t obvious where in the code illustrations were referenced, or how many times it was used across the application. Illustrations existed in multiple repositories or were used multiple times in the one codebase. There wasn’t a consistent file type, sometimes they were PNG, SVG, PDF, or WebP. Images also came from API requests that lived with server code. Looking at a page in the Shopify admin, it was difficult to determine which repository’s code an illustration file belonged to, or if it was reused across the application… It was gnarly.
Shopify is constantly deploying and making changes to our illustrations. An illustration found one week could change or not exist the next. Our illustrations also appear on specific application states or flows. For example, a shop without discounts shows a discount illustration or a homecard for increased return rates. We were low on confidence that we could find all the illustrations while browsing the admin.
We started with a list of repositories to search and the image file extensions (png, svg, and webp). We experimented with the GitHub API to find the different files with matching extensions across multiple repositories. This wasn’t great. Finding their usage in code wasn’t possible without hitting API limits or false positives.
There are a lot of considerations for searching code on GitHub. The most limiting being the wild card characters being removed from search: when trying to find hello+world.png in code GitHub searches for “hello world png”. The search also has a maximum of two fragments per file, but there may be more results.
Internally, we didn’t have any tools set up to monitor image downloads. Finding images with the highest visibility and which line of code they were referenced would help us easily prioritise which files to remove or update.
We needed a tool to:
- Find images in a repository that match our extensions
- Find the line number that the image is referenced in code
- Create an API with the results
- Create a website to browse, filter, and sort the results
Trial and Error
We started with a monorepo. Using Node.js we wrote a script to git clone down multiple repositories. Once we had the code locally we used globby to find the images with matching extensions. Finally, we searched every file and line of code for the images usage. We saved the results in a JSON file that we made accessible through json-server, and combined it with a lightweight Preact application to browse, sort, and filter the results. This allowed our illustrators to visually search our images to quickly find what they were looking for.
The application was built quickly and had some major flaws. It had to be rebuilt daily to have up to date information on illustration usage. Each time it was rebuilt, it needed to clone multiple repositories and search millions of lines of code. Bottlenecks started to appear and Node.js fs (file system) library was not performant in completing these large operations. We were running multiple scripts, maintaining a monorepo, and solving complex performance problems with Node.js. Even after a thorough optimization, it still took over 35 minutes to search 24 repositories.
The experience for our illustrators was suboptimal. The data was inaccurate, it couldn’t be run locally without technical knowledge, and adding new repos to search was complicated. Embracing this failure, it was time to give it a go and get out of our comfort zone.
Go Was the GoTo
As front-end developers we write a lot of JavaScript. Node.js was easy for us to pick up and prototype with, however the experience wasn’t meeting our expectations. Staying in our comfort zone, we had created a worse experience that wasn’t accessible. The performance was slow and made it hard to quickly get accurate information. As a team that often stresses the details of accessibility and performance to build the best experience we weren’t happy with the result.
We had to solve the hard problem. To start we reframed the problem to “How can anyone generate a website to browse visual assets from any GitHub repository?”
Through research we explored different ways we could remove complexity from our initial solution. We took inspiration from libraries like esbuild where thousands of file based operations are performed from one npx command. This would allow our users to perform similar Node.js file system functionality performantly with Go while still only needing Node.js installed.
Why Go? Go, allowed us to bring together a Preact website, json-server API, multiple Node.js scripts for finding images, and their usage into one executable file. The code that read millions of lines of code and found multiple illustration files now ran in parallel across multiple cores.
We removed the monorepo entirely, moving the cloning, searching, building, and starting the application logic into one Go binary file. We moved our Preact website components to Next.js. We could now remove json-server and create a simple API with the data from the JSON file.
The performance improvements were breathtaking. We reduced the previous 35+ minute search and website build down to 8 minutes. The only bottleneck was the amount of time it took to clone 24 monolithic repositories. Teams across Shopify now use @shopify/get-repo-images to quickly find images, remove them, or update them.
Getting the team out of our JavaScript comfort zone meant we learned a new framework and built a better experience for our users.
get-repo-images is Open Source
We hope that @shopify/get-repo-images will be useful for anyone maintaining images at scale. It’s an extremely fast repository crawler to find images and their usage across multiple repositories. It is written in Go for performance and can be run through NPM. Give it a go today!
$ npx @shopify/get-repo-images -repo golang/website
Search complete found 76 images
Your site is building, please wait...
Browse, sort and filter your images http://localhost:3000
@shopify/get-repo-images was built to be used by anyone regardless of their technical background so that illustrators and developers alike can investigate the state of illustrations within a repository.
The Polaris team will be maintaining the get-repo-images and we’re stoked to see how you will use it. You are also welcome to report bugs and open pull requests in accordance with our contribution guidelines.
If you are interested in solving challenging and creative technical problems that evolve our experiences. We are hiring front-end developers.
Alex Page is a front-end manager on the Polaris team. He builds communities and creates systems with modern technology and thoughtful design. He is obsessed with user experience and writes code that connects people to pixels. Follow Alex on Twitter.
Emma Orhun is a Dev Degree intern on the Polaris team as well as a full time computer science student. She loves solving creative problems by bridging the gap between code, design and art. On the side she is a maker of all things and works as a freelance artist. Follow Emma on Twitter.