Our engineering blog is the home for all our stories sharing technical knowledge and lessons learned. But that's only part of the story: we hire passionate people who love what they do and are invested in mastering their craft. Today we launch "The Side Hustle," an occasional series highlighting some of the side projects from our devs while off the Shopify clock.
When Gabriel O'Flaherty-Chan from our Frenzy team noticed quadcopter controllers on mobile mostly translated analog controls to digital, he took it upon himself to find a better design.
7 minute read
For under $50, you can get ahold of a loud little flying piece of plastic from Amazon, and they’re a lot of fun. Some of them even come with cameras and Wi-Fi for control via a mobile app.
Unfortunately, these apps are pretty low quality — they’re unreliable and frustrating to use, and look out of place in 2017. The more I used these apps, the more frustrated I got, so I started thinking about ways I could provide a better solution, and two months later I emerged with two things:
1. An iOS app for flying quadcopters called SCARAB, and
2. An open-source project for building RC apps called QuadKit
What’s SCARAB?
Quadcopters are notorious for making an obnoxious, high pitched buzzing sound while in flight, so it only made sense to name the app after the flying scarab, a beetle as majestic as it is loud.
One of the biggest issues I had with existing apps was the general disregard for touch input as a completely different interaction model from physical input. Rather than rethink from this perspective, designers of these apps more or less ported as much as they could from hardware RC transmitters:
A typical RC transmitter on the left, and the generic app on the right
Physical analog sticks offer the flexibility of having a z-axis (ie. depth), allowing everything to be crammed together much tighter. But without depth for tactility, it’s nearly impossible to use the software controller without looking at it. So I started by sorting each control by what I considered most important, giving priority to controls which required constant input (such as three dimensional movement), and reprioritizing controls which didn’t need to be in the forefront at all times (such as calibration).
After figuring out input priorities, I sketched out on graph paper a few layouts that felt comfortable and made it easier to access important movement controls without looking at the screen, pushing back less important controls that weren’t absolutely necessary. For example, the on/off switch was made intentionally small and pushed into the top left corner because it’s really only important right when the quadcopter is turned on, and then again when it’s landed and will be turned off. Seems like an obvious decision, but I noticed common controls schemes having on/off (or a kill switch) front and center at all times.
The UI system I came up with was extremely simple, relying on a dominant tone for indicating connection status (green for connected and gray for disconnected), and large touch-friendly control pads with varying degrees of opacity for suggesting non-existent depth. Yes, I borrowed heavily from how elevation is often portrayed in cartography.
Each control was made logarithmic (ie. exponentially higher sensitivity at larger values) in order to make it easier to provide both fine and coarse grain control. While subtle, the UI indicates this by leveraging the depth effect, positioning higher “altitude” rectangles closest to the user’s touch, and lower altitude rectangles closer to the control pad’s edges.
Shortly after building SCARAB for iOS, I had a small, crazy idea: what would it be like to fly a quadcopter from my Apple Watch? Having already worked with watchOS to build a FPS game several months prior, I figured it would be worth a shot.
And so, I spent an evening hacking together a watchOS extension which relied on the accelerometer for pitch and roll, digital crown for yaw, and had one button on screen for thrust.
The largest point of failure (and one I probably should’ve investigated beforehand) was Bluetooth. Because the most obvious way for watchOS to communicate with the quadcopter was through the iOS app via the WatchConnectivity framework (ie. Bluetooth), I discovered too late that the feedback loop was far too long to make the experience anything but frustrating.
The next logical approach was to remove WatchConnectivity entirely, and communicate with the quadcopter directly from the watch. But after a bit of research, it became apparent that Apple flat out discourages any sort of networking from the device, providing next to no documentation.
All that being said, I’m optimistic there’s a working solution somehow, but it’ll definitely take more than an evening’s worth of research!
The next half of this article will get slightly more technical, focusing on the tools used to build SCARAB, and how I packaged and open sourced them as QuadKit.
What’s QuadKit?
If you do a search on Amazon for “quadcopter” and filter by under $50, you’ll see nearly endless results for products which barely vary from one another, either by color scheme, or slight differences in frame design.
Now if you compare the apps that ship with each model, you’ll see they’re nearly identical. This suggests the hardware all comes from the same manufacturer, meaning unlocking one model may actually unlock many. QuadKit was built for communicating with quadcopters, but with a generic approach not coupled to the requirements of any one brand or model, should my one-size-fits-all hypothesis fail. By making it open source, there’s flexibility for developers to provide their own support for individual models.
My first approach was to reverse engineer the mobile apps to see how they worked.
Starting with what I knew, I attempted to get a sense of the app’s structure using class-dump to get a list of class interfaces. By SSH’ing into a jailbroken device, I could use various tools like dumpdecrypted and Clutch to produce a decrypted binary while the app ran in memory, then use class-dump to get a comprehensive list of headers. This gave me a decent sense of the app’s architecture.
By hosting an lldb-server to remotely debug the app while it ran in memory, I was able to use the class-dump headers to know where to send messages and where to record results. This gave me a sense of where and how data was sent and received.
Disassembling the decrypted binary using Hopper then allowed me to view something that kind of resembled source code. This revealed the front-end being written in native code, but all the useful networking logic in C (and possibly Assembly, but that may have just been part of the disassembly process). The result was too garbled to be of much use, mostly nightmarish stuff like this:
After spending about a week trying to rewrite everything in Objective-C (and not getting very far), I decided to do some research into looking for an approach which wouldn’t require source code. Unsurprisingly, the hardware hacking community has been all over toy quadcopters for a few years now, and I found a huge number of quality resources basically documenting what I was trying to do, usually using an Arduino. By finding the right radio frequency then reverse engineering the communication protocol, Michael Melchior was able to use a Wii remote to fly his toy quadcopter.
Thanks to the immense amount of available documentation and guidance, I was able to follow Jim Hung’s article on Reverse Engineering a Hubsan X4 Quadcopter (as well as his super handy protocol specification) to flesh out a Swift framework which followed the same protocol, but, instead of communicating over radio, sent data over socket connections.
An interesting point which reenforces the one-size-fits-all hypothesis from before is that although everything he wrote was for a completely different brand of quadcopter, it aligned perfectly with the data I was seeing. If this actually turns out to be the case, I imagine there could be a lot more potential for others to use QuadKit to help build quadcopter software.
What’s Next?
QuadKit is publicly available on Github, with step-by-step instructions for how to contribute, as well as how to record data from your quadcopter using free tools. In the future, I’d like to work on adding first-person video support, specifically for models which come with cameras, which could make the platform even more compelling.
As for SCARAB, I’d like to re-investigate watchOS in the near future, as there may be a lot of potential for interesting new ways to provide controller input outside of dual analog stick-style control schemes. For the time being, it can be downloaded for free now from the App Store.
If you have any questions or liked/disliked something here, feel free to post a comment here or on Twitter at @_gabrieloc!