Over the past decade, the internet and mobile devices became the dominant computing platforms. In parallel, the family of software architecture styles that support distributed computing are the ways we build systems to tie these platforms together. Styles fall in and out of favor as technologies evolve and as we, the community of software developers, gain experience building ever more deeply connected systems.
If you’re building an app to integrate two or more systems, you’ll need to bridge between two different domain models, communication protocols, and/or messaging styles. This is the situation that our team found itself in as we were building an application to integrate with Amazon’s online marketplace. This post talks about some of our experiences integrating two well-established but very different commerce platforms.
Shopify is a multi-channel commerce platform enabling merchants to sell online, in stores, via social channels (Facebook, Messenger and Pinterest), and on marketplaces like Amazon from within a single app. Our goals for the Amazon channel were to enable merchants to use Shopify to:
- Publish products from Shopify to Amazon
- Automatically sync orders that were placed on Amazon back to Shopify
- Manage synced orders by pushing updates such as fulfillments and refunds back to Amazon
At Shopify, we deal with enormous scale as the number of merchants on our platform grows. In the beginning, to limit the scale that our Amazon app would face, we set several design constraints including:
- Ensure the data is in sync to enable merchants to meet Amazon’s SLAs
- Limit the growth of the data our app stores by not saving order data
In theory, the number of orders our app processes is unbounded and increases with usage. By not storing order data, we believed that we could limit the rate of growth of our database, deferring the need to build complex scaling solutions such as database sharding.
That was our plan, but we discovered during implementation that the differences between the Amazon and Shopify systems required our app to do more work and store more data. Here’s how it played out.
Integrating Domain Woes
In an ideal world, where both systems use a similar messaging style (such as REST with webhooks for event notification) the syncing of an order placed on Amazon to the Shopify system might look something like this:
Each system notifies our app, via a webhook, of a sale or fulfillment. Our app transforms the data into the format required by Amazon or Shopify and creates a new resource on that system by using an HTTP POST request.
Reality wasn’t this clean. While Shopify and Amazon have mature APIs, each has a different approach to the design of these APIs. The following chart lists the major differences:
| Shopify API | Amazon’s Marketplace Web Server (MWS) API |
|
|
|
|
|
|
To accommodate, the actual sequence of operations our app makes is:
- Request new orders from Amazon
- Request order items for new orders
- Create an order on Shopify
- Acknowledge receipt of the order to Amazon
- Confirm that the acknowledgement was successfully processed
When the merchant subsequently fulfills the order on Shopify by the merchant, we receive a webhook notification and post the fulfillment to Amazon. The entire flow looks like this:
When our app started to receive an odd error from Amazon when posting fulfilment requests we knew the design wasn’t totally figured out. It turned out that our app received the fulfillment webhook from Shopify before the order acknowledgement was sent to Amazon. Therefore when we attempted to send the fulfillment to Amazon, it failed.
Shopify has a rich ecosystem of third-party apps for merchants’ shops. Many of these apps help automate fulfillment by watching for new orders and automatically initiating a shipment. We had to be careful because one of these apps could trigger a fulfilment request before our app sends the order acknowledgement back to Amazon.
Shopify uses a synchronous messaging protocol requiring two messages for order creation and fulfillment. Amazon’s messaging protocol is a mix of synchronous (retrieving the order and order items) and asynchronous messages (acknowledging and then fulfilling the order), which requires four messages. All six of these messages need to be sent and processed in the correct sequence. This is a message ordering problem: we can’t send the fulfillment request to Amazon until the acknowledgement request has been sent and successfully processed even if we get a fulfillment notification from Shopify. We solved the message ordering problem by holding the fulfillment notification from Shopify until the order acknowledgement is processed by Amazon.
Another issue cropped up when we started processing refunds. The commerce domain model implemented by Amazon requires refunds to be associated with an item sold while Shopify allows for more flexibility. Neither model is wrong, they simply reflect the different choices made by the respective teams when they chose the commerce use-cases to support.
To illustrate, consider a simplified representation of an order received from Amazon.
This order contains two items, a jersey and a cap. The item and shipping prices for each are just below the item title. When creating the order in Shopify, we send this data with the same level of detail, transformed to JSON from the XML received from Amazon.
Shopify is flexible and allows the merchant to submit the refund either quickly by entering a refund amount, or with more a detailed method specifying the individual items and prices. If the merchant takes the quicker approach, Shopify sends the following data to our app when the refund is created:
Notice that we didn’t get an item-by-item breakdown of the item or shipping prices from Shopify. This causes a problem because we’re required to send Amazon values for price, shipping costs, and taxes for each item. We solved this by retaining the original order detail retrieved from Amazon and using this to fill in missing data when sending the refund details back.
Lessons Learned
Our choices violated the design constraint that we initially set to not persist order data. Deciding to persist orders and all the detail retrieved from Amazon in our app’s database enabled us to solve our problems integrating the different domain models. Looking back, here are a few things we learned:
- It’s never wrong to go back and re-visit assumptions, decisions, or constraints put in place early in a project. You’ll learn something more about your problem with every step you take towards shipping a feature. This is how we work at Shopify, and this project highlighted why this flexibility is important
- Understand the patterns and architectural style of the systems with which you’re integrating. When you don’t fully account for these patterns, it can cause implementation difficulties later on. Keep an eye open for this
- Common integration problems include message ordering and differences in message granularity. A persistence mechanism can be used to overcome these. In our case, we needed the durability of an on-disk database
By revisiting assumptions, being flexible, and taking into account the patterns and architectural style of Amazon, the team successfully integrated these two very different commerce domains in a way that benefits our merchants and makes their lives easier.