Shopify's path to Rails 3

The TL;DR version

Shopify recently upgraded to Rails 3!

We saw minor improvements in overall response times but what we’re most happy with is the new API – it means we get to write cleaner code and get features out faster.

However, this upgrade wasn’t trivial – as one of the largest and oldest Rails apps around, the adventure involved jumping through a few hoops. Here’s what we did and what you might consider if you’ve got an established Rails app that you’re thinking of upgrading.

First, some numbers

The first svn check-in to Shopify was on the release date of Rails 0.5. That was in July of 2004, six years ago, which according to @tobi is “roughly 65 years in internet time”.

At that time Shopify had only two active developers. Today it has eleven full time devs working on it.

The Shopify codebase has over 300 files in the app/models directory, over 130 controllers, and almost 100 gem dependencies.

$ find app/models/ -type f | wc -l
     327
$ find app/controllers/ -type f | wc -l
     131
$ bundle show | wc  -l
      95

Over the past 6 years Shopify has been under constant development, amassing nearly 12000 commits. This makes Shopify one of the oldest, most active Rails projects in existence.

Our process

There are many Rails 3 upgrade guides out there, but we didn’t try to follow any of them. We focused on doing as much as we could ahead of time to prepare for Rails 3, and then giving one big final push when 3.0 final was released.

When upgrading a large app to a major release like this we found there are some things you can do to prepare yourself, but at a certain point you’ve just got to bite the bullet and make the final push to get things working.

Bundler

Shopify had been using Bundler in production for 9 months before making the move to Rails 3. Like most, we weren’t convinced of its utility at first, but as the code got more stable we saw how much it helped with deployments and managing development environments. We think Bundler was absolutely the right choice for managing dependencies.

It was pretty painless to use Bundler with Rails 2.3.x, the Bundler documentation has everything that is needed. We’d definitely recommend doing this step ahead of time as it removes one more obstacle in the Rails 3 migration.

XSS

This was a big one. Some more numbers: Shopify has about 100 helper modules and 130 views. The task of updating all of our views/helpers for the new ‘safe by default’ XSS behaviour was a separate migration all its own. This too, we completed a few months before the release of 3.0.

There was no secret way to go about this, just the obvious back-breaking way. Here’s the basic process I followed:

1. Run the functional tests. Fix any issues that show up there.
2. Boot up Shopify in my development environment and click around, fixing any issues I see there.
3. Manually scan through all of the modules in app/helpers, looking for anything suspicious.
4. Deploy the code to our staging server. Have the team try it out and report any errors to a shared Google spreadsheet (great for collaborative editing).
5. Code review.
6. Deploy the code to production and hope that no issues slipped through.

N.B. When new issues come in, do your best to use ack (or some other project search tool) to find any instances of that issue in other views/helpers and correct those as well.

The rest

After getting Bundler and XSS out of the way, the rest of the migration was done as one large chunk. Some of the work in upgrading to Rails 3 was actually going on in parallel to the XSS work.

The first commit to our rails3 branch was made back in February when the first Rails 3 beta was released. At that point we didn’t know how much work it would be to get Shopify running on Rails 3. We were excited about the launch of the beta and the prospect of getting Shopify using it soon.

After a few days of work we ran into some major blockers that were keeping the app from functioning. Work was abondoned on the rails3 branch for 5 months while the 3.0 release became more stable. When the first release candidate came out in July work we resurrected the rails3 branch.

From then (mid-July) until mid-October the rails3 branch saw pretty constant action, never going more than a few days without a commit. There was a lull during the XSS migration, and as devs took on other projects while doing the migration. We remained mindful of the fact that 3.0 final wasn’t yet released and didn’t want to put our changes into production until we had the confidence of that final release.

Since this whole process took several months there was a lot of activity going on in the master branch at the same time. The only advice to offer is merge early and merge often.

When the final release came out we once again underestimated how much work would be involved in getting Shopify the rest of the way on to Rails 3. The day that it was released @tobi put something like the following into our Campfire room “Let’s get Shopify running on Rails 3! Any devs who want to help join the Meeting Room [campfire room].” It was another few weeks before all was finished.

Major stumbling blocks

Routes

Shopify also has lots of routes.

$ rake routes | wc -l
     846

At the beginning of the upgrade process we used the routes rake task that comes with the rails_upgrade plugin but we were still plagued with missing routes throughout the upgrade.

Although our routes tripled in size, the increase was worth it because the new routing API is much nicer to work with.

The old

map.namespace :admin do |admin|
  admin.resources :products, :collection => { :inventory => :get,
    :count => :get },  
    :member => { :duplicate => :post, 
      :sort => :post,
      :reorganize => :any,
      :update_published_status => :post } do |products|        
    products.resources :variants, :controller => "product_variants", :collection => { :reorder => :post, :set => :post, :count => :get }
  end
end

The new

namespace :admin do
  resources :products do
    collection do
      get :count
      get :inventory
    end 

    member do
      post :sort
      post :duplicate
      post :update_published_status
      match :reorganize
    end 

    resources :variants, :controller => 'product_variants' do
      collection do
        get :count
        post :set
        post :reorder
      end 
    end 
  end
end

Libraries

Like everyone else we were tripped up by libraries in need of upgrades for Rails 3 compliance. There was a lot less of this than you’d expect because Shopify implements so much of what it needs internally. Lots of code in Rails core began in Shopify’s code base.

There were updates required to the plugins that Shopify maintains. Otherwise, when we found issues with libraries we were happy to discover that other maintainers were diligent and had already pushed fixes for Rails 3 compatibility, it was just a matter of updating library versions we were tracking.

helper :all

helper(:all) was a configuration option in Rails 2.x. You could add it to a controller and that controller would have access to all helpers modules defined in your application. In 2.x this was part of the default Rails template, but it could be removed for users who didn’t want it.

In Rails 3.0 this has been moved into ActionController::Base and it can no longer be turned off. This can create very weird behaviour like the following: https://gist.github.com/517669

This was causing issues for us since a lot of our helpers define methods with the same name. We ended submitting a patch to Rails that let us continue to use routes with the default naming scheme. The fix is to use the

clear_helpers

method in your

ApplicationController

class ApplicationController
  clear_helpers
  ...
end

Documentation

External services

Shopify integrates with a myriad of external services. Payment gateways through ActiveMerchant, fulfillment services through ActiveFulfillment, shipping providers through ActiveShipping, product search engines, Google Analytics, Google Checkout, the list goes on.

Ensuring that these integrations continued working was very important for us and we would have had issues had we not thoroughly tested them. Don’t overlook this step.

Looking ahead

Towards the end of the upgrade we (jokingly) asked ourselves if it was really worthwhile to upgrade to Rails 3. After all, we were doing just fine with Rails 2.x, and upgrading to 3.0 was not trivial.

To give you an idea of how much code was changed, here’s the diffstat from Github:

But we soon came to realize that there are a lot of exciting things coming in future releases in the 3.x series and this is the way forward. We’re really excited about getting to use stuff like Arel 2.0, Automatic Flushing, Identity Map, and lots of other goodies.

The Rails project and its surrounding ecosystem are moving ahead quickly. By staying on top of it, we can provide the best tools for our developers and the best experience for our customers.