Prognostication For Fun And Profit: States And Events

I'm Ben Doyle, research scientist and data prophet at Shopify. Over the next little while I'm planning to go into some detail about how we're calculating some common ecommerce metrics (like customer count, churn rate, lifetime value, etc.) here at Shopify.  Most of these metrics come down to fancy ways of counting, but the devil is in the details. 

You have a table of users and a boolean flag that says whether each of them is a paying customer. Counting your customers is as simple as counting the rows where the flag is set. Or is it? What if you want to plot your growth over time? You can replace the flag with a date range and count how many of these ranges include a date in question. What if your customer definition changes, perhaps due to a change in your business model? You have to go back through your whole history, which might be a problem if your new definition relies on information you have just started collecting. If you are doing exploratory work you might not even know what data you need yet. 

These problems have solutions but clearly there's work to do. We need to untangle our methods and sort out our definitions. I thought I'd kick things off by clarifying the elements most basic to counting: intervals and points.

 

 

In the diagram above the dark blue lines represent intervals and the light blue circles represent points (the points should really only be a single pixel but are large so we can see them). In general you need intervals to measure points and you need points to measure intervals. So in a) three of the four intervals overlap with the point and in b) three of the four points overlap with the interval.

States And Events

In more familiar terms the space we're usually talking about is time, our points are events and our intervals are states. Events can be found in server logs or rows in a transactional database.  So every time a purchase is made or a signup form is completed, an event is created. At minimum the event type and a timestamp will be recorded, though the event will usually be associated with a user or other entity.

A state history is often the result of a state machine as Willem discussed in a recent post. For example a subscriber can be subscribed to one or many of several subscription packages. Then keeping a history of those packages means having a start and end date for each package, for each subscriber. States must at minimum have a start and end timestamp and a type, though an entity is usually recorded as well.

Example: Accounting

Understanding states and events clarifies the distinction between different sorts of counting. For example standard accounting practise is to report both a balance sheet and an income statement.  

The balance sheet is a snapshot. This means it is an event in time being used to count states. Consider an account that had $100 in the interval between May 1st and June 18th.  It had $100 on an event date of June 1st, which lies within the interval. The state of every account can be assessed on a given date, and a total reported.

The income statement reports changes over a period of time. It uses an interval to count a collection of events.  So if there was a withdraw event on June 19th of $5 and deposits on the 20th and 21st of $10, the income statement for June would be $10 + $10 - $5 = $15. If these were the only events for June the balance sheet for July 1st should be $115. 

These are two different and complementary ways to look at the problem of counting, over time. In theory if you add up all of your income statements before a date and time the sum should be the same as your current balance at that date and time. So the results are equivalent. Unfortunately if there are errors or omissions in your data discrepancies will arise between the two methods.

Metrics

 

Perhaps the most basic metric to track is "How many customers did we have on date X?". If you have a software system already set up that consistently tracks the states of your users (e.g. visitor, customer, churned), and the events that can influence them (e.g. signups, payments, cancellations) the answer can be easy. Like in the finance example above you can present a balance sheet for any point in time, counting customer states at a point in time. In status quo situations this will probably be the preferred method.

It's nice to know that there is an alternative at hand for when things get complicated though. Counting the events that change customer states will give you greater flexibility to adapt to changes. So if you suddenly decide you want to count “happy” customers separately, looking at the events that indicate happiness - as opposed to "customer-ness" is a good place to start.

For an example of the above you could count the number of events signifying happiness (e.g. logins or site interactions) within 30 days of a particular date. This could serve as a happiness metric for that date. Since you are counting the states directly it's easy to add flourishes like having some events count more than others, having their weights decay over time, or even incorporating events from an entirely different source. This flexibility especially helps if you are trying to build up a metric to serve as a proxy for or to predict another metric. For example you could use the weights on your event types as adjustable parameters in fitting a model. Continuing with the example, our happiness metric could be constructed to predict customer churn. I'll go into more detail about these sorts of analyses in future posts.

When In Doubt, Start With Events

If you are running a store it's nice to know how many customers you have, but it's more important to know how many sales you've made. The definition of customer is abstract and can be arbitrary. Do they need to make a purchase? Several purchases? Do coupons or promotions count? When do they cease to be a customer? In contrast, sales events are much harder to argue about, so they make a great basis for metrics. I hope this article has left you with some insight into the seemingly simple act of counting and welcome questions or comments.