When I was a high school student in 2013, I co-founded a Shopify app company, Shopulse, with my friend Raphael. It was that year that I first wrote my Liquid code to fix our client’s Shopify theme. Now in 2022, I’m a developer at Shopify working on various projects like Metafields, Pricing, and Customer Accounts. Even nine years later, I haven’t forgotten my fun Liquid experiences. I’ve decided to learn more about Liquid and Liquid-C at Shopify, and try to make it faster! Since everyone is very reachable at Shopify, I got so much help to understand Liquid from the creator of Liquid-C, Dylan Thacker-Smith, VM expert, Marc-André Cournoyer, and even our CEO, Tobi Lütke!
Liquid was first introduced in 2006, and Liquid-C was introduced in 2014 to accelerate Liquid’s parsing performance by using C. Since then, Liquid-C started to evaluate Liquid AST nodes, then recently, it has introduced a VM to compile Liquid. Liquid-C works as an extension to Liquid gem, and it parses, tokenizes, and renders Liquid templates. However, not every feature of Liquid has been ported over to Liquid-C, such as the If tag and For Tag that are rendered in Liquid. Therefore, I started to look into how to compile Liquid’s If tag from Liquid-C.
In order to implement If Tag in Liquid-C, we need to first understand how Liquid-C works. In a nutshell, Liquid-C takes a Liquid template and converts it into bytecode instructions for its customer VM just like other languages like Ruby, Python, and Java.
With a Liquid template file, Liquid creates a Document object, and with a Tokenizer, it parses through the template to create Block objects which could be an Expression node or a Variable node.
With the Liquid-C extension, it creates the Block, Expression, and Variable objects in C environment. Parsing continues inside the Block, and with the tokenizer, it creates a corresponding byte instruction in the VM. The possible token types are:
- TOKENIZER_TOKEN_NONE
- TOKEN_INVALID
- TOKEN_RAW
- TOKEN_TAG
- TOKEN_VARIABLE
- TOKEN_BLANK_LIQUID_TAG_LINE.
With the TOKEN_RAW, Block object creates a write_raw instruction to VM with the given raw string.
Here’s a example of compiling a Liquid template into a Liquid-C VM:
It’s compiled into:
How Liquid-C VM Renders
When Liquid-C generates byte instructions, it also generates a constants array:
As Liquid-C renders, it moves the instruction pointer step by step, and when the instruction pointer is at a certain instruction, it moves the constant pointer. From the example diagram above, on the first PUSH_CONST instruction, it pushes a constant at the constant pointer to the VM stack and moves the constant pointer one step forward.
Let’s Remove the Constant Pointer
With Liquid-C’s current structure, it’s costly to jump forward or backward the instructions because Liquid-C has to calculate how many steps to move the instruction pointer and the constant pointer. In order to implement the If tag, and For tag in Liquid-C, we have to remove the constant pointer to freely move the instruction pointer.
Current Instruction structure:
Refactored Instruction structure:
With this new structure, Liquid-C stores the index of the constant in the constants array thus it no longer requires the use of the constant pointer.
Benefits of This Refactor
With this new structure, we store the constant’s index in a hash table, and we won’t store duplicate elements in the constants array. This Liquid template creates a constants array with duplicate values:
Here is the disassembled view of the example template:
After the refactor VM’s constants array’s size has been reduced. As an example performance/tests/dropify/cart.liquid stores 157 objects in VM’s constants array, but with the refactor it stores 104 objects instead.
Concerns Around the Refactor
Unfortunately, this refactor has not drastically improved Shopify's theme rendering speed. The refactor has slightly increased memory usage and parsing time.However, this refactor will be a stepping stone to implement If and For tags, which will improve Liquid-C rendering performance.
With this refactor, Liquid-C creates a hashtable to store the constants’ index value in the VM’s constant array. As a side effect, the refactor has increased the Liquid-C’s memory usage.
Secondly, with the refactor, Liquid template parsing time has been slightly increased.
What’s Next?
With this refactor, we should be able to introduce new operations such as GOTO or JMP that can move the instruction pointer forward and backward. With those, Liquid-C will be able to introduce If tag and For tag to greatly improve its rendering speed!
Liquid-C is an open source project, and it’s available at github.com/Shopify/liquid-c. Liquid-C still has many Liquid features to extend such as variable processing, template loading, and standard filters. As Liquid is used by numerous projects like Jekyll, we’re hoping to grow the Liquid-C community and supercharge Liquid’s performance!
Michael Go is a Developer at Shopify. He’s worked on several teams, including self-serve, since joining the company a year ago. To connect with Michael, follow him on Twitter.
If building systems from the ground up to solve real-world problems interests you, our Engineering blog has stories about other challenges we have encountered. Visit our Engineering career page to find out about our open positions. Join our remote team and work (almost) anywhere. Learn about how we’re hiring to design the future together—a future that is digital by default.