Navigating Recurring Payments in India: A Backend Perspective

In October 2021, a directive issued by the Reserve Bank of India (RBI) came into effect that introduced new requirements related to recurring payments in India. The requirements in the directive include the following:

• All recurring payments require a one-time authorization from the payer via an e-mandate that is registered through an Additional Factor of Authentication (AFA).
• Recurring payments up to and including the RBI threshold amount can be automatically collected with a successfully registered e-mandate in place, while recurring payments greater than the threshold amount require the payer to authorize the payment by completing an AFA. Each recurring payment will require a pre-debit notification to be sent to the payer at least 24 hours prior to any charge.

To address the new RBI framework for recurring payments in Shopify’s billing platform, we worked with a local payment provider that could accommodate both card-based payments as well as a local payment method called Unified Payments Interface (UPI).

From a backend perspective, all these complexities meant a few things for our internal payment service. Particularly, the system would need to:

• model the new e-mandate concept
• handle recurring and one-time payments
• work with cards and UPI
• do all of the above with a new payment provider

Conceptualizing an E-Mandate

In the context of the RBI’s regulation changes, an e-mandate is a standing instruction given by a user for Shopify to charge their payment method on either a recurring or an as-presented basis. This instruction is prompted with an AFA on a small, temporary authorization hold (typically ₹1). Upon the successful completion of the AFA, the authorization hold is released, and an e-mandate is registered with the payment provider.

With the registration of an e-mandate, Shopify’s payment service would receive a token from the payment provider. The system could then use this token to charge the user when necessary. Here’s a high-level sequence diagram of how an e-mandate would be registered in the happy path:

Our payment service during the implementation phase of this project had not formally conceptualized an e-mandate. However, the system had already modeled a source object: 

In the context of the system, a source is an object that encapsulates the underlying payment method. Once the source ends up in a chargeable state, it could then be used to charge the user. We felt the source object closely resembled an e-mandate, and associating the e-mandate’s token with its remote_reference would allow us to build this out quickly.

Another thing we had to figure out was whether the source was being verified or the charge. Depending on which way we went with this scenario, it would affect the payment service’s internal logic and how a request would be served with a response when onboarding a payment method.

For our payment provider, it was clear that a charge was being verified. However, the concept of verifying an authorization charge to then move a source to a chargeable state would be completely novel for the system. This is because our implementations up until now had authorization charges that would automatically expire after some time for payment method onboarding, meaning no money was being moved from the system’s perspective, and we would only deal with the verification, associating it to a source or payment instrument.

We ultimately opted to continue treating the source as the subject of verification since it made sense in our business logic—that the underlying payment method was being verified. However, we also opted to create a charge record for an audit trail on the money being moved. This charge would then be automatically refunded through an asynchronous background job so that no money would be moved by the time transactions were posted on the user’s bank statement.

Handling Recurring and One-Time Payments

Since we were integrating multiple payment methods in our billing platform, we needed our systems to deal with both recurring payments (i.e., monthly bills) and one-time payments (i.e., domain purchases).

We had only one option from our initial gatherings: use our payment provider’s native payment method onboarding flows to support both payment types. Per API-level constraints, this would return two separate tokens for recurring and one-time payments, with each flow requiring an AFA for card tokenization. For UPI, only one AFA would be executed.

While this would allow us to handle both payment types and work well for UPI, it also came with a few challenges that downgraded our user experience. For example, in our billing platform, users enter their card once and add it to make recurring and one-time payments. Using our payment provider’s native onboarding flows would mean the user must complete two consecutive AFA challenges when clicking the add button. This would leave our system in a tricky place. For example, the platform would have to gracefully handle edge cases where the first AFA is successful but the second one is not. Not to mention that this would add friction to card onboarding and increase latency by making more API calls.

After quite a bit of internal deliberation and multiple discussions with our payment provider, we decided that tokenizing once for recurring payments would work for both paths.

This proposal would have its own tradeoff. The benefit was that the solution would avoid the friction for our users of having to input their card details multiple times. On the other hand, it would result in an atypical user experience (UX) since the user would get a notification that they’ll be charged in 24-48 hours when trying to buy a theme within the admin dashboard.

We felt this tradeoff was worth it in the short term until our payment provider came up with a native solution that was more UX-friendly.

Implementing Cards

In the implementation phase, we had to ensure that this journey went through our Payment Card Industry (PCI) compliant server and eventually redirected the merchant to their bank’s page to complete the AFA:

 

 

 

Implementing UPI

UPI was introduced by the National Payments Corporation of India in 2016. It’s a local mobile-only payment method supported by a bank account under the hood. Since UPI was introduced, it has gained widespread adoption in the region for peer-to-peer and peer-to-merchant transactions.

Management of a UPI account and its corresponding payments is done through a UPI application, for which there are multiple competing players. Additionally, when making a payment with UPI, a notification is sent to the user, and they would either approve or reject the payment through their application’s UI. This meant that onboarding the payment method is more asynchronous than cards since the AFA notification will be triggered and completed on the user’s phone, which is why the payment service would be relying heavily on webhooks from our payment provider.

When registering an e-mandate with UPI, subsequent charges would go through the UPI AutoPay functionality. This would mean the user would get a notification at least 24 hours in advance of a payment being made. One of the trickier things we had to deal with in the backend was defining the e-mandate’s limit for UPI. This is because the API we worked with allowed payments greater than the maximum amount set when a card’s e-mandate was being registered. However, UPI was more of a hard limit as a payment higher than the maximum amount on the e-mandate would immediately fail with the API. Why was this the case? Because banks would set different recurring transaction limits depending on the payment method.

Our research found that the UPI application the user downloaded would also impact the e-mandate’s limit, among other things. As an example, let’s take a look at how the overall recurring transaction limit could be affected in UPI AutoPay:

UPI recurring transaction limit by bank	= x	₹10,000
Daily transaction limit by bank	= y	₹20,000
UPI recurring transaction limit by application	= z	₹5,000
Overall UPI recurring transaction limit	= min(x, y, z)	₹5,000

 

This was particularly problematic for us since there was no way for our system to determine these limits at runtime. After digging deeper, we concluded that, at the time of this article being written, banks and applications had a minimum recurring transaction limit of ₹5,000. As such, we decided this would be the maximum amount we’d set on the e-mandate. Meaning recurring payments up to this amount would automatically go through. However, if the amount were to be greater, we’d bring the user to their admin dashboard to complete their payment manually. In this case, the payment would be treated as a one-time payment since the user would be on-session. And this is what would allow the larger payment to go through, as the user would be actively accepting the payment through their mobile application.

 

Making payments up to ₹5,000 with UPI would end up being similar to the sequence diagrams for cards. However, payments greater than ₹5,000 would have a different flow:

 

Building Resiliency

When implementing the above functionalities for cards and UPI, our happy paths were secure in the payment service. However, we had to account for unhappy paths.

Since the payment service at Shopify is an API serving as the internal mechanism for products such as Billing to move money around, it needs to be resilient to downstream failure. This way, anomalies like double charges don’t go through. The system achieves that through idempotency, the concept of returning the same response for identical requests being made multiple times in a row. The main feature which makes a request idempotent is an idempotency key, which, when passed, will serve as the unique identifier of a request.

Creating a charge with our payment provider involved two steps:

1. creating an order, and
2. creating a payment

Our payment provider’s new APIs were not idempotent by default. However, they had a feature where orders were unique by the value passed in the Orders API’s receipt field, and if the value were passed twice, the API would return an error. This would essentially function as our idempotency key. Here’s how it would work in practice:

 

Here are code snippets of the logic involved to achieve the last few steps of the above sequence:

For more on idempotency at Shopify, read Building Resilient GraphQL APIs Using Idempotency on the blog.

What’s Next?

Collect Outstanding Invoices

Before implementing our new billing solution, our invoices were issued in U.S. dollars (USD). Going forward, they will be issued in Indian rupees (INR). To achieve this, we will reissue any outstanding USD invoices with a foreign exchange rate applied for INR conversion and update our logic for managing merchants who have not paid their bills. With these changes, merchants will see reissued invoices in their local currency, determine when they’re due, and be able to pay them each as they see fit, which will provide a better billing experience for Indian merchants.

Make Better Abstractions for the Future

In delivering our solution, we made some short-term decisions. However, each was made with a long-term solution in mind. Once we have enough use cases, we plan to explicitly define the concepts of an e-mandate object and tokenized card payment instrument within the payment service.

Monitor Success Rates and Reduce Friction

This regulation introduced friction for larger recurring payments, which was partly the incentive for bringing it into effect. India had been dealing with quite a few fraudulent automatic transactions, and the government decided to make such payments harder for bad actors to exploit.

As such, we intend to closely monitor our larger transactions and respond based on their success rates. If they’re going through steadily, we’ll continue with our current revision. If not, we’ll explore alternatives.

Since the RBI’s regulations are relatively new, we anticipate further changes within the payments ecosystem in India, such as e-mandate limits being increased to ₹15,000, which will help reduce friction, or how cards may be added within a UPI account. As these updates roll out, we intend to closely follow them to see how they’ll impact our product and take the necessary steps to make payments more straightforward for our merchants.

What We Learned

There were quite a few technical and non-technical takeaways from working on this effort at Shopify.

Let the Mission Drive the Details, Not Vice Versa

The ultimate goal in this context was for users to be able to pay their bills. Any further optimization would be a bonus. Early in this endeavor, we were highly self-critical because we amplified any minor degradation to user experience. It wasn’t until we got outside feedback that we realized that small inconveniences aren’t necessarily the end of the world for users. Instead, the more apt approach would be to aim high and make adjustments as you see fit.

Any Short-term Compromise Should Come with a Long-term Vision in Mind

Navigating the new regulatory framework in India involved a lot of timely decisions and compromises. Compromises will often result in deciding between multiple imperfect options from a user experience perspective; that’s nothing new to engineers, but navigating future projects makes it easier when there’s a long-term and sustainable strategy in mind. Approaching our project from this perspective helped us determine where the pain points in the new implementation existed and how we might address them a few months from now.

Extract Obvious Cases of Shared logic Into a Dependency (Even If It Takes a Bit of Effort)

In this project’s scope, we had a few systems communicating with our payment provider’s API. Noticing that this would end up being a pattern in multiple server applications, we decided to abstract these Shopify-specificities into an internal Ruby on Rails gem and de-duplicate code across multiple repositories. This set us up for future endeavors where another system in the Shopify ecosystem might need to communicate with the payment provider and made implementation simpler towards the latter half of the project for each system.

Working with Cross-border Teams Requires Significant Planning and Constant Communication

An overwhelming majority of our team was based out of the EST time zone, while our payment provider’s team was based out of the IST time zone. That’s a 9.5-hour difference! This meant meetings occurred at odd times for at least one team, and replies to direct messages required a minimum one business day turnaround time. Not only that, but since Shopify became a digital by design company, our go-to was messaging through Slack. Meanwhile, our payment provider’s team was more comfortable with communicating in a video call. This difference required a bit of learning from both sides. As we expand further globally, we must consider these things for future endeavors.

Expect New Patterns to Emerge as Payment Providers Change

This was a complex project due to a perfect storm of changes happening simultaneously. Having significant experience with North American and European markets, we couldn’t make assumptions or use existing patterns with a new payment provider in India. Each provider has different UX philosophies for their APIs​​ and each new implementation challenges the system’s design. As this project approached its completion, we concluded that we have to better abstract our systems for long-term simplicity and become versatile with providers as we scale across local markets.

New Regulations Require Due Diligence Before Moving to the Build Phase

When regulations change, it’s often the case that not everyone knows everything immediately. This was true for this project as we found that implementation would be more meticulous at the software layer than we originally suspected. We ended up spending quite a bit of time asking questions and exploring the problem space before we began seriously writing the code. Despite the time it took to gain complete clarity over API and other constraints, our efforts paid off.

Strategizing (or Parallelizing) Your Production Testing Phase Can Be Beneficial

When we implemented our cards solution and began testing it out in production, we spent quite a bit of time waiting for charges to go through due to the pre-debit notification and charge attempt taking 24-48 hours per quality assurance test. If a bug was caught, we’d have to retry and repeat the testing. When we went about implementing UPI afterward, we started our production testing as soon as it was possible during the build phase. This allowed us to get our backend solution to a stable and trustworthy state early on and greatly reduced the time we’d spend on iterating after everything else was built out.

Familiarity Doesn’t Necessarily Imply Simplicity

Let’s say x is a concept you’re familiar with, and y is a concept you’re not familiar with. Familiarity alone will not make x simpler to implement than y. We thought the regulation changes within the card space would be simple to implement, mainly because we’d worked with that payment method for so long. This was definitely not the case. The regulation change completely redefined how cards would work with recurring payments in our systems. Interestingly, introducing UPI, an asynchronous payment method unlike any other in our billing platform, was simpler to get through.

But Wait…there’s More!

Making a system adapt to a new regulatory environment takes time. In the months following the initial beta launch of our card solution, and now with it widely available, the success rates for billing payments in India have increased from the rates we observed prior to this project. And we’re soon going to make our UPI solution generally available to all Indian merchants in local currency pricing!

This was a major endeavor for our engineering team, and we’re glad we could navigate through this as best as possible. We’re excited to see how UPI will work with our merchants and anticipate a long road in fine-tuning the platform’s payment experience in India.

---

If building systems from the ground up to solve real-world problems interests you, our Engineering blog has stories about other challenges we have encountered. Visit our Engineering career page to find out about our open positions. Join our remote team and work (almost) anywhere. Learn about how we’re hiring to design the future together—a future that is digital by design.