Dmitry Alexeenko

Why Building Billing Systems is So Painful

August 19, 2024 (1y ago)499,540 views

What is billing

Billing might seem like a straightforward concept at first glance. When I talk about billing to friends, most of them bring up their monthly subscriptions, whether it's Netflix, Spotify, ChatGPT or something similar. How hard can it be to engineer a system to support this? After all, all you need to do is to charge a customer a fixed amount each month. You can probably set up a cron job to do that. Done. However, as any seasoned engineer will tell you, the billing space is a deceptively complex beast that becomes exponentially more intricate as your product scales. Let's talk about that complexity.

So what is billing exactly? First, let's talk about subscriptions. It's a revenue management model where businesses charge recurring payments for access to products or services. A common example is the good-better-best pricing model used by many SaaS startups. Customers choose a tier (good, better, or best) and pay a flat rate for it. Typically each tier has both monthly and yearly prices. For example, Netflix offers the following 3 tiers: (1) Basic: $7.99/month, (2) Standard: $10.99/month, and (3) Premium: $13.99/month. At Stripe we put together the following diagram to demonstrate the good-better-best approach:

Another popular pricing model example is per-seat pricing. In this case the user picks how many seats they plan to use, and the business then charges them based on that amount (another diagram from Stripe):

Sounds easy. However, each of these models comes with its own set of challenges. From managing trial periods that need to be billed at a reduced amount or offered for free, to handling upgrades and downgrades that can happen in the middle of the billing cycle, which then prompts the additional complexity of prorated charges or credits. From managing fixed contract periods with financial penalties for early cancellation, to supporting multiple currencies once you want to sell your product globally. From allowing customers to bundle several products together at a discounted rate, to managing entitlements to grant or revoke features based on the customer's active subscription plan.

In recent years, another pricing model found itself in the limelight — usage-based billing (I blame AI). You pay exactly for the resources you consume. That could be hours of CPU usage, number of function invocations at the edge, GB of egress for storage, number of tokens generated, etc. This all sounds great, yet, all of this introduces another layer of technical challenges:

  1. Real-time usage tracking. Real-time is the keyword here. In order to provide customers with visibility into resource consumption the billing system needs to work in real time. As your customers approach certain usage and cost thresholds, you may need to notify them, before they receive an unexpected $100,000 invoice. This typically means relying on a stateless event-based architecture, to both provide a real-time balance and scalable webhooks platform.
  2. Handling high ingestion volume (think hundreds of thousands of events per second). The whole billing system needs to be designed to handle extremely high throughput and scale with concurrent load.
  3. Adapting to changing pricing models. E.g., if an application developer defines a billable event as "GB used" and then decides to change it to "GB used per region", you shouldn't require the application developer to reimplement their pricing model and integration.
  4. Allowing for flexibility in the data model. If you change packaging, pricing, or decide to add a new product, it's important not to force application developers reimplement the entire integration: they shouldn't need to create new billable metrics, make changes to integration around subscriptions, products, prices, customers, or backfill all of their previous subscriptions.

These are just a few examples. The folks over at Lago have more stories about why billing is 100x harder than you think. If that's not enough to convince you, read this.

Why billing matters

Billing isn't just about sending an invoice to your customer. It's the lifeblood of your business. Full stop. Simply put, if you can't charge for your product, then there isn't really any economic value from it. Billing ultimately dictates how your business makes money. Typically billing isn't just invoicing, instead it spans an entire revenue management workflow: from contract and quote management, to provisioning, from product and price management, to usage metering, from managing entitlements, to invoice generation and delivery, from payment processing, to revenue recognition. There are different ways to slice and dice billing, but the high-level architecture typically resembles this:

[insert the diagram]

Billing is infrastructure

The deeper you dive into billing, the more complexity you uncover. What starts as a simple invoice generation task quickly expands into a web of interconnected challenges. At some point you'll need to manage multiple currencies (and decide whether you manage static or dynamic currency conversion). Then you'll need to manage thousands and tens of thousands tax jurisdictions with their peculiarities. Did you know that in New York bagels don't incur sales tax, however, bagels that are sliced and have toppings do get taxed? Chances are you'll need to handle prorations for mid-cycle upgrades and downgrades. Don't forget about ensuring compliance with financial regulations and supporting complex enterprise contracts.

You get where I'm going with this. Billing isn't like any other product. Billing is a critical piece of infrastructure. This realization impacts several things: your hiring strategy, the internal abstractions your team needs to build, the level of abstraction you need to provide externally, guarantees and service-level objectives, etc.

I often compare billing infrastructure to databases. As an application developer, I don't care how the database actually persits my data, how it does replication, and what needs to be done behind the scenes to provide certain durability guarantees. I have an interface to the database and I simply work with it to store and retrieve data. Billing is exactly the same. I can guarantee that your customers don't want to think about all of those billing challenges. They don't want to think about currencies, taxes, prorations, billable metrics, or ASC 606. They just want it to work.

When Billing is down or can't be relied on, the whole company notices. Heck, the market does too: in late 2019 Twilio had to re-report its full-year forecast after they had issues with billing. TWLO fell more than 5% that day.

Who is your customer

Another reason billing is tough is that you don't get to have a single set of customers. Instead an effective billing system needs to cater to several customer sets:

  1. Finance requires accurate revenue recognition and reporting.
  2. Sales and GTM depends on Billing's capability to actually charge the customer, so the business gets paid.
  3. Product needs to be able to change prices, launch new products, and experiment with different price points for new markets, products, etc.
  4. Engineering expects reliable APIs and scalable infrastructure, to make sure customers get access to and pay for the product they built.
  5. Data science relies on Billing for pricing and packaging decisions, forecasting, and optimization.
  6. End users want transparent and easy-to-understand usage data and invoices.

Most of these customers aren't technical. They typically have tremendous domain expertise, but their interface with the billing system can't be an API. Billing is both an infrastructure and a product at the same time, which makes it an extremely hard problem to solve.

Building the team

What does all of this mean for the kind of team that's required to make Billing a success?

  1. First, the entire team must recognize that Billing is not just a technical function but a strategic pillar of the business. While it's the backbone of your business and your customers experience it as an infrastructure (they expect it to always be up), Billing teams are often under-resourced despite their critical role.
  2. Successful billing teams must have principal engineers who bring technical depth and leadership and have influence and social capital across the organization. This is where internal mobility can work really well.
  3. Even though some of your customers aren't technical, they are deep domain experts. You will need principal engineers who are subject matter experts in both engineering and finance. These engineers must earn the trust of the Finance team, deeply understand their needs, and influence the right build out in the right sequence, to make the right path-dependent steps in the short-term. Building for Finance is complex, invoicing compliance and audit workflows, so having engineers with domain expertise in both finance and billing is essential to getting it right.

Be clear on priorities

With so many customer sets, you must be clear on your priorities as the Billing team. Trying to make progress on everything at once in parallel won't work. Believe me, I tried. So be crystal clear with yourself, your team and your customers on what you're prioritizing for the next 6-12 months. Is it correctness? Product launch velocity? Global expansion support? Or maybe it's product and price model experimentation? Compliance adherence? Auditability? It's critical to have a clear prioritization across all of these different sets of customers.

Your Product team will always want to prioritize what's best for the customer. Your GTM team will push for whatever is needed to make the numbers this quarter. Your Finance team will want to prioritize whatever is needed to minimize risk to the business. Fundamentally everyone wants the same thing. In practice though, these teams want sufficiently different things — and it's important to be decisive and clear on priorities here.

Domain model

It's important to establish a clear domain model and define core entities — customers, products, prices, subscriptions, invoices, payments — and strong interface boundaries between them.

Source of truth

Make sure to have a single source of truth for all of the core entities defined above. It's okay to enrich the source of truth with additional data from other sources (say, CRM), but your systems and processes need to operate with one source of truth. The last thing you want is for different teams to get different answers to the same question because they operate on their own versions of the source of truth.

Say no to grandfathering

Fight against grandfathering. It's important to be diligent about continuous pruning of your old pricing plans. Otherwise before you know it, you'll find yourself with dozens and hundreds of user cohorts that all have slightly different billing logic. Not only that, but then your team will need to spend hours on certifying hundreds of accounting statements explaining how different invoices with different prices are computed. The best way to combat this is to launch new versions of your product and incentivize your customers to upgrade to the new version: they get new functionality, you move them onto up-to-date pricing. Win-win.

Billing as a strategic asset

While building a robust billing system can be quite challenging, it's an investment that pays dividends. A well-designed billing infrastructure not only ensures accurate revenue capture but also enables quick product iteration, helps deter fraud, and provides valuable insights for business decisions. It's important to remind the business about it and shift the perspective from billing as a way to reduce the cost of servicing revenue to the one that billing is a strategic asset that can drive the company growth and success.