A business-configurable rebilling decision engine on Azure Databricks

The problem

Recurring billing fails constantly: insufficient funds, expired cards, chargebacks, cancellations. Every failure is a decision point — retry later? shift the rebill date? charge a reduced amount or an admin fee? cancel outright? For fifteen years those decisions sat in a legacy system bound to the enterprise data warehouse — logic frozen where no business analyst could reach it.

The mandate was a platform where analysts define and tune the rules themselves — “three consecutive insufficient-funds declines → charge a small admin fee, cut the membership price by 25%” — with the engine, not engineers, in the loop. Rule complexity is unbounded, so the engineering problem is containment: letting analysts express anything without letting them break billing.

Architecture

An event-driven loop between the internal flow orchestration layer and a decision engine running on Azure Databricks:

1. Biller events arrive (declines, chargebacks, refunds, cancels) via GCP Pub/Sub from the payment-system estate.
2. A bridge flow reconciles each event against the warehouse data the decision depends on, which lands asynchronously — the ordering problem at the heart of the system.
3. The decision engine queries subscription history, applies the analyst-defined exclusion and action rules, logs every decision to Delta Lake, and emits an operations plan (rebill shift, price change, admin fee, cancel…).
4. The main flow executes that plan against the external payment systems and feeds the outcomes back into the loop, so the engine always decides with full context.

Infrastructure is fully codified (Databricks asset bundles, Terraform), with separate non-production and production stacks.

My role

I carried this end-to-end as principal engineer and wrote effectively all of it myself: analysis and stakeholder alignment, the full technical proposal, then implementation through to turnkey delivery — flows, notebooks, infrastructure, documentation and QA guides.

Outcome

Live in production, handling more than 500,000 biller notifications a day, and projected to add seven figures in annual revenue. Rule changes that once meant quarters of legacy-system work now happen at the speed of a notebook edit, every decision auditable in Delta Lake.

The scheduling layer opened a door closed for fifteen years: just-in-time billing-cycle optimization. The engine owns when a rebill is attempted, not just whether — so analysts can tune the payment cycle itself against live outcomes, a class of revenue work the business had no mechanism to attempt before.