← Writing

· 16 min read · AWS

From a Broken Payment Flow to a Resilient Registration System: What I Learned Building Zeron Academy Payments from Scratch

Broken Payment Flow to a Resilient Registration System

For the last two days, I worked on something that looked simple on paper:

Let students register for a course, pay the registration fee online, and have the system automatically confirm the registration.

The first version sounded straightforward.

Student fills the form → payment gateway opens → payment succeeds → registration is confirmed.

In reality, building it from scratch exposed a much larger problem. The challenge was not just integrating a payment gateway. It was making the entire system reliable enough that a problem with a payment provider would not stop us from accepting registrations.

And then, to make things more interesting, while working on the payment system, I also had a deployment failure that temporarily took down the production server.

This article is a walkthrough of what happened when nobody watching, what broke, how I investigated it, how I redesigned the deployment process, why Razorpay became a problem, why I moved to Cashfree, and finally how I built a manual QR fallback so that payment-provider problems can no longer block student registrations.

The Goal Was Simple

Zeron Academy has a registration flow for students. A student receives a private registration link, opens it, fills in their details, and pays a registration fee.

The system needs to make sure that:

  • the amount cannot be manipulated by the student
  • the registration is tied to the correct course
  • the payment belongs to the correct registration
  • the same payment cannot be processed twice
  • duplicate webhooks do not create duplicate registrations
  • a failed payment does not consume the registration link
  • a successful payment marks the student as paid
  • the registration link becomes unusable after successful payment
  • administrators can see the registration and payment state

The architecture eventually became:

Registration Link
        ↓
Student Registration
        ↓
PENDING_PAYMENT
        ↓
Payment Provider
        ↓
Payment
        ↓
Webhook / Verification
        ↓
Transactional Finalization
        ↓
PAID
        ↓
Registration Link = USED

The important part is that payment confirmation is not trusted from the browser. The browser can say: “The payment succeeded.” That is not proof. The server needs to independently verify the payment.

The First Big Problem Was Not the Payment Gateway

Before I could even properly test the payment system, I had to make sure deployment itself was safe. The production server is an AWS Lightsail instance with:

2 GB RAM
2 vCPU
60 GB SSD

The original deployment approach was essentially:

git pull
npm ci
npm run build
pm2 restart academy

It works well until npm run build fails. Then you have a problem. A new deployment can destroy the working production state before the new version is actually ready. And that is exactly what happened.

The Production Server Became Unresponsive

While deploying the new payment system, the Next.js build took longer than expected. GitHub Actions eventually reported:

Run Command Timeout

At one point the server became difficult to access. The website was unavailable and SSH was also unavailable. That is a particularly unpleasant situation because you cannot even log into the machine to investigate the failure.

The obvious instinct is:

“The deployment failed.”

But that is only half the problem.

The real question is:

“Did the failed deployment damage the existing production application?”

This is where the deployment architecture became important.

The Release Directory Strategy

Instead of building directly inside the directory used by production, I changed the deployment model. The production system now looks conceptually like this:

/home/admin/
│
├── academy-repo/
│
├── academy-releases/
│   ├── release-A/
│   ├── release-B/
│   └── release-C/
│
├── academy-current -> academy-releases/release-C
│
├── academy-previous -> academy-releases/release-B
│
└── academy-shared/
    └── .env.production

The key idea is very simple. Production does not run from a directory that is being modified during deployment. Instead, every deployment gets its own release directory. For example:

academy-releases/a5bc9ebccfee3c16ecb15209f69c...

The new version is built there. Only after the build succeeds do we switch:

academy-current

to point to the new release. This is the same basic idea used by larger deployment systems: build first, switch later.

Why This Makes Production Safer

Suppose version A is currently running:

academy-current → release-A

We deploy version B.

Instead of modifying release A, we create:

academy-releases/release-B

Then:

npm ci
npm run build

happens inside release B.

If the build fails:

release-A ← still running
release-B ← failed

Nothing happened to production. The deployment simply stops.

If the build succeeds:

release-A
release-B ← healthy

we switch:

academy-current → release-B

This gives you a very important guarantee:

A failed build should not make a healthy application disappear.

That became one of the most important improvements in the entire system.

Deployment Flow After the Redesign

The deployment process eventually became:

GitHub
  ↓
SSH to Lightsail
  ↓
Fetch repository
  ↓
Create new release directory
  ↓
Copy source
  ↓
Link production environment
  ↓
npm ci
  ↓
npm run build
  ↓
Health check
  ↓
Switch academy-current
  ↓
Restart/reload PM2
  ↓
Verify production

If anything fails before the switch:

academy-current

is untouched.

That single design decision dramatically reduced deployment risk.

But We Still Had a Problem: The Server Was Only 2 GB

The release strategy protected the application from a bad build, but it could not protect the server from running out of resources. As the application became larger, the production build became more demanding.

One deployment reached:

Compiled successfully in 3.7min
Running TypeScript ...

and then stayed there.

Eventually:

Run Command Timeout

appeared.

And shortly afterward:

SSH unavailable
Website timeout

The instance had effectively become unhealthy.

After restarting the server, I checked:

free -h

and found:

Mem:   1.9Gi
Swap:  0B

There was no swap at all.

I added a 2 GB swap file:

sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

Then verified:

free -h

which showed:

Swap: 2.0Gi

I also made it persistent:

echo '/swapfile swap swap defaults 0 0' | sudo tee -a /etc/fstab

The next build succeeded.

The lesson was straightforward:

On a small production server, build workloads can be much heavier than runtime workloads.

The application itself was using relatively little memory. The build process was the expensive part.

Then the Actual Payment Integration Started Breaking

Once deployment was stable, I moved deeper into the payment flow. The first provider was Razorpay. The initial implementation included:

  • order creation
  • payment verification
  • signature verification
  • webhook verification
  • idempotent payment finalization
  • registration state updates
  • payment failure handling

The architecture itself was correct. But we hit a strange problem. The application returned:

Payment Failed

and server logs initially showed only:

err: "unknown"

That was not useful. So I improved the server-side logging.

Eventually the actual error became visible:

BAD_REQUEST_ERROR
Authentication failed
HTTP 401

That changed everything.

The Razorpay Problem Was Not the API Integration

I tested the Razorpay credentials directly from the server using curl. The first attempt failed:

{
  "error": {
    "description": "Authentication failed"
  }
}

The key ID in the application was also different from the current Razorpay key. The application was using an older key. I updated the environment, tested again, and successfully created a Razorpay order directly:

{
  "entity": "order",
  "id": "order_...",
  "amount": 100,
  "currency": "INR",
  "status": "created"
}

So Razorpay itself was working. But the application was still using the old key.

The PM2 Environment Trap

This turned out to be one of the most interesting lessons of the whole project. The production environment file contained the new credentials. The file was correct. The application release pointed to the correct file. But the running PM2 process still had the old environment.

I checked the running process:

PID=$(pm2 pid academy)

sudo tr '\0' '\n' < /proc/$PID/environ \
| grep -E '^(AWS_REGION|RAZORPAY_KEY_ID|NEXT_PUBLIC_RAZORPAY_KEY_ID)='

and saw:

RAZORPAY_KEY_ID=rzp_test_old...
NEXT_PUBLIC_RAZORPAY_KEY_ID=

while the shared production file contained the new value. The solution was not another pm2 restart. I recreated the process:

cd /home/admin/academy-current

pm2 delete academy
pm2 start ecosystem.config.cjs --only academy
pm2 save

After that, the application finally started using the correct credentials. The Razorpay flow worked. I tested successful and failed payments multiple times. At that point the core payment integration was stable.

Then I Ran Into Razorpay’s Operational Limitations

The next problem was not code. It was payment-method availability. Some payment instruments were not available for the merchant account. For a student-facing application, relying only on an inconvenient payment method like Netbanking is not ideal. The obvious requirement was:

Students should be able to pay using UPI.

This is where I started looking at Cashfree.

Moving to Cashfree

Instead of deleting the Razorpay integration, I introduced a payment provider abstraction.

Conceptually:

Payment Provider
├── Razorpay
└── Cashfree

The registration system no longer needed to know all provider-specific details.

It only needed to know:

Start a payment for this registration.

The provider handled the implementation details.

Cashfree Hosted Checkout worked smoothly in testing.

The student could reach the Cashfree checkout and use the available payment methods, including the QR experience displayed inside the Cashfree checkout.

That was a major improvement.

But then I tried to build a separate dynamic UPI QR flow through Cashfree’s Order Pay API.

And Cashfree returned:

POST/orders/pay is not enabled or approved

The Cashfree account required additional enablement for that capability.

This was another important lesson:

A payment API can be technically supported by the provider but still unavailable to your merchant account.

The Difference Between Hosted Checkout QR and Dynamic QR

This distinction was initially confusing.

There are two different concepts.

The first is the QR that appears inside Cashfree Hosted Checkout.

The student enters the hosted payment flow, selects UPI, and Cashfree presents the appropriate QR for that payment.

That worked.

The second is our application asking Cashfree directly:

Create a unique QR for this registration.

That required a different capability:

POST /orders/pay

and that was not approved for the merchant account.

So instead of rebuilding the architecture around a feature we did not yet have access to, I decided to add another fallback.

The Most Important Architectural Change: Runtime Payment Routing

This is probably the part of the project I am happiest with.

Initially, the active payment provider was effectively a configuration/deployment decision.

That meant:

Razorpay broken
→ change configuration
→ deploy
→ wait

That is not a good operational model.

Payment providers are external systems.

They can have:

  • outages
  • account configuration problems
  • disabled payment methods
  • approval delays
  • API problems
  • merchant restrictions

Your registration system should not stop because one payment provider is having a problem. So I moved payment routing into DynamoDB.

Runtime Payment Settings

A new singleton settings record was created:

AcademyPaymentSettings

The admin can now control:

Payment Collection
Cashfree Checkout
Razorpay Checkout
UPI QR
Custom QR
Default Payment Method
Fallback

For example:

Cashfree Checkout    ON
Razorpay Checkout    OFF
UPI QR               ON
Custom QR            OFF

Default: Cashfree

The important part is that this configuration is read from DynamoDB at runtime.

The application does not need a deployment to change it.

The lookup uses a consistent read for each request.

So if an administrator changes:

Cashfree ON
UPI QR ON

to:

Cashfree OFF
UPI QR ON

a new student request immediately follows the new configuration.

No:

git push
npm run build
GitHub Actions
PM2 restart

is required.

This Solved the Real Business Problem

Imagine tomorrow morning Cashfree stops working.

The old architecture would require a technical change.

The new architecture allows the admin to simply do:

Admin → Payment Settings

Cashfree Checkout = OFF
UPI QR = ON
Default = UPI QR

Save.

New registrations immediately use the alternative method.

This is much more valuable than simply having a second payment provider.

It gives the business control over payment availability.

Existing Payments Are Not Changed

There is another important design decision here.

Changing the active provider should affect new payment attempts, not historical payments.

Suppose:

10:00 AM
Student A starts Cashfree payment

At:

10:05 AM
Admin disables Cashfree

Student A’s existing Cashfree payment should not suddenly become invalid.

The system therefore keeps provider association with the existing payment attempt.

Historical Cashfree webhooks continue to work even if Cashfree is disabled for new registrations.

The same applies to Razorpay.

This separation is important because operational settings and historical transaction processing are different concerns.

And Then Came the Final Fallback: Custom QR

I wanted one more safety net.

Sometimes the gateway might work technically but still have some account-level problem.

Or maybe I simply don’t want registrations to stop because of a gateway issue.

So I added:

Custom QR

The administrator can upload a QR image from the admin panel.

For example:

Admin → Payment Settings

Custom QR
[✓] Enabled

Current QR:
[ image ]

[Update QR]

The QR is stored in S3. The payment settings record stores the QR metadata. The image itself is not stored inside DynamoDB.

How the Custom QR Flow Works

This is deliberately different from online gateway payments. Student sees:

Registration Fee
₹500

[QR IMAGE]

Scan using your UPI app.

Payment / UTR / Reference Number
[_____________________]

[Submit Payment Reference]

The student pays using the uploaded QR. Then submits the UTR/reference number. But there is one very important rule:

The system does not mark the registration as paid.

Instead:

PENDING_PAYMENT
        ↓
PENDING_VERIFICATION

An administrator then sees the payment under:

Admin → Registrations
→ Awaiting Verification

The admin can:

Verify Payment

or:

Reject Payment

Only after verification does the system perform the normal finalization:

PENDING_VERIFICATION
        ↓
PAID
        ↓
Registration Link = USED

This avoids one dangerous assumption:

A payment reference number is not proof that the money was actually received.

The student can enter anything into that field.

The admin must verify it.

Why We Did Not Create a Second Payment Finalization System

The existing application already had a shared payment finalizer.

Something conceptually like:

finalizePaidRegistration()

That function handles:

  • marking the registration paid
  • marking the registration link used
  • preventing duplicate finalization
  • maintaining transactional consistency

I reused that same mechanism for:

  • Cashfree
  • Razorpay
  • Custom QR

This matters because I do not want different payment methods to have different rules. The final state should always be governed by the same business rules.

The Final Architecture

After two days of debugging, rebuilding, and changing direction, the architecture looked like this:

                    Admin Payment Settings
                             |
                             v
                    Runtime configuration
                             |
          +------------------+------------------+
          |                  |                  |
          v                  v                  v
     Cashfree            Razorpay          Custom QR
     Checkout             Checkout          Manual
          |                  |                  |
          |                  |                  |
          +------------------+------------------+
                             |
                    Payment processing
                             |
                    Server verification
                             |
                      Shared finalizer
                             |
                    +--------+--------+
                    |                 |
                   PAID          Payment failed
                    |
              Link becomes USED

The important part is not the number of payment methods.

The important part is the separation of responsibilities.

The Deployment Architecture Also Became Better

By the end, the deployment system had its own resilience.

Instead of:

Production directory
        ↓
npm build
        ↓
restart

we now have:

GitHub
   ↓
Persistent repository mirror
   ↓
New release directory
   ↓
npm ci
   ↓
npm run build
   ↓
Health check
   ↓
Switch academy-current
   ↓
PM2

For example:

academy-current
    ↓
academy-releases/
    ├── df1d3ab...
    ├── 6a88af4...
    └── a5bc9eb...

If a5bc9eb fails to build, it never becomes active.

That is what happened during the incident.

The failed release was created, the build timed out, the production pointer remained on the previous release, and after the server reboot the application came back normally.

Commands That Became Useful During the Incident

Checking memory:

free -h

Checking disk:

df -h /

Checking PM2:

pm2 status

Checking the active release:

readlink -f /home/admin/academy-current

Checking the publicly deployed revision:

curl -s https://academy.zerontech.com/__release_revision

Checking processes:

ps aux | grep -E 'next build|tsc|node' | grep -v grep

Inspecting the application logs:

pm2 logs academy --lines 50

Checking the running process environment:

PID=$(pm2 pid academy)

sudo tr '\0' '\n' < /proc/$PID/environ

Testing AWS access directly:

aws sts get-caller-identity

Testing DynamoDB:

aws dynamodb get-item \
  --table-name AcademyPaymentSettings \
  --key '{"id":{"S":"production"}}' \
  --region ap-south-1

Testing a payment API directly was also extremely useful.

For Razorpay, a direct API request quickly proved whether authentication was working before blaming the application.

One of the Most Valuable Lessons: Make Errors Observable

Early in the debugging process, the application logged:

err: "unknown"

That was not enough. Later, we changed the server logs to include safe information such as:

errorName
errorMessage
errorStack
payment provider
amount
currency
order ID
HTTP status
provider error code

while never logging:

API secrets
AWS credentials
webhook secrets

That changed debugging from guesswork into evidence. For example, this:

Authentication failed
401

is actionable.

This:

unknown

is not.

Good error handling is not just about displaying nice messages to users. It is also about giving developers enough information to understand what actually happened.

What I Would Do Differently Next Time

There are several things I would design earlier.

  • First, I would introduce payment-provider abstraction from day one. Not because I expected Razorpay to fail, but because payment providers are external dependencies.
  • Second, I would make runtime payment configuration part of the initial design. A payment system should assume that providers can become unavailable.
  • Third, I would configure swap on a small production instance before running heavy Next.js builds.
  • Fourth, I would make deployment environment handling deterministic.

One of our hardest bugs was caused by stale environment values being retained by PM2 while the .env.production file had already been updated.

The lesson is that:

A correct environment file does not necessarily mean a correct running process.

Finally, I would never depend on one payment method for business-critical registration.

The Architecture Today

The final operational model is much more resilient. Normal situation:

Cashfree Checkout
        ↓
Student pays
        ↓
Automatic confirmation

If Cashfree Checkout has a problem:

Admin → Payment Settings

Cashfree OFF
Custom QR ON
Default = Custom QR

Students continue registering. If Cashfree later activates the dedicated dynamic QR capability:

Cashfree Checkout ON
Cashfree UPI QR ON
Custom QR ON
Default = Cashfree Checkout

And if Razorpay becomes useful again:

Razorpay ON

There is no need to redesign the application.

The Bigger Lesson

The biggest lesson from these two days was not about Razorpay. It was not about Cashfree. It was not even about AWS Lightsail. It was about designing for failure. A production system should assume:

  • payment providers will have problems
  • builds will fail
  • servers will become unhealthy
  • credentials will be misconfigured
  • APIs will return unexpected errors
  • webhooks will be duplicated
  • students will refresh pages
  • administrators will make configuration changes
  • external approvals will take time

The goal is not to prevent every failure. That is impossible. The goal is to make failure safe, visible, recoverable, and operationally manageable. That is why the two biggest architectural improvements from this exercise were actually very simple:

1. Never replace production until the new release is healthy.

2. Never make a business-critical payment method depend on a deployment.

The first led to the release-directory deployment strategy. The second led to runtime payment routing and the Custom QR fallback. Those two decisions transformed the system from:

“It works when everything is working.”

into:

“The system can continue operating when something breaks.”

And that is a much more useful definition of a production-ready system.

Final Architecture at a Glance

                       Zeron Academy
                            |
                    Student Registration
                            |
                    Secure Registration Link
                            |
                    Persisted Registration
                            |
                     Payment Settings
                      (DynamoDB)
                            |
        +-------------------+-------------------+
        |                   |                   |
        v                   v                   v
     Cashfree            Razorpay           Custom QR
     Checkout             Checkout          + UTR
        |                   |                   |
        |                   |                   |
        +-------------------+-------------------+
                            |
                    Payment Confirmation
                            |
                 Shared Finalization Logic
                            |
                    +-------+-------+
                    |               |
                   PAID          Not Paid
                    |
              Registration Link
                    ↓
                  USED


Deployment:

GitHub
  ↓
Persistent Repository Mirror
  ↓
New Release Directory
  ↓
npm ci
  ↓
npm run build
  ↓
Health Check
  ↓
academy-current switch
  ↓
PM2
  ↓
Production

For me, that was the real outcome of these two days. I started by trying to add a payment feature.

I ended up building a small payment platform with provider switching, fallback payment collection, manual verification, safer deployment, runtime configuration, webhook reconciliation, and a deployment process that can fail without immediately taking production down.

That is probably the more important engineering result.

← Writing