# Razorpay Route Linked Accounts & Split Payments Comprehensive technical documentation for the **Automatic Razorpay Route Linked Account Provisioning** and **Zero-Commission Split Payments** system implemented across the platform (Laravel 12, REST API, Redis, MySQL). --- ## Table of Contents - **[Part 1: Linked Account Provisioning](#part-1-linked-account-provisioning)** 1. [Overview](#1-overview) 2. [Business Rules & Policy Enforcement](#2-business-rules--policy-enforcement) 3. [Architecture & Workflow](#3-architecture--workflow) 4. [Database Design: linked_accounts](#4-database-design-linked_accounts) 5. [Core Services & Provisioning Job](#5-core-services--provisioning-job) 6. [Delivery App Blocking Rule](#6-delivery-app-blocking-rule) 7. [Admin Panel UI & Retry Feature](#7-admin-panel-ui--retry-feature) 8. [CLI Commands & Queue Workers](#8-cli-commands--queue-workers) 9. [Provisioning Test Suite](#9-provisioning-test-suite) - **[Part 2: Automatic Route Split Payments (Zero-Commission Model)](#part-2-automatic-route-split-payments-zero-commission-model)** 10. [Split Payment Money Model & Invariants](#10-split-payment-money-model--invariants) 11. [Endpoints & API Contracts](#11-endpoints--api-contracts) 12. [Lifecycle & Driver Assignment Timing](#12-lifecycle--driver-assignment-timing) 13. [Database Schema: Transfers, Ledger & Webhooks](#13-database-schema-transfers-ledger--webhooks) 14. [Admin Configuration & Webhook Secret Security](#14-admin-configuration--webhook-secret-security) 15. [Background Processing & Edge Case Handling](#15-background-processing--edge-case-handling) 16. [Proposal for Driver Reassignment Strategy](#16-proposal-for-driver-reassignment-strategy) 17. [Queue Workers & Operational Commands](#17-queue-workers--operational-commands) 18. [Verification & Full Automated Test Suite](#18-verification--full-automated-test-suite) - **[Part 3: Queue Architecture, Operations & Reliability Layer](#part-3-queue-architecture-operations--reliability-layer)** 19. [Queue Topology & Dedicated Job Matrix](#19-queue-topology--dedicated-job-matrix) 20. [Retry, Backoff & Anti-Double-Processing Guard](#20-retry-backoff--anti-double-processing-guard) 21. [Permanent Failure Auditing & Admin Alerts](#21-permanent-failure-auditing--admin-alerts) 22. [Queue Health Check Command & Scheduler](#22-queue-health-check-command--scheduler) 23. [Laravel Horizon & Linux Supervisor Configuration](#23-laravel-horizon--linux-supervisor-configuration) 24. [Non-Horizon Worker Fallbacks (Windows & cPanel)](#24-non-horizon-worker-fallbacks-windows--cpanel) 25. [Zero-Downtime Deployment & Maintenance Procedures](#25-zero-downtime-deployment--maintenance-procedures) 26. [Queue Hardening Test Inventory](#26-queue-hardening-test-inventory) - **[Part 4: Route Fund Release Layer (Settlement Hold Release)](#part-4-route-fund-release-layer-settlement-hold-release)** 27. [Hold Release Architecture & Workflow](#27-hold-release-architecture--workflow) 28. [Pre-Fetch & Verify Idempotency Pattern](#28-pre-fetch--verify-idempotency-pattern) 29. [Database Design: order_transfer_actions & Monotonic Transitions](#29-database-design-order_transfer_actions--monotonic-transitions) 30. [ReleaseTransferService & OrderPaymentLock](#30-releasetransferservice--orderpaymentlock) 31. [ReleaseOrderTransfersJob & Error Recovery](#31-releaseordertransfersjob--error-recovery) 32. [Release Layer Automated Test Suite](#32-release-layer-automated-test-suite) - **[Part 5: Refund & Transfer Reversal Layer](#part-5-refund--transfer-reversal-layer)** 33. [Automated Refund & Transfer Reversal Architecture](#33-automated-refund--transfer-reversal-architecture) 34. [State Machine Isolation: OrderPaymentStateMachine](#34-state-machine-isolation-orderpaymentstatemachine) 35. [Timeout Recovery & Pre-Flight Idempotency Pattern](#35-timeout-recovery--pre-flight-idempotency-pattern) 36. [Separate Accounting: Gateway Refund vs Promotional Wallet Credit](#36-separate-accounting-gateway-refund-vs-promotional-wallet-credit) 37. [Lifecycle Guards, Pickup & Partial Release Restrictions](#37-lifecycle-guards-pickup--partial-release-restrictions) 38. [Driver Restaurant Closed Reporting Endpoint](#38-driver-restaurant-closed-reporting-endpoint) 39. [Late Payment Capture on Canceled Orders](#39-late-payment-capture-on-canceled-orders) 40. [Admin Resolution: Retry Refund & Mark as Manually Refunded](#40-admin-resolution-retry-refund--mark-as-manually-refunded) 41. [Automatic Customer Cash Wallet Fallback & Double-Refund Safeguard](#41-automatic-customer-cash-wallet-fallback--double-refund-safeguard) 42. [Two-Connection Concurrency Proof & Honest Limitations](#42-two-connection-concurrency-proof--honest-limitations) 43. [Unverified Live Razorpay Sandbox Behaviors](#43-unverified-live-razorpay-sandbox-behaviors) 44. [Refund Automated Test Suite (21 Tests, 113 Assertions)](#44-refund-automated-test-suite-21-tests-113-assertions) 45. [Global Test Suite Summary (67 Tests, 409 Assertions)](#45-global-test-suite-summary-67-tests-409-assertions) --- # Part 1: Linked Account Provisioning ## 1. Overview Razorpay Route allows automated split payouts and transfers to sub-merchants (Restaurants and Delivery Partners). This implementation provisions Route Linked Accounts directly using existing database fields without introducing cumbersome onboarding forms: - **Zero New Forms:** Reuses bank records from the `vendors` table and `disbursement_withdrawal_methods` table. - **Asynchronous & Non-Blocking:** Dispatched on the dedicated `gateway` queue. - **Idempotent & Retry-Safe:** Avoids duplicate accounts on Razorpay via unique constraints and `X-Pgp-Idempotency-Key` headers. - **Real-Time Admin Visibility:** Live status badges and AJAX retry buttons across Restaurant and Delivery Man lists. --- ## 2. Business Rules & Policy Enforcement ### 2.1. Business Type Defaults To keep onboarding lightweight without requiring database schema changes for company structures: - **Delivery Men:** Hardcoded to **`individual`**. - **Restaurants / Vendors:** Hardcoded to **`proprietorship`** (simplest documentation tier under Razorpay Route). ### 2.2. Strict IFSC Code Validation (No Fallback) - Indian IFSC codes must strictly match the RBI format: ```regex ^[A-Z]{4}0[A-Z0-9]{6}$ ``` - Must be **exactly 11 characters**, case-insensitive, with `0` in the 5th position. - If the regex fails or Razorpay rejects the bank account, **no fallback is attempted**. The status is marked as `action_required` or `rejected`, and the delivery man or vendor is blocked from accepting orders until valid details are provided. ### 2.3. Delivery App Strict Blocking Rule - **Cannot Go Online:** Delivery partners without an active, verified Razorpay Linked Account are strictly prohibited from toggling their status to "Online" in the Flutter Delivery App. - **Cannot Accept Orders:** Direct order acceptance (`accept-order` endpoint) is blocked. - **Cannot Be Assigned:** The Admin and Vendor panels reject order assignments to delivery men who do not have an active linked account. - **Auto-Provisioning Trigger:** The moment a delivery man adds or updates a bank withdrawal method in the app, a provisioning job is automatically queued. --- ## 3. Architecture & Workflow ``` [ Restaurant Approved / Status Toggled ] [ Delivery Man Adds Bank Details / Approved ] │ │ ▼ ▼ VendorController / RestaurantController DeliverymanController │ │ └───────────────────────┬─────────────────────────────┘ │ ▼ [ ProvisionLinkedAccountJob ] (Queue: "gateway") │ ▼ [ RazorpayRouteService ] │ ┌───────────────────┴───────────────────┐ │ │ (Data Valid) (Data Invalid) │ │ POST /v2/accounts Mark status: "action_required" POST /v2/accounts/{id}/products (route) Store last_error in DB POST /v2/accounts/{id}/bank_accounts Block from Online & Orders │ Mark status: "active" ``` --- ## 4. Database Design: linked_accounts ### Table: `linked_accounts` Migration file: `database/migrations/2026_11_02_000001_create_linked_accounts_table.php` | Column | Type | Description | | :--- | :--- | :--- | | `id` | `BIGINT UNSIGNED` (PK) | Primary key | | `accountable_type` | `VARCHAR(191)` | Polymorphic model (`App\Models\Restaurant` or `App\Models\DeliveryMan`) | | `accountable_id` | `BIGINT UNSIGNED` | ID of the linked model | | `razorpay_account_id` | `VARCHAR(100)` (Nullable) | Razorpay Account ID (e.g., `acc_xxxxxxxxxxxxxx`) | | `legal_business_name` | `VARCHAR(191)` (Nullable) | Business / individual name passed to Razorpay | | `business_type` | `VARCHAR(50)` | `individual` (Delivery Man) or `proprietorship` (Restaurant) | | `status` | `ENUM` | `'unlinked'`, `'pending'`, `'active'`, `'rejected'`, `'action_required'` | | `kyc_status` | `VARCHAR(50)` | KYC verification state (`under_review`, `activated`, `rejected`) | | `banking_status` | `VARCHAR(50)` | Settlement bank account state (`pending`, `active`, `failed`) | | `last_error` | `TEXT` (Nullable) | Explanatory error message for rejected / invalid attempts | | `metadata` | `JSON` (Nullable) | Full JSON response payload from Razorpay | | `created_at` / `updated_at` | `TIMESTAMP` | Timestamps | --- ## 5. Core Services & Provisioning Job 1. **`AccountDataMapper`** (`app/Services/RazorpayRoute/AccountDataMapper.php`): - Maps Eloquent attributes to Razorpay Route V2 payload. - Cleans phone numbers (extracts 10 digits, strips country codes). - Validates IFSC format. 2. **`RazorpayRouteService`** (`app/Services/RazorpayRoute/RazorpayRouteService.php`): - Authenticates using credentials from `addon_settings`. - Calls `POST /v2/accounts`, `POST /v2/accounts/{id}/products`, and `POST /v2/accounts/{id}/bank_accounts`. 3. **`ProvisionLinkedAccountJob`** (`app/Jobs/ProvisionLinkedAccountJob.php`): - Dispatched to `gateway` queue on model approval or bank details update. --- ## 6. Delivery App Blocking Rule In `DeliverymanController.php`: ```php if (!$dm || !$dm->hasActiveLinkedAccount()) { return response()->json([ 'errors' => [ [ 'code' => 'bank_details_required', 'message' => translate('Please update your Bank Details in the app to receive orders.') ] ] ], 403); } ``` --- ## 7. Admin Panel UI & Retry Feature - **Badges:** - `Active` (Green): Account provisioned and KYC active. - `Action Required` (Red): Missing or invalid bank details. - `Pending` (Yellow): Account creation queued or KYC under review. - **One-Click AJAX Retry:** Allows administrators to re-dispatch provisioning after fixing details. --- ## 8. CLI Commands & Queue Workers ```bash # 1. Dry run - inspect data readiness without making any API calls php artisan razorpay:route-sync --dry-run # 2. Dispatch all eligible records to the 'gateway' queue php artisan razorpay:route-sync --queue # 3. Synchronously sync only Restaurants php artisan razorpay:route-sync --type=restaurants # 4. Synchronously sync only Delivery Men php artisan razorpay:route-sync --type=delivery-men ``` --- ## 9. Provisioning Test Suite ```bash php artisan test tests/Unit/RazorpayRouteTest.php ``` - `test_ifsc_code_validation`: Tests valid RBI codes vs invalid length/format codes. - `test_phone_cleaner`: Verifies 10-digit normalization. - `test_default_business_types`: Asserts `proprietorship` for Restaurants and `individual` for Delivery Men. - `test_delivery_man_app_blocking_rule`: Verifies HTTP 403 response when delivery man attempts to go online without an active linked account. - `test_razorpay_route_provisioning_with_mock`: Fakes Razorpay Route V2 endpoints, ensuring idempotency and status updating. --- # Part 2: Automatic Route Split Payments (Zero-Commission Model) This section details the end-to-end split payment implementation for orders paid online via Razorpay. Under this model, order amounts are split at the gateway level into on-hold transfers for the restaurant and delivery partner, with platform fees and taxes retained directly in the platform master balance. --- ## 10. Split Payment Money Model & Invariants All monetary calculations are performed strictly on the server in **integer paise** (1 INR = 100 paise) to prevent floating-point discrepancies. ### 10.1. Amount Allocation & 6-Component Arithmetic | Component | Destination | Settlement Mechanism | | :--- | :--- | :--- | | **Gross Food Amount & Packaging** | Restaurant Linked Account | Initial Razorpay Order `transfers[]` on hold | | **Delivery Charge** | Delivery Partner Linked Account | Post-assignment transfer via `POST /v1/payments/{id}/transfers` on hold | | **Driver Tips** | Delivery Partner Linked Account | Post-assignment transfer bundled with delivery charge on hold | | **Handling & Convenience Fee** | Platform Master Account | Retained directly in Razorpay master balance | | **Additional Charges** | Platform Master Account | Retained directly in Razorpay master balance | | **Order GST / Taxes (5%)** | Platform Master Account | Retained directly in Razorpay master balance per **Section 9(5) CGST Act** | #### Complete Sample Arithmetic Breakdown: ``` Order Total: ₹550.00 (55,000 paise) ├── Restaurant Food & Packaging: ₹455.00 (45,500 paise) ──► Restaurant Route Transfer (On Hold) ├── Delivery Charge: ₹40.00 (4,000 paise) ──┐ ├── Driver Tip: ₹10.00 (1,000 paise) ──┴─► Driver Route Transfer (On Hold) ├── Handling Fee: ₹15.00 (1,500 paise) ──┐ ├── GST (5% per Sec 9(5)): ₹25.00 (2,500 paise) ────► Retained in Platform Balance └── Additional Charge: ₹5.00 (500 paise) ──┘ Total Sum: 45,500 + 4,000 + 1,000 + 1,500 + 2,500 + 500 = 55,000 paise (₹550.00) ``` ### 10.2. Tax Remittance Under Section 9(5) CGST Act Under Section 9(5) of the Indian CGST Act, electronic commerce operators are legally mandated to collect and remit the 5% GST directly to the government. Consequently: - `total_tax_amount` **MUST NOT** flow to the restaurant linked account. - The restaurant transfer is strictly `Gross Food Amount - Discounts`. - Platform retains GST bundled with platform convenience fees (`config('fee.remit_restaurant_tax_by_platform', true)`). ### 10.3. Partial Wallet Payment Allocation & Non-Negative Formula When a customer pays partially using their promotional wallet balance: 1. **Priority Rule:** Driver Share (`delivery_charge + tips`) and Admin Share (`handling_fee + additional_charge + tax`) take absolute priority from the actual captured online funds. 2. **Non-Negative Formula:** The restaurant's Razorpay Route transfer is calculated with strict lower-bound protection so it can **never evaluate below zero**: $$\text{Restaurant Transfer} = \max(0, \min(\text{Restaurant Share}, \text{Online Paid} - \text{Driver Share} - \text{Admin Share}))$$ 3. **Negative Formula Edge Case:** If wallet payment covers almost the entire order such that $\text{Online Paid} < \text{Driver Share} + \text{Admin Share}$, the intermediate difference is negative, but $\max(0, \dots)$ caps the restaurant transfer at **0 paise**. 4. **Residual Balance:** Any remaining balance owed to the restaurant (covered by the customer's wallet) flows into the internal vendor payout ledger for settlement. ### 10.4. Mathematical Invariant & Code Assertion $$\sum \text{Transfers} = \text{Restaurant Transfer} + \text{Driver Transfer} \le \text{Online Paid Amount}$$ Enforced in [`OrderSplitCalculator.php`](file:///E:/Freelancer/app/Services/RazorpayRoute/OrderSplitCalculator.php): ```php if ($totalTransfersPaise > $onlinePayablePaise) { throw new \LogicException("Math invariant assertion failed: Total transfers ({$totalTransfersPaise} paise) exceed online payable ({$onlinePayablePaise} paise)."); } ``` --- ## 11. Endpoints & API Contracts Both `/api/payment/...` and `/api/v1/payment/...` routes are supported. ### 11.1. Create Payment Order `POST /api/payment/create-order` (Authenticated Customer / Guest with session) - **Request Body:** ```json { "order_id": 12345 } ``` *(Security rule: Any client-supplied amounts, fees, or account IDs are strictly discarded. All amounts are recalculated authoritatively server-side.)* - **Pre-Flight Checks:** - Order must exist, be unpaid, and not be canceled. - Restaurant must be open and active. - Restaurant must possess an active, verified linked account (`kyc_status = 'activated'`). - Cash on delivery (COD) orders are rejected with `COD_ORDER`. - **Success Response (HTTP 200):** ```json { "success": true, "data": { "razorpay_order_id": "order_Oq7JzQXXXXXXX", "key_id": "rzp_live_xxxxxxxxxxxxxx", "amount_paise": 55000, "amount_rupees": 550.00, "currency": "INR", "display": { "order_amount": 550.00, "wallet_deducted": 0.00, "payable_online": 550.00, "delivery_charge": 40.00, "handling_fee": 15.00, "tax": 25.00, "additional_charge": 5.00, "tips": 10.00 } } } ``` *(API Security Rule: Customer API responses strictly omit internal transfer amounts, splits, and linked account IDs).* - **Idempotency:** Re-calling this endpoint with an unchanged order returns the existing `razorpay_order_id` without creating duplicate Razorpay orders. ### 11.2. Client Signature Verification Callback `POST /api/payment/verify-signature` (Authenticated Customer / Guest) - **Request Body:** ```json { "razorpay_order_id": "order_Oq7JzQXXXXXXX", "razorpay_payment_id": "pay_Oq7K1AXXXXXXX", "razorpay_signature": "9b7f5...hmac_sha256..." } ``` - **Behavior:** - Verifies timing-safe HMAC SHA256 signature using the Razorpay API secret. - **State rule:** Sets order status to `processing` for responsive UI state. **Never** marks order as `paid` or `paid_held` on the client callback alone. Authoritative payment marking is handled exclusively by the webhook. ### 11.3. Authoritative Webhook Receiver `POST /api/payment/webhook/razorpay` (Public, CSRF-Exempt, Rate-Limited) - **Middleware:** Excluded from CSRF, protected with `throttle:60,1`. - **Headers:** `X-Razorpay-Signature: ` - **Behavior:** - Verifies HMAC SHA256 signature of the raw request body using `webhook_secret` fetched from `addon_settings`. - Deduplicates events using unique index on `gateway_webhook_events.event_id`. - Dispatches `ProcessRazorpayWebhookJob` onto the `webhooks` queue. - Returns HTTP 200 immediately (`{"message": "Webhook received successfully."}`). --- ## 12. Lifecycle & Driver Assignment Timing Since delivery partners are assigned dynamically after customer checkout: ``` [ Customer Checkouts ] │ ▼ POST /api/payment/create-order ──► Razorpay Order created with Restaurant Transfer (ON HOLD) │ ▼ Customer Completes Payment │ ▼ Razorpay Webhook ──► payment.captured ──► Order marked 'paid_held' & 'confirmed' ──► Restaurant notified via FCM │ [ Order Dispatched to Drivers ] │ ▼ Driver Accepts Order / Admin Assigns Driver / Vendor Assigns Driver │ ▼ TransferDriverDeliveryFeeJob (Queue: "gateway") │ ▼ POST /v1/payments/{payment_id}/transfers ──► Delivery Fee + Tips transferred to Driver (ON HOLD) ``` --- ## 13. Database Schema: Transfers, Ledger & Webhooks ### 1. Table: `order_transfers` Migrations: - `database/migrations/2026_11_03_000002_create_order_transfers_table.php` - `database/migrations/2026_11_03_000005_update_order_transfers_table.php` Tracks all gateway transfers associated with an order. | Column | Type | Description | | :--- | :--- | :--- | | `id` | `BIGINT UNSIGNED` (PK) | Primary key | | `order_id` | `BIGINT UNSIGNED` (Index) | Foreign ID to `orders.id` | | `type` | `VARCHAR(30)` | `'restaurant'` or `'delivery'` | | `razorpay_transfer_id` | `VARCHAR(100)` (Nullable) | Razorpay Transfer ID (`trf_xxxxxxxx`) | | `linked_account_id` | `VARCHAR(100)` | Razorpay Linked Account ID (`acc_xxxxxxxx`) | | `amount_paise` | `BIGINT UNSIGNED` | Transferred amount in paise | | `status` | `VARCHAR(50)` | `'created'`, `'processed'`, `'on_hold'`, `'released'`, `'reversed'`, `'failed'` | | `on_hold` | `TINYINT(1)` | `1` (True) while order is active | | `error_message` | `TEXT` (Nullable) | Gateway error message if transfer failed | | `created_at` / `updated_at` | `TIMESTAMP` | Timestamps | - **Unique Index:** `['order_id', 'type']` (Strictly guarantees at most 1 restaurant transfer and 1 delivery transfer per order). ### 2. Table: `payment_ledger` Migration: `database/migrations/2026_11_03_000003_create_payment_ledger_table.php` Immutable financial audit log recording all payment lifecycle events: - `razorpay_order_created` - `client_signature_verified` - `payment_captured` - `payment_failed` - `payment_amount_mismatch` - `driver_transfer_created` - `driver_transfer_blocked_no_account` - `transfer_processed` - `transfer_failed` ### 3. Table: `gateway_webhook_events` Migration: `database/migrations/2026_11_03_000004_create_gateway_webhook_events_table.php` Provides idempotent deduplication of all incoming Razorpay webhooks. ### 4. Columns Added to `orders` Table Migration: `database/migrations/2026_11_03_000001_add_razorpay_columns_to_orders_table.php` - `razorpay_order_id` (`VARCHAR(100)`, indexed) - `razorpay_payment_id` (`VARCHAR(100)`, indexed) - `paid_at` (`TIMESTAMP`, nullable) --- ## 14. Admin Configuration & Webhook Secret Security Admins can configure the Webhook Secret directly from the Admin Panel without modifying `.env`: 1. Navigate to **Admin Panel > Third Party > Payment Methods > Razorpay**. 2. **Encrypted Storage:** Secrets submitted through the UI are encrypted via `Illuminate\Support\Facades\Crypt::encryptString()` before saving to `addon_settings`. 3. **Password Masking:** The input is rendered as `type="password"` and masked with `••••••••`, ensuring plain text credentials are never displayed on screen. 4. **Zero Log Leaks:** The webhook secret is strictly excluded from logging statements. 5. The read-only **Webhook URL** is displayed directly beneath the field: ``` https://yourdomain.com/api/v1/payment/webhook/razorpay ``` 6. **Events to subscribe to in Razorpay Dashboard:** - `payment.captured` - `payment.failed` - `order.paid` - `transfer.processed` - `transfer.failed` --- ## 15. Background Processing & Edge Case Handling ### 15.1. `TransferDriverDeliveryFeeJob` (`gateway` queue) Dispatched whenever a driver is assigned in: - `DeliverymanController@accept_order` - `Admin\OrderController@add_delivery_man` - `Vendor\OrderController@add_delivery_man` **Edge Case Rules Enforced:** - **(a) Unlinked Driver:** If the driver has no active linked account, the job **does not call Razorpay**. It creates `OrderTransfer` with `status = 'failed'`, logs `driver_transfer_blocked_no_account` in `payment_ledger`, raises an Admin Alert, and retains the amount in the internal payout ledger. - **(b) Payment Not Yet Captured:** If the driver is assigned before the payment webhook arrives (`payment_status !== 'paid_held'`), the job releases itself with a 15-second delay (`$this->release(15)`) to wait and retry until capture. - **(c) Driver Reassignment Guard:** If an on-hold or processed delivery transfer already exists for this order, the job skips creating a second transfer to prevent duplicate payouts. - **(d) Idempotency:** Guarded against redundant queue events. ### 15.2. `ProcessRazorpayWebhookJob` (`webhooks` queue) - **Authoritative Amount Check:** If the captured amount does not match `online_payable_paise`, the job logs `payment_amount_mismatch` in `payment_ledger` and **refuses to mark the order as paid**. - **`payment.captured`:** Marks `payment_status = 'paid_held'`, sets `order_status = 'confirmed'`, saves `razorpay_payment_id` and `paid_at`, logs to `payment_ledger`, and sends FCM notifications. - **`payment.failed`:** Marks `payment_status = 'failed'` without canceling the order, leaving it retryable. - **`transfer.processed`:** Updates `OrderTransfer.status = 'on_hold'` and logs to `payment_ledger`. - **`transfer.failed`:** Updates `OrderTransfer.status = 'failed'`, records error in `payment_ledger`, and dispatches an Admin Alert notification. --- ## 16. Proposal for Driver Reassignment Strategy When Driver A is assigned and a delivery fee transfer is placed on hold, and the order is subsequently reassigned to Driver B: 1. **Option A: Internal Ledger Settlement (Recommended)** - Maintain the initial gateway transfer on hold until delivery. - Upon delivery completion, release the on-hold transfer to platform balance/payout, and credit Driver B inside the platform's internal earnings ledger (`delivery_man_wallets`). - *Benefits:* Eliminates gateway transfer reversal fees and reduces webhook/API race conditions. 2. **Option B: Razorpay Route Transfer Reversal API** - Call Razorpay's `POST /v1/transfers/{transfer_id}/reversals` to pull the funds back from Driver A to the master account. - Once reversed, initiate a new `POST /v1/payments/{payment_id}/transfers` to Driver B on hold. - *Trade-off:* Reversals are permitted only while funds are on hold, but introduce API latency and gateway failure dependencies. --- ## 17. Queue Workers & Operational Commands Ensure queue workers process both the `webhooks` and `gateway` queues: ```bash php artisan queue:work --queue=webhooks,gateway,default --tries=3 ``` --- ## 18. Verification & Full Automated Test Suite Run the full automated test suite covering all Route Split Payment mechanics and provisioning: ```bash # Run Split Payments Suite (17 tests, 100 assertions) php artisan test tests/Unit/RazorpayRouteSplitPaymentTest.php # Run Linked Account Provisioning Suite (5 tests, 34 assertions) php artisan test tests/Unit/RazorpayRouteTest.php ``` ### Full Test Inventory (22 Tests, 134 Assertions): #### `RazorpayRouteSplitPaymentTest.php` (17 Tests) 1. **`test_split_calculator_paise_math_and_tax_remittance`**: Verifies integer paise calculation, Section 9(5) platform tax retention, and invariant $\sum \text{transfers} \le \text{amount}$. 2. **`test_partial_wallet_payment_prioritizes_driver_and_admin_fees`**: Verifies driver and platform fee prioritization from captured funds and capped restaurant transfer. 3. **`test_wallet_payment_making_formula_negative_is_capped_at_zero`**: Verifies that when wallet coverage makes the formula difference negative, the transfer evaluates to 0 paise without throwing underflow. 4. **`test_create_order_preflight_checks`**: Verifies rejection of paid, canceled, or inactive restaurant orders. 5. **`test_create_order_successful_and_idempotent`**: Mocks Orders API, verifies DB updates, transfer records, and idempotency. 6. **`test_create_order_response_does_not_expose_transfer_data`**: Asserts customer API response strictly omits internal transfer amounts and accounts. 7. **`test_client_supplied_amount_fee_and_account_id_are_ignored`**: Verifies server recalculation ignores forged amounts, fees, and account IDs. 8. **`test_verify_signature_updates_processing_status_without_marking_paid`**: Tests client HMAC verification and asserts transition to `processing` rather than `paid_held`. 9. **`test_webhook_payment_captured_marks_paid_held_and_idempotent`**: Tests authoritative webhook payment capture and deduplication. 10. **`test_paid_amount_mismatch_is_rejected_and_logged`**: Verifies tampered payment amounts are rejected and logged in `payment_ledger`. 11. **`test_invalid_webhook_signature_returns_400_and_changes_nothing`**: Verifies bad HMAC signatures return 400 with no DB modifications. 12. **`test_webhook_payment_failed_leaves_order_retryable`**: Verifies payment failure marks `failed` while order remains retryable. 13. **`test_cod_orders_are_unaffected`**: Confirms COD orders reject payment creation and skip gateway transfers. 14. **`test_webhook_secret_read_from_addon_settings_and_never_logged`**: Verifies secret retrieval and ensures no log leaks. 15. **`test_driver_assignment_dispatches_fee_transfer`**: Verifies driver transfer on hold via `POST /v1/payments/{id}/transfers`. 16. **`test_driver_edge_cases_no_linked_account_and_idempotency`**: Tests unlinked driver failure handling, payout ledger retention, and reassignment duplication guard. 17. **`test_transfer_processed_and_transfer_failed_events`**: Tests webhook handlers for `transfer.processed` and `transfer.failed` (with admin alert). #### `RazorpayRouteTest.php` (5 Tests) 18. **`test_ifsc_code_validation`**: Validates 11-character Indian RBI IFSC format. 19. **`test_phone_cleaner`**: Verifies phone normalization. 20. **`test_default_business_types`**: Verifies `proprietorship` and `individual` defaults. 21. **`test_delivery_man_app_blocking_rule`**: Verifies HTTP 403 when unlinked delivery man attempts to go online. 22. **`test_razorpay_route_provisioning_with_mock`**: Verifies Route V2 account and bank account creation. --- # Part 3: Queue Architecture, Operations & Reliability Layer This section outlines the production-grade queue layer engineered to ensure high webhook delivery reliability, reliable transfer processing with database action ledger, observable health monitoring, and graceful degradation across hosting environments. --- ## 19. Queue Topology & Dedicated Job Matrix To prevent slow external communications (SMTP/FCM) from starving real-time payment webhooks and Route gateway transfers, jobs are isolated across 5 dedicated queues: | Queue | Primary Job(s) | Tries | Backoff (sec) | Timeout | Purpose & Priority | | :--- | :--- | :---: | :---: | :---: | :--- | | **`webhooks`** | `ProcessRazorpayWebhookJob` | 3 | `[10, 60, 300]` | 45s | High priority. Processes inbound webhook events (`payment.captured`, `transfer.processed`, etc.). Offloads notifications to keep execution fast. | | **`gateway`** | `TransferDriverDeliveryFeeJob`, `ProvisionLinkedAccountJob`, `ReleaseOrderTransfersJob` | 3 | `[10, 60, 300]` | 45s | Outbound Razorpay Route V2 API calls (Direct transfers, onboarding, KYC sync, hold release). | | **`notifications`** | `SendOrderNotificationJob` | 3 | `[5, 15, 30]` | 30s | External communications (Firebase FCM, Push, SMS, Mail). Non-blocking to payment pipeline. | | **`refunds`** | *Reserved for upcoming Refund jobs* | 3 | `[10, 60, 300]` | 45s | Gateway refunds & Route transfer reversals. | | **`default`** | General application jobs | 1 | None | 60s | Reports, exports, housekeeping. | ### Webhook Security & Rate Limiting - **Endpoints:** `POST /api/payment/webhook/razorpay` and `POST /api/v1/payment/webhook/razorpay`. - **CSRF Exemption:** Both URLs are explicitly excluded from CSRF validation in `App\Http\Middleware\VerifyCsrfToken`. - **Burst Rate Limiting:** Enforced at `throttle:120,1` (120 requests per minute per IP) to absorb webhook retries and payment flash surges from Razorpay. - **HMAC SHA-256 Verification:** Computed on the **raw body** (`$request->getContent()`) using timing-safe `hash_equals()` against the secret stored in admin `addon_settings`. Mismatched signatures are rejected with HTTP 400 immediately. --- ## 20. Retry, Backoff & Anti-Double-Processing Guard ### 20.1. Timeout < Retry-After Invariant In Laravel queues on Redis, when a worker picks up a job, Redis reserves it for `retry_after` seconds. If the worker does not finish and delete the job within `retry_after` seconds, Redis assumes the worker crashed and releases the job to a second worker. > [!IMPORTANT] > **Only job execution timeout matters for `retry_after`**, not the backoff delay. When a job is released with backoff, it is placed back onto Redis in the delayed set outside active worker execution time. To prevent duplicate worker execution while a job is still processing: - **Redis `retry_after`** is configured to **`360 seconds`** in `config/queue.php`. - **Job Timeouts** are capped at **`45 seconds`** for `gateway`, `webhooks`, and `refunds` jobs, and **`30 seconds`** for `notifications`. - Because $45\text{s} < 360\text{s}$, a hanging worker will be forcefully timed out and failed locally long before Redis releases the job reservation: $$\text{Job Timeout (45s)} < \text{Redis retry\_after (360s)}$$ > [!WARNING] > **ext-pcntl and Job Timeouts ($timeout):** > Laravel's job timeout enforcement (`$timeout = 45`) relies on PHP asynchronous process signals (`SIGALRM`), which require the `ext-pcntl` extension. In environments lacking `ext-pcntl` (e.g., Windows CLI, shared cPanel cron jobs), Laravel cannot forcefully terminate hanging in-flight PHP processes. Consequently, production workers MUST run on Linux environments with `ext-pcntl` and `ext-posix` installed (supervised by Supervisor or Horizon). Do NOT run continuous background daemons or payment queues from raw cron without a supervisor. ### 20.2. Fast-Fail on Non-Retryable 4xx / Business Constraint Errors Retrying unrecoverable requests (e.g. invalid IFSC, invalid bank format, rejected KYC, or captured amount tampering) wastes worker cycles. - The system throws **`App\Exceptions\NonRetryablePaymentException`**. - The job catches this exception and calls `$this->fail($exception)` immediately—failing fast without burning remaining retries. --- ## 21. Permanent Failure Auditing & Admin Alerts Every queued job implements the `failed(\Throwable $exception)` lifecycle hook: 1. **Immutable Audit Ledger:** Records a sanitized exception message (`strip_tags`, max 255 chars) in `payment_ledger`. **No stack traces, request data, or credentials** are stored. 2. **Real-Time Admin Push Notification:** Sends an alert via `Helpers::send_push_notif_to_topic($data, 'admin_message', 'order_request')` wrapped in error-handling so push failures never mask job termination. 3. **Database Notification:** Inserts an actionable alert row into the `notifications` table for immediate inspection in the Admin Panel. --- ## 22. Queue Health Monitoring & Automated Sweeper ### 22.1. Health Check Artisan Command Monitors only critical payment queues (`gateway`, `webhooks`, `refunds`), strictly excluding unrelated default-queue failures: ```bash php artisan queue:health-check --threshold=50 --failed-threshold=5 --age-threshold=300 ``` **Checks performed:** 1. **Failed Jobs:** Queries `failed_jobs` for entries in `['gateway', 'webhooks', 'refunds']` newer than the previous health check timestamp (or if total failures $\ge 5$). 2. **Backlog Depth:** Inspects waiting job counts; alerts if depth $\ge 50$. 3. **Queue Wait Age:** Inspects the oldest pending job payload in FIFO Redis queues; alerts if oldest job age $\ge 300\text{s}$ (5 minutes). 4. If issues are found, logs `CRITICAL` error, dispatches admin alert, and returns exit code `1` (`Command::FAILURE`). 5. If healthy, returns exit code `0` (`Command::SUCCESS`). ### 22.2. Stuck Webhook Sweeper Command Handles rare edge cases where an inbound webhook was received and stored in `gateway_webhook_events` with status `'received'`, but worker dispatch threw an exception (e.g., temporary Redis disconnection): ```bash php artisan razorpay:sweep-webhooks --minutes=2 ``` Scans for events with `status = 'received'` older than 2 minutes, re-dispatches them to the `webhooks` queue, and transitions them to `'queued'`. ### 22.3. Cron Schedule Configuration In `routes/console.php` (the single scheduler registration file loaded via `bootstrap/app.php`): ```php // Queue health check runs every 5 minutes Schedule::command('queue:health-check')->everyFiveMinutes(); // Stuck webhook event sweeper runs every minute Schedule::command('razorpay:sweep-webhooks')->everyMinute(); ``` **Required Server Crontab:** ```bash * * * * * cd /var/www/stackfood && php artisan schedule:run >> /dev/null 2>&1 ``` --- ## 23. Laravel Horizon & Linux Supervisor Configuration ### 23.1. Horizon Dashboard & Security Gate Laravel Horizon (`laravel/horizon:^5.50`) is installed and configured in `config/horizon.php`. - **Dashboard URL:** `https://your-domain.com/horizon` - **Security:** Access is strictly guarded in `App\Providers\HorizonServiceProvider` using `Auth::guard('admin')->check()`, ensuring only authenticated Admin Panel users have access. ### 23.2. Horizon Supervisor Configuration (`config/horizon.php`) Both `production` and `local` environment profiles map directly to supervisors: `supervisor-webhooks`, `supervisor-gateway`, `supervisor-notifications`, `supervisor-refunds`, and `supervisor-default`. ### 23.3. Production Linux Supervisor Config (`/etc/supervisor/conf.d/horizon.conf`) ```ini [program:stackfood-horizon] process_name=%(program_name)s command=php /var/www/stackfood/artisan horizon autostart=true autorestart=true user=www-data redirect_stderr=true stdout_logfile=/var/www/stackfood/storage/logs/horizon.log stopwaitsecs=360 ``` ### 23.4. Horizon Deploy Verification Check Before launching Horizon in CI/CD pipelines, container entrypoints, or deployment scripts, verify that both required Unix process extensions are present: ```bash php -r "extension_loaded('pcntl') && extension_loaded('posix') ? exit(0) : exit(1);" || { echo "ERROR: Horizon requires ext-pcntl and ext-posix."; exit 1; } ``` --- ## 24. Non-Horizon Worker Fallbacks (Windows & cPanel) Because native Windows and shared cPanel environments lack Unix extensions (`pcntl`, `posix`), full native fallbacks are provided: ### 24.1. Platform Configuration in Composer In `composer.json`: ```json "config": { "platform": { "ext-pcntl": "8.4.15", "ext-posix": "8.4.15" } } ``` This enables normal `composer install` on Windows and cPanel without needing platform ignore flags. If installing on older Composer versions without this config, use: ```bash composer install --ignore-platform-req=ext-pcntl --ignore-platform-req=ext-posix ``` ### 24.2. Windows Local Development Run the worker directly against the local Redis instance (`npx redis-local` / port 6379): ```powershell php artisan queue:work redis --queue=webhooks,gateway,notifications,refunds,default --tries=3 ``` ### 24.3. Linux Supervisor (`queue:work` without Horizon) File: `/etc/supervisor/conf.d/stackfood-worker.conf` ```ini [program:stackfood-worker] process_name=%(program_name)s_%(process_num)02d command=php /var/www/stackfood/artisan queue:work redis --queue=webhooks,gateway,refunds,notifications,default --sleep=3 --tries=3 --max-time=3600 autostart=true autorestart=true user=www-data numprocs=4 redirect_stderr=true stdout_logfile=/var/www/stackfood/storage/logs/worker.log stopwaitsecs=360 ``` ### 24.4. cPanel / Shared Hosting Cron Fallback (with `flock` Overlap Protection) When process supervisors are unavailable on shared hosting, configure a 1-minute crontab using `flock -n` to prevent concurrent process overlap: ```bash * * * * * flock -n /tmp/stackfood_queue.lock -c "php /home/user/public_html/artisan queue:work redis --queue=webhooks,gateway,refunds,notifications,default --stop-when-empty --tries=3" >> /dev/null 2>&1 ``` --- ## 25. Zero-Downtime Deployment & Maintenance Procedures ```bash # 1. If using Horizon: php artisan horizon:terminate # 2. If using standard Supervisor queue:work: php artisan queue:restart # 3. Inspect failed jobs: php artisan queue:failed # 4. Retry all failed payment jobs: php artisan queue:retry all ``` --- ## 26. Queue Hardening Test Inventory ### Test Suites Execution ```bash # Run Queue & Webhook Hardening Suite (6 tests, 36 assertions) php artisan test tests/Unit/QueueAndWebhookHardeningTest.php # Run Split Payments Suite (17 tests, 100 assertions) php artisan test tests/Unit/RazorpayRouteSplitPaymentTest.php # Run Linked Account Provisioning Suite (5 tests, 34 assertions) php artisan test tests/Unit/RazorpayRouteTest.php ``` ### Tests in `QueueAndWebhookHardeningTest.php`: 23. **`test_jobs_route_to_designated_queues`**: Fakes the queue and asserts `ProcessRazorpayWebhookJob` dispatches to `webhooks`, `TransferDriverDeliveryFeeJob` and `ProvisionLinkedAccountJob` dispatch to `gateway`, and `SendOrderNotificationJob` dispatches to `notifications`. 24. **`test_job_retry_backoff_and_timeout_parameters`**: Asserts `tries = 3`, `backoff = [10, 60, 300]`, job timeouts (45s/30s), and verifies timeouts are strictly shorter than Redis `retry_after` (360s). 25. **`test_webhook_fast_fails_on_amount_mismatch_exception`**: Tests that non-retryable payment mismatches trigger immediate fast-fail and ledger audit without burning tries. 26. **`test_failed_handlers_record_ledger_audits`**: Verifies that when any job encounters permanent failure, the `failed()` hook logs to `payment_ledger` and marks accounts rejected. 27. **`test_webhook_controller_deduplication_and_atomic_catch`**: Tests atomic unique constraint catch and 200 return during concurrent duplicate webhook deliveries. 28. **`test_queue_health_check_command`**: Asserts `queue:health-check` returns exit code 0 on clean queues and returns exit code 1 (`Command::FAILURE`) when a failed job is detected. --- # Part 4: Route Fund Release Layer (Settlement Hold Release) ## 27. Progressive Hold Release Architecture & Workflow Under the zero-commission split payment architecture, customer payments are routed into separate vendor and driver transfers with `on_hold = 1` (true). Rather than releasing all funds at final completion, releases are **progressive and independent**, executing at two distinct stages of the order lifecycle: 1. **Stage 1 — Restaurant Transfer Release:** - **Trigger:** Driver taps **'Reached'** at the restaurant and confirms the **'Open'** Yes/No prompt. - **Purpose:** Guarantees the restaurant's payout is settled before food packaging and handover. - **Invoked via:** `ReleaseTransferService::releaseRestaurantTransfer($order)` or `ReleaseOrderTransfersJob::dispatch($order->id, 'restaurant')`. - **Order Payment Status:** Remains `paid_held` because the driver fee is still on hold. 2. **Stage 2 — Delivery Man Transfer Release:** - **Trigger:** Order transitions to **'delivered'** (Completed). - **Purpose:** Releases the delivery partner's delivery fee and tips upon confirmed delivery. - **Invoked via:** `ReleaseTransferService::releaseDeliveryTransfer($order)` or `ReleaseOrderTransfersJob::dispatch($order->id, 'delivery')`. - **Order Payment Status:** Transitions from `paid_held` to `paid` once all order transfers are released. ``` [ Driver Taps 'Reached' & Confirms 'Open' ] [ Order Delivered to Customer ] │ │ ▼ ▼ ReleaseOrderTransfersJob(order, 'restaurant') ReleaseOrderTransfersJob(order, 'delivery') │ │ ▼ ▼ OrderPaymentLock (FOR UPDATE) OrderPaymentLock (FOR UPDATE) │ │ ▼ ▼ ReleaseTransferService ReleaseTransferService (Filters type = 'restaurant') (Filters type = 'delivery') │ │ Pre-fetch GET /v1/transfers/{id} Pre-fetch GET /v1/transfers/{id} │ │ PATCH /v1/transfers/{id} PATCH /v1/transfers/{id} {"on_hold": false} {"on_hold": false} │ │ ▼ ▼ Restaurant: RELEASED Driver: RELEASED (Order status: paid_held) (Order status: PAID) ``` --- ## 28. Pre-Fetch & Verify Idempotency Pattern ### Razorpay API Constraint: PATCH Does Not Support Idempotency Headers Unlike `POST /v1/payments/{payment_id}/transfers` (which accepts `X-Transfer-Idempotency`), Razorpay's settlement hold release endpoint: ```http PATCH https://api.razorpay.com/v1/transfers/{transfer_id} Content-Type: application/json {"on_hold": false} ``` **does not accept an idempotency key header**. If a worker dispatches a `PATCH` request and experiences a network timeout or packet loss before receiving Razorpay's HTTP 200 response, retrying the `PATCH` blindly risks unhandled gateway errors or race conditions. ### The Solution: Pre-Fetch & Post-Error Ground Truth Verification To achieve strict idempotency: 1. **Pre-Fetch Verification:** Before executing `PATCH`, the service executes: ```http GET https://api.razorpay.com/v1/transfers/{transfer_id} ``` If Razorpay reports `on_hold === false`, the transfer was already released. The system records the transition to `'released'` in the local database and skips the `PATCH` call entirely. 2. **Post-Timeout Ground Truth Check:** If a `PATCH` call fails due to a network timeout (`GuzzleHttp\Exception\ConnectException` or `RequestException`), the service immediately re-executes `GET /v1/transfers/{transfer_id}`: - If `on_hold === false`, the PATCH succeeded remotely before the connection dropped. The service completes successfully. - If `on_hold === true`, the request truly did not arrive at Razorpay. The service throws an exception so Laravel's queue retries with backoff. --- ## 29. Database Design: order_transfer_actions & Monotonic Transitions ### 29.1. Table: `order_transfer_actions` Migration: `database/migrations/2026_11_03_000008_create_order_transfer_actions_table.php` Tracks in-flight and completed actions on order transfers to eliminate duplicate parallel operations across workers: | Column | Type | Description | | :--- | :--- | :--- | | `id` | `BIGINT UNSIGNED` (PK) | Primary key | | `order_id` | `BIGINT UNSIGNED` (Index) | References `orders.id` | | `transfer_type` | `VARCHAR(30)` | `'restaurant'` or `'delivery'` | | `stage` | `VARCHAR(30)` | `'release'`, `'reverse'`, etc. | | `status` | `ENUM` | `'in_progress'`, `'success'`, `'failed'`, `'skipped'` | | `razorpay_reference` | `VARCHAR(100)` (Nullable) | Gateway transfer identifier (`trf_xxxxxxxx`) | | `error` | `VARCHAR(255)` (Nullable) | Sanitized error reason if failed | | `created_at` / `updated_at` | `TIMESTAMP` | Timestamps | - **Unique Composite Index:** `['order_id', 'transfer_type', 'stage']` Guarantees that across any number of parallel workers, only one worker can acquire the action lock for a specific order and stage. If a duplicate worker attempts to run, the unique index and action state prevents duplicate gateway calls. ### 29.2. Monotonic State Machine Status transitions on `order_transfers.status` are strictly one-way and monotonic: $$\text{created} \longrightarrow \text{on\_hold} \longrightarrow \text{released} \longrightarrow \text{reversed}$$ Enforced by `App\Services\RazorpayRoute\OrderTransferStatus::canTransition()`: - Once a transfer is `'released'`, it **cannot** be downgraded to `'on_hold'`, `'processed'`, or `'created'`. - If an out-of-order webhook arrives (e.g. delayed `transfer.processed` event), `OrderTransfer::transitionTo()` detects the invalid backward transition, safely discards it, and writes a `stale_webhook_event_ignored` audit record to `payment_ledger`. --- ## 30. ReleaseTransferService & OrderPaymentLock ### 30.1. Pessimistic Row Locking (`OrderPaymentLock`) To prevent race conditions between incoming webhooks (`ProcessRazorpayWebhookJob`), driver assignments (`TransferDriverDeliveryFeeJob`), and release triggers (`ReleaseOrderTransfersJob`): ```php OrderPaymentLock::run($orderId, function (Order $order) { // Executed inside a DB transaction with InnoDB SELECT ... FOR UPDATE }); ``` On MySQL InnoDB, `lockForUpdate()` ensures exclusive row-level locking. Concurrent workers wait for the lock to release, guaranteeing serial processing. ### 30.2. Service Guard Conditions & Progressive Handlers `ReleaseTransferService` exposes dedicated methods for the two stages: - **`releaseRestaurantTransfer(Order $order)`**: Releases only `type = 'restaurant'`. Executed when the driver confirms arrival at the restaurant. - **`releaseDeliveryTransfer(Order $order)`**: Releases only `type = 'delivery'`. Executed when the order transitions to `delivered`. - **`release(Order $order, ?string $transferType = null)`**: Core release logic with pessimistic row locking. **Guard Rules (`canRelease`):** 1. `order_status` is not in terminal error states (`canceled`, `failed`, `refund_requested`). 2. `payment_status` is `paid_held` or `paid`. 3. If releasing `delivery`, `order_status` must strictly be `delivered`. 4. No active reversal/refund action in progress for the specified transfer type. 5. Transfer has not already been released or reversed. --- ## 31. ReleaseOrderTransfersJob & Error Recovery ### 31.1. Job Configuration - **Class:** `App\Jobs\ReleaseOrderTransfersJob` - **Constructor:** `__construct(int $orderId, ?string $transferType = null)` - **Queue:** `gateway` - **Tries:** `3` - **Backoff:** `[10, 60, 300]` (seconds) - **Timeout:** `45` seconds - **Unique Lock:** Implements `ShouldBeUnique` with `$uniqueFor = 300`. `uniqueId()` evaluates to `"{orderId}_{transferType}"`, ensuring restaurant and delivery release jobs do not collide. ### 31.2. Failure Auditing (`failed()`) When all 3 attempts fail, the `failed()` hook: 1. Logs a sanitized error summary in `payment_ledger` with event `transfer_release_job_permanently_failed` and `transfer_type`. 2. Marks the corresponding `order_transfer_actions` record as `'failed'`. 3. Dispatches an admin notification to the Admin Panel. --- ## 32. Release Layer Automated Test Suite A comprehensive test suite in `tests/Unit/ReleaseTransferTest.php` (9 tests, 78 assertions) verifies: ```bash php artisan test tests/Unit/ReleaseTransferTest.php ``` ### Test Case Breakdown 1. **`test_same_payload_delivered_twice_produces_one_release_call`**: Asserts that sending multiple release requests for an order calls Razorpay's `PATCH /v1/transfers/{id}` exactly once and marks the database status as `'released'`. 2. **`test_pre_fetch_on_hold_false_skips_patch_call`**: Simulates Razorpay `GET /v1/transfers/{id}` returning `on_hold: false`. Asserts `PATCH` is completely skipped and local DB transitions to `'released'`. 3. **`test_timeout_then_retry_leads_to_fetch_before_second_call`**: Simulates a network timeout (HTTP 504) on `PATCH`. The recovery re-fetch reveals `on_hold: false`, resolving the release successfully without duplicate calls. 4. **`test_release_is_blocked_for_rejected_closed_and_refunded_orders`**: Asserts `canRelease()` guard rejects canceled orders, refunded payments, unpaid orders, and orders with active reversal actions. 5. **`test_out_of_order_events_do_not_downgrade_status`**: Replays a late `transfer.processed` webhook on a released transfer. Asserts status remains `'released'` and logs `stale_webhook_event_ignored` to `payment_ledger`. 6. **`test_two_workers_processing_same_release_produce_exactly_one_call`**: Verifies pessimistic row locking (`SELECT ... FOR UPDATE`) serializes parallel workers, preventing duplicate releases across workers. 7. **`test_webhook_duplicate_event_stuck_in_received_is_redispatched`**: Verifies that a duplicate webhook event stuck in `'received'` status is re-dispatched to the `webhooks` queue, whereas events in `'queued'` or `'processed'` are safely deduplicated. 8. **`test_release_order_transfers_job_parameters_and_failure_hook`**: Asserts job queue configuration (`gateway`), timeout (45s), backoff, unique key deduplication (`ShouldBeUnique`), sanitized ledger logging, and admin alert notification upon permanent failure. 9. **`test_progressive_release_stages_restaurant_then_delivery`**: Tests progressive split releases: releases restaurant transfer at restaurant arrival (order status remains `paid_held`), refuses premature delivery release, and releases delivery transfer at `delivered` (order status transitions to `paid`). --- # Part 5: Refund & Transfer Reversal Layer ## 33. Automated Refund & Transfer Reversal Architecture When an online-paid order cannot be fulfilled (e.g. restaurant rejection, restaurant closed, driver reports restaurant closed, or customer cancellation during payment processing), the platform guarantees: 1. **Route Transfers Never Settle to Linked Accounts:** Funds held on linked accounts are automatically reversed back to the platform master account. 2. **Customer Is Refunded Fully:** Online payment is refunded to the original payment method via Razorpay, and promotional wallet debits are credited back to the customer's wallet balance. 3. **Pessimistic Concurrency:** All refund mutations run strictly inside `OrderPaymentLock::run($orderId)` using MySQL InnoDB row-level locking (`SELECT ... FOR UPDATE`). 4. **Queue Isolation:** Handled asynchronously via `RefundOrderJob` dispatched to the dedicated `refunds` queue (`tries: 3`, `backoff: [10, 60, 300]`, `timeout: 45`, `ShouldBeUnique`). ### Razorpay Route Refund API Specification - **Endpoint:** `POST https://api.razorpay.com/v1/payments/{payment_id}/refund` - **Parameter `reverse_all: 1`:** > Quote from Razorpay Official Documentation: *"Setting this parameter to `true` instructs Razorpay to automatically recover/reverse the amount for every transfer made on the payment from the Linked Account(s) before processing the customer's refund."* - **Speed:** `"speed": "optimum"` (requests instant refund where supported by issuer bank, with fallback to standard processing). - **Receipt:** Deterministic identifier `"receipt": "order_refund_{order_id}"` for idempotent correlation and timeout recovery. --- ## 34. State Machine Isolation: OrderPaymentStateMachine To prevent invalid state interleaving, payment status transitions are strictly governed by `App\Services\RazorpayRoute\OrderPaymentStateMachine`. ### Disallowed Values `canceled` and `failed` are **strictly forbidden** as `payment_status` values. They belong exclusively in `orders.order_status`. Attempting to set them on `payment_status` throws `InvalidPaymentStateTransitionException`. ### Valid Payment States | State | Type | Description | | :--- | :--- | :--- | | `unpaid` | Initial | Order created, online payment capture not yet confirmed. | | `processing` | Ephemeral | Payment initiation in progress. | | `paid_held` | Active | Payment captured; Route transfers held on linked accounts. | | `partially_released` | Active | Restaurant transfer released; delivery partner transfer still held. | | `paid` | Terminal | All Route transfers released to linked accounts. | | `refund_pending` | In-Flight | Gateway refund and wallet credit recorded; awaiting webhook confirmation. | | `refund_failed` | Exceptional | Gateway refund rejected by issuer bank or permanent job failure. | | `refunded` | Terminal | Authoritatively confirmed refunded via `refund.processed` webhook or admin manual settlement. | ### Allowed State Transitions Graph ``` ┌─────────┐ │ unpaid │ ─────────┐ (Late capture on canceled order) └────┬────┘ │ │ │ ▼ ▼ ┌──────────────┐ ┌────────────────┐ │ processing │──>│ refund_pending │<───── (Admin retry) └───────┬──────┘ └───────┬────────┘ ▲ │ │ │ ▼ ├────── (refund.failed) ────> ┌───────────────┐ ┌──────────────┐ │ │ refund_failed │ │ paid_held │───────────┤ └───────┬───────┘ └───────┬──────┘ │ │ │ ▼ (refund.processed ONLY) │ (Admin manual │ ┌──────────┐ │ settlement) ├────────────>│ refunded │<──────────────────────────────┘ │ └──────────┘ ▼ ┌────────────────────┐ │ partially_released │ ────> ┌──────┐ │ (REFUND BLOCKED) │ │ paid │ └────────────────────┘ └──────┘ ``` > [!IMPORTANT] > **Webhook-Driven Terminal Confirmation:** In accordance with user requirements, `RefundOrderJob` and `RefundOrderService` leave the order in `refund_pending`. The terminal `payment_status = 'refunded'` is set **ONLY** upon receiving the verified `refund.processed` webhook. If Razorpay issues a `refund.failed` event, the state transitions from `refund_pending` to `refund_failed`. --- ## 35. Timeout Recovery & Pre-Flight Idempotency Pattern Because HTTP POST requests can experience connection drops, socket timeouts (cURL error 28), or gateway 504s after the payment gateway has already processed the refund, `RefundOrderService` implements pre-flight recovery: 1. **Deterministic Receipt Matching:** Every refund sends `"receipt": "order_refund_{order_id}"`. 2. **Pre-Flight Query:** Before issuing `POST /v1/payments/{payment_id}/refund`, the service calls `GET /v1/payments/{payment_id}/refunds`. 3. **Recovery Discovery:** If any refund entity in the collection has `receipt === "order_refund_{order_id}"`, the service reuses that refund's `id` without calling POST, avoiding duplicate refund attempts. 4. **Post-Timeout Catch Recovery:** If the POST call throws `ConnectionException`, the service queries `GET /v1/payments/{payment_id}/refunds` again to inspect whether the request reached Razorpay before failing. --- ## 36. Separate Accounting: Gateway Refund vs Promotional Wallet Credit When an order was paid partially with online gateway funds and partially with promotional wallet balances (`orders.partially_paid_amount > 0`), the refund layer isolates each step on the `order_refunds` table: ``` ┌────────────────────────────────────────────────────────┐ │ order_refunds │ ├────────────────────────────────────────────────────────┤ │ order_id : unsignedBigInteger (UNIQUE) │ │ razorpay_refund_id : string (nullable) │ │ receipt : string ("order_refund_{id}") │ │ amount_paise : unsignedBigInteger (online INR) │ │ wallet_amount_rupees : decimal(10, 2) (wallet INR) │ │ status : enum(pending,success,failed,...)│ │ speed : string ("optimum") │ │ gateway_refund_at : timestamp (nullable) │ │ wallet_credited_at : timestamp (nullable) │ │ manually_refunded_by : unsignedBigInteger (nullable) │ │ manually_refunded_at : timestamp (nullable) │ │ admin_notes : text (nullable) │ └────────────────────────────────────────────────────────┘ ``` - **Order of Execution:** The gateway refund API call executes first and writes `gateway_refund_at`. - **Wallet Credit Guard:** Customer wallet balance is credited via `CustomerLogic::create_wallet_transaction($userId, $amount, 'order_refund', (string)$orderId)` **ONLY AFTER** `gateway_refund_at` is confirmed. - **Reference Integrity:** `WalletTransaction.reference` is strictly verified to hold the string order ID `(string)$order->id`. --- ## 37. Lifecycle Guards, Pickup & Partial Release Restrictions 1. **No Refund from `partially_released`:** If the restaurant transfer has already been released (`payment_status === 'partially_released'`), automated gateway refund and transfer reversal are permanently blocked. `RefundOrderService` throws `NonRetryablePaymentException`. 2. **Cancellation Blocked After Pickup:** In `VendorController`, `Vendor/OrderController`, and `Admin/OrderController`, canceling an order that is in `picked_up` or `delivered` is blocked for paid orders (`paid_held`, `partially_released`, `paid`). Operators receive an explicit error message directing them to dispute settlement. 3. **Driver "Reached" Does Not Release:** Reaching the restaurant location does not trigger automated fund release. Holds remain on hold until food handover/completion. --- ## 38. Driver Restaurant Closed Reporting Endpoint - **Endpoint:** `POST /api/v1/delivery-man/report-restaurant-closed` - **Route Name:** `api.v1.delivery-man.report-restaurant-closed` - **Middleware:** `dm.api`, `actch:deliveryman_app` - **Strict Guards:** 1. *Assigned Driver Only:* `$order->delivery_man_id === $delivery_man->id` (returns 403 on mismatch). 2. *Before Pickup Only:* Refuses orders in `picked_up`, `delivered`, `canceled`, or `refunded` (returns 400). 3. *Paid-Held Only:* `$order->payment_status === 'paid_held'` (returns 400 on unpaid or invalid states). 4. *Mandatory Reason:* Validates `reason` (string, min 3, max 255). 5. *Server-Side Status Check / Admin Review Flag:* If the restaurant was marked active/open in the database, the order is annotated with `[FLAG: Admin Review Required - Driver reported restaurant closed while system showed open]`. - **Action:** Under row lock, marks order `canceled`, sets `canceled_by = 'deliveryman'`, transitions `payment_status` to `refund_pending`, and dispatches `RefundOrderJob` to the `refunds` queue. --- ## 39. Late Payment Capture on Canceled Orders When customers abandon checkout or cancel an order during payment processing, a race condition can cause Razorpay to fire `payment.captured` after the order is already in `canceled` or `failed` status. - **Handling in `ProcessRazorpayWebhookJob`:** ```php if (in_array($order->order_status, ['canceled', 'failed'], true)) { $order->razorpay_payment_id = $paymentId; $order->paid_at = now(); OrderPaymentStateMachine::transition($order, OrderPaymentStateMachine::REFUND_PENDING, "Late capture on {$order->order_status} order"); PaymentLedger::record($order->id, 'late_capture_on_canceled_order', (int)$amountPaise, $paymentId); RefundOrderJob::dispatch($order->id, "Late capture on {$order->order_status} order"); return; } ``` - **Invariants Enforced:** The canceled order is never revived or confirmed; funds are automatically queued for refund on the `refunds` queue. --- ## 40. Admin Resolution: Retry Refund & Mark as Manually Refunded When an order ends in `refund_failed` (e.g. customer's bank account was frozen, or Razorpay rejected the reversal), the Admin Panel provides two explicit operational resolution actions: 1. **Retry Refund (`POST /admin/order/retry-refund/{id}`):** - Resets order `payment_status` to `refund_pending`. - Re-dispatches `RefundOrderJob` to the `refunds` queue to attempt automated recovery or re-issuance. - Audited in `payment_ledger` with event `admin_retry_refund`. 2. **Mark as Manually Refunded (`POST /admin/order/mark-manually-refunded/{id}`):** - For orders that cannot be processed via gateway (e.g., cash settlement or offline bank transfer). - Validates mandatory `notes` describing the manual payout reference (NEFT, UPI, or cash voucher). - Under `OrderPaymentLock`, updates `order_refunds.status = 'manually_refunded'`, stores admin ID (`manually_refunded_by`) and timestamp (`manually_refunded_at`), transitions `orders.payment_status` to `refunded`, and logs an audit record to `payment_ledger`. --- ## 41. Automatic Customer Cash Wallet Fallback & Double-Refund Safeguard When a payment gateway refund definitively fails, the system automatically falls back to crediting the customer's cash (non-promotional, real-money) wallet: ### 1. Definitive Failure Triggers vs Transient Errors - **Definitive Failures:** 1. `refund.failed` webhook received from Razorpay. 2. 4xx HTTP client error / business rejection from the Razorpay refund API (`POST /v1/payments/{payment_id}/refund`), confirmed by `GET /v1/payments/{payment_id}/refunds` verifying that no refund entity exists on Razorpay. - **Transient Failures (Never Fallback):** - Socket timeout (cURL error 28 / `ConnectionException`) or 5xx server errors are treated strictly as transient. The system recovers via pre-flight `GET` queries and re-queues via `RefundOrderJob`. ### 2. Strict Invariants Before Crediting Wallet Inside `fallbackToWalletRefund()`, executed under InnoDB `OrderPaymentLock`: 1. **Reverse Held Transfers First:** Reverses any linked account transfers that are still on hold and records `OrderTransferAction` reversal records. 2. **Verify Release Stays Blocked:** Checks `ReleaseTransferService::canRelease()` to confirm funds cannot be released to vendors or drivers. 3. **Credit Once Only Guard:** Verifies `order_refunds.wallet_fallback_at === null`, `order_refunds.refund_method !== 'wallet'`, and inspects existing `WalletTransaction` records for `transaction_type = 'order_refund'` and `reference = (string)$orderId` to prevent duplicate wallet credits under retries or parallel webhooks. 4. **Credit Real-Money Cash Wallet:** Invokes `CustomerLogic::create_wallet_transaction($userId, $onlineAmountRupees, 'order_refund', (string)$orderId)`. 5. **Terminal State Isolation:** Transitions `orders.payment_status` to `refunded_to_wallet` via `OrderPaymentStateMachine`. This status is strictly terminal. 6. **Preserve Mobile App Status:** `orders.order_status` remains strictly `'canceled'` for mobile client apps. Existing API response contracts remain untouched. 7. **Audit & Notifications:** Writes an audit record to `payment_ledger` (`event = 'wallet_refund_fallback'`) and dispatches a customer notification job on the `notifications` queue. 8. **Driver Compensation Manual Review:** If a driver was assigned, records `driver_compensation_manual_review` in `payment_ledger` with the driver's ID and reason so support can compensate them manually. ### 3. Late `refund.processed` Double-Refund Safeguard If a delayed `refund.processed` webhook subsequently arrives for an order that is already in `payment_status = 'refunded_to_wallet'`: - The system **does not** modify the order state. - The system **does not** reverse the wallet credit. - The webhook handler logs an audit record to `payment_ledger` (`event = 'possible_double_refund'`). - The handler dispatches an admin alert notification titled `"possible double refund"` to the `notifications` queue for administrative investigation. --- ## 42. Two-Connection Concurrency Proof & Honest Limitations ### What the Concurrency Test Proves The test `RefundOrderTest::test_release_vs_refund_race_with_two_real_connections` utilizes two distinct MySQL PDO connections (`mysql` and `mysql_second`) with different MySQL thread IDs (`SELECT CONNECTION_ID()`) connected to `stackfood_test`. - **Pessimistic Row Lock Serialization:** When Connection 1 begins a transaction and executes `SELECT ... FOR UPDATE`, Connection 2 attempting to lock the exact same row with `SET innodb_lock_wait_timeout = 1` encounters lock contention (Error 1205: Lock wait timeout exceeded). - **Mutual Exclusion:** Proves that simultaneous operations (e.g., release worker vs refund job) cannot interleave or corrupt order state; Connection 2 must wait until Connection 1 commits. ### What the Concurrency Test Does NOT Prove - It does **not** simulate multi-region network partitioning or distributed cross-datacenter clock drift. - It does **not** test asynchronous latency or eventual consistency inside Razorpay's internal banking microservices. --- ## 43. Unverified Live Razorpay Sandbox Behaviors The following edge-case behaviors cannot be fully verified in sandbox mocks and must be validated under live production credentials with real linked accounts: 1. **`reverse_all: 1` with on-hold Route transfers:** Razorpay documentation specifies that `reverse_all: 1` automatically recovers transfer amounts from linked accounts before issuing customer refunds. However, whether on-hold transfers are instantly canceled without debiting linked account balances or whether a reversal entity is generated cannot be proven without active live settlement. 2. **Partial refunds with Route transfers:** Razorpay Route documentation explicitly documents `reverse_all: 1` for full payment refunds. For partial refunds on payments with multiple linked account transfers, Razorpay requires individual transfer reversals via `POST /v1/transfers/{id}/reversals` specifying exact transfer shares. 3. **Instant refund fallback speed:** Behavior when `speed: 'optimum'` cannot be fulfilled by the customer's card network and Razorpay falls back to normal speed (5-7 business days). --- ## 44. Refund Automated Test Suite (19 Tests, 99 Assertions) ```bash php artisan test tests/Unit/RefundOrderTest.php ``` | Test Method | Requirement / Focus | | :--- | :--- | | `test_driver_reports_restaurant_closed_success` | Driver report cancels order, transitions to `refund_pending`, dispatches `RefundOrderJob`. | | `test_driver_reports_restaurant_closed_guards` | Enforces assigned driver, before-pickup, `paid_held` order, and mandatory reason. | | `test_refund_order_service_reverses_transfers_and_credits_wallet` | Calls Razorpay with `reverse_all: 1`, reverses transfers, credits wallet, records timestamps. | | `test_webhook_refund_processed_marks_payment_status_refunded` | Webhook `refund.processed` authoritatively marks order as `refunded`. | | `test_webhook_refund_failed_marks_payment_status_refund_failed` | Webhook `refund.failed` sets `refund_failed` and triggers fallback wallet refund. | | `test_timeout_recovery_recovers_refund_id` | Pre-flight `GET /v1/payments/{id}/refunds` recovers existing refund and prevents duplicate POST. | | `test_refund_blocked_if_transfer_already_released` | Guard rejects refund if any transfer is already released to linked accounts. | | `test_refund_blocked_from_partially_released` | Guard rejects automated refund if order is in `partially_released` state. | | `test_late_capture_on_canceled_order_triggers_automatic_refund` | Late `payment.captured` on canceled order initiates refund without reviving order. | | `test_admin_mark_as_manually_refunded` | Admin manual refund updates `order_refunds`, logs audit trail, and marks `refunded`. | | `test_release_vs_refund_race_with_two_real_connections` | Two real database connections prove InnoDB `lockForUpdate` row lock serialization. | | `test_refund_failed_webhook_triggers_automatic_wallet_credit_and_sets_refunded_to_wallet` | Requirement 8.1: `refund.failed` webhook credits customer cash wallet, sets `refunded_to_wallet`, and marks `refund_method = 'wallet'`. | | `test_duplicate_refund_failed_webhook_credits_wallet_only_once` | Requirement 8.2: Duplicate `refund.failed` webhook executed under row lock credits wallet exactly once. | | `test_timeout_or_5xx_does_not_credit_wallet_and_retries` | Requirement 8.3: Gateway timeout / 5xx does NOT credit wallet; job throws retryable exception for queue retry. | | `test_4xx_rejection_with_no_refund_found_credits_the_wallet` | Requirement 8.4: Gateway 4xx error with GET refunds confirming no refund exists immediately credits wallet fallback. | | `test_transfers_still_on_hold_are_reversed_before_wallet_credit_and_release_blocked` | Requirement 8.5 & 8.6: On-hold transfers are reversed before wallet credit; release is strictly blocked afterwards. | | `test_late_refund_processed_after_wallet_credit_raises_alert_and_changes_nothing` | Requirement 8.7: Late `refund.processed` on `refunded_to_wallet` order leaves state unchanged and dispatches "possible double refund" alert. | | `test_partial_wallet_order_refund_failed_credits_only_gateway_share_to_wallet` | Requirement 8.8: For ₹500 split (₹100 wallet, ₹400 gateway), failed gateway refund credits only ₹400 to wallet, avoiding over-crediting. | | `test_cod_or_unpaid_orders_are_immune_to_wallet_fallback` | Requirement 8.9: COD / unpaid orders are strictly rejected by fallback wallet guard. | | `test_fallback_to_wallet_refund_calls_razorpay_transfer_reversal_api` | Requirement 1: `fallbackToWalletRefund` calls Razorpay `GET /v1/transfers/{id}` and `POST /v1/transfers/{id}/reversals` for each on-hold transfer. Fails if reversal call is missing. | | `test_refunded_to_wallet_is_restricted_to_refund_pending_and_refund_failed` | Requirement 2: Restricts `refunded_to_wallet` to `refund_pending` and `refund_failed`. Disallows direct jumps from `paid_held` and `paid`. | --- # Part 6: Restaurant Confirmation Model & Go-Live Checklist ## 46. Restaurant Confirmation Model Architecture Under the restaurant-driven confirmation model (replacing the legacy driver Yes/No prompt): 1. **Restaurant Accept / Reject Gate:** - After online payment is captured and held (`payment_status = 'paid_held'`), the order appears immediately in the Restaurant App under "New / Pending Orders". - The kitchen can tap **Accept** (order transitions to `confirmed` or `processing`, recorded idempotently in `order_restaurant_confirmations` with `answer = 'yes'`). - The kitchen can tap **Reject** (order transitions to `canceled` with mandatory reason recorded in `order_restaurant_confirmations` with `answer = 'no'`; automatically initiates full refund and transfer reversal through `RefundOrderService::cancelWithFullRefund`). 2. **Driver Assignment & Notification:** - Delivery men are notified via topic push notification (`sentDeliveryManNotification`) once the restaurant accepts (whether app sends `confirmed` or `processing`). - Driver pickup (`picked_up`) is gated on `restaurant_confirmed = true` and `payment_status` valid. 3. **Unanswered Order Sweep & 3-Minute Warning:** - `razorpay:cancel-unanswered-orders` runs every minute via the Laravel scheduler. - If an order remains unanswered after 3 minutes (180s), a push warning is sent once to the restaurant and recorded in `payment_ledger` (`event = 'restaurant_unanswered_warning_sent'`). - If unanswered after the configurable timeout (default 5 minutes), the command auto-cancels the order and triggers `RefundOrderService::cancelWithFullRefund`. 4. **Progressive Hold Release:** - Restaurant transfer is released upon **`picked_up`**. - Delivery partner fee transfer is released upon **`delivered`**. - If an order reaches `delivered` and the restaurant transfer was not yet released (e.g. driver skipped straight to delivered), a catch-up release is dispatched automatically. 5. **Driver Cancellation Restrictions:** - Driver cancel is strictly blocked for paid online orders with error code `DRIVER_CANCEL_BLOCKED`. - If driver reports restaurant physically closed before pickup (`report-restaurant-closed`), full refund is triggered and logged with an admin review flag if restaurant is shown open server-side. --- ## 47. Production Go-Live Deployment Checklist Follow this checklist sequentially during production deployment: ### Step 1: Pre-Deployment Database Backup ```bash # Take a full transactional dump of the production database mysqldump -u -p --single-transaction --quick --routines --triggers stackfood > stackfood_pre_deployment_$(date +%Y%m%d_%H%M%S).sql ``` ### Step 2: Run Database Migrations ```bash php artisan migrate --force ``` Verifies creation of: - `order_restaurant_confirmations` (unique on `order_id`) - `order_transfers` & `order_transfer_actions` - `order_refunds` - `payment_ledger` - Additions to `orders` (`captured_amount_paise`, `online_payable_paise`, `driver_payment_notified_at`) ### Step 3: Business Settings & Configuration Configure the following keys in `business_settings` (via Admin Panel or CLI): ```bash # 1. Feature activation timestamp (set to current UTC/local deploy time) php artisan tinker --execute="App\Models\BusinessSetting::updateOrCreate(['key' => 'feature_enabled_from'], ['value' => now()->toDateTimeString()]);" # 2. Restaurant unanswered timeout (minutes before auto-cancel & refund, default: 5) php artisan tinker --execute="App\Models\BusinessSetting::updateOrCreate(['key' => 'restaurant_unanswered_timeout_minutes'], ['value' => 5]);" # 3. Scheduled order lead time (minutes before schedule_at when order activates, default: 30) php artisan tinker --execute="App\Models\BusinessSetting::updateOrCreate(['key' => 'schedule_order_slot_duration'], ['value' => 30]);" # 4. Verify wallet status is enabled for customer wallet refunds php artisan tinker --execute="App\Models\BusinessSetting::updateOrCreate(['key' => 'wallet_status'], ['value' => 1]);" ``` Configure `config/razorpay.php` (or `.env`): - `RAZORPAY_WALLET_FALLBACK_ENABLED=true` - `RAZORPAY_REFUND_SPEED=optimum` - `RAZORPAY_WALLET_FALLBACK_ALLOWED_DESCRIPTIONS=[]` (keep empty by default until specific non-refundable card/account patterns are identified). ### Step 4: Backfill Existing In-Flight Orders (Optional / Hybrid) For orders placed before deploy time that are already in `processing`, `handover`, `picked_up`, or `delivered`: - By default, `Order::getRestaurantConfirmedAttribute()` derives `true` for all orders where `paid_at < feature_enabled_from`. - To create permanent audit records for in-flight orders: ```bash php artisan tinker --execute=" \$cutoff = App\Models\BusinessSetting::where('key', 'feature_enabled_from')->value('value'); App\Models\Order::whereIn('order_status', ['processing', 'handover', 'picked_up', 'delivered']) ->where('paid_at', '<', \$cutoff) ->whereDoesntHave('restaurantConfirmation') ->chunk(100, function (\$orders) { foreach (\$orders as \$ord) { App\Models\OrderRestaurantConfirmation::create([ 'order_id' => \$ord->id, 'restaurant_id' => \$ord->restaurant_id, 'vendor_id' => \$ord->restaurant?->vendor_id, 'answer' => 'yes', 'reason' => 'backfilled', 'answered_at' => \$ord->paid_at ?? now(), 'admin_review' => false, ]); } }); " ``` ### Step 5: Crontab / Task Scheduler Setup Ensure the standard Laravel scheduler runs every minute in the server crontab: ```bash * * * * * cd /path/to/stackfood && php artisan schedule:run >> /dev/null 2>&1 ``` The schedule (`routes/console.php`) automatically executes: - `razorpay:cancel-unanswered-orders` every minute - `razorpay:sweep-transfer-releases` every 2 minutes - `queue:health-check` every 5 minutes (monitors `gateway`, `refunds`, and `webhooks` queues, alerts on stuck `refund_pending` orders) ### Step 6: Warning-Sent Flag Storage & Deduplication - The 3-minute warning is stored in `payment_ledger` with `event = 'restaurant_unanswered_warning_sent'`. - The sweeper checks `PaymentLedger::where('order_id', $order->id)->where('event', 'restaurant_unanswered_warning_sent')->exists()` before sending, ensuring it is dispatched **at most once per order**. ### Step 7: Emergency Rollback Procedure If the feature needs to be deactivated instantly: ```bash # Deactivate feature via artisan command php artisan razorpay:configure-restaurant-model --enable-from=disable # Or via tinker php artisan tinker --execute="App\Models\BusinessSetting::updateOrCreate(['key' => 'feature_enabled_from'], ['value' => null]);" ``` Setting `feature_enabled_from = null`: - Deactivates the pickup gate restaurant confirmation requirement immediately. - Causes `razorpay:cancel-unanswered-orders` to exit immediately without canceling any order. - Restores standard un-gated flow without database migrations or code reverts. ### Draining In-Flight Orders During Rollback When rolling back to the legacy model while orders are in-flight: 1. **Unanswered Pending Orders:** Orders that were captured and currently waiting for restaurant confirmation in `paid_held` will no longer be auto-canceled by `razorpay:cancel-unanswered-orders` (since `feature_enabled_from` is null). Administrators or operators can allow restaurants to accept them or cancel them manually via the admin panel/API; `cancelWithFullRefund()` will cleanly initiate gateway refunds. 2. **Accepted / In-Progress Orders:** Orders already confirmed by restaurants will proceed to pickup and completion normally. Because the progressive gates are bypassed when the feature is disabled, drivers can deliver without hindrance. 3. **Transfer Releases:** Any transfers still on hold will be released either upon terminal delivery status or by the scheduled background sweeper `razorpay:sweep-transfer-releases`. ### Deadlock Prevention & Concurrency Model Note Global concurrency protection relies on `OrderPaymentLock` utilizing `orders` row locks (`SELECT ... FOR UPDATE`). Codebase analysis revealed no known occurrences of locking `users` before `orders`, though third-party plugins or raw queries could conceivably exist in unmanaged environments. Transactions acquiring user balances strictly lock the `users` row inside the order lock scope. --- ## 48. Configuration Management CLI (`razorpay:configure-restaurant-model`) The system provides a dedicated CLI tool to inspect and update runtime operational parameters without direct database manipulation: ```bash # View current status php artisan razorpay:configure-restaurant-model --status # Enable with immediate cutoff php artisan razorpay:configure-restaurant-model --enable-from=now # Set specific timestamp cutoff, unanswered timeout, and dedicated scheduled lead time php artisan razorpay:configure-restaurant-model --enable-from="2026-11-05 00:00:00" --timeout=5 --scheduled-lead-time=30 # Disable restaurant confirmation model php artisan razorpay:configure-restaurant-model --enable-from=disable ``` | Parameter | Business Setting Key | Default | Description | | :--- | :--- | :---: | :--- | | `--enable-from` | `feature_enabled_from` | `null` | Deploy cutoff timestamp. Orders paid prior are exempt from gates. | | `--timeout` | `restaurant_unanswered_timeout_minutes` | `5` | Minutes restaurant has to accept/reject before automated refund. | | `--scheduled-lead-time` | `restaurant_schedule_order_lead_time_minutes` | `30` | Dedicated activation lead time for scheduled orders (independent of slot duration). | --- ## 49. Concurrency Test Proof & Written Tables The release-vs-refund race test (`test_release_vs_refund_race_with_two_real_connections` in `tests/Unit/RefundOrderTest.php`) proves: - **Two real database threads:** Verified using `SELECT CONNECTION_ID()` on connections `mysql` and `mysql_second` (verifying `$threadId1 !== $threadId2`). - **Pessimistic serialization:** Connection 1 acquires `SELECT ... FOR UPDATE` row lock on `orders`. Connection 2 invokes `ReleaseTransferService` with `innodb_lock_wait_timeout = 1` and throws MySQL error 1205 (lock wait timeout). - **Mutual exclusion:** After Connection 1 executes full refund, Connection 2's subsequent release is strictly rejected by guard checks. - **Zero double-payout:** Exactly 1 POST refund call made; exactly 0 PATCH release calls made. ### Tables Written and Cleaned Up in `finally`: The concurrency test writes and cleanly removes records from the following 8 tables: 1. `order_transfer_actions` 2. `payment_ledger` 3. `order_refunds` 4. `order_transfers` 5. `orders` 6. `razorpay_linked_accounts` 7. `restaurants` 8. `users` --- ## 49. Global Test Suite Summary All tests run strictly against the dedicated MySQL test database `stackfood_test` (port 3306). ```bash # Run complete test suite php artisan test tests/Unit ``` | Suite | File | Tests | Assertions | Focus Areas | | :--- | :--- | :---: | :---: | :--- | | **Restaurant Confirmation** | `tests/Unit/RestaurantConfirmationModelTest.php` | 29 | 216 | E2E capture-list-accept flow, multi-status formatting, pickup/delivery gates for all payment statuses, DRIVER_CANCEL_BLOCKED, driver payment push deduplication, catch-up releases, unanswered timeout sweep, 3-min warning, customer cancel, backfill & cutoff exemption, multi-tenant vendor isolation | | **Refund Layer** | `tests/Unit/RefundOrderTest.php` | 38 | 179 | `reverse_all: 1`, transfer reversals, cash wallet fallback with description substrings, insufficient merchant balance alerts, two-connection race test, timeout recovery, admin retry/manual refund | | **Release Layer** | `tests/Unit/ReleaseTransferTest.php` | 10 | 83 | Progressive release stages, pre-fetch, timeout recovery, refusal guards, monotonic status, row locking, closed/inactive restaurant non-blocking release | | **Queue Hardening** | `tests/Unit/QueueAndWebhookHardeningTest.php` | 9 | 51 | Queue topology, retry/backoff/timeout invariants, fast-fail on mismatch, health monitoring, sweeper, stuck refund alerts | | **Split Payments** | `tests/Unit/RazorpayRouteSplitPaymentTest.php` | 17 | 100 | Section 9(5) GST tax remittance, integer paise math, non-negative formula, pre-flight checks, webhooks | | **Linked Accounts** | `tests/Unit/RazorpayRouteTest.php` | 5 | 34 | RBI IFSC validation, phone sanitization, business types, delivery app blocking rule, account provisioning | | **Fee Calculator** | `tests/Unit/FeeCalculatorTest.php` | 5 | 32 | Handling fee slabs, BCMath half-up rounding, continuity rules, zone override & global fallback | | **Example** | `tests/Unit/ExampleTest.php` | 1 | 1 | Basic health check | | **Total** | | **114** | **696** | **All 114 Tests Passing (100% Green)** |