# Wallet Statements

**Portal:** Manage · **Nav:** Sales hub → **Payment** main tab → **Wallet Statements** pill (Operations suite; the Project / Subsale suites reach it from their standalone Payment entry, [`SectionTabs.vue`](/resources/js/Components/SectionTabs.vue) `payments` section) · **Permission:** `view-sales` (read) / `manage-sales` (upload, dismiss a line)

## What it does

Reads the owner's **Touch 'n Go eWallet statement PDF** and shows what the wallet actually did.

This is the **money half** of the manual-transfer lane. The customer's half — [Transfer Receipts](/docs/modules_handbook/manage/payments/transfer-receipts/readMe.md) — records what a buyer *says* they paid, which is a screenshot and therefore forgeable. A wallet statement is what the wallet itself reports, which is not. **Neither is trusted alone**, and an admin joins them.

> **Phase D reads and shows. Nothing here reaches the ledger.** `tng_statement_lines.purchase_history_id` is NULL throughout — turning a statement line into a payment is Phase E, and it carries a hazard recorded at the bottom of this page.

## How it works

### Getting the text out of the PDF

`pdftotext` cannot read this document, in any of its modes. Verified against a genuine statement: one Reference cell comes out split across **five separate lines**, and a row's timestamp lands beside a different row's date. `-layout` and `-table` both scramble it; `-raw` additionally drops rows at page boundaries. Any line-based grammar built on that output is guessing.

So the text is read **geometrically**, in pure PHP ([`Src\Common\Pdf`](/src/Common/Pdf/PdfTextExtractor.php)). The generator (`iText 2.1.7` driving a JasperReports template) emits an **absolute `Tm` before every single cell** — 768 of them on the real file, and not one kerning array — so the table's coordinates are already in the document. Reading them is not reconstruction; it is declining to throw them away.

**Zero new server packages.** Nothing was added to `scripts/server-setup.sh`, deliberately: `scripts/deploy-update.sh` runs no `apt` at all, so a binary dependency means the owner SSH-ing into production by hand. The live counter-example is [`BrochureExtractionService.php:176`](/app/Services/FacebookLeadGenerator/BrochureExtractionService.php#L176) — `shell_exec('pdftotext …')` with the return unchecked, on a server where poppler is not installed. It returns `''`, which reads as *"this PDF has no text"*. On a bank statement that sentence becomes **"no money came in today."**

The supported PDF profile is deliberately narrow — classic xref, `FlateDecode`, `/Pages /Kids`, Type1/WinAnsi — and everything outside it (object streams, xref streams, CID fonts, a public-key handler) is **refused by name**. A named refusal costs one clear message; a half-read document costs a wrong number that looks exactly like a right one.

**The statement PDF's password is the wallet number WITHOUT its leading zero** — `010-868 5352` opens with `108685352`. `expectedPdfPassword()` derives it, and the settings screen shows the operator live what it will try, because a wrong password does not fail loudly: it decrypts to bytes holding no text, which reads downstream as *"this statement had no transactions"*. Every other spelling of the same number is still attempted as a fallback — the extra attempts cost microseconds.

**Encryption is handled in-process** ([`PdfDecryptor`](/src/Common/Pdf/PdfDecryptor.php)), revisions 2/3/4/6. Two reasons it is not shelled out to `qpdf`: the password *is* the owner's mobile number, and passing it in `argv` puts it where any user on the box can read it from `ps`; and the binary is not installed. RC4 is hand-written because **OpenSSL 3 dropped it from the default provider**, so `openssl_decrypt('rc4', …)` returns `false` on Ubuntu 24.04 — it is pinned against the RFC 6229 known-answer vectors at all four key lengths, which is the only thing that makes writing a cipher by hand acceptable.

⚠️ **Every revision verifies the password against the stored `/U` before returning.** A decryptor that accepts anything and emits noise is the worst failure available here: the noise contains no text, so the statement reads as empty, and an empty statement means nobody paid.

### The direction problem — the most dangerous inference in the feature

**The Amount column is UNSIGNED.** RM299 received and RM299 spent are the identical string on the page. Getting it backwards books the owner's own grocery shopping as a customer's payment.

Two **independent** lines of evidence, and when they disagree **no winner is chosen**:

| | Evidence | Source |
|---|---|---|
| **A** | The transaction type, looked up in an allow-list | `config/tng.php` |
| **B** | `balance[i] − balance[i+1]` must equal the row's signed amount, **exactly, in integer cents** | the statement's own running balance |

⚠️ **The order is not interchangeable. The type DECIDES; the balance CHECKS.** Deriving the sign from the delta and then "verifying" it against the same delta is one number agreeing with itself.

**The arbitration table** ([`TngDirectionResolver`](/app/Support/TngStatement/TngDirectionResolver.php)) is asymmetric on purpose — recording money in is the dangerous action, failing to record it is the safe one:

| Type says | Balance says | Result |
|---|---|---|
| in | in | ✅ `source=both`, chain verified — **the only state safe to act on** |
| out | out | ✅ `source=both` |
| in | out (or the reverse) | 🚫 **quarantined.** No vote is taken |
| anything | neither ±amount | 🚫 quarantined — a gap, or a mis-read cell |
| in | nothing | ⚠️ `type_only`, needs review — not actionable |
| out | nothing | ⚠️ `type_only` — warning only; not recording a debit costs nothing |
| unlisted | in | 🚫 quarantined — **this is the day Touch 'n Go renames something, on the side where money is at stake** |
| unlisted | out | ⚠️ `balance_only` — warning only |

**The case that justifies the whole mechanism**, taken from the real statement:

```
4/10/2021  PayDirect Payment  amount RM1.70  balance 1,759.93 → next 59.63  delta = +RM1,700.30
```

An implementation that takes `sign(delta)` records that row as **money in, RM1,700.30**. The type says it is a payment out; the delta is neither +1.70 nor −1.70; the sources contradict; the row goes to a person.

**Reading order is MEASURED, not assumed.** A statement may print newest-first or oldest-first and the sign of every delta flips with it. Both orders are scored and compared as a **ratio** (1.5×) rather than against a pass mark — the real file only reconciles 55 of 60, so any absolute threshold near 90% would reject a perfectly good statement, while 55 : 27 is unambiguous. Two orders scoring the *same non-zero* number is refused outright; two orders scoring **zero** is not, because a zero score means no row can be chain-verified either way, so a wrong guess cannot produce a wrong payment.

### The money-in vocabulary is a LIST, not a substring match

⚠️ **Never `str_contains($type, 'Received')`.** That rule is wrong three ways on the real vocabulary:

- `Receive from Wallet` — a customer paying. **No "d".** Missed.
- `DUITNOW_RECEIVEFROM` — a customer paying from a bank. Missed.
- `Money Packet Received` — a **gift packet**. Caught, wrongly.

`config('tng.types')` is an **allow-list**: an unlisted type gets no direction at all. A stale table costs a payment recorded *late*; a guess costs one recorded *wrong*. `customer_payment` separates further — a reload, a cashback and a gift packet are all money in, and none of them is a sale.

Keys are normalised (uppercase, alphanumerics only), which is what makes a type split across two cells (`DUITNOW_RECEI` + `VEFROM`) heal itself with no hard-coded repair to go stale.

### ⚠️ A statement may be a FILTERED SUBSET, and that is not a bug

The app exports *"the last 90 days **or the current search result**"*. An owner who searched before tapping Send emails a file containing only the rows they were looking at — while the Wallet Balance column still shows the **true** running balance, which is how the gap becomes visible at all. On the one genuine statement available, the chain breaks in **5 places out of 60**, the largest being a RM1,700.30 credit that is simply not printed.

Three consequences, all load-bearing:

1. **No document-level "the chain must close" gate.** It would reject every real statement. The chain is checked per row; the document merely reports how many links held.
2. **Rows either side of a gap fall back to `type_only`** and are not actionable.
3. **The screen must say so.** `likelyFiltered()` drives a banner: *"a payment you cannot find here may still have happened."* Without that sentence an admin takes a filtered export as proof that an honest customer is lying.

### What the chain cannot check — the one silent channel

The balance arithmetic proves the date, the amount, the direction and the ordering. It says **nothing** about `counterparty_name`. A row can reconcile perfectly and still carry the wrong sender.

The only defence is a person's eyes, so the Show page prints the **eight raw cells verbatim** beside the interpretation. That is a requirement, not a debugging aid.

### Identity of a row

```
dedupe_key = sha256(wallet_id | date | type_key | amount_cents | balance_after_cents | counterparty)
```

⚠️ **Every input is a property of the ROW ITSELF** — never its neighbours, never the file. That rules out:

- the **line number**, the file hash and the statement period (all change on a re-send);
- the **resolved direction**, which is subtler and was the trap: direction is an *arbitration outcome* that changes with whether a neighbouring row was missing, so the same transaction in two overlapping exports would mint two identities and be recorded — and granted — twice. The transaction **type** is the row's own text and cannot drift;
- the **reference**, which has no arithmetic protecting it. On the real statement its assembly is demonstrably imperfect (the header label leaks into the first row; a date prefix migrates to the end) while the chain still reconciles at 55/60. It stays a human-readable bridge to the customer's receipt, and nothing more.

**One scheme, never a fallback.** A "reference if confident, fingerprint otherwise" design lets a row hop between schemes when the parser changes, minting a fresh identity — and a second ledger row for money that arrived once means the buyer is granted their product twice.

Two rows this statement genuinely cannot tell apart are **both quarantined** — not merged, and not given a serial number, which would change the moment the other row is absent from an export.

### Privacy — the redaction is in the repository, not the parser

This is the owner's **personal** wallet. The money-out rows are their groceries, their tolls, transfers to family, and this database is readable by sales staff. So [`TngStatementRepository`](/src/Payment/Repositories/TngStatementRepository.php) nulls `description`, `counterparty_name`, `details_text` and `raw` on every debit, keeping only what the chain needs (when, how much, the balance after).

⚠️ It happens at **persist** time on purpose. The parser keeps full fidelity, so re-reading a retained PDF after a parser fix is still possible; a parser that redacted would have destroyed what a re-parse needs.

### Refuse-to-guess rules

Whole-document refusals throw [`TngStatementException`](/src/Payment/Exceptions/TngStatementException.php) with a named reason and produce **zero rows**:

| Reason | Prevents |
|---|---|
| `NOT_A_WALLET_STATEMENT` | some other PDF read as money |
| `CARD_STATEMENT` | a physical-card statement, whose toll rows look exactly like the wallet's |
| `EPORTAL_VARIANT` | the web-portal export, which has **no running balance** — direction cannot be checked at all |
| `LAYOUT_CHANGED` | a renamed or reordered column shifting every cell by one. The message prints expected **and** found |
| `PASSWORD_REFUSED` | a wrong password → noise → "no text" → "no money came in" |
| `NO_TEXT_LAYER` | a scan mistaken for an empty month |
| `NO_TABLE_BODY` | a broken read mistaken for a period with no transactions |
| `METADATA_AMBIGUOUS` | filing one person's payments under another's wallet |
| `ORDER_UNDECIDABLE` | half the rows verified backwards |

⚠️ **There is no fallback that reads the table upward.** A layout with rows above their header produces zero rows and a loud named refusal — obviously wrong to whoever sees it. Trying the other side instead reads the document's own title block as a row of payments and reports success, which is the single outcome this design exists to prevent.

## Reference usage

```php
use Src\Payment\Services\TngStatementParser;
use Src\Payment\Exceptions\TngStatementException;

try {
    $statement = app(TngStatementParser::class)->parse(
        $bytes,
        $file->getClientOriginalName(),
        // The password is the wallet number; only the INDEX that worked is recorded.
        TngStatementParser::passwordCandidates($walletNumber),
    );
} catch (TngStatementException $e) {
    flash()->error($e->describe());
    return back();
}

$statement->ingestable();     // safe for a machine — chain-verified customer payments
$statement->moneyIn();        // everything that arrived; the difference is the admin's queue
$statement->likelyFiltered(); // ⚠️ show the banner if true
```

`php artisan tng:parse <path> [--all] [--store]` does the same from the shell. `--dry` is the default; writing has to be asked for.

## Testing

⚠️ **No real statement is committed, ever.** A genuine one carries a person's name, wallet id, every road they drove and everyone who sent them money. [`TngPdfBuilder`](/tests/Support/Tng/TngPdfBuilder.php) synthesises files in the same shape — `/Rotate 90`, one FlateDecode stream per page, an absolute `Tm` per cell, `Amount ` and `(RM)` drawn as two fragments at the identical coordinate — and can produce deliberately broken ones (renamed column, missing row, wrong balance) so every refusal has a file that triggers it.

Six rules are **mutation-verified** (disable the rule → the named test goes red): the arbitration table, gap quarantine, the type allow-list, the per-page header cut, integer-cent parsing, and the identity key's independence from direction.

The amounts in the cents test are chosen, not arbitrary: `(int)(((float) '0.29') * 100)` is **28**. 1,145 of the first 20,000 sen values truncate short that way, and because the chain is an exact equality, one wrong cent sends a perfectly good payment to quarantine.

## Related files

- [src/Common/Pdf/](/src/Common/Pdf/PdfTextExtractor.php) — `PdfTextExtractor` (`inspect()` / `extract()`), `PdfDocument`, `PdfDecryptor`, `ContentStreamReader`, `TextFragment`, `PdfProfile`, `PdfText`, 5 named exceptions
- [app/Support/TngStatement/](/app/Support/TngStatement/TngTableReader.php) — `TngTableReader` (geometry → cells), `TngRowMapper` (cells → values), `TngDirectionResolver` (the arbitration)
- [src/Payment/Services/TngStatementParser.php](/src/Payment/Services/TngStatementParser.php) — orchestration, refusals, the identity key
- [src/Payment/Support/Tng/](/src/Payment/Support/Tng/ParsedStatement.php) — `ParsedStatement`, `ParsedStatementRow`, `TngParseReason`
- [config/tng.php](/config/tng.php) — the type allow-list, the expected columns, the limits
- [src/Payment/TngStatement.php](/src/Payment/TngStatement.php) · [TngStatementLine.php](/src/Payment/TngStatementLine.php) · [TngStatementRepository.php](/src/Payment/Repositories/TngStatementRepository.php)
- [resources/js/Pages/Manage/Payment/Statements/](/resources/js/Pages/Manage/Payment/Statements/Index.vue) — `Index.vue` (upload box + statements list), `Show.vue` (one statement's lines), `Partials/WaitingLines.vue` (the cross-statement queue of money waiting on a person), `Partials/MailboxLog.vue` (what the poller found in the mailbox — outcome per message, and whether an unreadable one **will be tried again**, which is the fact that makes fixing the wallet number worth doing)
- [app/Http/Controllers/Manage/Payment/PaymentStatementsController.php](/app/Http/Controllers/Manage/Payment/PaymentStatementsController.php) · [Pages/Manage/Payment/Statements/](/resources/js/Pages/Manage/Payment/Statements/Index.vue)
- [app/Console/Commands/Tng/ParseTngStatement.php](/app/Console/Commands/Tng/ParseTngStatement.php)
- [tests/Feature/Payment/TngStatementParserTest.php](/tests/Feature/Payment/TngStatementParserTest.php) (25) · [TngStatementUploadTest.php](/tests/Feature/Payment/TngStatementUploadTest.php) (8) · [tests/Support/Tng/TngPdfBuilder.php](/tests/Support/Tng/TngPdfBuilder.php)

---

## Phase E — the mailbox and the ledger

Reading a statement is only half of it. **[Phase E — the convergence rule](/docs/modules_handbook/manage/payments/phase-e-convergence.md)** covers how a statement line and a customer's receipt become ONE payment rather than two, the mailbox poller, and why the machine is never allowed to match them by itself.

## ⚠️ Handover notes written during Phase D

1. **`PaymentClaimsController::resolve()` already CREATES a ledger row** ([:148](/app/Http/Controllers/Manage/Payment/PaymentClaimsController.php#L148)). The moment Phase E also creates one from a statement line, the same money has **two** rows, and attributing both grants the product twice. The fix — `resolve()` accepting an optional statement line and *attaching* an existing unattributed row instead of making a new one — is genuinely required, and belongs to Phase E in its own commit.
2. **Pre-check with `withTrashed()` before inserting.** MySQL unique indexes ignore soft deletes; the house idiom is [`HostedLinkImportPreview.php:85-94`](/app/Services/Payment/HostedLinkImportPreview.php#L85-L94).
3. ⚠️ **CORRECTED 2026-08-07 — the earlier advice here was wrong.** It said to copy `dedupe_key` into `payment_reference`. Verified against the live schema, `purchase_histories` carries **four** unique indexes, not one:
   ```
   uuid
   payment_reference
   (payment_provider, provider_payment_id)
   (payment_provider, provider_charge_id)
   ```
   Putting the statement's key in `payment_reference` leaves the CLAIM's reference (`TNG-DUFB26`) with nowhere to live, and the two lanes then share no key at all. Put the statement line's `dedupe_key` in **`provider_payment_id`** instead — it has its own composite unique index with the provider — which leaves `payment_reference` free for the claim. **One row can then carry BOTH identities**, and the second lane to arrive finds the row by the other key and UPDATES it rather than inserting a twin.
   ⚠️ The index alone does not save you: MySQL allows unlimited NULLs in a unique index, so `('tng', NULL)` and `('tng', 'tng:a3f9…')` do not collide. The reconciliation has to be written, not assumed.
4. **`counterparty_name` is create-only** on `PurchaseHistoryRepository`; it is absent from `update()`'s whitelist. Not supplied at insert means never.
5. **Touch 'n Go is a `READ_ONLY_FACT_PROVIDER`** — amount, currency and `purchased_at` cannot be edited afterwards. That is the whole reason the staging tables exist: fixing a parser bug means bumping `TngStatementParser::VERSION`, re-parsing the retained originals, and diffing.
6. **`TngSettingRepository::markPolled()` already exists and has no caller.** Phase E uses it; do not invent a second stamping path.
7. **The "may be incomplete" warning must travel with the statement into Phase E's screens.**
8. **`tng_statement_lines` must never gain `softDeletes`** — see the model docblock. Use `state`.
9. **Still unknown, and worth closing first:** nobody has seen a real money-in row. Ask the owner to have someone transfer RM1, export that single day, and upload it. Everything above is built to refuse rather than guess, but a real sample turns one assumption into a fact.