# UAE v2 onto the master — the Hub import

**Written 2026-10-13 as the hand-off from wk to the CTO.** It is the step the UAE v2 docs kept
calling "still to do": every row of the UAE v2 data set lives only in wk's own database today, and
this is how it reaches the shared master so every deployment reads it and wk can retire its copy.

> **Done on the Hub 2026-10-14/15.** Both data sets are on the master, their photos are copied and
> the Layout Analysis UAE tab is built (step 7, 2026-10-15) — results in
> [*The Hub run*](#the-hub-run-2026-10-14) below. Three traps this runbook did not foresee are now
> written into steps 1, 4 and 7. The old feed was deleted 2026-10-15 (below). Still open: wk
> retiring its own copy.

## What it does

The UAE v2 feature on Analyze Property (the *United Arab Emirates* New Projects tab, the project
page in AED with Units / Investment / Buying Guide / Amenities, the UAE tab of Layout Analysis) runs
on two data sets, loaded with two commands:

| Role | Provider | File | Records | What the engine does with it |
|---|---|---|---|---|
| **Evidence** (the comparison market) | `bayut` | `bayut_subsale_merged_20261009.json` (49.7 MB) | 12,247 ready buildings UAE-wide, `building:*` only — 12 months of DLD sales and Ejari rents for ~6,000, building guides, building GPS | Market value and rental for a subject come from the ring of these within 2 km (Dubai only: the DLD ledger is Dubai's) |
| **Subjects** (what a member analyses) | `propertyfinder-nl` | `pf_new_projects_20261009_final.jsonl.gz` (29.6 MB, version 3) | 2,990 Property Finder new projects, matched to Bayut's building GPS; every developer unit type with price, size, photos, brochure; owner's display rules baked in (`display.showable`) | Becomes the catalogue projects + their layouts; one project per line |

Both files are on wk and bundled with a checksum manifest at
`storage/app/catalogue/handoffs/uae-v2-master-import-20261013.zip` (35 MB; 79 MB unpacked). Their checksums:

```
924ebabe4708819d0daecb90444266fccaecb21640546ccf3187b5ba3ce05f0c  bayut_subsale_merged_20261009.json
8b6158b3be89edcaf3d7fb480ad3c695929b41e4ae4765dd908bbdbaa3d87474  pf_new_projects_20261009_final.jsonl.gz
```

### Where things stand (2026-10-13)

| | Code | Data |
|---|---|---|
| **master / Hub** | On `master` since PR #199 (merged 2026-10-12): both providers, both importers, the tab, the project page, Buying Guide, Amenities, the Layout Analysis UAE tab. **One migration is still only on `dev-wk`** — see step 0. | **None.** Master `data_providers` ends at id 9; there is no `bayut` or `propertyfinder-nl` row, no source, no project. The OLD UAE feed (provider `propertyfinder`, 951 projects, the old `/ae` page) is untouched and stays. |
| **wk** (`wk.propertylab.com.my`, reads the master LIVE with a SELECT-only account) | `dev-wk` = master + the HK Layout Analysis commits + this runbook and its migration | 15,085 own AE rows (`origin = own`, ids ≥ 1,000,000,000) in `petav3wk`, loaded through a process-local redirect of the `catalogue` connection: the 12,247 buildings + the 2,990 projects (152 attached to a kept tower), 9,351 Property Finder layouts, 55,532 photos copied to our storage, 6,904 Layout Analysis rows for Dubai. The federation shows them beside the master's rows. |

wk cannot write the master, and `catalogue:mirror-from-master --apply` on wk would DELETE every
one of those own rows — so the order is fixed: the master gets the data through the Hub first, then
wk retires its copy.

## How it works — the steps, on the Hub

The Hub (`propertylabglobal.com`) is the only deployment whose `catalogue` connection may write
`master_projects`. Every command below is the stock one; wk ran the same commands through a redirect
wrapper, which the Hub does not need.

### 0. Ship the code

The master lacks one file: `database/migrations/catalogue/2026_10_13_100000_seed_propertyfinder_nl_provider.php`
(on `dev-wk`). Deploy the Hub from `master` **after** the next `dev-wk → master` PR merges, and run
nothing below before that deploy. (Merged 2026-10-13, `07f96ac54`.) Why it matters: the earlier Property Finder seed
(`database/migrations/2026_10_09_180000_…`) sits in the SITE directory, so it only ever creates the
row in a deployment's own `data_providers`. `CatalogueIngestionService` reads the provider id on
the **default** connection and writes it into `catalog_project_sources` on the **catalogue** one, so
without the master row every source the Hub writes would carry an id the master cannot explain,
and the UAE v2 rule (`CatalogueDetailService::UAE_V2_PROVIDER`) would never match on a site reading
the master.

### 1. Migrate the master, then the Hub's own tables

```bash
php artisan migrate --path=database/migrations/catalogue --database=catalogue   # the master — Hub only
php artisan migrate --path=database/migrations/catalogue                        # the Hub's own copy (an ordinary migrate also runs it)
php artisan config:cache
```

Pending on the master on 2026-10-13 (`php artisan migrate:status --path=database/migrations/catalogue
--database=catalogue` shows them): five provider seeds — `2026_09_27 seed_propin_pg_provider`,
`2026_09_27 seed_nuprop_provider`, `2026_10_08 seed_bayut_provider`, `2026_10_12
seed_landregistry_provider`, `2026_10_13 seed_propertyfinder_nl_provider`. They must take ids
**10 propin-pg, 11 nuprop, 12 bayut, 13 propertyfinder-nl, 14 landregistry** — the ids wk's local
table already holds, which is what every LIVE site needs (the id is read on the site's own table and
compared against the master's sources).

⚠️ **Not with one `migrate`.** Each seed takes the next free id, and `migrate` runs files in filename
order, so `landregistry` (10-12) would take 13 and `propertyfinder-nl` (10-13) 14 — swapped against
wk. The Hub ran the seeds one file at a time, `propertyfinder-nl` before `landregistry`, then the
two directory commands above for anything else pending:

```bash
for f in 2026_09_27_190000_seed_propin_pg_provider 2026_09_27_220000_seed_nuprop_provider \
         2026_10_08_120000_seed_bayut_provider 2026_10_13_100000_seed_propertyfinder_nl_provider \
         2026_10_12_120000_seed_landregistry_provider; do
    sudo -u www-data php artisan migrate --database=catalogue --path=database/migrations/catalogue/$f.php
done
```

**Check before loading anything — on BOTH databases:**

```sql
SELECT id, code FROM master_projects.data_providers ORDER BY id;
SELECT id, code FROM petav3.data_providers ORDER BY id;   -- the Hub's own table: the same list
-- expect … 9 realtycheck, 10 propin-pg, 11 nuprop, 12 bayut, 13 propertyfinder-nl, 14 landregistry
```

If the ids come out different, stop and tell wk: a site whose local id differs from the master's
reads the wrong provider for those sources. The new migration realigns a site's
`propertyfinder-nl` id to the master's when nothing references it yet; `bayut`'s does the same.

### 2. Put the files on the Hub

Copy the bundle (or the two files) to the Hub's `storage/app/catalogue/handoffs/` and verify the
checksums. **Not** into `storage/app/catalogue/inbox/<provider>/` — `catalogue:sync` without
`--file` loads the oldest `*.json` in that inbox root, and both loads below name their file.

### 3. Load the evidence — Bayut's ready buildings (wk: 55 min)

```bash
php -d memory_limit=4G artisan catalogue:sync bayut \
    --file=/path/to/bayut_subsale_merged_20261009.json --no-interaction
```

No `--full`. The file holds `building:*` records only; `project_catalogue.providers.bayut.new_projects`
stays `false`, so Bayut new projects are never loaded (owner rule, 2026-10-09: Bayut is the subsale
evidence, Property Finder the new projects). Expected row in `catalog_sync_runs` (`provider_code`
`bayut`, `country_iso2` `AE`): `status` success, `stats.created` 12,247, `stats.failed` 0. (wk's own
run read 12,034 created + 213 updated because 213 Business Bay buildings already existed there.) The
merge's resolve step mints the subsale layouts from the sales and rents — rent median + market value,
no price; wk minted 37,077.

### 4. Load the subjects — Property Finder's new projects (wk: 34 min)

```bash
php -d memory_limit=4G artisan catalogue:sync propertyfinder-nl \
    --file=/path/to/pf_new_projects_20261009_final.jsonl.gz --no-interaction
```

No `--full`. Expected: `status` success, `created` 2,990, `failed` 0 → 9,351 layouts on 2,689
projects (every one priced AND sized; 7,729 with a drawing), ~152 projects attached by exact name +
coordinates to a Bayut tower kept in step 3, and **57,029 media rows ingested INACTIVE** — nothing
shows until step 5 copies it (owner rule 3: never hot-link another portal's images).

⚠️ **On the master, the stock command above overwrites 154 live pages — the Hub did not run it.**
The attach rule (`NewLaunchArtifactAdapter::resolveCanonicalId`: exact `project_name` + coordinates
to 5 decimals, same country) matches ANY project, and the master holds the OLD feed (`propertyfinder`,
951 projects, all published on the public `/ae` page), which wk's own rows never saw. A read-only
dry run of the file against the master: 154 of the 2,990 attach to a published old-feed project.
`field_priorities` has no entry for either provider, so the newer source's values win and the public
page changes; and since `rendersAsUaeV2` requires NO active `propertyfinder` source, those 154 would
never show as v2 either.

The Hub loaded through a wrapper instead: it binds a subclass of `PropertyFinderNewProjectsAdapter`
whose `resolveCanonicalId` drops every project carrying a `propertyfinder` source from the name +
coordinate candidates — an existing `propertyfinder-nl` link still wins and a Bayut tower still
attaches, exactly as on wk — then calls the stock `catalogue:sync`. Kept on the Hub at
`storage/app/catalogue/pf-nl-sync-hub-20261014.php`:

```bash
sudo -u www-data php -d memory_limit=4G storage/app/catalogue/pf-nl-sync-hub-20261014.php check   # read-only
sudo -u www-data php -d memory_limit=4G storage/app/catalogue/pf-nl-sync-hub-20261014.php run
```

`check` printed 2,990 read · 2,838 create · 152 attach, all onto `[bayut]` — wk's numbers. Until
the old feed was deleted (2026-10-15), those 154 names existed twice on the master, but never on one
screen: the portal lists only v2, the public site only the old feed. With the old feed gone, the stock
command is safe again.

### 5. Copy the photos onto our storage (hours; both commands resume)

```bash
nohup php artisan catalogue:store-pf-media > storage/logs/catalogue-pf-media.log 2>&1 &
# heroes → floor plans → gallery → documents; wk: 55,532 stored in ~4.5 h, 87 held for good (assets answering 403/404, unsupported formats)

nohup php artisan catalogue:store-bayut-media --kind=hero --kind=gallery --kind=document --kind=floor_plan \
    > storage/logs/catalogue-bayut-media.log 2>&1 &
# all FOUR kinds — the first wk pass listed three and left the drawings on Bayut URLs
```

Both write through `MediaService` (GCS). A Property Finder row stays hidden until stored; a Bayut
row shows from its Bayut URL until copied, so that pass is about not hot-linking, not visibility.
Dry-run either with `--dry-run` to see the eligible row counts first.

### 6. Flush the listing cache

Tab counts cache for 10 minutes (`BuildNewProjectListing::AGGREGATE_TTL`, keys `np-listing:*` on
the `cache` Redis connection). Wait, or delete those keys.

### 7. The Layout Analysis UAE tab (~4 h on wk and on the Hub)

Not optional: without it the master holds none of wk's Layout Analysis rows and the UAE tab of
Layout Analysis is empty on every deployment.

```bash
nohup php artisan layouts:project --country=AE --state=Dubai --run > storage/logs/layouts-project-ae-dubai.log 2>&1 &
```

⚠️ **On the Hub this stock command writes nothing to the master** — the console write gate reads
production's `APP_URL`. Run it through the wrapper instead (see
[*The Hub run*](#the-hub-run-2026-10-14)): `check` first, then `run`. The command prints nothing
while it works; count the master's AE `layout_analyses` rows to follow it.

Dubai only — elsewhere every layout fails "no comparables" (no DLD ledger). wk and the Hub alike:
6,904 rows on 1,956 projects, 771 villa/townhouse enclaves skipped with *No comparable projects
found within 2 km of the given location*. `--slug=` runs one project; `--fresh` recomputes.

### 8. Verify

- In the **portal**, signed in as an admin — `/analyze-property/new-projects?country=ae`, not the
  public site's *Analyze Property* menu — one **United Arab Emirates** tab reads **2,990**. The public
  `/ae/new-projects` still reads **951**: the old feed, unchanged by design (v2 is unpublished).
- Master AE projects = old feed + 12,247 + (2,990 − attaches): **16,036** on the Hub.
- **Golf Grand** → FAIR PRICE (AED 2,246 vs market 2,458 psf, rent AED 8,948, yield 7.0 %);
  **The Meriva Collection** → OVERPRICED; **Al Ghadeer Gardens** (Abu Dhabi) renders its layouts and
  the analysis answers *No comparable projects within 2 km* — by data, not a bug.
- `catalog_sync_runs`: both rows `success`, `failed` 0.
- **Then tell wk.** wk retires its own AE copies (otherwise the federation shows each project twice:
  the master's and wk's own — seen on the UAE tab on 2026-10-14, the morning after this import ran) —
  that step is wk's, not the Hub's: `php artisan catalogue:retire-own-rows --country=AE` (dry-run;
  `--apply` backs every row up to `storage/app/catalogue/retired-own/` and deletes 15,085 projects
  with their 47,586 floor plans, 66,289 media rows, 15,548 sources, 918 own-minted developers AND the
  6,904-row Dubai layout table). It refuses while the master lacks the providers. **Before applying,
  run `layouts:project` for the UAE on the Hub** — the import left the master with 0 `layout_analyses`
  rows for AE, so the Layout ROI Summary for Dubai is empty on every deployment reading the master
  until that runs, and wk's own copy of it goes with its own rows. Until the retirement is done, wk
  never runs `catalogue:mirror-from-master --apply`.

## The Hub run (2026-10-14)

Production and the Hub are one deployment — one server, one `.env`, `app.propertylab.com.my` and
`propertylabglobal.com` as two Apache vhosts — reading the master LIVE (`catalogue` →
`master_projects`, default → `petav3`). Every command ran as `sudo -u www-data` from
`/var/www/html/peta`, after a dump of `master_projects`.

| Step | Result |
|---|---|
| 0 · code | `dev-wk` merged into `master` (`07f96ac54`), Hub pulled |
| 1 · seeds | one file at a time (step 1's loop); master and `petav3` `data_providers` identical, ids 1–14 |
| 3 · Bayut | Run #13 success: 12,247 created, 0 failed |
| 4 · Property Finder | Run #14 success, through the wrapper: 2,990 created (2,838 new projects + 152 onto Bayut towers), 0 failed |
| 5 · photos | Property Finder: 87 held for good (wk's 87). Bayut: 7,083 stored, 5 held (4 × HTTP 404, 1 not a plausible image/PDF) |
| 7 · layouts (2026-10-15) | through the wrapper, after the fix below: 2,422 projects read, 7,675 layouts; wrote **6,904** (6,895 run by the engine + 9 already saved), 771 skipped — all *No comparable projects found within 2 km*. Master AE: `layout_analyses` 6,904 on 1,956 projects, current `unit_analysis` snapshots 6,904 — wk's figures exactly |
| 8 · verify | master AE projects 16,036 = 951 old feed + 12,247 + 2,838; public `/ae/new-projects` 951 (before the old feed was deleted, below); 15,085 after it |

Step 7 (Layout Analysis) was not run on 2026-10-14 — the master held **0** of wk's 6,904 UAE layout
rows, so the Layout Analysis UAE tab was empty on the Hub. Two things stopped it on this box, and
both are fixed or worked around; it ran 2026-10-15:

- **The console write gate.** `CatalogueFederationService::consoleMayWriteMaster()` checks the
  `APP_URL` host (`app.propertylab.com.my` — production's `.env` is the Hub's) against
  `CATALOGUE_EDIT_DOMAINS` (`propertylabglobal.com`), so every master project is skipped. The Hub
  runs it through `storage/app/catalogue/uae-layouts-project-20261015.php`, which sets `app.url` to
  `https://propertylabglobal.com` for its own process only (the `.env`, cached config and the web
  gate are unchanged) and refuses unless the gate then answers yes. `check` = `--missing`, writes
  nothing; `run` = `--run`.
- **A bug, fixed 2026-10-15.** `ProjectLayoutAnalyses::projects()` read the master side with
  `publiclyListed()` and gave only the OWN side the UAE v2 rule (`uaeV2Scope()`) — written when v2
  rows were wk's own. On the master the v2 projects are unpublished, so `--country=AE` printed
  *No projects matched*. Both sides now use the same rule
  (`tests/Feature/Console/ProjectLayoutAnalysesTest.php`).

### Deleting the old feed (2026-10-15)

The card: *"the current UAE data in master_projects can be deleted"* — the OLD feed (`propertyfinder`,
951 AE projects, 1,988 sources, every one published: the whole public `/ae` listing). Since v2 is
unpublished, **`/ae/new-projects` now lists nothing** and the old project pages 404; the portal's
2,990 are untouched. Master AE projects: 16,036 − 951 = **15,085** — wk's own count.

No command does this: `CatalogueIngestionService::retireSources()` refuses to retire every active
source of a provider, by design. The Hub used a one-off script (kept at
`storage/app/catalogue/uae-oldfeed-delete-20261015.php`; its read-only reference scan beside it as
`uae-oldfeed-check-20261015.php`). Per project, in one transaction on BOTH connections: switch the
old-feed sources and their media off, delete the `catalog_project_developers` rows
(`ProjectCatalogueMergeService::deleteIfSafelyOrphaned()` does not — 951 would have been left
behind), move a lead's `lead_project_views` row to the v2 project of the same name, then
`deleteIfSafelyOrphaned()`, which re-checks every blocker and deletes the project with its sources,
floor plans, media and AI content. A refused project rolls back whole. `check` mode runs it all and
rolls every project back.

- **Reference scan first** (every column on both databases that can hold a catalogue project, plan
  or source reference): on `petav3` only 4 `lead_project_views` rows; on wk's own `petav3wk`, 5 — left
  as they are (dev server; the lead page reads the project null-safely and shows a blank name).
- **Result:** `catalog_sync_runs` #15 — 951 deleted, 0 kept, 1,988 sources and 6,547 media switched
  off, 951 developer links removed, 4 lead views moved (Arancia Yards 2 → #66908, Golf Trails →
  #67055, SVN Residences → #67858).
- **Rollback** is the `master_projects` dump taken before step 1.
- **Slugs stay suffixed.** `CatalogProject::mintSlug()` suffixes a taken slug, so every v2 project
  that shares a name with an old-feed one was created as `<name>-2`; the delete frees the base slug
  but nothing renames the v2 row.
- Nothing schedules `catalogue:sync propertyfinder`, but a manual run would re-create the old feed
  from `ih_uae_*`.

## What this import does NOT do

- **Show UAE on `app.propertylab.com.my`.** `market_sites` scopes that domain to Malaysia
  (`all_markets = 0`), so HK and UAE pages 404 there by design — an owner decision, not an import
  step. Production reads the master LIVE (above), so nothing needs mirroring.
- **Renew the Google Maps key.** wk's expired 2026-10-10; the UAE Amenities tab and the HK location
  panel fall back to straight-line Mapbox places until a valid key is set.
- **Gate the absurd rows.** ~60 of the 6,904 Layout Analysis rows carry Property Finder sizes or
  prices that are not sizes or prices (a 5.5 M sqft 4BR, a 1BR at AED 117k) or sit on rings of 1–3
  contracts. The proposed gates need the owner's numbers ([uae-layout-table notes in the memory and
  `propertyfinder-new-projects.md`](/docs/modules_handbook/shared/project-catalogue/propertyfinder-new-projects.md)).
- **Refresh either crawl.** Bayut and Property Finder captcha wk's IP; both crawls ran in a Claude
  Code cloud session and were handed over as files. A later refresh is a re-sync of a newer file
  (additive by provider + external id; a source's `fields` are REPLACED, so a later file must carry
  everything the earlier one did).

## Traps

- **Two artifacts naming the same keys: merge first.** `ProjectCatalogueMergeService::upsertSource`
  replaces a source's `fields` on update, so the last file wins wholesale. That is why the Bayut file
  is a MERGE of the buildings crawl and the building-guide file — never load the two originals one
  after the other (the guide file has no evidence and would blank the sales and rents). Runbook rule 6
  in [bayut-uae-v2-runbook.md](/docs/modules_handbook/shared/project-catalogue/bayut-uae-v2-runbook.md).
- **`--full`** deactivates every source missing from the file. Both files are `snapshot: partial`,
  which the sync refuses to treat as complete, but do not reach for it on a first import anyway.
- **The two Property Finder seeds are both needed.** The site-directory one still runs on every site
  and creates the local row; the catalogue-directory one creates the master's and aligns a site's id.
  Neither is deleted — a migration on `master` is never edited.
- **Provider ids are compared across two databases.** Nothing errors when they differ; the page just
  stops recognising the source. Step 1's check is the only place that catches it — and a single
  `migrate` over the seeds produces exactly that mismatch (step 1).
- **wk's numbers are not the master's.** wk loaded into its own rows, which never held the old feed;
  the master does. Dry-run a load against the master before trusting wk's attach counts (step 4).
- **On the Hub box:** `storage/` belongs to `www-data`, so artisan runs as `sudo -u www-data`;
  tinker needs `sudo -u www-data env HOME=/tmp php artisan tinker` (psysh cannot write
  `/var/www/.config`). A DataGrip dump of the master must not "lock all tables" — that is the whole
  Cloud SQL instance.

## Related files

- `database/migrations/catalogue/2026_10_13_100000_seed_propertyfinder_nl_provider.php` (this hand-off), `database/migrations/catalogue/2026_10_08_120000_seed_bayut_provider.php`, `database/migrations/2026_10_09_180000_seed_propertyfinder_nl_provider.php`
- `app/Console/Commands/SyncCatalogue.php`, `app/Services/Property/Ingestion/CatalogueIngestionService.php` (`$providerId` on the default connection), `app/Services/Property/Ingestion/Adapters/BayutAdapter.php`, `…/PropertyFinderNewProjectsAdapter.php`
- `app/Console/Commands/StoreBayutCatalogueMedia.php`, `app/Console/Commands/StorePropertyFinderNewProjectMedia.php`, `app/Console/Commands/ProjectLayoutAnalyses.php`
- The Hub's step-4 wrapper (excludes old-feed projects from the attach match): `storage/app/catalogue/pf-nl-sync-hub-20261014.php` on the Hub
- wk's redirect wrappers, for reference only: `storage/app/catalogue/bayut-sync-20261009.php`, `storage/app/catalogue/pf-nl-sync-20261009.php`, `storage/app/catalogue/pf-store-media-20261009.php`
- The feature these rows feed: [uae-v2.md](/docs/modules_handbook/shared/project-catalogue/uae-v2.md); the subjects' field map: [propertyfinder-new-projects.md](/docs/modules_handbook/shared/project-catalogue/propertyfinder-new-projects.md); the evidence: [bayut.md](/docs/modules_handbook/shared/project-catalogue/bayut.md); the two-database rules: [databases-and-distribution.md](/docs/modules_handbook/shared/project-catalogue/databases-and-distribution.md) §6.1, `database/migrations/catalogue/README.md`
