# Runbook — fill UAE v2 with the rest of the Bayut data

> 📍 Part of the project catalogue doc set — the map is
> [start-here.md](/docs/modules_handbook/shared/project-catalogue/start-here.md). Background:
> [bayut.md](/docs/modules_handbook/shared/project-catalogue/bayut.md) (the provider) and
> [uae-v2.md](/docs/modules_handbook/shared/project-catalogue/uae-v2.md) (the page it feeds).
>
> **Written for an AI agent (Claude Code) to execute step by step.** Every step says what to run,
> what a good result looks like, and when to STOP and report instead of improvising.

## 0. Where things stand (2026-10-09)

wk.propertylab.com.my runs the UAE v2 code (branch `dev-wk`), is in catalogue **LIVE mode**
(`CATALOGUE_USE_DEFAULT_CONNECTION=0`: the models read the shared master, SELECT-only), and holds the
Bayut data as **wk's OWN rows** (`origin = own`, loaded through a process-local connection redirect —
[uae-v2.md](/docs/modules_handbook/shared/project-catalogue/uae-v2.md) *On wk*). It has loaded two
Bayut crawl files:

| Loaded file | Contains |
|---|---|
| `bayut_partial_20261008_224008_588061.json` | 2,998 UAE new projects — basic facts only |
| `bayut_partial_20261009_011112_162391.json` | Business Bay only — 74 projects, 470 layouts, 909 floor-plan drawings, 213 GPS buildings, 12 months of DLD sales + Ejari rents |
| `bayut_subsale_merged_20261009.json` (12:08 MYT) | **Subsale, UAE-wide** — 12,247 ready buildings (`building:*` only, `market_segments: ["subsale"]`), 6,003 of them with 12 months of DLD sales (24,148) + Ejari rents (227,250), 2,975 with a building guide. ONE file merged on wk from the cloud zip's two (rule 6) |
| `pf_new_projects_20261009_final.jsonl.gz` (provider **`propertyfinder-nl`**, 20:57 MYT, `inbox/propertyfinder-nl/processed/`) | **The UAE v2 subjects** — 2,990 Property Finder new projects with the developer's unit types, pinned at the matched Bayut building (rule 7). Run #15: 2,990 created, 0 failed; 10,705 floor plans (7,198 priced), 58,306 media rows; tab 2,990. Loaded with `storage/app/catalogue/pf-nl-sync-20261009.php` — [propertyfinder-new-projects.md](/docs/modules_handbook/shared/project-catalogue/propertyfinder-new-projects.md) *Loading it* |

**2026-10-09 — the subsale set landed first.** The cloud session held the new-project file back for
the Property Finder merge and sent `bayut_subsale_load.zip` instead (wk Uploads
`/file/343aed1a-a732-4d23-b41a-693ba257f7ce`): the ready buildings with their evidence, plus a second
file of building guides for a subset. Its `LOAD-SUBSALE.md` said to sync the two files one after the
other — that would have erased the evidence (rule 6), so they were merged first and loaded with the §7
command. **Result (Run #13, 56 min): 12,034 sources created, 213 updated, 0 failed** → 14,934 own AE
rows (11,738 new; 296 buildings attached by name to a Bayut project that is the same tower, e.g. Island
Park 1), building sources 12,247 (5,966 with rent rows, 1,997 with sales). The merge's resolve step
(`ProjectRepository::syncCatalogFloorPlans`, the same one that builds Malaysian layouts from listings)
minted 37,077 subsale layouts for 5,853 buildings from their sales/rent rows — each with `rent_median` and
`market_value`, no price, no floor-plan drawing. The UAE v2 tab still counts 2,537 (new-project segment
only); the rows are the neighbours every Dubai project's Market Value and Rental read within 2 km
(Altitude de GRISOGONO after the load: FAIR PRICE on 6,348 listings). The 10 saved analyses were
cleared and the guide photos (8,090 media rows) copied with the §9 script.

**The gap this runbook closes:** only the 74 Business Bay projects have layouts, Market Value and
Rental. 2,143 UAE projects have off-plan listings on Bayut but no layouts (e.g. **10 Oxford**,
JVC — its page shows "Analysis not available: no unit pricing yet"). Of those, 1,710 are in Dubai
and can also get Market Value + Rental. Projects outside Dubai get layouts only (the sales/rent
ledger is the Dubai Land Department's). ~855 projects have no listings on Bayut at all and will
stay empty — that is not a failure.

## 1. Ground rules — do not break these

1. **Never pass `--full` to `catalogue:sync` unless the file's `meta.snapshot` is `"full"`.** A
   full sync deactivates every Bayut record the file does not contain. This runbook produces
   `partial` files; sync them WITHOUT `--full`.
2. **Never run `catalogue:mirror-from-master --apply` on wk** while the Bayut data lives only in
   wk's own rows — it deletes every local-only row the master does not have, i.e. all Bayut rows.
3. **Do not flip wk to copy mode for this** (`CATALOGUE_USE_DEFAULT_CONNECTION`). Its copy lacks the
   951 PropertyFinder UAE rows and every publication date, so the other country tabs read 0 the
   moment it flips (seen and reverted 2026-10-09). If the line is ever removed, DELETE it — never
   set it to `=false` (that string reads as true).
4. **If Bayut blocks you (captcha / 403 / 429), stop and report.** Do not add proxies, rotate
   user agents or retry in loops. Bayut's terms restrict automated collection; volume is the
   owner's decision.
5. No paid API is involved (no ScraperAPI, no RapidAPI). Don't add one.
6. **Two files that name the SAME keys must be merged into ONE file before loading.** The ingestion
   REPLACES a source's `fields` on every update (`ProjectCatalogueMergeService::upsertSource` →
   `$source->update(['fields' => …])`) — it does not merge them. A guides-only file synced after the
   evidence file erases the DLD sales and rent lists of every building both files name (1,424 of
   them on 2026-10-09). `storage/app/catalogue/bayut-merge-subsale-20261009.py` is the merge that was
   used: file 1's record, file 2's fields overlaid, the evidence keys kept from file 1, file 2's media
   and raw payload (the newer crawl of the same guide). One artifact, one §7 command.
7. **Bayut is subsale only (owner, 2026-10-09).** Never load Bayut new projects (`project:*`) and never
   set `project_catalogue.providers.bayut.new_projects` to true — the loader skips them while it is
   false. UAE new projects come from Property Finder (facts, developer unit types, prices, photos,
   description) — provider `propertyfinder-nl`, loaded from the matched hand-off with
   `catalogue:sync propertyfinder-nl --file=…jsonl.gz` (on wk: the redirect wrapper
   `storage/app/catalogue/pf-nl-sync-20261009.php`), and THOSE are the UAE v2 subjects
   (`CatalogueDetailService::UAE_V2_PROVIDER`) —
   [propertyfinder-new-projects.md](/docs/modules_handbook/shared/project-catalogue/propertyfinder-new-projects.md)
   *Loading it*. Bayut's own new-project crawl is only the input for matching Property Finder to
   Bayut's GPS ([pf-bayut-matching.md](/docs/modules_handbook/shared/project-catalogue/pf-bayut-matching.md)).
   The 2,998 Bayut new projects loaded on 2026-10-09 were retired the same evening with
   `catalogue:retire-bayut-new-projects --apply`, run through the redirect wrapper
   `storage/app/catalogue/bayut-retire-new-projects-20261009.php` (on wk the bare command would
   address the read-only master). **Result (Run #14, 7 min): 2,998 sources deactivated, 19,175 media
   deactivated, 2,687 projects removed; the 311 towers that are also a Bayut building keep the row as a
   subsale building.** Afterwards: 0 active `project:*` sources, 12,247 building sources (5,966 with
   rents, 1,997 with sales), 12,247 own AE rows, 38,235 layouts; the UAE v2 tab read 0 until the
   Property Finder load later that night (Run #15: 2,990 created, tab 2,990 — the file's §0 row); a Dubai ready building's page (e.g. Aykon City
   Tower C) still opens with its layouts, rents and sales.
8. **The owner's display rules for Property Finder unit types (2026-10-09 evening). Never break them.**
   The final data set is `propertyfinder_new_projects_final_v3.zip`, stored on wk at
   `storage/app/catalogue/propertyfinder-projects/final/` (earlier zips under `old/`; the v2 copy that
   Run #15 loaded stays in `inbox/propertyfinder-nl/processed/` as the record of that run). Every
   developer unit type carries a `display` block — `price_aed`, `size_sqft_from`/`_to`,
   `layout_photos`, `showable` — with gaps filled from the project's own recent sales, then listings.
   1. **Show a unit type only when it has a price AND a size** (`display.price_aed`,
      `display.size_sqft_from`). The data set's own `showable` flag also demands a layout photo, and
      the first wording of this rule did too; the owner settled it the same night ("allow price,
      size without photo"): 1,622 unit types on 486 projects had both figures and no drawing
      anywhere, and they are listed and analysed, with an empty drawing slot. A type without a price
      or a size is never shown.
   2. **No source labels on screen.** Price and size are plain numbers; the price is called
      **"Price"**, never "Starting from" (some shown prices are recent sales, 5–7 % above the
      developer's figure). Never show `_provenance`, `bayut_match` or `bayut_names`; never name Bayut
      or any other portal as a source anywhere on the portal.
   3. **Copy every layout photo to our own storage (MediaService) before showing it.** Never link
      another site's image address.
   The rules are enforced in code since the same night: `catalogue:sync propertyfinder-nl` reads
   `display` (showable only), ingests every Property Finder asset INACTIVE, and
   `catalogue:store-pf-media` copies each file to our disk and only then shows it (on wk:
   `storage/app/catalogue/pf-store-media-20261009.php`). The owner approved the v3 reload the same
   night (option 1); the interim retire of the v2 rows was not needed because Analyze Property is
   locked to admins on wk. Result: Run #17 [success] propertyfinder-nl/AE: 0 created, 2,990 updated, 0 failed (34 min, 23:48–00:22 MYT; Run #16, the same file under the stricter first wording of rule 1, was stopped mid-run and marked aborted) → 9,351 layouts on 2,689 projects, every one priced AND sized, 7,729 of them with a drawing; 57,029 media rows held inactive until `catalogue:store-pf-media` copies them (started 00:25 MYT, heroes first); the UAE v2 tab reads 2,990; 08 Life Residences (sizes filled from recent sales) now analyses as FAIR PRICE. See
   [propertyfinder-new-projects.md](/docs/modules_handbook/shared/project-catalogue/propertyfinder-new-projects.md)
   *The final data set and the owner's display rules* and *Loading it*.

## 2. Pick the machine — 1-minute test

The crawl must run where Bayut answers. wk's server got a captcha on www.bayut.com before. Test
on the machine you intend to use:

```bash
curl -s -o /dev/null -w "%{http_code}\n" -X POST https://www.bayut.com/api/transactions \
  -H "Content-Type: application/json" -H "Origin: https://www.bayut.com" -H "Referer: https://www.bayut.com/" \
  -d '{"purpose":"for-rent","locationExternalIDs":["5002"],"locationLevels":[1],"categoryExternalID":"1","completionStatus":"any","timeSinceCreation":"12m","contractRenewalStatus":"All","page":1,"hitsPerPage":20,"v":"5"}'
```

- `200` → this machine can crawl. Continue.
- anything else → STOP and report the code. The owner will run steps 3–5 on a laptop / office
  machine instead (it needs PHP 8.4, the repo on `dev-wk`, `composer install`; it does **not**
  need a database — the crawl only writes a JSON file), then copy the file to the server (§6).

## 3. Preconditions (on the crawl machine)

```bash
git rev-parse --abbrev-ref HEAD                               # dev-wk (or claude/eloquent-mendel-xkbkhs)
php artisan catalogue:scrape-bayut --help | grep -E "community|skip|sales-pages"   # all three must print
```
If the options are missing, the checkout is too old: `git pull` the branch first.

## 4. Proof run — JVC (10–15 min)

```bash
php -d memory_limit=4G artisan catalogue:scrape-bayut --community="Jumeirah Village Circle (JVC)" --skip=guides
```

The last line prints the file path and `N failures`. Check the file:

```bash
F=$(ls -t storage/app/catalogue/inbox/bayut/bayut_*.json | head -1)
php -r '$d=json_decode(file_get_contents($argv[1]),true); $m=$d["meta"]; $p=$d["projects"];
 echo json_encode($m["stats"]),"\nsnapshot=",$m["snapshot"]," failed=",$m["detail_failed"],"\n";
 echo "projects with layouts: ",count(array_filter($p,fn($r)=>!empty($r["floor_plans"]))),"\n";
 echo "records with rents: ",count(array_filter($p,fn($r)=>!empty($r["fields"]["asking_rental_listings"]))),"\n";' "$F"
```

Good result: `projects` ≈ 227, `layouts` in the hundreds, `evidence_locations` in the hundreds,
`failed=0` (a handful is tolerable; dozens means throttling — STOP and report). Then load it on wk
(§7) and confirm 10 Oxford has floor plans before spending hours on the rest.

## 5. The rest of the UAE (estimated 2–3 hours)

One run, in the background, without the slow building-guide stage (guides add facts, not prices):

```bash
nohup php -d memory_limit=4G artisan catalogue:scrape-bayut --skip=guides > storage/logs/bayut-scrape.log 2>&1 &
echo $! > storage/logs/bayut-scrape.pid
```

Check progress occasionally with `tail -n 5 storage/logs/bayut-scrape.log` (progress bars; the
stages print in order: units → floor plans → GPS buildings → transaction evidence). Do not
sleep-poll in a tight loop. It is finished when the log's last line names the output file.

Expected scale (estimates from the Business Bay run, ~943 requests in 3 minutes): ~3,000
floor-plan requests, ~35,000 sales/rent requests. Memory peaks with the evidence stage; 4 GB is
ample.

**If it dies or Bayut starts refusing mid-run**, do NOT restart the whole UAE. Run the big Dubai
communities one at a time instead — each is its own small partial file, and loading several is
fine (later files update the same records):

```bash
for c in "Jumeirah Village Circle (JVC)" "Dubai Islands" "Dubai South" "Dubai Land Residence Complex (DLRC)" \
         "Al Furjan" "Dubai Hills Estate" "Arjan" "Business Bay"; do
  php -d memory_limit=4G artisan catalogue:scrape-bayut --community="$c" --skip=guides || break
done
```
(`--community` is repeatable and matches the community name exactly as Bayut writes it — the
`area` column on `catalog_projects`.)

Validate the output file with the §4 snippet. Good result for the whole UAE: `projects` = ~2,998,
`layouts` in the thousands, `evidence_locations` in the thousands, `failed` 0 or a handful.

## 6. Copy the file to wk (only if the crawl ran elsewhere)

Either `scp` it straight into the inbox:

```bash
scp storage/app/catalogue/inbox/bayut/<file>.json ubuntu@<wk host>:/var/www/html/peta/storage/app/catalogue/inbox/bayut/
```

or, without shell access, zip it and drop it on wk's **Manage → Uploads** page (any type, 500 MB
per file; 2026-10-09 the first handoff went this way as `/file/{uuid}`). On wk the agent streams
it out with `MediaService::stream()` into a fresh directory and unzips into the inbox. A JSON that
gunzips to more than ~500 MB is split by community (§5's per-community loop) rather than uploaded.

## 7. Load into wk

On the wk server, in `/var/www/html/peta`. **Not the bare command** — on wk `catalogue:sync` writes
the `catalogue` connection, which is the read-only master, so it is run through the process-local
redirect that the first load used (the script also clears the saved analyses of §8 and flushes the
listing cache, then prints the §8 counts):

```bash
BAYUT_FILE=/var/www/html/peta/storage/app/catalogue/inbox/bayut/<file>.json \
  php -d memory_limit=4G artisan tinker --execute="require '/var/www/html/peta/storage/app/catalogue/bayut-sync-20261009.php';"
```

The script refuses unless the redirected connection resolves to `petav3wk`. No `--full` (rule 1).
The sync prints `Run #N [success] bayut/AE: X created, Y updated, 0 failed.`
**How it matches what is already there:** every record is keyed by provider `bayut` plus
`external_id` — `project:{Bayut project id}` or `building:{Bayut location id}` — the same keys the
two loaded files used. Existing projects are UPDATED in place (layouts, sales, rents added), new
buildings are CREATED, nothing is duplicated, and nothing absent from the file is removed. The
whole-UAE file also re-covers Business Bay, so the older Business Bay file needs no reload.
A handoff of SEVERAL files for the same keys is merged into one first — rule 6.

Expected time: minutes per thousand records (each is matched against wk's ~37,700 projects).

## 8. Clear saved analyses, then verify

Each layout's first analysis is saved and served again while the hash of its inputs matches
(`UnitAnalysisStore::inputHash`: coordinates, price, size, bedrooms). The sales/rent evidence is
NOT in that hash, so a Business Bay layout with an unchanged price would keep its old Market Value.
The §7 script clears the saved analyses of wk's own UAE plans (10 rows on 2026-10-09) and prints the
counts below. **On wk the model classes point at the master**, so the Eloquent snippets below only
work inside the same redirect; by hand, query the default connection instead:

```bash
php artisan tinker --execute='
$own = DB::table("catalog_projects")->where("origin","own")->where("country_id",3)->select("id");
echo "UAE projects with layouts: ", DB::table("catalog_floor_plans")->whereIn("catalog_project_id",$own)->distinct()->count("catalog_project_id"), "\n";
$ox = DB::table("catalog_projects")->where("slug","10-oxford")->where("origin","own")->value("id");
echo "10 Oxford floor plans: ", DB::table("catalog_floor_plans")->where("catalog_project_id",$ox)->count(), "\n";'
```

Good result: projects with layouts in the low thousands (182 before), 10 Oxford > 0. Then
open `https://wk.propertylab.com.my/ae/projects/10-oxford?chrome=portal&tab=invest` as an admin —
Units · Price lists layouts in AED, Investment Analysis shows Market Value and Rental.

## 9. Optional — store photos on our own storage

Pages already show photos straight from Bayut's URLs (`CatalogMedia::resolveUrl()`), so this is not
needed for the demo. A re-sync keeps rows already copied (`CatalogMedia::upsertScraped` never turns a
stored row back into a URL). To copy the rest (tens of GB, resumable, safe to stop) run **all four
kinds** — on wk through the same redirect, see `storage/app/catalogue/bayut-store-media-20261009.php`:

```bash
php artisan catalogue:store-bayut-media --kind=floor_plan --dry-run   # count first
php artisan catalogue:store-bayut-media --kind=hero --kind=floor_plan --kind=document --kind=gallery
```

## 10. Report back

Give the owner: the machine used and the §2 status code; each output file name with its §4
numbers; each sync line from §7; the §8 counts; anything skipped or stopped, and why.
