# Property Finder new projects — the UAE new-project reference (`catalogue:scrape-pf-new-projects` → `catalogue:sync propertyfinder-nl`)

> 📍 Part of the project catalogue doc set — the map is
> [start-here.md](/docs/modules_handbook/shared/project-catalogue/start-here.md). Companion to
> [bayut.md](/docs/modules_handbook/shared/project-catalogue/bayut.md) and
> [uae-v2.md](/docs/modules_handbook/shared/project-catalogue/uae-v2.md).

## What it does

Reads every UAE new project on propertyfinder.ae/en/new-projects (2,990 on 2026-10-09: 1,519 sold by
the developer, 1,471 resale) — the search list, 125 pages of 24, then each project's own page — into
one gzipped JSON Lines file. **The scrape writes no catalogue rows**; joining it to Bayut is a separate,
reviewed step (below), and the matched file is what `catalogue:sync propertyfinder-nl` loads — these
projects are the **UAE v2** set (owner, 2026-10-09: Property Finder is the UAE new-project reference,
Bayut the subsale evidence) — see *Loading it* below.

What Property Finder has that Bayut does not: the developer's **unit types with a starting price
per type** ("Units — from developer" on the page). Sampled 2026-10-09: 23 of 24 projects carry unit
types, ~69 % of types priced (Bayut's developer types ~31 %), plus stock status, resale "from"
price, construction timeline.

```bash
php -d memory_limit=2G artisan catalogue:scrape-pf-new-projects          # ~1 hour, no ScraperAPI
php artisan catalogue:scrape-pf-new-projects --pages=1                   # trial: first 24 projects
```

## How it works

- `ScrapePropertyFinderProjects` reads each page's Next.js payload (`NextData`). The list page now
  and then answers with an empty project array — retried up to 5 times.
- `PropertyFinderProjectNormalizer::project()` maps one project: developer, prices, stock, sales
  phase, handover, construction, ownership, payment plans, amenities, images, brochure, master
  plan, timeline, FAQs, the community summary, and `unit_types` (flattened by `unitTypes()`: per
  bedroom group — size range, bathrooms, starting price, total units, layouts with floor plans). A
  `0` price is "not published", stored as null. The page's own payload is kept under `raw`.
- Output: `storage/app/catalogue/propertyfinder-projects/pf_new_projects_<stamp>.jsonl.gz` and a
  `_summary.json` (projects, with unit types, priced types, failed pages).

## Before merging it with Bayut — two facts

1. **The map pin is the community, not the building.** `location_type` says which level (COMMUNITY,
   SUBCOMMUNITY, sometimes TOWER); Talay's pin is Saadiyat Island's, shared by its neighbours, 1.8 km
   from the project. Nearby-market analysis must keep using Bayut's GPS.
2. **No DLD project number.** Matching is by name + developer + community + GPS —
   [pf-bayut-matching.md](/docs/modules_handbook/shared/project-catalogue/pf-bayut-matching.md).

Property Finder's per-building pages (area insights) carry reviews and average asking price per
bedroom but none of Bayut's guide facts (year, floors, units); its transaction pages are Dubai only,
the same DLD data Bayut's ledger gives.

## The final data set and the owner's display rules (2026-10-09)

The importer for Property Finder new projects (`catalogue:sync propertyfinder-nl`, *Loading it* below) reads `propertyfinder_new_projects_final_v3.zip`, stored on wk at `storage/app/catalogue/propertyfinder-projects/final/` (earlier zips in `old/`), and enforces the three rules in code: only priced, sized unit types become layouts, every asset is held off the page until `catalogue:store-pf-media` has copied it, and no provenance or portal name reaches a displayed field (owner's go-ahead for the v3 reload, 2026-10-09 late evening):
every project matched to Bayut for its building GPS (`location_used`, see
[pf-bayut-matching.md](/docs/modules_handbook/shared/project-catalogue/pf-bayut-matching.md)), and every developer unit type
carrying a `display` block (`price_aed`, `size_sqft_from`/`_to`, `layout_photos`, `showable`) — gaps filled from the project's own recent
sales, then market listings, then matching floor-plan drawings. 7,729 of 10,627 unit types are
showable; 2,203 of 2,990 projects have at least one. The owner's rules for showing it on the portal:

1. **A unit type is shown only when it has a price AND a size** (`display.price_aed`,
   `display.size_sqft_from`). The data set's `showable` flag also demands a layout photo, and so did
   the first wording of this rule; the owner settled it the same night ("allow price, size without
   photo") — 1,622 unit types on 486 projects had both figures and no drawing anywhere, and they are
   listed and analysed, with an empty drawing slot. Without a price or a size, never shown.
2. **No source labels on screen** (owner, 2026-10-09: "no labels"): price and size are plain
   numbers, and the price is called "Price", never "Starting from" — about 1,900 shown prices are a
   recent sale or market listing, a median 5–7 % above the developer's starting price. The origin
   lives only in `_provenance`, which is never displayed; nor are `bayut_match` / `bayut_names`, and
   no other portal is ever named as a source.
3. **Every layout photo is copied to our storage (MediaService) before it is shown** — never link
   another site's image address.
4. A filled figure is a stand-in: recent-sale sizes fall inside the developer's range 76–84 % of the
   time and their prices run a median 7 % above the starting price; listings are looser (70 %, −24 %
   to +50 %). The zip's README carries the field list and counts.

**History.** Run #15 (2026-10-09 evening) loaded version 2 of the file before the rules arrived:
every priced unit type was a layout and the photos were linked from Property Finder's addresses.
The owner chose to reload all 2,990 from version 3 the same night (option 1; the interim retire was
not needed because Analyze Property was locked to admins on wk) — the §*Loading it* run below.

## Loading it — `catalogue:sync propertyfinder-nl --file=…jsonl.gz` (2026-10-09)

```bash
php artisan catalogue:sync propertyfinder-nl --file=/path/pf_new_projects_<stamp>.jsonl.gz
# on wk (live mode): the redirect wrapper instead — storage/app/catalogue/pf-nl-sync-20261009.php
```

Provider **`propertyfinder-nl`** (`DataProvider::CODE_PROPERTYFINDER_NL`, seeded by migration
`2026_10_09_180000`; separate from the older `propertyfinder` feed for the reason `edgeprop-nl` is
separate from `edgeprop` — source identity is (provider, external_id), and the UAE v2 rule keys off
this code). Registered in `config/project_catalogue.php` under `providers`, `launch_providers` and
`plan_derived_price_providers`; never `publish_on_ingest`.

**What it reads.** The FINAL hand-off (version 3) — `pf_new_projects_<stamp>.jsonl.gz` after
`catalogue:match-pf-bayut` / `judge-pf-bayut` ([pf-bayut-matching.md](/docs/modules_handbook/shared/project-catalogue/pf-bayut-matching.md))
and the unit gap fill, one project per line with `location_used`, `bayut_actual_location`,
`bayut_match` and a `display` block on every unit type. `PropertyFinderNewProjectsAdapter`
streams it line by line (`fgets`, never `gzfile()` — lines run to 170 KB), turns each line into the
artifact entry every crawl delivers (`PropertyFinderProjectNormalizer::catalogueRecord()`) and hands
it to the shared contract (`NewLaunchArtifactAdapter::normalizedRecord`: canonical whitelist,
source-only fields, https-only media, exact name + 5-decimal-coordinate attachment). A
`<file>_summary.json` beside it supplies the watermark (`built_at`) and, with `--full` and
`failed_pages` = 0, the full-snapshot flag; without the sidecar a run is partial. The inbox is
`storage/app/catalogue/inbox/propertyfinder-nl/` (`*.jsonl`, `*.jsonl.gz`).

**Field map** (`external_id` = Property Finder's project uuid, `pf_id`):

| Catalogue | From the line |
|---|---|
| `project_name`, `developer`, `description` (HTML → text, block breaks kept) | `title`, `developer`, `description` |
| `latitude` / `longitude` | **`location_used`** — the matched Bayut building's pin (2,228), else Property Finder's own (756); 6 projects have none and get no coordinates |
| `state` / `area` / `address` | `location_tree[0]` (emirate) / `[1]` (community) / the tree, most specific first |
| `property_type` | first of `property_types`: apartment, penthouse, duplex → Condominium/Apartment; villa → Detached House; townhouse → Town House; land → Land |
| `tenure` | `ownership` freehold / leasehold |
| `sale_status` | `construction_phase` completed → Completed, else New Launch (985 / 2,005 on 2026-10-09) |
| `completion_date` / `completion_year` | `delivery_date` |
| `price_min` | `starting_price_aed`; the merge then derives the range from the priced floor plans (`plan_derived_price_providers`) and falls back to this when none is priced |
| `facilities` | `amenities` |
| `market_segments` | `["new_project"]` always — the tab counts this segment |
| Floor plans — one per `unit_types` row with a **price and a size** (rule 1 as settled; the `showable` flag's photo condition is not applied) | name `{Studio|N Bedroom} {Property type}` (+ ` · {building}` when the project has several; ordinal on a repeat); `sqft` = `display.size_sqft_from`; `price` = `display.price_aed` (the developer's figure, else a recent sale, else a listing — the origin stays in `raw_payload.unit_provenance`, never in a field); `bathrooms_from`; `layout_description` "AED … · size range · N units · layout types" (plain figures, no "From", no portal named — rule 2); `source_key` `unit|{name}|{sqft}|{n}`; the size range, units and layout types as `source_fields`. Version 3 under the settled rule: 9,351 plans on 2,689 projects (7,729 with a drawing; v2 had made 10,705 plans, 2,331 of them sizeless) |
| Media — every entry **held** (`hold: true`) | `images[0]` hero, the rest gallery, `master_plan_image` gallery (label *Master plan*), `brochure_url` document (*Brochure*), every showable type's `display.layout_photos` a `floor_plan` row attached to its plan (deduplicated by URL: 32,873). A held row is ingested **inactive** (`CatalogueIngestionService::syncRecordMedia`) and the page never renders it until **`catalogue:store-pf-media`** has copied the file onto our disk and activated the row (rule 3); a held row the next file no longer lists is DELETED rather than deactivated, because an inactive never-stored row is what "held" looks like and the store command would otherwise copy and show it (1,522 such rows from the v2 load were pruned by hand after Run #17, `storage/app/catalogue/pf-prune-stale-media-20261010.php`) — heroes first, then floor plans, gallery, documents; resumable; on wk through `storage/app/catalogue/pf-store-media-20261009.php`. Version 3: 57,029 entries (2,984 hero, 18,748 gallery, 2,424 brochures, 32,873 drawings) |
| Source-only fields (`PropertyFinderProjectNormalizer::SOURCE_FIELDS`) | `pf_id`, `pf_developer_id`, `starting_price_aed`, `min_resale_price_aed`, `stock_availability`, `sales_phase`, `sales_start_date`, `construction_phase`, `construction_progress`, `property_types`, `down_payment_pct`, both pins (`pf_latitude`/`pf_longitude`, `bayut_actual_location`) with `location_type` / `location_source` / `location_precision`, `bayut_names`, `bayut_match_status`, `bayut_ids`, `payment_plans`, `unit_types` (every type, its `display` with a photo COUNT in place of the addresses, never `_provenance`), **`unit_bedrooms`** (the card's bedroom mix — the SHOWN types only), `showable_unit_types`, `timeline`, `brochure_url`, `master_plan_image` |
| `raw_payload` | `bayut_match` minus its candidate list and AI transcripts (those stay in the matching files) and `unit_provenance` (each unit type's `_provenance`, for audit — raw_payload is never rendered); the page payload is dropped |

**Attachment.** Exact name + 5-decimal coordinates against existing rows of the country, else a new
row. A Property Finder project whose match pinned it on a Bayut tower that survived the retirement
(the 311 kept as subsale buildings) can attach to that tower's row when the names agree — the row then
carries both the building's evidence and the developer's unit types, which is the right identity.

**A re-sync** is additive by provider + `pf_id`: existing projects are updated in place (a source's
`fields` are REPLACED, so a later file must carry everything the earlier one did), new ones created,
nothing deactivated without `--full` + a clean sidecar. Two hand-offs for the same keys: merge them
first (runbook rule 6).

**On wk** (live mode, master read-only) the sync runs through the process-local redirect wrapper
`storage/app/catalogue/pf-nl-sync-20261009.php` (`PF_FILE=…`): it refuses unless the catalogue
connection resolves to `petav3wk`, checks the provider row and the cached config, loads, clears the
own-AE saved analyses, flushes the `np-listing:*` cache and prints the counts. **Reload from
version 3 (same night, in place by `pf_id`): Run #17 [success] propertyfinder-nl/AE: 0 created, 2,990 updated, 0 failed (34 min, 23:48–00:22 MYT; Run #16, the same file under the stricter first wording of rule 1, was stopped mid-run and marked aborted) → 9,351 layouts on 2,689 projects, every one priced AND sized, 7,729 of them with a drawing; 57,029 media rows held inactive until `catalogue:store-pf-media` copied them — done by 05:00 MYT: 55,532 on our storage and showing, 87 held for good (78 removed assets answering 403/404, 9 unsupported formats), the 1,410 gallery videos deleted (a video is not a photo; dropped at ingestion since), 606 brochures and 245 Bayut-hosted borrowed drawings copied once their hosts were allowed; the UAE v2 tab reads 2,990; 08 Life Residences (sizes filled from recent sales) now analyses as FAIR PRICE.** First load (version 2) earlier that
evening: Run #15 [success] propertyfinder-nl/AE: 2,990 created, 0 updated, 0 failed (32 min, 20:57–21:29 MYT) → 15,085 own AE rows (2,990 Property Finder projects, 152 of them attached by exact name + coordinates to a Bayut tower kept as a subsale building), 2,984 with coordinates, 10,705 floor plans (7,198 priced, 6,147 priced AND sized; 2,490 projects with a priced plan), price_min on 2,862, 58,306 media rows (2,984 hero, 18,748 gallery, 2,424 brochures, 34,150 drawings) on Property Finder's CDN; the UAE v2 tab reads 2,990, the UAE tab still 951.

## Related files

- `app/Console/Commands/ScrapePropertyFinderProjects.php`
- `app/Services/Property/Scraping/PropertyFinderProjectNormalizer.php` (`project`, `unitTypes`, `catalogueRecord`)
- `app/Services/Property/Ingestion/Adapters/PropertyFinderNewProjectsAdapter.php`, `database/migrations/2026_10_09_180000_seed_propertyfinder_nl_provider.php`
- `app/Console/Commands/StorePropertyFinderNewProjectMedia.php` (`catalogue:store-pf-media`), `CatalogueMediaStorageService::storePropertyFinderNewProject`, `PropertyFinderAssetFetcher` (images + brochure PDFs); the `hold` flag in `CatalogueIngestionService::syncRecordMedia`
- Tests: `tests/Unit/Scraping/PropertyFinderProjectNormalizerTest.php`, `tests/Feature/Console/SyncPropertyFinderNewProjectsTest.php`, `tests/Feature/Console/StorePropertyFinderNewProjectMediaTest.php`
- The page these projects render on: [uae-v2.md](/docs/modules_handbook/shared/project-catalogue/uae-v2.md)
- The OLDER Property Finder feed the current UAE page uses is a different thing — provider
  `propertyfinder`, `PropertyFinderUaeAdapter`, read from the `reference` connection.
