# Bayut — UAE new projects and ready buildings

> 📍 Part of the project catalogue doc set — the map and routing table is
> [start-here.md](/docs/modules_handbook/shared/project-catalogue/start-here.md).


> **Bayut is the SUBSALE reference only (owner, 2026-10-09).** It supplies ready buildings, building
> guides, DLD sales, Ejari rents and building-accurate GPS; **UAE new projects come from Property
> Finder** ([propertyfinder-new-projects.md](/docs/modules_handbook/shared/project-catalogue/propertyfinder-new-projects.md),
> matched to Bayut by [pf-bayut-matching.md](/docs/modules_handbook/shared/project-catalogue/pf-bayut-matching.md)).
> Enforced in code: `BayutAdapter` skips `project:*` records on load while
> `project_catalogue.providers.bayut.new_projects` is `false` (the default), and
> `catalogue:retire-bayut-new-projects` (dry-run; `--apply`) retires ones already loaded — their
> sources deactivate, a project left with no other source is deleted when safely orphaned, one that
> is also a Bayut building keeps that building source. The UAE v2 tab lists Property Finder's new
> projects (`uaeV2Scope`: an active `propertyfinder-nl` source and no older `propertyfinder` one —
> [propertyfinder-new-projects.md](/docs/modules_handbook/shared/project-catalogue/propertyfinder-new-projects.md)),
> never Bayut rows. The crawler still collects `project:*` records — they are the matching input.
## What it does

Provider **`bayut`** (2026-10-08) brings Bayut's UAE inventory into the catalogue, every emirate,
with its photographs and brochures stored on **our own** media disk:

| Family | External id | Segment | Comes from | Size (2026-10-08) |
|---|---|---|---|---|
| New projects | `project:{bayut id}` | `new_project` (under construction) or `subsale` (completed) | Bayut's public project search index | 2,998 projects — 2,536 under construction, 462 completed |
| Ready buildings | `building:{bayut location id}` | `subsale` | Bayut's building guides, `/buildings/ready/{emirate}/` | ~3,170 guides (Dubai 2,489, Sharjah 466, Abu Dhabi 148, Ajman 56, RAK 14, Fujairah 1) |

A project record carries developer, launch price, handover date, payment-plan split, unit mix,
amenities, GPS, the DLD project number, every photo and the developer's brochure PDF. A building
record carries the guide's "In a Nutshell" facts (developer, completion year, floors, units,
ownership, amenities), its description, exact GPS and the guide's photographs.

**No ScraperAPI.** Both sources answer plain requests. The project index is the same Algolia index
Bayut's own `/new-projects/` page queries from the browser, with the search-only key that page
publishes — re-read from the page on every run, so a rotated key does not fail silently.

## How it works

Three steps, the standard catalogue path for a crawled provider — nothing writes the canonical
directly and nothing publishes itself:

```bash
php artisan catalogue:scrape-bayut                      # → storage/app/catalogue/inbox/bayut/bayut_full_….json
php artisan catalogue:sync bayut --full                 # artifact → sources → merge (writes the master: run on the Hub)
php artisan catalogue:store-bayut-media --kind=hero     # then --kind=document, then --kind=gallery
```

1. **`catalogue:scrape-bayut`** (`ScrapeBayut`) — ~4 requests for all projects, then one request
   per guide-list page and per guide (~3,400 for every emirate; `--delay` ms between guides,
   default 200). `--scope=projects|buildings|all`, `--emirate=` to narrow, `--limit=` to cap.
   Then the stages below (*Mapping notes*), each skippable with `--skip=units|floor-plans|guides|geo-buildings|comparables`;
   `--community=` narrows projects and buildings to communities, `--radius=` (km, default 2),
   `--sales-pages=` / `--rent-pages=` (20 rows a page) cap the ledger per location. The
   artifact is **full** only for `--scope=all`, no `--limit`, `--community` or `--skip`, and zero failures — a full sync
   deactivates every Bayut source the artifact does not carry. A guide that answers **404** between
   the listing and its own fetch was taken down by Bayut and is skipped, not counted as a failure
   (five were, on 2026-10-08); any other failure makes the snapshot partial.
   Bayut's guide **list** has broken pages too — Dubai's pages 81, 87, 92, 98, 100, 103, 105–107 and
   134 answered 404 on 2026-10-09 while the pages after them worked. The crawl skips such a page and
   carries on (`guide_pages_broken` in the artifact's stats); until that fix it stopped at the first
   one and collected 1,885 of Bayut's 3,174 guides. The ~150 guides on broken pages stay unreachable;
   their buildings still arrive as GPS-only `building:` records.
2. **`BayutAdapter`** (a `NewLaunchArtifactAdapter`) reads it. Bayut facts that have no canonical
   column — payment-plan split, DLD project number, units listed, floors, nearby buildings — are
   whitelisted as **source-only** fields (`BayutNormalizer::SOURCE_FIELDS`), the way
   `PropertyFinderUaeAdapter` keeps its extras. Canonical attachment is the crawl rule: an existing
   `(bayut, external_id)` link, else exact name **and** five-decimal coordinates; never a name alone.
3. **`catalogue:store-bayut-media`** (`StoreBayutCatalogueMedia`) downloads each `catalog_media` row
   still pointing at a Bayut URL — hero, gallery, building photographs and **brochures (`document`
   kind, PDF)** — stores it through `MediaService`, writes the `MasterMedia` row on the owning row's
   connection and flips the row to it under a lock. Idempotent and resumable; same contract as
   `catalogue:store-uae-media`, now one shared `CatalogueMediaStorageService::storeAndLink()`.

### Mapping notes

- `state` is the emirate and `area` the community, as `PropertyFinderUaeAdapter` does.
- `sale_status` is taken verbatim: `Listed` (under construction, matching PropertyFinder) or
  `Completed`. Bayut is deliberately **not** in `launch_providers`, `plan_derived_price_providers` or
  `derived_sale_status_providers` — it mixes launches with ready buildings, and its headline
  `price_min` is the launch price as Bayut publishes it.
- Project photos are taken at `images.bayut.com/thumbnails/{id}-1066x800.jpeg` (1422 × 800), the
  largest preset the host answers; larger ones 403.
- A guide's coordinates are a PostGIS EWKB point in its payload (`BayutNormalizer::point()`).
- Community guides (e.g. Discovery Gardens' "Cactus" cluster) are listed among ready buildings and
  write free-form "In a Nutshell" bullets, so their facts come back empty — by design, not a bug.
- **Layouts** come from the project's off-plan listings (`BayutNormalizer::layouts()`): sqm × 10.7639,
  grouped per bedroom, sizes within 5 % form one layout, single-listing sizes dropped when the
  bedroom count has a real cluster. Price = lowest asking; the median and the listing count ride in
  `layout_description` / `source_fields`. Bayut's floor-plan drawings (`/api/floorPlans/`) attach as
  `floor_plan` media to the layout with the same bedroom count.
- **Comparables** (`BayutNormalizer::comparables()`): per location, 12 months of DLD sales and
  Ejari rent contracts from Bayut's transactions ledger, in the engine's listing-row shape
  (`price_rm` / `psf_rm` hold AED). Rent = contract amount ÷ contract months. Only projects and the
  buildings within `--radius` km of one are fetched; a building whose location a project in the
  same run already covers gets none (it is the same building — fetching both listed it twice in
  every comparable table).
- **Geo buildings** — the ~11,800 ready buildings that have GPS on Bayut's locations index but no
  guide become `building:{location id}` records (`locationBuilding()`), so nearby evidence exists
  beyond the guided ~3,170.
- What this feeds: [uae-v2.md](/docs/modules_handbook/shared/project-catalogue/uae-v2.md).

### Asking prices and rents — every listing (`catalogue:scrape-bayut-listings`)

A separate command, because it answers a different question. The transactions ledger above is what
units **sold and rented for** — registered with the Dubai Land Department, so **Dubai only** (Bayut's
ledger returns nothing for Abu Dhabi, Sharjah or any other emirate). Listings are what sellers and
landlords are **asking**, and they exist in **every emirate**: on 2026-10-09, 192,514 for sale (ready
and off-plan) and 363,824 for rent.

```bash
php -d memory_limit=4G artisan catalogue:scrape-bayut-listings              # both purposes, ~15–30 min
php artisan catalogue:scrape-bayut-listings --purpose=for-rent --limit=3    # quick trial
```

- Bayut's search index returns at most 1,000 hits a query, so each purpose is split into price
  ranges, halved until each holds ≤ 1,000; a single crowded price (AED 1,000,000; AED 50,000 a year)
  is split further by emirate, bedrooms, then furnishing (`ScrapeBayutListings::SPLITTERS`). A slice
  still over the cap keeps its first 1,000 and is counted as `capped_ranges` in the summary.
- Output: `storage/app/catalogue/bayut-listings/bayut_listings_{sale,rent}_<stamp>.jsonl.gz` — JSON
  Lines, one listing per line (`--format=csv` for spreadsheets) — plus `_summary.json` (Bayut's
  total vs rows written per purpose). Bayut's total is an **estimate** for large queries (its index
  answers `exhaustiveNbHits: false`), so `written` can exceed it; where Bayut's count is exact the
  two match (2026-10-09: Abu Dhabi rents 9,007 = 9,007, Dubai Marina 6,466 vs 6,452). One record per listing
  (`BayutNormalizer::listing()`): asking price, rent frequency and `annual_rent_aed`, sqft, beds,
  baths, type, ready / off-plan, furnishing, emirate → community → sub-community → building, the
  leaf `location_id` (the same id a `building:{id}` catalogue record carries), GPS, agency, dates, URL.
- **It writes no catalogue rows.** It is a data set; feeding it to the engine (asking beside
  transacted on the Market Value tab, asking rent outside Dubai) is a separate change.
- No ScraperAPI: same search-only key as the project crawl.

### What to know before running it

- **Volume.** ~18,300 project media rows (2,936 heroes, 13,011 gallery, 2,320 brochures of a few
  MB each) plus a few photographs per guide — tens of GB. Batch with `--kind` and `--limit`.
- **The asset allowlist is exact.** `BayutAssetFetcher` accepts HTTPS to `images.bayut.com`,
  Bayut's own CloudFront host `d3ob0s3rxbjyep.cloudfront.net` and its brochure bucket
  `bayut-production.s3.eu-central-1.amazonaws.com` — never `*.cloudfront.net`, which would admit
  anyone's. Images and PDFs only, streamed to disk, capped at 60 MB.
- **Terms of use.** Bayut's terms restrict automated collection. Whether to run this at all is a
  business decision, not a technical one.

## Related files

- `app/Console/Commands/ScrapeBayut.php`, `app/Console/Commands/ScrapeBayutListings.php`, `app/Console/Commands/StoreBayutCatalogueMedia.php`
- `app/Services/Property/Scraping/BayutNormalizer.php`
- `app/Services/Property/Ingestion/Adapters/BayutAdapter.php`
- `app/Services/Property/BayutAssetFetcher.php`, `app/Services/Property/CatalogueMediaStorageService.php`
- `config/project_catalogue.php` (`providers.bayut`, `staleness_threshold_hours.bayut`)
- `database/migrations/catalogue/2026_10_08_120000_seed_bayut_provider.php` — run on the master
  first (`--database=catalogue`), then locally, as for every provider seed
- Tests: `tests/Unit/Scraping/BayutNormalizerTest.php`, `tests/Feature/Console/ScrapeBayutCommandTest.php`,
  `tests/Feature/Console/StoreBayutCatalogueMediaTest.php`
