# VR Bake Worker (production setup)

The process that turns a queued [VR360](/docs/modules_handbook/shared/project-catalogue/vr360/readMe.md) row into a panorama. It runs on a **dedicated box**, not the web server.

## Why a separate machine

One bake is:

- **minutes to hours** of fully pegged CPU (rendering is software WebGL via SwiftShader — there is no GPU path in headless Chromium here);
- **1–3 GB resident**, depending on `VR_BAKE_FACE_SIZE`; and
- **metered Google spend** — it streams a large number of 3D tiles.

**Measured on the web box** (2 vCPU, 4 GB, no GPU, `cache_bytes` 256 MB, KLCC at 200 m):

| Face size | Output | Per face | Whole bake |
| --- | --- | --- | --- |
| 512 | 2048×1024 | ~75–110 s | **~11 min** |
| 1024 | 4096×2048 | — | ~40 min (4× the tiles) |
| 2048 | 8192×4096 | — | 2 h+ |

Those are the numbers that justify a separate machine: a single 1024 bake would
hold two vCPUs flat for most of an hour on the box that also serves the site.
A GPU-capable host collapses this — SwiftShader is the dominant cost, not the
network.

The web box (`petav3-dev`, 2 vCPU / 4 GB) has previously been OOM-killed into a 522 and a hard reboot by a *Vite build*, which is lighter than this. Hence: the `vr-bake` queue has **no Horizon supervisor**, so the web box physically cannot pick a bake up. A machine only bakes if you deliberately start the worker below on it.

## What the box needs

| Requirement | Notes |
| --- | --- |
| The app checkout | Same repo/branch as production — the worker calls `base_path('workers/vrbake/…')` and reads the same config. |
| PHP 8.4 CLI + composer deps | It runs `artisan queue:work`; no web server needed. |
| **Node.js 20+** | `VR_BAKE_NODE` must point at it. |
| **Chromium** | Prefer a **standalone** build over the distro/snap package — snap confinement blocks paths outside `$HOME`, and the worker writes scratch under `storage/`. What worked here: `cd workers/vrbake && npx @puppeteer/browsers install chrome@stable` (~290 MB, gitignored), then point `VR_BAKE_CHROMIUM` at the printed path. |
| Chrome's system libraries | The standalone build ships no dependencies. On Ubuntu 24.04: `libatk1.0-0t64 libatk-bridge2.0-0t64 libcups2t64 libxkbcommon0 libasound2t64 libgbm1 libpango-1.0-0 libxcomposite1 libxdamage1 libxfixes3 libxrandr2 libnss3 libnspr4 libdrm2 libcairo2`. Verify with `ldd <chrome> \| grep "not found"` — it must print nothing. |
| `poppler-utils` | `pdftoppm`, for brochure rendering, **if** you also run the `ai` lane here. |
| Worker npm deps | `cd workers/vrbake && npm install` (installs `puppeteer-core` + `sharp`, ~300 MB, gitignored). |
| Shared **Redis** | Same instance as the web box — that is how the job arrives. |
| Shared **GCS** credentials | Finished panoramas are written to the same private bucket the web box reads. |
| **RAM** | ≥ 8 GB recommended. 4 GB only survives `VR_BAKE_FACE_SIZE=512`–`1024` with `VR_BAKE_CACHE_BYTES` reduced to match. |

Disk is scratch only — each bake writes a few files under `storage/app/vr-bake/{uuid}` and deletes them when it finishes.

## Google Cloud key

`GOOGLE_MAPS_API_KEY` must have, on a project **with billing enabled**:

- **Map Tiles API** (this is the 3D Tiles one — the usual thing people forget),
- Maps JavaScript API,
- Elevation API.

If the key is restricted by HTTP referrer, `VR_BAKE_REFERER` must match one of its allowed referrers exactly, or Google returns 401 for the tiles and the panorama comes out with **black patches** — which reads as a rendering bug, not an auth failure.

There is deliberately **no fallback key in config**. A blank key fails the job at dispatch with a clear message instead of silently baking six black faces.

## Running it

```ini
# /etc/systemd/system/vr-bake-worker.service
[Unit]
Description=VR360 panorama bake worker
After=network.target

[Service]
Type=simple
User=ubuntu
WorkingDirectory=/var/www/html/peta
# --max-time recycles the process periodically so a leaked Chromium cannot
# accumulate. --tries=1 because a bake BILLS: never auto-retry one.
ExecStart=/usr/bin/php artisan queue:work redis-vr-bake --queue=vr-bake \
          --sleep=5 --tries=1 --timeout=2100 --max-time=86400
Restart=always
RestartSec=10
# One at a time. Two concurrent bakes will OOM anything short of a very large box.
[Install]
WantedBy=multi-user.target
```

```bash
sudo systemctl enable --now vr-bake-worker
sudo journalctl -u vr-bake-worker -f
```

**Concurrency is 1 by design** — do not add a second unit or `--max-jobs` parallelism without sizing the RAM for it.

### The timeout chain — one tunable end

Three ceilings must stay ordered, and only the innermost one is meant to be tuned:

| Ceiling | Where | Value |
| --- | --- | --- |
| Node process | `vr_bake.timeout` ← **`VR_BAKE_TIMEOUT`** | per box (1800 default, ~5400 for face 2048) |
| Job | `BakeProjectVr::$timeout` | **derived**: process + 300 s, capped 5700 |
| Lane re-delivery | `redis-vr-bake.retry_after` | 6000 |

`--timeout` on the `ExecStart` above must match the **job** ceiling, so set it from
`php artisan tinker --execute="echo (new App\Jobs\Property\BakeProjectVr(1))->timeout;"`
rather than hardcoding it.

Only `VR_BAKE_TIMEOUT` should ever be edited; the other two follow. That is deliberate —
the job ceiling used to be a hardcoded 2100 while the process ceiling was env-tunable, so
raising the latter for a face-2048 bake **inverted** the order silently. The symptom is
not an obvious error: the supervisor kills a *healthy* bake at the job ceiling while the
worker still believes it is running, and the row sits in `processing` having already paid
Google for every tile it fetched.

**Raise `VR_BAKE_STUCK_AFTER` alongside it.** `vr_bake.stuck_after_minutes` must exceed
the process ceiling in minutes, or the sweep fails a healthy long bake under itself — at
the default 45 against a 90-minute face-2048 bake, it always would. *(As of 2026-08-08
nothing actually reads `stuck_after_minutes`: the sweep it documents is not implemented,
so a bake whose process is hard-killed stays in `processing` indefinitely with no
recovery path but a manual status reset.)*

## Verify the geometry first (no Chromium, no Google, no cost)

Before spending a real bake, prove the three projection files still agree:

```bash
cd workers/vrbake && npm install
node verify.mjs                    # fast — synthetic faces at 256px
node verify.mjs --face-size 1024   # once per box, at production resolution
```

18 checks: each cube face lands where the projection says, the nadir round-trips
back to `rotate180(down)`, and a composited floor plan appears at the nadir
without touching any other face. Exit 0 means the maths is sound and anything
that then goes wrong is Chromium, the key, or the tiles — which is a much
smaller search space than "the panorama looks a bit off".

Run it after ANY edit to `stitch.mjs`, `extract_face.mjs` or `overlay.mjs`.

## Verifying it end to end

1. On the web box, queue a bake from a project's **VR360** tab (a Klang Valley project has the best tile coverage).
2. Watch `journalctl -u vr-bake-worker -f` — you should see the six `Capturing face n/6` progress lines.
3. The admin tab polls every 4 s; progress should climb through `init → loading → capture → stitch → upload → done`.
4. On completion the row shows a thumbnail. Open it: the horizon should be sharp all the way round, and the ground beneath the camera as sharp as the horizon.

A one-off bake without the queue, for debugging:

```bash
php artisan tinker --execute="App\Jobs\Property\BakeProjectVr::dispatchSync(<bakeId>);"
```

Or drive the Node worker directly, which isolates whether a problem is Laravel's or Chromium's:

```bash
node workers/vrbake/bake.mjs \
  --lat 3.0584 --lng 101.6041 --height 120 --face-size 1024 \
  --output-dir /tmp/vrtest --api-key "$GOOGLE_MAPS_API_KEY" \
  --chromium /usr/bin/chromium --referer https://wk.propertylab.com.my/
```

## How to tell it is down

- Bakes sit at **Queued** and never reach Baking → the worker is not running, or is pointed at a different Redis.
- Bakes sit at **Baking** past `vr_bake.stuck_after_minutes` (45) → the process died (OOM, reboot). `BakeProjectVr::failed()` catches queue-level failures; a hard kill needs the sweep.
- `journalctl` shows `Failed to launch the browser process` → `VR_BAKE_CHROMIUM` is wrong, or it is a snap build refusing to write scratch.

## Tuning

| Env | Default | Effect |
| --- | --- | --- |
| `VR_BAKE_FACE_SIZE` | 1024 | Output is `×4` by `×2` (1024 → 4096×2048). The single biggest lever on both time and memory; 2048 captures ~4× the tiles. |
| `VR_BAKE_CACHE_BYTES` | 1 GiB | Cesium's tile budget. **Size against the host, not the image.** Too small → blurry centre; too large → OOM. |
| `VR_BAKE_TILE_WAIT` | 150000 | Per-face ceiling on waiting for tiles to settle. Raise if a side comes out blurry. |
| `VR_BAKE_TIMEOUT` | 1800 | Whole-bake ceiling. Face 2048 needs ~5400. Raise `VR_BAKE_STUCK_AFTER` with it; the other two ceilings follow automatically. |
| `VR_BAKE_STUCK_AFTER` | 45 | Minutes before a non-advancing bake is failed. Must exceed `VR_BAKE_TIMEOUT` in minutes. |
| `VR_BAKE_CHROMIUM` | `/usr/bin/chromium` | **The default rarely exists** — the prescribed install is the standalone `@puppeteer/browsers` build under `workers/vrbake/chrome/…`. Leaving it unset fails the bake at browser launch, only when one is first dispatched from the UI. |
| `VR_BAKE_REFERER` | `APP_URL` | Sent on every tile request. |