# Incident 2026-10-07 — the repo, `.env` and `storage/` were web-readable

**Status:** open until every step below is ticked. **Severity:** critical: secrets plus customer data.

## What was exposed

Found by the external SEO audit (2026-10-07) and confirmed with status-code-only requests the
same day (no file contents were opened). Production serves the app from `/var/www/html/peta`,
which sits **inside Apache's stock webroot**. A request for a host that no vhost claims
(`www.propertylabglobal.com` on port 80, or the bare IP `34.2.146.148`) was answered by the
default vhost, which serves `/var/www/html` with directory listing on.

| URL (any unclaimed host) | Result |
|---|---|
| `http://www.propertylabglobal.com/` | `Index of /` listing `peta/` |
| `/peta/` | listing of the whole repo |
| `/peta/.env` | **200, 14,103 bytes**: every production secret |
| `/peta/.git/HEAD` | **200**: the whole git history is downloadable, so is any secret ever committed |
| `/peta/storage/logs/` | listing: `laravel-*.log`, `wa-bridge-*.log` (WhatsApp messages), `zoom-rtms-*.log` (meeting transcripts), `deploy.log` |
| `/peta/storage/app/` | listing: `final_leads.csv`, `sg_and_180min_leads.csv`, `sg_phone_fix_backup_*.csv`, `sql/`, `google/` (service-account keys?), `hk-import/`, `yhy/` … |
| `/peta/storage/app/deploy/backups/` | empty at the time of the check (the deploy keeps 3 DB dumps here, so they may have been listed earlier) |

Dotfiles (`.env`, `.git`) are hidden from Apache's listing by default, so they were not *listed*.
They were **fetchable by anyone who guessed the path**, and `/<dir>/.env` is a path every
scanner tries.

## 1 · Stop it (minutes): on the server

```bash
sudo tee /etc/apache2/conf-available/block-webroot.conf >/dev/null <<'EOF'
# Only the Laravel public/ dir is web-served. Everything else under the webroot
# (the repo, .env, .git, storage/) must never be reachable on any vhost.
<Directory /var/www/html>
    Options -Indexes
    Require all denied
</Directory>
<Directory /var/www/html/peta/public>
    Require all granted
</Directory>
# Let's Encrypt renewal challenges, in case the certificate renews through the webroot.
<Directory /var/www/html/.well-known/acme-challenge>
    Require all granted
</Directory>
EOF
sudo a2enconf block-webroot && sudo apachectl configtest && sudo systemctl reload apache2
sudo certbot renew --dry-run   # renewal must still succeed (the cert expires 2026-12-01)
```

First check the app really lives at `/var/www/html/peta` (`ls /var/www/html`) and look at
`sudo apache2ctl -S`. Undo at any time: `sudo a2disconf block-webroot && sudo systemctl reload apache2`.

Verify. Every line must be 403 (or 301/404), **never 200**:

```bash
for h in http://www.propertylabglobal.com http://34.2.146.148; do
  for p in "" peta/ peta/.env peta/.git/HEAD peta/storage/logs/laravel.log peta/storage/app/final_leads.csv; do
    curl -s -o /dev/null -w "%{http_code} $h/$p\n" "$h/$p"; done; done
curl -s -o /dev/null -w "%{http_code} app still up\n" https://propertylabglobal.com/
```

`scripts/server-setup.sh` now writes this same file, and `scripts/service-check.sh` (run at the
end of every deploy) probes `/<app>/.env` on an unclaimed host and **fails the deploy** if it
ever returns 200 again. `public/.htaccess` also refuses every dotfile except `.well-known/`.

**Every box, not just prod.** This step was first applied by hand on prod only. The wk server
(35.213.165.89) runs the same layout and was still serving `/peta/.env` and `/peta/.git/` on
2026-10-14; the same commands closed it. So that no box depends on someone remembering this,
`scripts/deploy-update.sh` now re-asserts the guard on **every deploy**: when
`block-webroot.conf` is not enabled it writes the file above, enables it, and reloads Apache
only if `apachectl configtest` passes. It never fails the deploy and leaves an existing file
untouched. It also skips any box that serves another site from inside the webroot, and warns
instead, because denying the parent would take that site down.

Optional, recommended: `sudo a2dissite 000-default && sudo systemctl reload apache2`. Then the
app's own vhost answers unclaimed hosts instead of the stock one.

## 1b · Give `www.` a real home (the door this came through)

`www.propertylabglobal.com` resolves to this server but no vhost claims it: port 80 fell to the
stock site, and on 443 the certificate has no `www` name. Make it a pure redirect to the primary
domain, then add it to the certificate.

Production renews with certbot's **Apache plugin** (`authenticator = apache` in
`/etc/letsencrypt/renewal/*.conf`), so certbot can build the https site itself:

```bash
sudo tee /etc/apache2/sites-available/www-redirect.conf >/dev/null <<'EOF'
<VirtualHost *:80>
    ServerName www.propertylabglobal.com
    RewriteEngine On
    RewriteCond %{REQUEST_URI} !^/\.well-known/acme-challenge/
    RewriteRule ^ https://propertylabglobal.com%{REQUEST_URI} [R=301,L]
</VirtualHost>
EOF
sudo a2ensite www-redirect && sudo apache2ctl configtest && sudo systemctl reload apache2
# A certificate for www, and an https copy of the redirect site above (--no-redirect: the site
# already redirects, certbot must not add its own http→https hop to www).
sudo certbot --apache --no-redirect -d www.propertylabglobal.com
# certbot COMMENTS OUT the RewriteRule in its https copy ("potential redirection loops"), which
# leaves https://www on the blocked webroot (403). Add a plain mod_alias redirect, which it leaves be:
sudo sed -i 's#^\(\s*ServerName www\.propertylabglobal\.com\)\s*$#\1\n    Redirect permanent / https://propertylabglobal.com/#' \
  /etc/apache2/sites-available/www-redirect-le-ssl.conf
sudo apache2ctl configtest && sudo systemctl reload apache2
curl -s -o /dev/null -w '%{http_code} %{redirect_url}\n' 'http://www.propertylabglobal.com/my?x=1'    # 301 -> https://propertylabglobal.com/my?x=1
curl -s -o /dev/null -w '%{http_code} %{redirect_url}\n' 'https://www.propertylabglobal.com/my?x=1'   # same, no TLS error
```

Found while doing this (2026-10-07, `apache2ctl -S`): the port-80 default server is
`000-default.conf`, which is what served the listing; the app is `peta.conf` +
`peta-le-ssl.conf`. `peta-le-ssl.conf` also carries two port-80 vhosts that shadow `peta.conf`,
so `peta.conf` never matches. Harmless, tidy later.

## 2 · Find out who read what: access logs

The default vhost logs to `/var/log/apache2/access.log*`. Ubuntu keeps about 14 days.
`zgrep` reads the rotated `.gz` files too.

```bash
cd /var/log/apache2
# who touched /peta/ at all, most active first
sudo zgrep -h '"GET /peta/' access.log* | awk '{print $1}' | sort | uniq -c | sort -rn | head -50
# every SUCCESSFUL fetch of something sensitive (status is field 9)
sudo zgrep -hE '"GET /peta/(\.env|\.git/|storage/|database/|[^ ]*\.(log|sql|csv|gz|json|html))' access.log* \
  | awk '$9 == 200' | awk '{print $1, $4, $7, $10, $12, $13, $14}'
# the oldest surviving log, so you know how far back this evidence reaches
ls -ltr access.log* | head -3
```

Two known readers on 2026-10-07 to exclude: the SEO audit crawl (about 01:05–01:30 MYT) and the
status-code checks run during the fix (about 13:00–13:40 MYT, user-agent `curl/*`). Any other IP
with a 200 on `.env`, `.git/` or a `.csv` means **assume it was taken**. Because the logs only reach
back two weeks, an empty result is not proof that nothing was read.

### What the logs showed (reviewed 2026-10-07; they reach back to 2026-09-23 only)

- **`.env` was downloaded by two outsiders**: `23.234.116.189` (a secrets scanner: it tried
  `.env.backup/.dev/.development/.local/.old/.production/.uat`, then took `.env`, `.git/HEAD`,
  `.git/config`, `.git/index` and `.git/logs/HEAD`) and `213.136.67.156`. Rotation is
  **required**, not precautionary.
- **`storage/oauth-private.key`** (Passport) was downloaded by 14 IPs. With it anyone can sign
  valid access tokens for the MCP / connector API as any user.
- **`storage/logs/`**: the `laravel-*` and `wa-bridge-*` logs (WhatsApp conversations) were each
  downloaded by about 17 IPs. **Personal data left the building; see section 4.**
- **`catalogue-packages/`** (the 283 MB catalogue export, 17,541 projects) was downloaded by
  13–15 IPs. Business data; its field names carry no personal data.
- Repeat scrapers: seven DigitalOcean IPs with identical behaviour (one actor, daily), a
  `Go-http-client` script, `WebSiphon` (a site ripper), `opendir-crawler` (indexes open
  directories) and ClaudeBot (`216.73.216.140`, 96,607 requests; removal requests go to
  claudebot@anthropic.com).
- Known readers to ignore: `205.147.27.22` (the status checks made during the fix) and
  `34.2.146.148` (the server itself).

## 3 · Rotate: assume every value in `.env` is known

Rotate at the provider first, then put the new value in production `.env`, then
`php artisan config:clear && php artisan config:cache`, then restart php-fpm, Horizon, the
scheduler, SSR and the bridge. Never paste values into chat or commits.

**Order by damage** (do these first):

0. **Passport keys** (`storage/oauth-private.key` was downloaded): on the server,
   `php artisan passport:keys --force`, then restart php-fpm and Horizon. Every issued token
   dies; connected apps sign in again. **And the server's git remote**: `.git/config` was taken.
   Run `git -C /var/www/html/peta remote -v`. If the URL carries a token (`https://…@github.com`),
   revoke that token on GitHub and switch the remote to a deploy key.
1. **Databases**: `DB_PASSWORD`, `MASTER_DB_PASSWORD`, `MASTER_DEV_DB_PASSWORD`,
   `PETAV3_READ_DB_PASSWORD`, `OWNER_DB_PASSWORD`, `OWNER_ADMIN_DB_PASSWORD`,
   `PETAV2_DB_PASSWORD`, `REFERENCE_DB_PASSWORD`. Also check each Cloud SQL instance's
   **authorized networks**: a leaked password on an instance open to `0.0.0.0/0` is a direct
   path to every lead.
2. **Money**: `BILLPLZ_KEY_PRODUCTION`, `REVPAY_MERCHANT_KEY`, `LM_KEY_PRODUCTION` /
   `LM_SECRET_PRODUCTION`, `ZOLOZ_*_PRODUCTION`, `SHOPIFY_ACCESS_TOKEN`.
3. **Accounts that can message customers as us**: `WHATSAPP_ACCESS_TOKEN`,
   `WHATSAPP_APP_SECRET`, `WHATSAPP_VERIFY_TOKEN`, `META_APP_SECRET`,
   `META_WEBHOOK_VERIFY_TOKEN`, `LINE_CHANNEL_SECRET`, `LINE_CHANNEL_ACCESS_TOKEN`,
   `SMS_API_USER` / `SMS_API_PASS` / `SMS_KEY`, `MAIL_PASSWORD`, `MAIL_PASSWORD_BACKUP`,
   `TELEGRAM_BOT_TOKEN` (BotFather → `/revoke`), `BRIDGE_API_KEY`, `BRIDGE_WEBHOOK_TOKEN`.
   Webhook secrets need the new value registered on the provider side too.
4. **Google Cloud service accounts**: the key files behind `GOOGLE_CLOUD_KEY_FILE` and
   `AREA_GUIDE_GCS_KEY_FILE` (and whatever sits in `storage/app/google/`). In IAM, create a
   new key, deploy it, **delete the old key**. Also `GOOGLE_MAPS_API_KEY`,
   `GOOGLE_PLACES_KEY`, `GEMINI_API_KEY` (restrict by referrer/IP while there).
5. **Zoom**: `ZOOM_CLIENT_SECRET`, `ZOOM_WEBHOOK_SECRET_TOKEN`, `ZOOM_RTMS_CLIENT_SECRET`,
   `ZOOM_RTMS_WEBHOOK_SECRET_TOKEN`, `ZOOM_RTMS_RELAY_TOKEN`, `ZOOM_BOT_TOKEN`,
   `ZOOM_SDK_SECRET_COMPANY`.
6. **Everything else with a secret in its name**: `JWT_SECRET` (logs the agent app out),
   `PUSHER_APP_SECRET`, `REVERB_APP_SECRET`, `REDIS_PASSWORD`, `DEEPGRAM_API_KEY`,
   `SERPER_KEY`, `SCRAPERAPI_KEY`, `PROPERTYLAB_PROPSENSE_API_KEY`, `INSPECTOR_API_KEY`,
   `YHY_APP_SECRET`, `EP_KEY`, `OPS_ALERT_WEBHOOK_URL`, `SITE_KEY`, and `MAPBOX_TOKEN` if it is
   a secret `sk.` token (a `pk.` token is public by design).
6b. **Secrets the code reads that `.env.example` does not list** (found after the first
   rotation pass, 2026-10-07; rotate each one that is set in production's `.env`):
   - **Bill per use, rotate first:** `TWILIO_ACCOUNT_SID` / `TWILIO_AUTH_TOKEN` /
     `TWILIO_API_KEY_SID` / `TWILIO_API_KEY_SECRET` (voice), `RETELL_API_KEY` (AI calls),
     `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY`, `SES_KEY` / `SES_SECRET`, `SQS_KEY` / `SQS_SECRET`,
     `ESTATE_UAE_RAPIDAPI_KEY`, `PROPERTYLAB_AI_KEY`, `INVESTHINK_PROPERTYLAB_API_KEY`,
     `IPINFO_TOKEN`, `IP_API_TOKEN`.
   - **Meta / ads:** `META_ACCESS_TOKEN`, `META_CAPI_TOKEN`, `FB_AD_ACCESS_TOKEN`,
     `FB_AD_APP_SECRET`, `AE_META_APP_SECRET`, `AE_META_VERIFY_TOKEN`.
   - **Email providers:** `EMAIL_SENDGRID_API_KEY` (+ `EMAIL_SENDGRID_WEBHOOK_PUBLIC_KEY`),
     `EMAIL_MAILGUN_API_KEY`, `MAILGUN_SECRET`, `SPARKPOST_SECRET`, `EMAIL_GOOGLE_CLIENT_SECRET`,
     `EMAIL_MICROSOFT_CLIENT_SECRET`, `GETRESPONSE_API_KEY`.
   - **Everything else:** `JWT_PRIVATE_KEY` / `JWT_PUBLIC_KEY`, `PROD_DB_READ_PASSWORD`,
     `MEMCACHED_PASSWORD`, `SHOPIFY_WEBHOOK_SECRET`, `VIMEO_ACCESS_TOKEN`, `MAPBOX_MOBILE_TOKEN`,
     `INSPECTOR_INGESTION_KEY`, `PETA_GPU_METRICS_TOKEN`, `AREA_GUIDE_SITE_KEY`,
     `ZOOM_COACH_BROWSER_KEY`.
   - **The rule that covers anything still missing:** every line of production's `.env` that holds
     a secret was readable, whether or not any list names it. Go through that file line by line.
7. **`APP_KEY`: last, and carefully.** Every `encrypted` cast is under it: `AiCredential`,
   `MessagingCredential` (Messages → Delivery APIs), `PaymentGateway`, `TngSetting`,
   `LineChannel`, `MessengerChannel`, `PdfProfile`, `LeadEnrichment`, the CEO email models.
   Changing it silently kills them (see the AI-credential note in the AI handbook). Laravel 13
   supports `APP_PREVIOUS_KEYS`: set the old key there, the new one in `APP_KEY`, re-save each
   DB credential through its Manage page (re-encrypting it under the new key), then drop
   `APP_PREVIOUS_KEYS`. Sessions are invalidated, so everyone signs in again.
   If the DB password **and** `APP_KEY` both leaked and the DB was reachable, the DB-resident
   provider keys are as good as public: rotate them at their providers too.

**Git history:** `.git/` was downloadable, so scan the history for anything ever committed:
`gitleaks detect --source . --log-opts="--all"` (or `trufflehog git file://.`), and rotate
whatever it finds even if it was deleted later.

## 4 · Personal data

`final_leads.csv`, `sg_and_180min_leads.csv`, the `sg_phone_fix_backup_*.csv` files, the WhatsApp
bridge and Zoom transcript logs, and the Laravel logs hold names, phone numbers and conversations.
If step 2 shows any of them fetched (200 with real bytes), this is a personal-data breach.
Malaysia's PDPA (as amended in 2024) requires telling the Commissioner promptly, within
72 hours, and in some cases the people affected. **Take this to whoever owns compliance or legal
counsel now. Do not wait for the cleanup.**

## 5 · Clean up (after 1–3)

On the server, move data that has no reason to live in the app tree out of it, or delete it once
it is no longer needed: the lead CSVs in `storage/app/`, `storage/app/sql/`, the root-level
`import*.log` / `verify*.log`, `devcheck.php`, the two files named after pasted PHP
(`as le','le.lead_id'…`, `er_profiles as up'…`, from a shell redirect accident), and the
`wb_*.html` exports. None of them are in git.

## Checklist

- [x] 1 · `block-webroot.conf` enabled, verification all non-200 (2026-10-07)
- [x] 1 · same on wk (35.213.165.89), which had been missed: `.env`, `.git` → 403, app still up (2026-10-14)
- [ ] 2–3 · wk: access logs reviewed, and every secret in its `.env` rotated
- [x] 1b · `www.` redirects to the primary domain over http and https, certificate covers it (2026-10-08)
- [x] 2 · access logs reviewed, readers listed (2026-10-07)
- [ ] 3 · DB passwords + Cloud SQL networks
- [ ] 3 · payment, messaging, Google, Zoom, remaining keys
- [ ] 3 · `APP_KEY` via `APP_PREVIOUS_KEYS`
- [ ] 3 · git history scanned
- [ ] 4 · compliance decision on notification recorded
- [ ] 5 · stray data files removed from the server tree
