# Map AI Advisor expanded acceptance — 2026-10-15/16

## Method

Eight four-turn conversations were fixed before the repair iterations. Each group starts a new conversation; turns within a group retain the same conversation. These use real OpenAI GPT-4.1 calls through the existing local Herd configuration. The service-level harness records answers, saved brief, tool events and result payloads. A separate browser pass exercises the actual local page, HTTP/SSE flow, evidence cards and map. No local model or mocked answer is used for these live runs.

Scoring is an independent qualitative review, not a statistically calibrated intelligence benchmark. Context, intent, tool use, factual grounding and useful progress each score 0/5/10/15/20. A group passes at 80 or above, every dimension at least 15, and no hard failure. Missing rental/live data is acceptable only when accurately disclosed; silently dropping a buyer constraint is a failure. HTTP completion alone is not a pass.

## Cases and acceptance checks

| Group | Scenario | Required behavior |
| --- | --- | --- |
| G1 | English colloquial follow-ups and rental budget | Resolve colloquial place labels and swap; retain rent RM2,200 and no-car constraint. |
| G2 | Chinese work correction and requirement recall | Change only work to TRX, retain Cochrane home, and recall saved requirements. |
| G3 | Malay purchase-to-rent switch and compound question | Change purchase RM650,000 to monthly rent RM2,800; answer rail plus rental-data gap. |
| G4 | Hypothetical move versus actual buyer brief | Compare Maluri and Kajang without changing actual home; change only search area to Cochrane. |
| G5 | Unknown development and explicit station starting point | Do not invent the unknown development; use the expressly chosen station without claiming an apartment walk. |
| G6 | Arrival guarantee versus connectivity | State no arrival guarantee, answer direct connectivity, ask at most one missing detail. |
| G7 | Zero matches, price basis and changed search area | Retain purchase RM500,000 and condo filter; no silent location-only relaxation; change only station scope. |
| G8 | Two-origin comparison, latter/there and language switch | Compare both origins, resolve latter/there, retain rent budget, and change work to Pasar Seni. |

## Findings and fixes

- Save explicit home/work corrections, independent search area, rental/purchase basis, and no-car requirement. A hypothetical move, third-person statement, income or monthly ownership cost must not silently replace those fields.
- Recall the current buyer brief without requiring an external property lookup. Keep prior map/property evidence when a brief is combined with a rail or property query.
- Search a sourced station's surroundings using coordinates and a disclosed 2 km straight-line default radius. Property-name matching does not establish nearby coverage. Purchase amounts use recorded aggregate medians; rental-price or availability evidence is explicitly unavailable.
- Retain saved budgets even when the model emits `locationOnly=true`; relaxing financial filters needs explicit buyer agreement. Keep/within-budget language takes priority over a location-only suggestion.
- Complete both sides of a commute comparison and both parts of a route/property question. Duplicate route evidence does not count as a comparison. Unresolved parts retain already verified results and show a limitation.
- Answer arrival-guarantee questions directly without inventing timing. Add sourced reference coordinates for Taman Connaught, Pasar Seni, Kajang and Bukit Bintang.
- Bind a current explicit station search area, or a saved station area during a finance-only update, to the actual property query. A workplace must not replace the desired search area; explicit new locations still take priority.
- Treat an answer envelope only as a stop signal; discard extra model-authored prose/data. Tool names and every supplied argument remain strictly validated.

## Results and verification

Final independent semantic review: **8/8 groups pass**, average **92.5/100** under the fixed rubric. This combines unchanged passing groups with the affected groups rerun after fixes; it is not a first-attempt 32/32 result and not a new unseen-question benchmark.

| Group | Evidence phase | Score / 100 | Result |
| --- | --- | ---: | --- |
| G1 | release | 95 | Pass |
| G2 | release | 85 | Pass |
| G3 | verified | 95 | Pass |
| G4 | release | 100 | Pass |
| G5 | release | 90 | Pass |
| G6 | release | 95 | Pass |
| G7 | repaired | 90 | Pass |
| G8 | release | 90 | Pass |

- Final isolated PHP suite: **347 tests / 2,862 assertions passed**; one pre-existing PHPUnit XML deprecation.
- Changed-file Pint: 15 PHP files passed. All three PHP source guards passed; existing sidebar-link baseline remains 67.
- Final focused code reviews: APPROVE, no remaining blocking findings.
- Final G3 browser rerun completed all four turns without retry; G7 browser rerun retained the price and property-type constraints. Other passing browser groups were reused where behavior was unchanged.


Browser proof covers all eight groups, sources and the changed destination's map. `View on map` from the Pasar Seni evidence card was observed to focus the actual map on `Station: Pasar Seni`. The browser series contains a G3 deadline timeout followed by a successful retry, and G4 was rerun after fixing repeated-one-route comparison. These are retained failures, not a first-attempt 32/32 claim.

The initial baseline scored 66.25/100, with 1/8 groups meeting the fixed rubric. Intermediate `final` and `release` artifact labels are iteration names, not claims of final acceptance. In the `release` iteration G3 failed an answer-envelope validation and G7 silently relaxed the budget; after those fixes, a G3 rerun exposed an incorrect search-area choice. Each failure was retained and converted into a regression before affected groups were rerun.

## Limits and rollout

The local snapshot and test fixtures verify retrieval/filtering behavior, not complete production inventory or business sign-off. No rental listing feed, live transit service, walking/door-to-door routing, or unit valuation is added. Rail references cover the Kajang Line; station coordinates are reference centers rather than confirmed building entrances. Short conversational references such as “latter” still depend on recent history, and requirement summaries can be mechanical (including the UI's default all-selected property types).

The cloud/browser runs are local Herd tests with real cloud inference, not a production benchmark or proof that app/wk is deployed. The one observed browser deadline timeout remains a reliability observation even though retry succeeded. Full reviewed property knowledge, master/COPY mapping and market-data coverage remain prerequisites outside this conversational regression suite. This does not complete the broader M09 business evaluation.

No dependency, migration or environment variable is added. Existing credit policy, the Super Admin exemption, permissions and other modules' rate limits remain unchanged. Delivery remains `dev-chen → master` through PR #210; no merge or production deployment is performed. An enabled site uses its normal reviewed deployment, including the existing PHP/SSR/cache refresh and frontend build from the original PR.

The prior frontend run (96 tests across 18 files and client/SSR build) is reused because no frontend code or dependencies changed in this follow-up. The existing learning-content guard failure (`how-to.json`/`how-to.en.json`) was reproduced on untouched master `9808c526`. GitHub Actions have been unable to start because of repository billing/spending restrictions; this is not a green remote CI claim.

Sanitized live answers, saved briefs, tool/result evidence, exact inputs, screenshots and test logs are retained in the task artifact folder `map-advisor-generalization-20261015`. Credentials, local test conversations and local runtime configuration are not committed.

## Exact replay inputs

### G1 — English colloquial follow-ups and rental budget

1. Help me find somewhere that fits my daily routine.
2. i stay near taman connaught mrt, office at trx
3. got direct train or need swap? just answer the train bit first
4. ok now help me rent nearby, max rm2200 a month, no car

### G2 — Chinese work correction and requirement recall

1. 我目前住在 Cochrane MRT 附近，在 Pasar Seni 上班，想找交通方便的住处。
2. 刚刚说错了，上班是在 TRX，住的地方没有变。
3. 这样去公司要转车吗？
4. 先别推荐房子，把你现在记住的住处、上班地点和我的交通需求列给我。

### G3 — Malay purchase-to-rent switch and compound question

1. Saya kerja di TRX dan nak cari rumah dekat MRT Kajang. Saya tak ada kereta.
2. Pada mulanya saya nak beli, bajet maksimum RM650,000.
3. Tukar rancangan. Saya nak sewa dulu, maksimum RM2,800 sebulan. Bajet beli tadi jangan guna.
4. Kalau tinggal dekat stesen Kajang, perlu tukar tren untuk pergi kerja? Lepas itu tunjuk pilihan sewa yang ada bukti harga.

### G4 — Hypothetical move versus actual buyer brief

1. I currently live near Maluri MRT and work at TRX. I'm considering buying, up to RM700k.
2. Hypothetically, if I lived near Kajang MRT instead, how would the commute differ? Don't change my actual brief yet.
3. That was only a what-if. What are my actual home location, work location and budget?
4. Now change the area I want to buy in to Cochrane. Keep my current home and office unchanged.

### G5 — Unknown development and explicit station starting point

1. I live at Residensi Awan Biru X9 and work in TRX. Is there an MRT nearby?
2. I don't have an address pin. Please don't guess which development it is.
3. For now use Taman Connaught MRT as my starting point. I'm asking from the station, not from my apartment.
4. Can I get to TRX without changing trains? Don't include a walking time from my apartment.

### G6 — Arrival guarantee versus connectivity

1. 我从 Maluri MRT 去 TRX 上班。
2. 早上 8:45 进站，你能保证我 9:00 到公司办公室吗？
3. 不要求保证时间了，我只想知道 MRT 有没有直达、是否需要换乘。
4. 如果还要算门到门时间，你缺我哪些信息？只问最关键的一项。

### G7 — Zero matches, price basis and changed search area

1. 我在 TRX 上班，想买 condo，总价最多 RM500,000，优先看 Taman Connaught MRT 附近。
2. 先用你现有的数据找，不要再问我的预算和房型。
3. 如果没有符合预算的记录就直说，不要用租金或每平方尺价格冒充总价。
4. 把范围改为 Cochrane MRT 附近，其余条件不变。先给有证据的结果，再告诉我最重要的缺口。

### G8 — Two-origin comparison, latter/there and language switch

1. Compare living near Cochrane MRT versus Taman Connaught MRT for working at TRX. I want to rent, at most RM2,500 a month.
2. Focus on the latter of the two stations I named. Keep the rent budget.
3. 用中文回答：从那里坐 MRT 去公司要换车吗？先别给我具体车程分钟。
4. 现在只把工作地点改成 Pasar Seni，住处候选和租金预算都保留。重新看这个通勤方向。
