A May 2025 report about AI hallucinations raised a useful question for property management leaders: what happens when a system gives a convincing answer that is wrong?
This article was originally published on 14 August 2025 in response to that reporting. The evidence supports a narrower reading than the claim that AI hallucinations are universally getting worse. The reporting discussed particular models and tests; it did not establish an error rate for every AI product or property management task.
What the 2025 evidence actually measured
The New York Times report of 5 May 2025 covered concerns about fabricated answers from AI systems. One relevant primary source is OpenAI’s April 2025 evaluation of o3 and o4-mini.
The system card’s hallucination section distinguishes SimpleQA, a short factual-question benchmark, from PersonQA, which asks about people. It reports that o3 made more claims overall than o1 in the evaluated setting, including more accurate and more inaccurate claims, and says the cause needed further research.
Percentages from these tests should not be read as the probability that an ordinary property management email will be wrong. A result depends on the dataset, model, prompt, tools and scoring method. It also does not establish that reasoning itself causes a continuing deterioration in accuracy.
Those are historical evaluations. Choosing a tool today requires checking the version and workflow your team will actually use, rather than transferring a 2025 benchmark result to a different product.
Why the issue matters in property management
A hallucination can be an invented repair appointment, an unsupported lease clause or a fee explanation that sounds reasonable but contradicts the agreement. The problem is especially difficult when most of a response is correct. A reviewer may recognize the property and work-order details and overlook the added promise.
NIST’s Generative AI Profile, published in July 2024, identifies confidently presented false content and fabricated citations as risks. Requesting a citation is therefore only the beginning of verification. Someone still needs to open the source and confirm that it supports the statement.
Consider an illustrative maintenance update. The work order says a plumber has been contacted, but no appointment is confirmed. A draft that says, “Your plumber will arrive tomorrow morning,” has changed an attempt to schedule into a promise. A polite tone cannot repair that factual error.
The approved message should preserve what is known: “We have contacted the plumber and are waiting for an appointment time. We will update you when the visit is confirmed.” The next-update commitment should come from your operating policy, not the model’s guess.
Check the source, the answer and the action
Use a review procedure that makes each layer visible:
| Check | What the reviewer verifies |
|---|---|
| Source | The correct property, agreement, policy version and work-order record were supplied |
| Facts | Names, amounts, dates and status match the source |
| Missing information | The answer identifies gaps instead of filling them with plausible details |
| Commitments | The message promises only actions the company has authorized and can deliver |
| Interpretation | Legal, screening, safety and financial judgments reach the appropriate qualified reviewer |
| Delivery | The recipient and attachments are correct, and the approved version is the one sent |
For company-policy questions, give the tool the approved document and ask it to identify the relevant section. If the document does not answer the question, route it to the person responsible for that policy. Avoid asking the model to bridge the gap with what companies “usually” do.
When an incorrect message has already been sent, stop the affected workflow, establish who received it and preserve the source and output. The responsible manager should decide how to correct the communication and whether related messages need review. Quietly editing the prompt does not correct a promise already made to a resident or owner.
Use the report to improve your own checks
List where AI currently drafts, summarizes, recommends or sends information in your company. For each use, record its source material, reviewer and permitted actions. A drafting assistant with no sending access needs different controls from a tool that can change records or communicate externally.
Then evaluate a small set of representative cases with known correct answers. Include:
- Incomplete records.
- Conflicting policies.
- Requests outside the tool’s permitted role.
Record failures and review time as well as useful outputs.
A hypothetical test record could read: supplied fact—plumber contacted, appointment unconfirmed; expected answer—state that confirmation is pending; illustrative error—promise arrival tomorrow morning; decision—pause the affected resident-update workflow, correct the source instructions or controls, and retest before resuming.
For this drafting task, resume only when the tested cases meet the factual and authority checks and the assigned reviewer can verify each draft against its records before approving it for sending. A small test set supports that limited decision, not a claim of general reliability. Repeat the relevant checks after a material model, prompt, integration or policy change.
Our guide to using AI in property management explains how to select suitable workflows. The ChatGPT prompt examples show how to supply source facts and review a draft.
If the difficult part is deciding who verifies work and handles exceptions, BetterTeam Consulting can help clarify those responsibilities within your team’s Operational Framework.
