Written by Kiwi Dynamics Team 1What real estate agencies have you actually shipped this for
Ask for specific examples of real estate agencies, ideally comparable in size to yours, where the system is actually running in production today, not a pilot, not a proof of concept from eighteen months ago that quietly stalled. A vendor with real deployments will talk about specifics: what the workflow was, what changed, what the agency measured. A vendor without them will talk in generalities.
2Will it connect to our existing CRM
Your agency already has a CRM full of listings, contacts and history, and an AI tool that does not read from or write to it just creates a second system nobody keeps updated. Ask exactly how the integration works, what data flows in both directions, and what happens if your CRM changes. If the answer is vague, the integration is probably an afterthought.
3How is success measured once it is live
Before signing, agree on what success actually looks like: response time, hours saved per week, appraisal conversion, whatever is relevant to the workflow being automated. A vendor who cannot tell you how they will measure the result is a vendor who cannot tell you if it worked. This should be a number you both check, not a vibe you both hope for.
4What happens when the AI does not know the answer
Every AI system encounters questions and situations it was not built to handle, an unusual enquiry, an edge-case request, a genuinely upset caller. Ask what happens in that moment: does it hand off cleanly to a human, does it guess, does it just fail. The answer tells you a lot about whether the vendor has actually run this in the real world.
5Who owns the data it is trained and run on
Real estate data, buyer contact details, sale history, vendor information, is sensitive, and you should know exactly where it lives, who can access it, and whether it is used to train anything beyond your own agency's system. This matters both for compliance and for basic trust with your vendors and buyers.
6How fast can this actually go live
Ask for a realistic timeline to a working first version, not a full platform, one scoped workflow live and measurable. A vendor who cannot commit to a concrete go-live date for something specific is often selling a roadmap rather than a product. Speed to a real, working thing is a good signal of how the whole engagement will go.
7What does it cost against the hours it saves
Get a clear picture of pricing against the actual hours or leads it is expected to save, not just a flat monthly fee with no reference point. A good vendor will help you do this math honestly, including the scenario where the tool underperforms. If they will not engage with the cost-benefit conversation directly, that is worth noting.
8Can we see it running, not just a slide about it
Ask to see the system actually running, ideally on a workflow close to your own, rather than sitting through a slide deck of logos and adjectives. A vendor confident in what they have built will show it working. One who keeps the conversation at the concept level usually has less to show than the pitch suggests.
9Who fixes it when something breaks
Find out what ongoing support looks like once the system is live: who monitors it, who you call when something goes wrong on a Friday afternoon, and what the response time on a fix actually is. A system that runs a core part of your enquiry handling needs a real answer here, not "we will look into it."
10What happens if we want to walk away
Ask what happens to your data, your workflows and your team's process if you decide to end the engagement. A vendor who has built something you are genuinely locked into, with no clean exit, is a different kind of risk than one whose system you could unwind if it stopped working for you. Clarity here upfront avoids a much harder conversation later.
A vendor who answers these 10 questions plainly, with specifics rather than reassurance, is usually one worth working with. Kiwi Dynamics builds production AI for real estate agencies in New Zealand and Australia and is happy to be asked every one of these questions directly, because the answers are the whole point. Success is measured in hours given back and dollars saved, never the size of the contract.