If you are building a fitness app, coaching platform, CRM, or any software with an AI chat layer — the part where a member or customer can talk to the product, ask questions, receive guidance, and expect it to remember what was discussed — there is a question that is easy to overlook:
How do you know it still works after the first conversation?
It is relatively easy to test whether an AI coach or assistant can respond to a message. It is much harder to know whether that chat experience remembers what happened last week, follows up on an important decision, notices a repeated problem, and changes its advice when circumstances change.
Testing an AI product one conversation at a time is not enough. That is the problem we have been working on while building our own AI health and accountability coach.
What is a synthetic user?
A synthetic user is a fictional person created for testing. They have a profile, goals, habits, preferences, good days, bad days, and reasons for not always doing what they planned.
They are not real customers. They are more useful than a simple test script because they can move through the product like a member would: asking questions, accepting or rejecting suggestions, changing their mind, missing a day, returning later, and creating a history for the app to remember.
The important part is that these fictional users go through the real application experience. They are not just shown a pretend version of the system. They use isolated test accounts and exercise the same important pathways a real member would use.
For a fitness app, that might mean logging food, recording a workout, reviewing health information, and responding to a coach. For a CRM, it might mean creating a lead, changing its status, scheduling a follow-up, and checking whether the next salesperson sees the right history. That is the same class of problem as a living GTM operations system: the next person, or the next conversation, has to inherit the real record, not a stale copy of it.
The problem with testing one conversation at a time
Imagine testing a personal trainer app by asking it one question:
“What should I do if I am tired today?”
The answer might be excellent.
But what happens when the same person asks about tiredness three more times? Does the coach keep repeating generic advice, or does it notice a pattern and help test one practical change?
What happens when the person agrees to train on Tuesday but misses Tuesday? Does the app remember the commitment and ask what got in the way, or does that promise disappear into an old conversation?
What happens when the person has not logged their food? Does the app correctly say that the information is missing, or does it mistakenly assume they ate nothing?
Those are not single-message problems. They are time problems. A one-shot answer can look finished and still leave the product broken, the same way a Ralph Loop is different from asking an agent to guess at a patch: the work is not done until the longer cycle holds.
Running a month overnight
To test those longer journeys, we created a Time Warp for synthetic users on an isolated development server. It allows approximately 10 member-days to pass during one Chicago wall day.
In practical terms, we can examine a multi-week coaching relationship overnight instead of waiting a month to find out whether it is working.
The first serious run produced 20 synthetic member-days and 56 completed Coach conversations. The confirmed cost for running four to eight active synthetic users through hundreds of daily interactions was approximately $1.10–$2.05 per day.
That is roughly the price of a coffee for a small team to exercise a large number of real workflows involving memory, decisions, and saved actions before releasing them to customers.
The most valuable result was not a perfect score
The test showed that the clock was working. The dates advanced, the conversations ran, and the individual conversations passed their basic checks.
But the Coach had not actually learned the fictional member in the way we wanted.
Important seven-day and fourteen-day reviews were missing. No meaningful habit loop had been completed. The synthetic user sometimes ignored an open question from the Coach and jumped to the next scheduled topic. The Coach often gave sensible answers, but did not reliably turn repeated problems into a small plan, a commitment, a follow-up, and an adjustment.
That gave us one of the most important lessons from the experiment:
An app can appear to work one interaction at a time while still failing the customer over time.
We also found an issue with missing information. If a meal has not been logged, the system must say “not logged” or “unknown.” It must never treat missing information as proof that nothing was eaten.
Because the test ran quickly and saved the conversations and results, we did not have to rely on memory or opinion. We could review the complete journey, see where it broke down, and identify what needed to change.
What else this kind of testing can find
Across the wider synthetic testing program, we found problems that would have been easy to miss in ordinary manual testing:
- A quantity such as “half a cup” was attached to the wrong food in a multi-item meal.
- A rejected food suggestion appeared later in an unrelated conversation about knee pain.
- An evening meal could be assigned to the wrong day because of a date change at midnight.
- A temporary network problem could cause an entire synthetic day to fail instead of being retried safely.
These are the sorts of problems that frustrate customers because the app appears mostly fine until one small detail matters.
Why this matters beyond fitness apps
The same problem appears anywhere software is expected to remember, act, and make decisions over time.
A CRM needs to remember what happened with a prospect and show the next employee the right context. A healthcare platform needs to distinguish missing information from negative information. A customer-support assistant needs to recognize when a problem has already been discussed. A sales or operations tool needs to avoid creating duplicate records when someone retries an action.
This is also why AI SDR tools are not a strategy on their own. An assistant can produce a pleasant next message and still fail the longer job: memory, timing, judgment, and whether the record is still true after interruptions.
The details differ, but the testing question is the same:
Does the system still make the right decision after weeks of real-looking activity, interruptions, corrections, and incomplete information?
What synthetic users can and cannot tell you
Synthetic users are not a replacement for real customers. They cannot tell us whether people will love a product, keep using it, or pay for it.
They can tell us whether the product behaves consistently, whether information is carried forward correctly, whether actions happen only when they should, and whether a long-running experience breaks in ways that are difficult to reproduce manually.
That makes them especially valuable before a public launch or whenever an app is changing quickly.
The confirmed cost is low enough that this does not have to be reserved for large technology companies. A small team can run a meaningful test fleet every day, find problems overnight, and spend its human testing time on the questions only real people can answer.
The broader lesson
The biggest benefit of Time Warp was not simply that it made testing faster.
It made long-term failure visible while there was still time to fix it.
That is the kind of testing we believe more AI products will need: realistic fictional users, real application behavior, clear evidence, and tests that measure whether the product is getting better over time — not just whether it produced a pleasant answer.
If you are building an AI product, fitness platform, CRM, workflow tool, or customer-facing application that needs to remember and act over time, All Great Things can help you design those tests before your customers have to find the problems for you.
What is a synthetic user?
A fictional person with a profile, goals, and messy habits, used to exercise the real product. They are not real customers. They exist so the app has a history to remember, interrupt, and get wrong.
Why isn’t one-conversation testing enough for an AI product?
A single reply can be excellent and the product can still fail later: missed follow-ups, forgotten commitments, duplicated actions, or missing data treated as a fact. Those failures only show up across days and weeks.
What did the Time Warp test actually find?
The clock worked. Dates advanced and 56 Coach conversations completed across 20 synthetic member-days. The Coach had not reliably learned the member: reviews were missing, repeated problems did not become a plan and a follow-up, and unlogged meals could be misunderstood.
