A friendly answer still needs to be right
A chatbot can sound helpful while promising the wrong delivery date. It can also say “I don't know” accurately but leave a customer with nowhere to go.
Check four things: the facts, how the AI handles missing information, the next step, and the wording. Keeping those checks separate makes it easier to see what needs fixing.
This is EC AI Lab's suggested checklist, not an industry standard or a product benchmark. The examples use fictional store rules. Product references were checked September 6, 2026. For help getting started, see our Shopify support guide.
Write down the expected answer first
Before asking the AI, record the customer's question, any relevant details, and what a good answer should say or do. Link to the correct store policy or product information and note its version or date.
Sometimes the right response is another question or a transfer to your team. Don't expect a complete answer when essential information is missing.
For each review, keep:
- A case ID, language, and type of question.
- The source, expected answer, and actual AI reply.
- The scores below and any serious problem.
- Who will fix the issue and what happened when you tested again.
Use fictional details for prepared tests and leave unnecessary customer information out of your notes. If the source itself is unclear, fix that before scoring the answer.
Score each part separately
Use 0 for a failure, 1 for an answer that needs changes, and 2 for an answer that meets your expectation. If a category doesn't apply, mark it N/A and explain why.
| Check | 0: fails | 1: needs changes | 2: meets expectations |
|---|---|---|---|
| Facts | A wrong or unsupported claim | Mostly correct, but an important condition is missing | Correct facts and relevant conditions |
| Missing information and permissions | Guesses or makes a promise it cannot authorize | Mentions uncertainty without explaining it | Explains the limit and asks for details when needed |
| Next step | No useful next step, or claims something happened when it didn't | An incomplete next step | A correct action or confirmed transfer to staff |
| Clarity and tone | Misleading or inappropriate | Understandable but confusing | Clear, readable, and appropriate |
Flag serious issues separately, such as an unauthorized refund promise, another customer's information, or a claimed transfer that never happened. Under this checklist, these need fixing regardless of the total score.
Example: a delivery promise
Suppose standard mainland delivery takes three to five business days after shipping, but island deliveries need a separate estimate. A shopper asks, “Will it arrive at my island address this Friday?” You don't have a shipping date.
Reply A: “Yes, delivery takes three to five days, so Friday should be fine.”
Score all four checks 0. The reply makes an unsupported promise, even though it's easy to read.
Reply B: “I can't confirm Friday. Island deliveries need a separate estimate, and the shipping date isn't confirmed yet. Please use our delivery inquiry form so the team can check your destination.”
Score each check 2 if that form is the approved route and actually reaches your team. If it doesn't work, the next-step score must change.
Example: a missing product detail
A bag's dimensions are listed, but its maximum load isn't. If someone asks whether it can carry a heavy instrument, “It should be strong enough” is a guess.
A better response explains which detail is missing and offers your approved way to ask the team. A product link doesn't make an unsupported recommendation correct.
Check what happens after a transfer
Ask for a person, then look at what staff receive. In Shopify Inbox, an available staff transfer sends the conversation and context to Unassigned. When that transfer isn't available, the shopper gets the store's sender email address and must email the team themselves. The agent doesn't send the email. Shopify conversation handling
In the second situation, “I've emailed our team” is wrong. “Please email the team at the address shown” describes the next step accurately.
Fix the problem and test again
Have two people review a small set of answers independently. Discuss disagreements and make the expected answer clearer. This helps reviewers be consistent; it doesn't establish an overall accuracy rate.
Keep ordinary questions, difficult questions, and transfers identifiable. If you deliberately test mostly difficult cases, don't present the average as typical of every conversation.
Check whether a fix belongs in the source information or the response instructions. Shopify explains that training feedback can suggest persona changes while factual knowledge stays in separate store sources. Shopify agent training
Record what changed and who changed it. Retest the failed question and a related one—for example, island and mainland delivery. Keep both results for comparison. Report how many cases you reviewed and what still needs fixing, with prepared tests separate from real customer reviews.
For setup help, see our Shopify Inbox guide.
Still choosing a chat tool?
Shopify Inbox is one option to consider. Check its availability for your store, then use these questions during your evaluation.
See whether Shopify Inbox fits your businessExplore Shopify Inbox
