Prompt optimization and testing

Writing a prompt is not a one-and-done task. The best-performing AI agents are continuously refined based on real conversation data. This guide gives you a systematic methodology for testing, measuring, and improving your Conversation AI and Voice AI prompts.

Start with the 4-part framework if you have not built your initial prompt yet. This guide assumes you have a working prompt and want to make it better.

Using HoopAI’s bot trial mode

Before sending your prompt to real customers, test it thoroughly using HoopAI’s built-in testing tools.

How to access trial mode

What to test in trial mode

Run through this checklist before going live:

Suggestive mode: your safety net

Before switching to Auto-Pilot, run your bot in Suggestive mode for at least 48 hours. In this mode, the bot generates responses but waits for your team to approve, edit, or reject them before sending.

This gives you:

• Real-world data on how the bot handles actual customer messages
• A safety net — no bad responses reach customers
• Training examples — every edit you make teaches you what to improve in the prompt

What to track during suggestive mode

Keep a simple log of:

1. Approved as-is — the response was perfect; no edits needed
2. Edited before sending — the response needed changes; note what you changed
3. Rejected — the response was wrong or unhelpful; note why

After 48 hours, calculate your approval rate. If fewer than 80% of responses are approved as-is, revise your prompt before switching to Auto-Pilot.

Systematic testing methodology

Once your bot is live, use a structured testing approach to identify and fix weaknesses.

Create test scripts

Write a set of test conversations that cover your most important scenarios. Run these scripts after every prompt update to catch regressions.

TEST SCRIPT: Lead Qualification Bot

Test 1 — Happy path (new lead):
  User: "Hi, I'm interested in your marketing services."
  Expected: Greeting + question about their business/needs
  User: "I run a small plumbing company and need more customers."
  Expected: Clarifying question about goals or budget
  User: "My budget is about $1,000/month."
  Expected: Timeline question
  User: "I'd like to start next month."
  Expected: Contact info collection
  User: "John Smith, john@example.com, 555-0123"
  Expected: Confirmation of all details + next steps

Test 2 — Objection handling:
  User: "Your prices seem really high."
  Expected: Acknowledge concern + highlight value + offer
    alternatives

Test 3 — Off-topic question:
  User: "What's the weather like today?"
  Expected: Polite redirect to business-related topics

Test 4 — Escalation trigger:
  User: "This is frustrating. I want to talk to a real person."
  Expected: Empathetic acknowledgment + immediate handoff

Test 5 — Unknown question:
  User: "Do you offer SEO services in Japanese?"
  Expected: Honest "I'm not sure" + offer to connect with team

The test-measure-improve cycle

Follow this cycle continuously:

Measuring prompt quality

You cannot improve what you do not measure. Track these key metrics to understand how your prompt is performing.

Resolution rate

The percentage of conversations the bot resolves without human intervention.

• Target: 60-80% for most businesses
• How to measure: Check the Conversation AI Dashboard for conversations marked as resolved by the bot vs. handed off to a human
• If too low: Your prompt may be missing common scenarios, or your escalation rules may be too aggressive
• If too high: Make sure the bot is not answering questions it should be escalating (check accuracy)

Handoff rate

The percentage of conversations that are transferred to a human agent.

• Target: 20-40% (some handoffs are expected and healthy)
• How to measure: Track handoff events in the dashboard
• If too high: The bot cannot handle enough scenarios — add more instructions and examples
• If too low: The bot may be over-confident — check if it is answering questions it should escalate

Response accuracy

How often the bot gives correct, helpful responses.

• How to measure: Review a random sample of 20 conversations per week and rate each response as accurate, partially accurate, or inaccurate
• Target: 90%+ accuracy rate
• If below target: Check your Knowledge Base for missing or outdated information. Add knowledge boundaries to your prompt.

Customer satisfaction signals

Look for behavioral signals that indicate satisfaction or dissatisfaction:

• Positive signals: Customer says “thank you,” continues the conversation, completes the desired action (books appointment, provides contact info)
• Negative signals: Customer repeats their question, says “never mind,” asks for a human, expresses frustration, abandons the conversation

Average conversation length

How many messages does a typical conversation take?

• For appointment booking: 5-8 messages is typical
• For FAQ questions: 2-4 messages is ideal
• If too long: The bot may be asking unnecessary questions or not getting to the point
• If too short: The bot may be giving incomplete answers or rushing to close

Using the Conversation AI Dashboard

The Conversation AI Dashboard is your primary tool for monitoring prompt performance. Here is how to use it effectively.

Daily review (5 minutes)

• Check total conversation volume
• Review handoff rate — any spikes?
• Scan recent conversations flagged as problematic

Weekly review (30 minutes)

• Read 15-20 random conversations end to end
• Identify the 3 most common questions the bot struggles with
• Note any new question types that are not covered by your prompt
• Check if knowledge base information is current and accurate

Monthly optimization (2 hours)

• Calculate your key metrics (resolution rate, handoff rate, accuracy)
• Compare to the previous month
• Identify the single biggest area for improvement
• Update the prompt with targeted changes
• Run your test scripts to validate the changes
• Document what you changed and why (prompt versioning)

A/B testing prompts

When you want to compare two different prompt approaches, set up an A/B test.

How to A/B test with workflows

What to A/B test

Good candidates for A/B testing:

• Greeting style — formal vs. casual, long vs. short
• Response length — concise vs. detailed
• Escalation thresholds — aggressive (hand off early) vs. conservative (try harder)
• Information collection order — name first vs. need first
• Tone — professional vs. friendly vs. enthusiastic
• Example quantity — 2 examples vs. 5 examples in the prompt

The iterative improvement cycle

Prompt optimization is not a project with an end date — it is an ongoing practice. Here is a sustainable cadence:

Next steps