Automated scoring options in evaluation forms

1. Create a question group. For more information, see Create and publish an evaluation form.
2. When you add a new question, you can set automated scoring for each question.
   
   The options are:
   • No Automation
   • AI Scoring
   • Evaluation Assistance
3. Save the question and finish the evaluation form setup. For more information, see Create and publish an evaluation form.

You can enable AI scoring per question within the evaluation form editor to have Genesys Cloud use AI to prefill the answer on an evaluation.

The AI prompt updates, but does not lose the existing scoring settings during the following editorial actions:

• Reorder questions or question groups.
• Delete questions or question groups, which updates the AI prompt.
• Add some questions or question groups. These additions update the AI prompt.

Create question prompts

The prompt is constructed automatically from the text of the question group, the question, the answers, and the help text. If you rephrase unclear questions and add context to these fields, you can improve the effectiveness of your questions for AI scoring.

Prompt filters

Configured questions for AI scoring use a prompt filter, so Genesys Cloud only provides answers to questions where the AI model’s answer has a high confidence level.

• Answer limit: You cannot save an AI scoring configuration for questions that have more than three answer options.
• N/A or no evidence: In this case, the AI model has no sufficient information to answer a question. Genesys Cloud displays the context when the AI model cannot provide an answer.
  
  
• Rolling accuracy: Genesys Cloud examines saved data for the AI-generated answers, and evaluates how accurate the data is. Accuracy is calculated based on how many times a human being edited an answer. If the accuracy measure is sufficient, Genesys Cloud displays the answer on future evaluations with the respective question.
  If this accuracy measure gets too low, Genesys Cloud reports a low confidence error on the question until the accuracy begins to improve. 

Guidelines for AI scoring

When you use AI scoring, Genesys recommends that you consider these guidelines.

Construct questions 

• Focus on transcript-driven questions. Use AI scoring for questions that can be answered directly from the transcript. Avoid questions requiring information not available in the transcript or beyond the conversation context.
  
  Example
  “Did the agent greet the customer at the beginning of the conversation using a standard greeting such as ‘hi,’ ‘hello,’ or ‘good morning’?”
  
• Avoid subjective queries that rely on subjective interpretations. You can rephrase subjective questions with specific, measurable criteria, which are supported by detailed help text. If you provide clear and actionable criteria, subjective questions become measurable and easier for AI to assess.
  
  Example

  • A question with subjective interpretation: “Did the agent display patience throughout the interaction?”
  • Improved question: “Did the agent allow the customer to finish speaking without interruptions?”
  • Help text example: The agent should let the customer complete their statements before responding. Interruptions are defined as speaking over the customer mid-sentence.

Improve question clarity

• Use complete sentences. Phrase questions clearly and avoid shorthand.
  
  Example:
  Replace “Greeting protocol” with: “Did the agent greet the customer at the beginning of the conversation using a standard greeting such as ‘hi,’ ‘hello,’ or ‘good morning’?”
  
• Provide relevant context to questions.
  
  Example:
  For a question like, “Did the agent confirm the customer’s identity?”, include help text such as:
  Agents must verify the customer’s phone number and order ID before resolving the issue.
  
• To avoid ambiguity, clarify business terms and business processes.
  
  Example
  For a question like, “Did the agent explain the account escalation process?”, clarify with:
  Did the agent clearly explain the steps involved in escalating a concern, including who to contact, the required information, and expected response times?
  
• Standardize terminology. Use consistent terms across all questions. For example, consistently use ‘agent’ and ‘customer’ instead of alternatives, like ‘staff’ or ‘client.’

Best practices

• Focus on transcript-derived answers: Ensure that all questions are directly answerable based on transcript data.
• Keep the instructions straightforward: Use simple, actionable language to guide AI scoring.
• Refine continuously: Use AI feedback to improve the clarity and relevance of questions iteratively.

When using AI Scoring in evaluations, you may occasionally encounter errors. These messages are intended to help you understand what went wrong and how to resolve the issue. Below are the most common AI Scoring error types:

Rate limit error

Message:
AI Scoring limit for your organization has been reached. Genesys AI could not process this evaluation.

When it occurs:
This error appears when an agent reaches the daily maximum of 50 AI scoring requests, including reprocessing attempts.

Duplicate evaluation error

Message – This evaluation is a duplicate of an existing evaluation associated with this interaction. Genesys AI did not process this evaluation.

When it occurs – This happens when there is already an evaluation for the same conversation, evaluator ID, and evaluation form.

Processing failure error

Message – Genesys AI failed to process this evaluation.

When it occurs – An unexpected failure occurs, often related to how a question is defined in the evaluation form. If this happens frequently, review and adjust the AI scoring questions using the AI scoring guidelines.

Low confidence on question

Message – Genesys AI could not accurately answer the question.

When it occurs – Genesys AI identifies low confidence in answering a question accurately. This could be due to specific evaluation conditions or based on historical low-confidence patterns for that question.

An evaluation assistance condition is made up of topics. Topics are collections of phrases that indicate a business-level intent. For example, if you want to identify interactions where the customer wants to cancel a service, create a topic named Cancellation and include several phrases, such as “close out my account” or “I want to cancel.” In addition, topics help to boost the recognition of specific words and phrases in the interaction because they adapt the underlying language model to look for organization-specific language in conversations.

When you add an evaluation assistance condition to a form question, speech and text analytics must locate interactions with the topics included in the condition. If an interaction with the specific topic is located, an answer to the specific question is automatically generated. For more information, see Understand programs, topics, and phrases and Work with a topic.

Add an evaluation assistance condition to a form question

1. Create an evaluation form question.
2. Click Add Evaluation Assistance Condition that is associated with the answer for which you want an answer to be automatically generated. 
3. From the select if conversation list, select if you want the conversation to include or exclude one or more topics.
4. From the these topics list, select one or more topics.
5. (Optional) Click the Add Evaluation Assistance Condition option again to add another condition to the same answer.