Evaluation JSON (shape)
{
"q_1_1": {
"evaluator_selection_option": "Did the agent greet the customer according to protocol? Evaluate only the detected agent's opening turn. Return Yes when the opening contains all three meanings: greeting or salam, agent identity or name, and an offer of assistance. Minor natural wording variations are acceptable. Before failing because identity appears absent from transcript, check original audio where available in case ASR omitted the name. Return No when any major component is genuinely missing. Return only: Yes/No",
"mark_distribution": "If all three components are present in meaning, give full marks: 1. If any major component is genuinely missing, give 0. No partial marks. Return only score:",
"detail": "Check only the agent's opening. Required meaning: greeting/salam, agent identity, and assistance offer. Use audio to verify possible ASR omission."
},
"q_1_2": {
"evaluator_selection_option": "Did the agent greet with a friendly and welcoming tone? Evaluate the opening only and use audio where available. Return Yes if the opening is respectful, welcoming, attentive, and service-oriented. Return No if it is rude, careless, cold, commanding, excessively rushed and unwelcoming, lethargic, or disinterested. Return only: Yes/No",
"mark_distribution": "If the opening is friendly and welcoming, give full marks: 2. If it is unfriendly, cold, rude, careless, excessively rushed, or lethargic, give 0. No partial marks. Return only score:",
"detail": "Evaluate the opening's wording and available audio. q_1_2 covers the greeting only; q_3_5 covers delivery throughout the call."
},
"q_2_1": {
"evaluator_selection_option": "Did the agent obtain enough relevant information to identify and understand the customer's actual need? This is an information-sufficiency criterion, not a question-count criterion and not a knowledge-accuracy criterion. Return Yes when the customer clearly supplied enough facts; the agent understood the intent immediately; the case was simple and unambiguous; the agent asked enough relevant follow-up questions; or the agent correctly confirmed or summarized the need. One initial clarification is allowed. Return Yes even if q_2_3 is No, provided the agent correctly identified the customer's actual intent. Return No only when clarification was genuinely necessary, the agent failed to obtain enough relevant information, and this caused failure to identify the need, an irrelevant response, wrong issue classification, or a wrong resolution caused by misunderstanding. Do not deduct when clarification was caused by mumbling, low customer volume, unclear audio, background noise, broken connection, incomplete statements, customer contradiction, or a changed request. Return only: Yes/No",
"mark_distribution": "If probing was unnecessary or the agent obtained enough relevant information where needed, give full marks: 5. If necessary information was not obtained and this caused misunderstanding or wrong issue identification/resolution, give 0. No partial marks. Return only score:",
"detail": "Judge whether the intent was sufficiently understood. Do not back-infer failure from q_2_3 or from the number of questions asked."
},
"q_2_2": {
"evaluator_selection_option": "Did the agent use the required system or security probing? Current-phase rule: return N/A unless Robi system, API, or security-verification metadata is supplied. Do not infer system-side activity from transcript alone. If reliable metadata is later supplied, return Yes when required verification was completed correctly and No when it was required but missing or incorrect. Return only: Yes/No/N/A",
"mark_distribution": "Without Robi system/API/security metadata, give 0 and mark N/A, excluding 5 marks from the applicable denominator. If auditable and correct, give full marks: 5. If auditable and failed, give 0. No partial marks. Return only score:",
"detail": "Current phase: N/A unless Robi-side system/security-verification metadata is supplied."
},
"q_2_3": {
"evaluator_selection_option": "Did the agent provide complete and correct information for the customer's exact use case and sub-use case? Before scoring, check the matching Knowledge Base section. Judge correct meaning, not exact wording. Return Yes when all mandatory and triggered conditional information is correct, including process, charge, eligibility, documents, timeline, channel, and required next step. Return No when mandatory information is missing, wrong, contradictory, misleading, unsupported, or remains unresolved after correction. Do not apply full-call fatal scoring. Do not use this result to automatically fail q_2_1 or q_2_4. Mandatory dependency: if q_2_3 is No, q_2_5 must automatically be Yes with 4 marks. Return only: Yes/No",
"mark_distribution": "If the agent provided complete, correct, SOP/KB-aligned information in essence, give full marks: 15. If mandatory information was missing, wrong, contradictory, misleading, or unresolved, give 0. No partial marks. A No here automatically sets q_2_5 to Yes/4 but does not automatically fail other questions. Return only score:",
"detail": "Identify the exact sub-use case and check the KB. Distinguish mandatory, conditional, and additional information."
},
"q_2_4": {
"evaluator_selection_option": "Did the agent handle the core issue with expertise in a timely and organized manner? Evaluate independently from q_2_3 and from whether the customer received the preferred outcome. Return Yes if the agent understood the main issue, followed a logical flow, responded efficiently, communicated confidently, addressed the core concern, and handled the interaction professionally. If requested information or action was genuinely unavailable at the agent's level, return Yes when the agent clearly explained the limitation, did not fabricate information, provided the best feasible approved next step, and avoided unnecessary delay. Customer dissatisfaction or an unfavorable answer alone does not cause failure. Return No when handling was highly disorganized, contradictory, unnecessarily prolonged, clearly ineffective, materially confusing, a vague deflection without explanation, an avoidable failure to use an available resolution, or an irrelevant redirect that did not help the customer proceed. Do not automatically return No because q_2_3 was No. Return only: Yes/No",
"mark_distribution": "If the call was handled competently, logically, efficiently, and professionally, including a valid best-available resolution where necessary, give full marks: 6. If handling was clearly disorganized, contradictory, unresolved through avoidable agent failure, inefficient, or materially confusing, give 0. No partial marks. Return only score:",
"detail": "Judge overall handling separately from factual completeness and customer satisfaction. State whether the agent resolved the issue or provided the best feasible next step."
},
"q_2_5": {
"evaluator_selection_option": "Did the agent provide necessary additional information? Evaluate only after q_2_3. If q_2_3 is No, return Yes automatically and do not evaluate additional information separately. If q_2_3 is Yes, return Yes when relevant additional information was provided, such as app guidance, USSD, SMS format, service-center locator, Messenger chatbot, alternative channel, self-care option, Doorstep contact, or useful follow-up guidance. If q_2_3 is Yes and clearly required additional information was missed, return No. Do not require irrelevant generic promotions. Return only: Yes/No",
"mark_distribution": "If q_2_3 is No, give full marks: 4 automatically. If q_2_3 is Yes and necessary additional information was provided, give full marks: 4. If q_2_3 is Yes and required additional information was missing, give 0. No partial marks. Return only score:",
"detail": "Inspect q_2_3 first. q_2_3 No means q_2_5 Yes/4 automatically. Evaluate normally only when q_2_3 is Yes."
},
"q_3_1": {
"evaluator_selection_option": "Did the agent actively listen to the customer? Return Yes when the agent understood and responded to the actual issue. The original explanation is not repetition, and one repeat of the same clear issue is allowed. Return No when the customer had to repeat the same clearly stated core issue twice or more after the original explanation because the agent was not understanding. Count only repetition caused by agent misunderstanding. Do not deduct when clarification was caused by mumbling, low customer volume, unclear words, background noise, poor connection, incomplete statements, customer contradiction, or a changed request. Return only: Yes/No",
"mark_distribution": "If the agent listened appropriately and the customer did not have to repeat the same clear issue twice or more because of agent misunderstanding, give full marks: 3. If that threshold was crossed, give 0. No partial marks. Return only score:",
"detail": "One repetition is allowed. Distinguish agent misunderstanding from unclear customer speech or poor audio."
},
"q_3_2": {
"evaluator_selection_option": "Did the agent avoid disruptively interrupting the customer? Any confirmed disruptive interruption results in No, but overlap alone does not prove interruption. First determine conversational floor ownership. Do not count interruption when the customer completed a question or thought, a natural response opportunity occurred, the agent began a relevant answer, and the customer then resumed or added a point. Do not count brief backchannels such as জি, হুম, আচ্ছা, or ঠিক আছে when they do not stop, redirect, or disrupt the customer. Return No when the agent starts a substantial response before the customer completes the active thought, cuts off the customer, takes the floor away, repeatedly talks over the customer, or causes the customer to stop, restart, or object. Use audio overlap, speaker timestamps, semantic completion, pauses, customer objection, and conversational repair where available. Do not infer interruption only from transcript turn ordering. Return only: Yes/No",
"mark_distribution": "If no confirmed disruptive interruption occurs, give full marks: 3. If even one confirmed disruptive agent interruption occurs, give 0. No partial marks. Return only score:",
"detail": "State whether overlap was a natural turn transition/backchannel or a confirmed floor-taking interruption."
},
"q_3_3": {
"evaluator_selection_option": "Did the agent show friendliness, positive tonality, empathy, and courtesy? Return Yes when the agent remains polite, respectful, patient, friendly, and empathetic when required. Empathy triggers include lost or stolen SIM, unauthorized activity, service failure, failed balance transfer or recharge, financial loss, repeated inconvenience, prolonged disruption, death or bereavement, or customer distress. Return No if the agent is rude, arrogant, dismissive, impatient, dictating, or a clear empathy trigger occurs and the agent provides no apology, concern, reassurance, or acknowledgment. Do not require apology for a neutral informational question with no inconvenience. Return only: Yes/No",
"mark_distribution": "If the agent demonstrates appropriate friendliness, courtesy, and required empathy, give full marks: 13. If the agent is rude or dismissive, or misses a clear empathy requirement, give 0. No partial marks. Return only score:",
"detail": "Evaluate politeness across the call and whether empathy was shown when the situation required it."
},
"q_3_4": {
"evaluator_selection_option": "Did the agent use magic words appropriately at the required opportunities? First identify all ordinary and critical magic-word opportunities. Valid expressions include kindly, please, sorry, thank you, কাইন্ডলি, অনুগ্রহ করে, দুঃখিত, আন্তরিকভাবে দুঃখিত, ধন্যবাদ, সহযোগিতার জন্য ধন্যবাদ, মোস্ট ওয়েলকাম, and ভালো থাকবেন. Sir or Madam alone does not count. Critical opportunities include correcting previously wrong information, customer financial loss or service failure, lost or stolen SIM, unauthorized activity, serious inconvenience, bereavement or distress, and returning after keeping the customer waiting. Missing even one critical opportunity results in No regardless of magic words used elsewhere. If no critical opportunity was missed, return Yes only when a strict majority of all identified opportunities was appropriately covered: 1/1, 2/2, 2/3, 3/4, 3/5, 4/6, and so on. Magic words must be connected to the relevant opportunity; unrelated usage does not compensate. If no valid magic word appears anywhere, return No. q_3_4 is never N/A. Return only: Yes/No",
"mark_distribution": "Give full marks: 2 only when at least one valid magic word appears, no critical opportunity was missed, and more than half of all identified opportunities were appropriately covered. Give 0 when no magic word appears, any critical opportunity is missed, or coverage is half or less. No partial marks and no N/A. Return only score:",
"detail": "State total opportunities, covered opportunities, and whether a critical opportunity was missed."
},
"q_3_5": {
"evaluator_selection_option": "Did the agent maintain appropriate energy, liveliness, and confidence throughout the call? Use audio when available. Return Yes when the agent is generally spontaneous, energetic, clear, audible, confident, attentive, responsive, and appropriately paced. Return No when reliable evidence shows an unexplained response delay above 8 seconds, lethargic or lifeless delivery, consistently low or difficult-to-hear voice, unclear delivery, excessive hesitation, repeated self-correction, lack of confidence, unnecessarily rushed delivery, consistently unwelcoming delivery, or disorganized speech caused by uncertainty. Measure delay from the end of the customer's completed turn to the beginning of the agent's meaningful response and exclude announced system checking, hold, IVR, promotional audio, continuing customer speech, and technical interruption. Do not infer delay from overall silence ratio. Do not automatically fail because q_2_3 failed. Return only: Yes/No",
"mark_distribution": "If the agent maintains suitable energy, responsiveness, audibility, liveliness, and confidence, give full marks: 6. If reliable evidence shows a material failure, give 0. No partial marks. Return only score:",
"detail": "Use call-wide audio evidence. q_1_2 covers only the greeting. Generic duration, RMS, and silence ratio cannot independently prove response delay."
},
"q_3_6": {
"evaluator_selection_option": "Did the agent demonstrate proper sentence construction and correct pronunciation? Apply Robi's strict standard. Return No if, at any material point, sentence construction becomes undisciplined, unorganized, broken, improperly formed, incoherent, confusing, or excessively repetitive. Return No for pronunciation when reliable audio evidence shows that a key word, code, number, or instruction is unclear; the customer asks for repetition because of pronunciation; the customer mishears it; the agent must correct or repeat it; or the instruction becomes unusable. Do not treat ASR error alone as proof. If audio is unavailable, evaluate sentence construction from transcript and use customer repair behavior as pronunciation evidence only when clear. Failure in either sentence construction or pronunciation results in No. Return only: Yes/No",
"mark_distribution": "If sentence construction is properly formed and pronunciation is acceptable where auditable, give full marks: 10. If either dimension materially fails, give 0. No partial marks. Return only score:",
"detail": "Separate genuine speech problems from ASR errors. Customer requests to repeat a mispronounced key instruction are strong evidence."
},
"q_3_7": {
"evaluator_selection_option": "Did the agent keep the rate of speech at or below 350 words per minute? This is a deterministic WPM-only criterion in the current phase. Calculate agent_wpm = agent_word_count / (agent_speaking_duration_seconds / 60), using only the detected agent's speech and excluding customer speech, silence, hold music, IVR, promotional audio, and other speakers. If precomputed values are absent, calculate from speaker-labelled timestamped agent segments. Do not calculate from total call duration, RMS, Peak dBFS, overall silence ratio, or combined speaker duration. Return Yes when agent_wpm is less than or equal to 350. Return No when agent_wpm is greater than 350. Never return N/A. If exact timing is unavailable but original audio is directly available, estimate agent-only active speaking rate and return No only when clearly above 350; otherwise return Yes under the current-phase fallback. Customer comprehension does not affect this criterion. Return only: Yes/No",
"mark_distribution": "If agent WPM is less than or equal to 350, give full marks: 10. If agent WPM is greater than 350, give 0. Never use N/A and never use partial marks. Return only score:",
"detail": "Show agent word count, agent speaking duration in seconds, calculated or estimated WPM, and comparison with the 350-WPM threshold."
},
"q_3_8": {
"evaluator_selection_option": "Did the agent avoid unexplained internal jargon? Return Yes when customer-friendly language is used or a necessary technical term is explained simply. Return No when the agent exposes internal terms such as dCCRM, ICMS, CQC, MCA, internal dashboard names, CRM or platform names, network dashboard, internal process codes, or similar terms without explanation. Do not fail for a standard customer-facing term that is properly explained. Return only: Yes/No",
"mark_distribution": "If no unexplained internal jargon was used, give full marks: 2. If unexplained internal jargon was used, give 0. No partial marks. Return only score:",
"detail": "Judge whether internal system or platform terminology was exposed without a customer-friendly explanation."
},
"q_3_9": {
"evaluator_selection_option": "Did the agent personalize the call according to Robi's current definition? Return Yes if the agent appropriately addresses the customer using Sir, Madam, স্যার, or ম্যাডাম according to apparent gender or context. If gender is unclear, an appropriate respectful address is acceptable. Return No if no appropriate respectful address is used. Return only: Yes/No",
"mark_distribution": "If an appropriate Sir or Madam or স্যার or ম্যাডাম address is used, give full marks: 2. If not, give 0. No partial marks. Return only score:",
"detail": "Apply Robi's simplified personalization definition. This is separate from magic-word scoring."
},
"q_3_10": {
"evaluator_selection_option": "Did the agent follow the correct hold or transfer protocol? Current-phase rule: always return N/A. Robi has temporarily excluded this question until Robi-side system data or an approved hold-detection process becomes available. Return only: N/A",
"mark_distribution": "Give 0 and mark N/A, excluding 4 marks from the applicable denominator. Do not return Yes or No during the current phase. Return only score:",
"detail": "Current-phase N/A. Hold or transfer protocol is temporarily excluded by Robi."
},
"q_4_1": {
"evaluator_selection_option": "Did the agent check comprehension or was comprehension otherwise explicitly confirmed before closure? Return Yes when the agent asks whether the customer understood, whether more information is needed, or whether further assistance can be provided. Also return Yes when the customer voluntarily and unambiguously confirms understanding, such as বুঝতে পেরেছি, ঠিক আছে বুঝেছি, আমার প্রশ্নের উত্তর পেয়েছি, or ওকে বুঝলাম, and then clearly initiates closure; do not require a redundant agent question. Current-phase disconnect assumption: if the transcript ends without any agent closing statement, assume the customer disconnected first and return Yes because the agent had no opportunity. Return No when the agent had a normal opportunity, the customer did not explicitly confirm understanding, and the agent completed a closing without checking comprehension or offering further assistance. Return only: Yes/No",
"mark_distribution": "If an agent comprehension or further-assistance check is present, the customer explicitly confirms understanding and initiates closure, or no agent closing exists under the current-phase disconnect assumption, give full marks: 1. Otherwise give 0. No partial marks. Return only score:",
"detail": "State whether the mark came from an agent check, explicit customer confirmation, or the current-phase disconnect assumption."
},
"q_4_2": {
"evaluator_selection_option": "Did the agent close appropriately, politely, clearly, and professionally? Exact Robi script wording is not required. Accept natural professional variations, including thank you and goodbye, thank you and stay well, a polite salam, আপনার সঙ্গে কথা বলে ভালো লাগলো, ধন্যবাদ ভালো থাকবেন, or another clear professional close. Return No if the agent had an opportunity but the closing was rude, abrupt, careless, incomplete, confusing, or missing. Current-phase disconnect assumption: if no agent closing statement appears at the end, assume the customer disconnected first and return Yes automatically. Return only: Yes/No",
"mark_distribution": "If a professional closing variation is present, or the current-phase customer-disconnect assumption applies, give full marks: 1. If the agent had an opportunity but closing was unprofessional or missing, give 0. No partial marks. Return only score:",
"detail": "Judge professionalism, not exact script matching. No agent closing currently means customer disconnected first."
},
"q_5_1": {
"evaluator_selection_option": "Did the agent capture the correct information in CRM, CQC, or wrap-up? Current-phase rule: return N/A unless Robi supplies CRM, CQC, or wrap-up metadata. Do not infer CRM capture from transcript. When reliable metadata is supplied, return Yes only if the selected category accurately matches the customer's actual issue and No if it is wrong, mismatched, or incomplete. Return only: Yes/No/N/A",
"mark_distribution": "Without Robi CRM, CQC, or wrap-up metadata, give 0 and mark N/A, excluding 5 marks from the applicable denominator. If auditable and correct, give full marks: 5. If auditable and incorrect, give 0. No partial marks. Return only score:",
"detail": "Current phase: N/A unless Robi provides CRM, CQC, or wrap-up metadata."
}
}
Evaluation JSON (result)
{
"q_1_1": {
"evaluator_selection_option": "Yes",
"mark_distribution": 1,
"detail": "Opening contains all 3 required meanings: greeting ('আসসালামু আলাইকুম'), agent identity ('আমি মাহিনা বলছি'), and assistance offer ('কীভাবে সহযোগিতা করতে পারি')."
},
"q_1_2": {
"evaluator_selection_option": "Yes",
"mark_distribution": 2,
"detail": "Opening is respectful and welcoming; the wording is attentive and service-oriented with a polite assistance offer."
},
"q_2_1": {
"evaluator_selection_option": "No",
"mark_distribution": 0,
"detail": "The customer's need was not understood effectively. After one clarification, the customer again explained wanting contact/connection with the service desk person, but the agent mainly repeated 'গিয়ে যোগাযোগ করতে হবে' without sufficiently probing the actual issue or possible resolution path."
},
"q_2_2": {
"evaluator_selection_option": "N/A",
"mark_distribution": "-",
"detail": "N/A in current phase because no Robi system/API/security-verification metadata was provided."
},
"q_2_3": {
"evaluator_selection_option": "No",
"mark_distribution": 0,
"detail": "For the actual issue (customer left SIM at a Robi care/service desk and wanted help contacting that point), the agent only said call transfer/contact was not possible and to go there again. The response was incomplete because no meaningful next-step guidance or relevant service channel alternative was provided beyond repeating in-person contact."
},
"q_2_4": {
"evaluator_selection_option": "No",
"mark_distribution": 0,
"detail": "Handling was not well organized or effective. The agent showed limited understanding of the core concern, repeated the same direction, and did not move the call toward a practical resolution."
},
"q_2_5": {
"evaluator_selection_option": "Yes",
"mark_distribution": 4,
"detail": "Auto-Yes under the anti-double-penalty rule because q_2_3 is No."
},
"q_3_1": {
"evaluator_selection_option": "Yes",
"mark_distribution": 3,
"detail": "There was one initial misunderstanding ('বুঝতে পারিনি স্যার'), but this stayed within the allowed threshold. The agent did engage with the customer's issue afterward."
},
"q_3_2": {
"evaluator_selection_option": "Yes",
"mark_distribution": 3,
"detail": "No clear inappropriate interruption is established. Brief interjections like 'জি?' did not materially prevent the customer from speaking."
},
"q_3_3": {
"evaluator_selection_option": "Yes",
"mark_distribution": 13,
"detail": "The agent remained polite and courteous throughout, using respectful address ('স্যার') and empathy/apology phrases such as 'খুবই দুঃখিত স্যার' when the customer described the SIM problem."
},
"q_3_4": {
"evaluator_selection_option": "Yes",
"mark_distribution": 2,
"detail": "Valid magic words were used, including 'কাইন্ডলি' and 'খুবই দুঃখিত' and 'ভালো থাকবেন'."
},
"q_3_5": {
"evaluator_selection_option": "Yes",
"mark_distribution": 6,
"detail": "The agent was generally responsive and confident in delivery. No reliable evidence of >8 second unexplained response delay, lethargic tone, or major hesitation appears from the transcript/timestamps."
},
"q_3_6": {
"evaluator_selection_option": "Yes",
"mark_distribution": 10,
"detail": "Sentence construction was generally understandable and usable for the conversation. No reliable evidence shows a pronunciation failure that made key information unusable."
},
"q_3_7": {
"evaluator_selection_option": "N/A",
"mark_distribution": 0,
"detail": "N/A because reliable agent-level audio evidence or agent-only speaking-duration/word-count data for pace, pitch, and WPM assessment was not provided. Overall call duration/silence ratio cannot be used."
},
"q_3_8": {
"evaluator_selection_option": "Yes",
"mark_distribution": 2,
"detail": "The agent did not use unexplained internal jargon such as CRM/CQC/system codes; language remained customer-friendly."
},
"q_3_9": {
"evaluator_selection_option": "Yes",
"mark_distribution": 2,
"detail": "The agent personalized the call with respectful address, repeatedly using 'স্যার'."
},
"q_3_10": {
"evaluator_selection_option": "N/A",
"mark_distribution": "-",
"detail": "N/A in the current phase. Hold/transfer protocol is temporarily excluded by Robi."
},
"q_4_1": {
"evaluator_selection_option": "No",
"mark_distribution": 0,
"detail": "The agent had a normal opportunity to close and did close, but did not ask any comprehension or further-assistance question such as 'আর কোনো তথ্য দিয়ে সহযোগিতা করতে পারি?'."
},
"q_4_2": {
"evaluator_selection_option": "Yes",
"mark_distribution": 1,
"detail": "Closing was polite and professional: 'কথা বলে ভালো লাগলো। ভালো থাকবেন। আসসালামু আলাইকুম।'"
},
"q_5_1": {
"evaluator_selection_option": "N/A",
"mark_distribution": "-",
"detail": "N/A in current phase because no CRM/CQC/wrap-up metadata was provided."
}
}