Overview

The industry that has undergone the most disruption with the rise of Generative AI is EdTech. While these advancements have caused concerns they also bring huge opportunities. Instead of fearing these changes, EdTech companies can embrace GenAI to improve learning experiences, save time, and enhance engagement.

Role of GenAI in EdTech

• Faster content creation by using GenAI tools, reducing manual efforts for educators.

• Personalized learning by analyzing student’s performance to provide tailored learning paths.

• Better engaging experiences by gamified learning and instant feedback.

• GenAI can offer 24/7 support in answering student’s queries or explaining complex topics.

• Providing instant automated grading and feedback on assignments.

• Breaking the language barrier with instant real-time translation.

and so on.

Understanding EdTech’s KPI

To quantify success in any EdTech company, different Key Performing Indicators (KPI) are used. These KPIs can include, but are not limited to:

1. Customer Satisfaction and Engagement

• Measures how satisfied and engaged users are with the platform through metrics like feedback, ratings, and active usage.

• High satisfaction and engagement levels indicate that the platform meets user needs, fostering loyalty and increasing the likelihood of referrals.

2. Learning Outcomes

• Tracks the knowledge and skills acquired by users through scores, completion rates, and progress metrics.

• Demonstrates the platform’s effectiveness in achieving educational goals, improving its value proposition for users and stakeholders.

3. User Retention

• Measures how well the platform retains its users over time by using retention and churn rates.

• High retention rates reflect user trust and satisfaction, ensuring consistent revenue streams and lower acquisition costs.

4. Product Quality

• Measures the usability, functionality, and overall value of the product through metrics like usability tests, feature usage, and error reports.

• High product quality ensures a seamless user experience, builds credibility, and reduces operational costs from recurring issues.

Problem Statement

While EdTech platforms strive to achieve KPIs like customer satisfaction, learning outcomes, and retention, they face significant challenges when leveraging Generative AI. Validating the AI generated educational content, for accuracy, context alignment, and engagement is a time-intensive process that often lacks scalability and consistency. The need for an efficient, automated solution to streamline content evaluation is critical to helping EdTech platforms maintain quality, scale operations, and meet their KPIs effectively.

What Future AGI Offers

Future AGI offers EdTech platforms tools that not only streamline processes but also elevate their key performance metrics. These tools can be accessed through Future AGI’s interactive dashboard or through SDK. Let’s now see what Future AGI has to offer to EdTech sector to improve their KPI:

Consider an EdTech company that collaborates with educators to transform their educational materials into engaging quizzes for students. An educator might provide notes or lesson plans, and this EdTech platform generates quizzes tailored to specific topics. The goal is to ensure that these quizzes are accurate, engaging, and aligned with the intended learning objectives. However, validating the quality and relevance of these generated quizzes can be a daunting task without robust evaluation tools.

This is where Future AGI steps in, offering a comprehensive evaluation suite that ensures quiz content meets high standards of quality, relevance, and engagement. By identifying errors, ensuring contextual alignment, and validating answer relevance, Future AGI tools empower EdTech platforms to monitor and improve their key performance metrics (KPIs).

The dataset used in this case study is a sample data simulating the EdTech workflow we talked about, which is transforming educator-provided lesson notes or material into AI-generated quizzes with questions, options, and correct option.

1. Context Adherence Evaluation

When quizzes are generated, ensuring that the questions align with the educational material is crucial. Future AGI’s Context Adherence Evaluation examines whether the quiz questions faithfully reflect the context provided by the educator. Future AGI evaluates the question against the provided context to ensure it neither deviates from the topic nor introduces unrelated information. This evaluation prevents student confusion and ensures the platform maintains its credibility, positively impacting KPIs like customer satisfaction and learning outcomes.

from fi.testcases import TestCase
from fi.evals.templates import ContextAdherence

adherence_scores = []
adherence_reasons = []
def evaluate_context_adherence(dataset, context_column, question_column):
    for _, row in dataset.iterrows():
        test_case = TestCase(
            output=row["question"],
            context=row["context"]
        )
        adherence_template = ContextAdherence(config={
            "required_keys": ["output", "context"],
            "output": "adherence_score",
        })

        response = evaluator.evaluate(eval_templates=[adherence_template], inputs=[test_case])

        adherence_result = response.eval_results[0].metrics[0].value
        reason = response.eval_results[0].reason

        adherence_scores.append(adherence_result)
        adherence_reasons.append(reason)

    dataset["context_adherence"] = adherence_scores
    dataset["adherence_reason"] = adherence_reasons
    return dataset

quiz_data_with_adherence = evaluate_context_adherence(
    dataset=dataset,
    context_column="context",
    question_column="question"
)

2. Grammatical Error Deterministic Evaluation

Poor grammar or awkward phrasing can reduce credibility of the content. Future AGI's deterministic eval lets you create custom evaluation metric for grammatical errors, where you can assesses the language quality of each quiz question and its context.

By ensuring clear and professional language, the platform enhances user satisfaction and retention, as students and educators find the platform more reliable and effective.

from fi.testcases import MLLMTestCase

class DeterministicTestCase(MLLMTestCase):
    context: str
    question: str

from fi.evals import Deterministic

deterministic_eval = Deterministic(config={
    "multi_choice": False,
    "choices": ["Pass", "Fail"],
    "rule_prompt": "context : {{input_key1}}, question : {{input_key2}}. Given the context and question, choose Pass if the question is grammatically correct, well-structured, and free of errors; choose Fail otherwise",
        "input": {
        "input_key1": "context",
        "input_key2": "question",
    }
})

complete_result = {}

options = []
reasons = []
for index, row in dataset.iterrows():
  test_case = DeterministicTestCase(
        context=row["context"],
        question=row["question"]
  )
  result = evaluator.evaluate([deterministic_eval], [test_case])
  option = result.eval_results[0].data[0]
  reason = result.eval_results[0].reason
  options.append(option)
  reasons.append(reason)

complete_result["Grammatical-Error-Eval-Rating"] = options
complete_result["Grammatical-Error-Eval-Reason"] = reasons

complete_result_df = pd.DataFrame(complete_result)

new_dataset = pd.concat([dataset, complete_result_df], axis=1)

3. Answer Relevance Evaluation

What if the quiz options or answers provided aren’t relevant to the context? This is where Future AGI's Answer Relevance Evaluation comes in. It checks whether the answer is logically connected to the provided context and query. This boosts learning outcomes by eliminating misleading or irrelevant options, fostering a more productive learning environment.

Just like above evals, you do this using SDK. But lets see how you can use Future AGI’s interactive dashboard for it. Select the Answer Relevance eval from evaluate section and fill in the required inputs. After that, choose what model you are going to be using to perform this eval, here we are using gpt-4o-mini. Then click on “Save and Run”

Evaluation is now added successfully and is running.

Impact

• Faster Turnaround Time: Future AGI reduces quiz validation time by 10x compared to manual processes.

• Reduction in Human Effort: The platform minimises reliance on manual validation by an estimate of 80%, significantly decreasing operational costs.

• Scalability: Enables evaluation of thousands of quizzes daily while maintaining consistent quality standards.

• Improved Quality: Ensures accuracy, contextual alignment, and answer relevance in quiz content.

Conclusion

Future AGI empowers EdTech companies to enhance quiz validation with efficiency, scalability, and quality. By streamlining processes, it boosts user engagement and learning outcomes, helping platforms confidently achieve their KPIs and deliver innovative, high-quality educational experiences.