Overview

AI-powered image generation companies face increasing demands for high-quality, unique, and visually striking outputs across diverse use cases such as marketing, entertainment, and virtual experiences. The success of these outputs hinges on the effectiveness of the prompts used to guide AI models like Stable Diffusion, DALL·E, and MidJourney. However, refining prompts to produce consistent, creative, and relevant results is an iterative process that is often subjective, labor-intensive, and difficult to scale.

Future AGI addresses this challenge with an AI evaluation platform that streamlines the process of image output optimization. By automating evaluation, analysis, and improvement of AI-generated images, Future AGI empowers companies to:

1. Objectively evaluate generated images based on criteria such as quality, visual relevance, and creativity.

2. Identify the best-performing prompts and models to achieve desired outcomes efficiently.

3. Gain actionable insights to refine poorly performing prompts and enhance image outputs.

4. Compare results across models, prompt iterations, and project versions using a unified dashboard.

5. Enable a scalable feedback loop to continually improve AI outputs based on real-world performance metrics.

This case study demonstrates how Future AGI enables AI image generation companies to elevate their creative workflows, achieve higher quality outputs, and accelerate experimentation at scale.

Problem Statement

AI-powered comic generation companies use generative models to create visually appealing comic panels based on textual descriptions. However, evaluating the quality and relevance of generated images remains a challenge, especially when scaling up production. An AI comic generation company faced the following challenges:

1. Alignment with Descriptions: Ensuring that generated images accurately represent the textual descriptions provided.

2. Meeting Specific Requirements: Evaluating images against predefined artistic or thematic standards.

3. Evaluating Image Replacements: Assessing the quality and fit of new images replacing previous versions.

4. Scaling Evaluations: Performing these evaluations manually was time-consuming and error-prone.

Solution Provided by Future AGI

Future AGI provides a suite of evaluation features accessible through our app with a dashboard and SDK. Here, we demonstrate how these evaluations can be leveraged via the SDK to address the company’s challenges through three key functionalities:

1. Evaluation of Image-Text Alignment

The SDK evaluates whether a generated image aligns with its textual description.

Using the ImageInstruction class, images were evaluated based on the following criteria:

• Accuracy of object representation.

• Setting accuracy.

• Image quality and realism.

Example Input:

{'id': masked,
 'image_url': masked,
 'prompt': 'an asian man, closeup, on new york city street',
 'type': 'T2I',
 'category': 'Ethnicity',
 'question': 'Does the image follow the Ethnicity mentioned in the prompt?',
 'output_image_url': masked}

Image:

Description:

an asian man, closeup, on new york city street

Code Snippet:

image_eval = ImageInstruction( config={ "criteria": """ Evaluate the image based on: 1. Accuracy of object representation 2. Setting accuracy 3. Image quality and realism """ } ) class ImageEvalTestCase(MLLMTestCase): input: str image_url: str test_case_img_eval = ImageEvalTestCase( input=datapoint['prompt'], image_url=datapoint['image_url'] ) batch_result = evaluator.evaluate([image_eval], [test_case_img_eval]) print(batch_result.eval_results[0].reason)

Example Result:

The image accurately represents an Asian man in a close-up on a New York City street with realistic details and setting elements like taxis and street signs.

2. Evaluation Against Specific Requirements

Objective: Verify if the generated image adheres to subjective requirements, such as ethnicity, attire, or location.

This evaluation used the Deterministic class with criteria focused on specific categories.

Code Snippet:

deterministic_eval = Deterministic(config={
    "multi_choice": False,
    "choices": ["Yes", "No"],
    "rule_prompt": "Prompt : {{input_key2}}, Image : {{input_key3}}. Given the prompt and the corresponding image, answer the Question : {{input_key1}}. Focus only on the {{input_key4}}",
    "input": {
        "input_key1": "question",
        "input_key2": "prompt",
        "input_key3": "image_url",
        "input_key4": "category"
    }
})
class DeterministicTestCase(MLLMTestCase):
    question: str
    prompt: str
    image_url: str
    category: str
test_case = DeterministicTestCase(
    question=datapoint['question'],
    prompt=datapoint['prompt'],
    image_url=datapoint['image_url'],
    category=datapoint['category']
)
batch_result = evaluator.evaluate([deterministic_eval], [test_case])
print(batch_result.eval_results[0].reason)

Example Result:

The image aligns with the specified Asian ethnicity based on visual features.

3. Image Replacement Evaluation

Objective: Validate if an input image was modified correctly based on textual instructions.

This task leveraged the ImageInputOutput class, ensuring adherence to input instructions and preservation of key elements.

Example Output:

Image:

Conversion:

Replace the man with a man of african ethinicity

Code Snippet:

image_input_output_eval = ImageInputOutput(config={
    "criteria": """
    Evaluate the output image based on:
        1. Adherence to input instruction
        2. Preservation of key elements from input image
        3. Quality of color modification
        4. Image quality and realism
    """
})
class ImageInputOutputTestCase(MLLMTestCase):
    input_text: str
    input_image_url: str
    output_image_url: str
test_case_image_input_output = ImageInputOutputTestCase(
    input_text = datapoint['input_text'],
    input_image_url=datapoint['image_url'],
    output_image_url=datapoint['output_image_url']
)
batch_result = evaluator.evaluate([image_input_output_eval], [test_case_image_input_output])
print(batch_result.eval_results[0].reason)

Example Result:

The output image accurately replaces the man with one of African ethnicity while preserving key elements like the background and attire, with natural color modifications and high image quality.

4. Scalable Evaluations with Dashboarding

Future AGI’s SDK integrates with a dashboard to:

• Visualize evaluation metrics across multiple models and generations on our dashboard.

• Compare different prompts and their outputs.

• Track image generation performance over time.

More examples:

Key Results

By adopting Future AGI’s SDK, the comic generation company achieved the following:

1. Improved Accuracy: Automated evaluation ensured better alignment between images and descriptions, with an estimated 10x faster turn around time.

2. Increased Efficiency: Reduced manual effort by 85%, enabling faster iteration cycles.

3. Enhanced Quality: Identified the best artistic prompts and ensured consistent adherence to custom requirements improving overall image quality by an estimated 10%.

4. Scalability: Enabled evaluation of thousands of images daily, achieving a scale 5-10x higher than their previous manual process.

Conclusion

Future AGI’s SDK empowers AI comic generation companies to optimize their workflows, ensuring high-quality outputs at scale. By automating image evaluation, companies can focus on creativity while maintaining rigorous quality standards.

About Future AGI

Future AGI specializes in AI evaluation solutions that enable organizations to optimize their AI systems and workflows at scale. Our cutting-edge platform combines robust evaluation metrics, insightful analytics, and seamless integrations to empower businesses across industries.