Gemini API × Playwright — Building an AI-Powered E2E Test Generation System

Why Use AI to Auto-Generate E2E Tests?

End-to-end testing remains one of the most reliable methods for ensuring web application quality. Yet in practice, writing test code is expensive, and maintaining tests through UI changes creates a significant burden on development teams.

By combining Gemini API's powerful multimodal analysis capabilities with Playwright's browser automation, we can fundamentally solve this problem. The system we build below feeds web page screenshots and HTML structures to Gemini, generating test scenarios, producing test code, and even repairing tests on its own when the UI changes (self-healing).

Here's the technology stack we'll be working with:

Gemini 2.5 Pro: Core engine for page analysis and test code generation
Playwright: Cross-browser E2E test execution framework
Node.js / TypeScript: Implementation language for the entire system
GitHub Actions: CI/CD pipeline integration

This guide targets intermediate to advanced frontend/QA engineers who are already familiar with Playwright basics.

Architecture Overview

The system is composed of three distinct layers.

1. Page Analysis Layer (Page Analyzer)

Playwright navigates to the target page and captures screenshots, extracts the DOM structure, and retrieves the accessibility tree. This data is sent to the Gemini API, which identifies the page's components and possible user interactions.

2. Test Generation Layer (Test Generator)

Based on the analysis results, Gemini generates Playwright test code in TypeScript. Rather than producing simple snapshot tests, it constructs meaningful test scenarios that follow actual user flows.

3. Test Execution & Repair Layer (Test Runner & Self-Healer)

The generated tests are executed, and when failures occur, the error details along with the current page state are sent back to Gemini for automatic test code repair. This is the self-healing mechanism.

Environment Setup

Start by initializing the project:

mkdir gemini-e2e-generator && cd gemini-e2e-generator
npm init -y
npm install @google/generative-ai playwright typescript tsx
npx playwright install chromium
npx tsc --init

Configure tsconfig.json as follows:

{
  "compilerOptions": {
    "target": "ES2022",
    "module": "ESNext",
    "moduleResolution": "bundler",
    "strict": true,
    "outDir": "./dist",
    "rootDir": "./src",
    "esModuleInterop": true
  },
  "include": ["src/**/*"]
}

Set your API key in the .env file:

GEMINI_API_KEY=YOUR_GEMINI_API_KEY

Implementing the Page Analysis Module

First, create the module that collects information from target pages.

// src/page-analyzer.ts
import { chromium, type Page, type Browser } from "playwright";
 
interface PageAnalysis {
  url: string;
  title: string;
  screenshot: Buffer;
  htmlStructure: string;
  accessibilityTree: string;
  interactiveElements: InteractiveElement[];
}
 
interface InteractiveElement {
  role: string;
  name: string;
  selector: string;
  type?: string;
}
 
export class PageAnalyzer {
  private browser: Browser | null = null;
 
  async init(): Promise<void> {
    this.browser = await chromium.launch({ headless: true });
  }
 
  async analyze(url: string): Promise<PageAnalysis> {
    if (!this.browser) throw new Error("Browser not initialized");
 
    const context = await this.browser.newContext({
      viewport: { width: 1280, height: 720 },
    });
    const page = await context.newPage();
    await page.goto(url, { waitUntil: "networkidle" });
 
    // Capture screenshot
    const screenshot = await page.screenshot({ fullPage: true });
 
    // Extract simplified HTML structure (to save tokens)
    const htmlStructure = await page.evaluate(() => {
      function summarize(el: Element, depth: number): string {
        if (depth > 4) return "";
        const tag = el.tagName.toLowerCase();
        const id = el.id ? `#${el.id}` : "";
        const cls = el.className
          ? `.${String(el.className).split(" ").slice(0, 2).join(".")}`
          : "";
        const text = el.textContent?.trim().slice(0, 50) || "";
        const children = Array.from(el.children)
          .map((c) => summarize(c, depth + 1))
          .filter(Boolean)
          .join("\n");
        const indent = "  ".repeat(depth);
        return `${indent}<${tag}${id}${cls}>${text ? ` "${text}"` : ""}\n${children}`;
      }
      return summarize(document.body, 0);
    });
 
    // Retrieve accessibility tree
    const accessibilityTree = await this.getAccessibilityTree(page);
 
    // List interactive elements
    const interactiveElements = await this.getInteractiveElements(page);
 
    await context.close();
 
    return {
      url,
      title: await page.title(),
      screenshot,
      htmlStructure,
      accessibilityTree,
      interactiveElements,
    };
  }
 
  private async getAccessibilityTree(page: Page): Promise<string> {
    const snapshot = await page.accessibility.snapshot();
    return JSON.stringify(snapshot, null, 2).slice(0, 8000);
  }
 
  private async getInteractiveElements(
    page: Page
  ): Promise<InteractiveElement[]> {
    return page.evaluate(() => {
      const selectors = [
        "button",
        "a[href]",
        'input:not([type="hidden"])',
        "select",
        "textarea",
        '[role="button"]',
        '[role="link"]',
        '[role="tab"]',
      ];
      const elements: InteractiveElement[] = [];
 
      for (const selector of selectors) {
        document.querySelectorAll(selector).forEach((el) => {
          const htmlEl = el as HTMLElement;
          elements.push({
            role: el.getAttribute("role") || el.tagName.toLowerCase(),
            name:
              el.getAttribute("aria-label") ||
              htmlEl.innerText?.trim().slice(0, 60) ||
              "",
            selector: generateSelector(el),
            type: el.getAttribute("type") || undefined,
          });
        });
      }
 
      function generateSelector(el: Element): string {
        if (el.id) return `#${el.id}`;
        const testId = el.getAttribute("data-testid");
        if (testId) return `[data-testid="${testId}"]`;
        const ariaLabel = el.getAttribute("aria-label");
        if (ariaLabel)
          return `${el.tagName.toLowerCase()}[aria-label="${ariaLabel}"]`;
        return `${el.tagName.toLowerCase()}:text("${(el as HTMLElement).innerText?.trim().slice(0, 30)}")`;
      }
 
      return elements.slice(0, 50);
    });
  }
 
  async close(): Promise<void> {
    await this.browser?.close();
  }
}

The key design decision here is providing three types of information to Gemini: screenshots (visual information), simplified HTML structure (structural information), and the accessibility tree (semantic information). Combining these allows Gemini to accurately understand the page's intent and functionality.

Gemini-Powered Test Code Generation

Next, implement the module that sends collected page data to the Gemini API and generates test code.

// src/test-generator.ts
import { GoogleGenerativeAI } from "@google/generative-ai";
import type { PageAnalysis } from "./page-analyzer";
 
const genAI = new GoogleGenerativeAI(process.env.GEMINI_API_KEY!);
 
const SYSTEM_PROMPT = `You are an expert Playwright E2E test engineer.
Given information about a web page (screenshot, HTML structure, accessibility tree,
and interactive elements), generate comprehensive E2E test code.
 
## Test Code Generation Rules
 
1. Write tests using Playwright + TypeScript
2. Choose selectors in this priority order:
   - data-testid attributes (highest priority)
   - aria-label / role-based selectors
   - Text content-based selectors
   - CSS selectors (last resort)
3. Include clear descriptions for each test (describe/it)
4. Use specific assertions: toBeVisible, toHaveText, toHaveURL, etc.
5. Include proper waitForURL / waitForLoadState for navigation tests
6. Cover both happy paths and edge cases (empty inputs, invalid values)
7. Make tests independently executable (no inter-test dependencies)
 
## Output Format
 
Output only the TypeScript code block. No explanatory text.
Include a filename comment at the top (e.g., // tests/login.spec.ts).`;
 
export async function generateTests(
  analysis: PageAnalysis
): Promise<string> {
  const model = genAI.getGenerativeModel({
    model: "gemini-2.5-pro",
    systemInstruction: SYSTEM_PROMPT,
  });
 
  const prompt = `Generate E2E tests for the following web page.
 
## Page Information
- URL: ${analysis.url}
- Title: ${analysis.title}
 
## HTML Structure (simplified)
${analysis.htmlStructure.slice(0, 6000)}
 
## Accessibility Tree
${analysis.accessibilityTree.slice(0, 4000)}
 
## Interactive Elements
${JSON.stringify(analysis.interactiveElements, null, 2)}
 
Analyze the above information along with the screenshot comprehensively,
and generate test code that covers the primary user flows for this page.`;
 
  const result = await model.generateContent([
    prompt,
    {
      inlineData: {
        mimeType: "image/png",
        data: analysis.screenshot.toString("base64"),
      },
    },
  ]);
 
  const text = result.response.text();
 
  // Extract TypeScript code from code blocks
  const codeMatch = text.match(/```typescript\n([\s\S]*?)```/);
  return codeMatch ? codeMatch[1].trim() : text;
}

A critical detail here is the explicit selector priority order in the system prompt. By prioritizing data-testid, tests become resilient to visual UI changes. Additionally, passing the screenshot as multimodal input allows Gemini to understand visual layout and content intent that can't be determined from HTML structure alone.

Implementing Self-Healing (Auto-Repair) Functionality

The maintenance phase is where E2E tests consume the most resources. Even minor UI changes can break tests, requiring manual fixes. Here we implement a self-healing mechanism where Gemini automatically repairs test code when failures occur.

// src/self-healer.ts
import { GoogleGenerativeAI } from "@google/generative-ai";
import { PageAnalyzer } from "./page-analyzer";
import * as fs from "fs/promises";
 
const genAI = new GoogleGenerativeAI(process.env.GEMINI_API_KEY!);
 
interface TestFailure {
  testFile: string;
  errorMessage: string;
  failedLine: number;
  pageUrl: string;
}
 
export class SelfHealer {
  private analyzer = new PageAnalyzer();
  private maxRetries = 3;
 
  async heal(failure: TestFailure): Promise<string | null> {
    await this.analyzer.init();
 
    try {
      // Re-fetch current page state
      const currentState = await this.analyzer.analyze(failure.pageUrl);
 
      // Read the failing test code
      const testCode = await fs.readFile(failure.testFile, "utf-8");
 
      const model = genAI.getGenerativeModel({
        model: "gemini-2.5-pro",
      });
 
      const prompt = `The following Playwright test has failed.
Analyze the latest page state and fix the test code.
 
## Failure Details
- Error: ${failure.errorMessage}
- Failed line: ${failure.failedLine}
 
## Current Test Code
\`\`\`typescript
${testCode}
\`\`\`
 
## Latest Page State
- URL: ${currentState.url}
- Interactive elements:
${JSON.stringify(currentState.interactiveElements, null, 2)}
 
## Latest Accessibility Tree
${currentState.accessibilityTree.slice(0, 4000)}
 
## Repair Rules
1. If the fix only requires changing selectors, change only the selectors
2. If the page structure has changed significantly, update the test logic as well
3. If an element has been completely removed, mark that test as skip (test.skip)
4. Output the complete fixed code
 
Output only the fixed TypeScript code.`;
 
      const result = await model.generateContent([
        prompt,
        {
          inlineData: {
            mimeType: "image/png",
            data: currentState.screenshot.toString("base64"),
          },
        },
      ]);
 
      const text = result.response.text();
      const codeMatch = text.match(/```typescript\n([\s\S]*?)```/);
      return codeMatch ? codeMatch[1].trim() : null;
    } finally {
      await this.analyzer.close();
    }
  }
 
  async healAndRetry(failure: TestFailure): Promise<boolean> {
    for (let attempt = 1; attempt <= this.maxRetries; attempt++) {
      console.log(
        `🔧 Heal attempt ${attempt}/${this.maxRetries}: ${failure.testFile}`
      );
 
      const healed = await this.heal(failure);
      if (!healed) {
        console.log("❌ Could not generate repair code");
        continue;
      }
 
      // Write repaired code
      await fs.writeFile(failure.testFile, healed, "utf-8");
      console.log("📝 Test code updated");
 
      // Re-run to verify the fix
      const success = await this.runTest(failure.testFile);
      if (success) {
        console.log("✅ Self-healing succeeded!");
        return true;
      }
    }
 
    console.log("❌ Self-healing failed — manual fix required");
    return false;
  }
 
  private async runTest(testFile: string): Promise<boolean> {
    const { execSync } = await import("child_process");
    try {
      execSync(`npx playwright test ${testFile} --reporter=list`, {
        stdio: "pipe",
        timeout: 60000,
      });
      return true;
    } catch {
      return false;
    }
  }
}

The critical aspect of self-healing is re-fetching the latest page state before attempting repairs. Rather than simply examining the error message, the system performs multimodal analysis of the page's current structure, enabling it to handle UI refactoring and design changes effectively.

Main Orchestrator

Create the main script that ties all modules together.

// src/index.ts
import { PageAnalyzer } from "./page-analyzer";
import { generateTests } from "./test-generator";
import { SelfHealer } from "./self-healer";
import * as fs from "fs/promises";
import * as path from "path";
 
interface GeneratorConfig {
  urls: string[];
  outputDir: string;
  selfHeal: boolean;
}
 
async function main(config: GeneratorConfig): Promise<void> {
  const analyzer = new PageAnalyzer();
  await analyzer.init();
 
  // Create output directory
  await fs.mkdir(config.outputDir, { recursive: true });
 
  for (const url of config.urls) {
    console.log(`\n🔍 Analyzing page: ${url}`);
 
    try {
      // Step 1: Page analysis
      const analysis = await analyzer.analyze(url);
      console.log(
        `  📊 Elements detected: ${analysis.interactiveElements.length}`
      );
 
      // Step 2: Test code generation
      console.log("  🤖 Generating tests with Gemini...");
      const testCode = await generateTests(analysis);
 
      // Step 3: Write to file
      const slug = new URL(url).pathname
        .replace(/\//g, "-")
        .replace(/^-|-$/g, "")
        || "index";
      const testFile = path.join(
        config.outputDir,
        `${slug}.spec.ts`
      );
      await fs.writeFile(testFile, testCode, "utf-8");
      console.log(`  💾 Test file created: ${testFile}`);
 
      // Step 4: Run tests
      console.log("  ▶️ Running tests...");
      const { execSync } = await import("child_process");
      try {
        execSync(
          `npx playwright test ${testFile} --reporter=list`,
          { stdio: "inherit", timeout: 120000 }
        );
        console.log("  ✅ All tests passed");
      } catch {
        if (config.selfHeal) {
          console.log("  🔧 Starting self-healing...");
          const healer = new SelfHealer();
          await healer.healAndRetry({
            testFile,
            errorMessage: "Test execution failed",
            failedLine: 0,
            pageUrl: url,
          });
        }
      }
    } catch (error) {
      console.error(`  ❌ Error: ${error}`);
    }
  }
 
  await analyzer.close();
  console.log("\n🏁 All pages processed");
}
 
// Execute
main({
  urls: [
    "https://example.com",
    "https://example.com/login",
    "https://example.com/dashboard",
  ],
  outputDir: "./tests/generated",
  selfHeal: true,
});

Improving Test Generation Quality with Prompt Techniques

Several practical techniques help stabilize the quality of Gemini-generated tests.

Page Type-Specific Prompt Branching

Classify pages by type (login form, list page, detail page, etc.) beforehand and use optimized prompt templates for each.

// src/prompt-templates.ts
export const PROMPT_TEMPLATES: Record<string, string> = {
  form: `This page contains a form. Generate the following tests:
- Happy path: Fill all fields with valid values and submit
- Validation: Submit with empty required fields, verify error messages
- Boundary values: Maximum character lengths, special character inputs
- Accessibility: Tab key focus order verification`,
 
  list: `This is a list/index page. Generate the following tests:
- Pagination: Navigate to next/previous pages
- Sorting: Toggle sort on each column
- Filtering: Apply and clear filters
- Empty state: Verify display when no data exists
- Item click: Navigate to detail pages for each item`,
 
  auth: `This is an authentication page. Generate the following tests:
- Successful login: Authentication flow with valid credentials
- Error cases: Invalid password, non-existent user
- Validation: Email format checking
- Password reset: Reset flow navigation
- Security: Defense against XSS input`,
};
 
export function detectPageType(
  analysis: PageAnalysis
): string {
  const { interactiveElements, htmlStructure } = analysis;
  const formCount = interactiveElements.filter(
    (e) => e.role === "input" || e.role === "textarea"
  ).length;
 
  if (htmlStructure.includes("login") ||
      htmlStructure.includes("sign-in")) {
    return "auth";
  }
  if (formCount >= 3) return "form";
  if (htmlStructure.includes("pagination") ||
      interactiveElements.length > 15) {
    return "list";
  }
  return "general";
}

Controlling Test Structure with Function Calling

You can also use Gemini's Function Calling feature to strictly control test structure through a JSON schema.

// src/structured-generator.ts
import { GoogleGenerativeAI, type FunctionDeclaration } from "@google/generative-ai";
 
const testSuiteSchema: FunctionDeclaration = {
  name: "create_test_suite",
  description: "Generate a Playwright E2E test suite in structured format",
  parameters: {
    type: "object",
    properties: {
      fileName: {
        type: "string",
        description: "Test file name (e.g., login.spec.ts)",
      },
      imports: {
        type: "array",
        items: { type: "string" },
        description: "Required import statements",
      },
      testSuites: {
        type: "array",
        items: {
          type: "object",
          properties: {
            description: {
              type: "string",
              description: "Description for the describe block",
            },
            beforeEach: {
              type: "string",
              description: "beforeEach code (optional)",
            },
            tests: {
              type: "array",
              items: {
                type: "object",
                properties: {
                  name: { type: "string" },
                  priority: {
                    type: "string",
                    enum: ["critical", "high", "medium", "low"],
                  },
                  steps: {
                    type: "array",
                    items: { type: "string" },
                  },
                },
                required: ["name", "priority", "steps"],
              },
            },
          },
          required: ["description", "tests"],
        },
      },
    },
    required: ["fileName", "imports", "testSuites"],
  },
};

This approach instructs Gemini to output structured data conforming to a predefined schema, rather than generating free-form code. This significantly improves output consistency and simplifies the downstream code assembly process.

CI/CD Pipeline Integration

For production use, automating test generation and execution through GitHub Actions is ideal.

# .github/workflows/e2e-ai-tests.yml
name: AI-Generated E2E Tests
 
on:
  pull_request:
    branches: [main]
  schedule:
    - cron: "0 6 * * 1" # Weekly on Mondays at 6:00 UTC
 
env:
  GEMINI_API_KEY: ${{ secrets.GEMINI_API_KEY }}
 
jobs:
  generate-and-run:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
 
      - uses: actions/setup-node@v4
        with:
          node-version: "20"
 
      - name: Install dependencies
        run: npm ci
 
      - name: Install Playwright browsers
        run: npx playwright install --with-deps chromium
 
      - name: Start application
        run: |
          npm run build
          npm start &
          npx wait-on http://localhost:3000 --timeout 30000
 
      - name: Generate E2E tests with Gemini
        run: npx tsx src/index.ts
 
      - name: Run generated tests
        run: npx playwright test tests/generated/ --reporter=html
 
      - name: Upload test report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: e2e-report
          path: playwright-report/
 
      - name: Comment PR with results
        if: github.event_name == 'pull_request'
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require('fs');
            const report = fs.existsSync('test-results.json')
              ? JSON.parse(fs.readFileSync('test-results.json', 'utf8'))
              : { passed: 0, failed: 0 };
            const emoji = report.failed === 0 ? '✅' : '⚠️';
            await github.rest.issues.createComment({
              owner: context.repo.owner,
              repo: context.repo.repo,
              issue_number: context.issue.number,
              body: `${emoji} **AI-Generated E2E Test Results**\n\n` +
                `- Passed: ${report.passed}\n` +
                `- Failed: ${report.failed}\n\n` +
                `Full report is available in the workflow artifacts.`
            });

This workflow auto-generates and runs E2E tests for changed pages whenever a PR is created, reporting results as PR comments. The weekly scheduled run performs regression testing across all pages.

Cost Optimization Strategies

Combine three strategies to keep Gemini API costs under control.

1. Caching

Skip test regeneration when the page structure hasn't changed. Store a hash of the page's HTML structure and only call Gemini when changes are detected.

import * as crypto from "crypto";
 
function computePageHash(analysis: PageAnalysis): string {
  const content = analysis.htmlStructure +
    JSON.stringify(analysis.interactiveElements);
  return crypto.createHash("sha256").update(content).digest("hex");
}
 
// Only regenerate if the hash differs from the cached value
const currentHash = computePageHash(analysis);
const cachedHash = await loadCachedHash(url);
if (currentHash === cachedHash) {
  console.log("No page changes detected — skipping test regeneration");
  return;
}

2. Model Selection

Use Gemini 2.5 Pro for initial test generation, and Gemini 2.5 Flash for self-healing (minor changes like selector updates). This dramatically reduces costs.

3. Batch Processing

Bundle multiple page test generation requests together to reduce API call frequency. Leveraging Gemini's [Batch Processing API]((/articles/gemini-api/gemini-batch-processing-api) can achieve an additional 50% cost reduction.

Practical Test Generation Example

Here's what Gemini generates when analyzing a login page:

// tests/generated/login.spec.ts
// Auto-generated by Gemini API
 
import { test, expect } from "@playwright/test";
 
test.describe("Login Page", () => {
  test.beforeEach(async ({ page }) => {
    await page.goto("/login");
  });
 
  test("should display login form with all required elements", async ({
    page,
  }) => {
    await expect(
      page.getByRole("heading", { name: /Sign In|ログイン/i })
    ).toBeVisible();
    await expect(page.getByLabel(/Email|メール/i)).toBeVisible();
    await expect(page.getByLabel(/Password|パスワード/i)).toBeVisible();
    await expect(
      page.getByRole("button", { name: /Sign In|ログイン/i })
    ).toBeVisible();
  });
 
  test("should show validation errors for empty submission", async ({
    page,
  }) => {
    await page.getByRole("button", { name: /Sign In|ログイン/i }).click();
    await expect(
      page.getByText(/required|必須/i)
    ).toBeVisible();
  });
 
  test("should show error for invalid credentials", async ({ page }) => {
    await page.getByLabel(/Email|メール/i).fill("test@example.com");
    await page.getByLabel(/Password|パスワード/i).fill("wrongpassword");
    await page.getByRole("button", { name: /Sign In|ログイン/i }).click();
    await expect(
      page.getByText(/invalid|incorrect|認証に失敗/i)
    ).toBeVisible();
  });
 
  test("should navigate to password reset page", async ({ page }) => {
    await page
      .getByRole("link", { name: /Forgot|パスワードを忘れた/i })
      .click();
    await expect(page).toHaveURL(/reset|forgot/);
  });
});

Gemini automatically generates selectors that handle both English and Japanese, using regex patterns for stability across multilingual sites.

Summary

By combining Gemini API's multimodal analysis with Playwright's browser automation, you can dramatically automate E2E test creation and maintenance. Embedding the cycle of page analysis, test code generation, execution, and self-healing into your CI/CD pipeline allows development teams to focus on improving product quality rather than writing tests.

While this guide uses TypeScript, the same architecture can be reproduced with Python's Playwright and Gemini SDK. Start with a small project, validate the quality of generated tests, and gradually expand the scope.