Vibe Coder · Unit 13: Testing and Security · Lesson 4 of 11

Testing AI features

An AI feature gives different answers each time, so it needs a different kind of test: a set of real inputs and a clear standard.

Read the short notes and pictures.Do everything inside a blue frame.

Tests for a feature that varies

You cannot check an AI feature by trying it once. A small evaluation set, kept and reused, tells you whether a change to the prompt or the model made things better or worse.

A 3D scene of coloured blocks and cards. A test set: 20 real inputs, what good looks like.
Twenty real inputs and a standard for each.

Building the set

  1. Collect real inputs

    Twenty is enough to start. Include hard and odd ones.

  2. Write the standard

    For each: what must be in a good answer, what must not.

  3. Run and score

    Pass or fail, with a note.

  4. Keep it

    Add every real failure you see later.

A 3D scene of coloured blocks and cards. Score against the standard: before, after.
Score before and after every change.

What to include

Typical cases

The everyday inputs.

Edge cases

Empty, very long, another language.

Hostile cases

Inputs that try to make it misbehave.

ActivityBuild stepA tested, locked-down app

Make a test set for your feature

Collect twenty inputs for your AI feature, write the standard for each, run them and record the score. Improve the prompt once and run again.

Done when

Run the set before every release from now on.

Keep these

  • AI features need a test set.
  • Define what good looks like per input.
  • Re-run on every prompt or model change.