Does prompt wording change what CLIP finds?

CLIP matches images and text by embedding both in the same space, so a text query can find images it was never trained to label. That makes the wording of the query part of the model. I wanted to measure how much it mattered.

I kept everything else fixed: the model, the images and the random seed. Only the prompts changed. Alongside the single canonical prompt, I wrote three sets of five templates: natural rewordings, deliberately misleading descriptions and descriptions of poor image quality. A set's five prompts are averaged into one query per class.

The images are encoded once and reused for every condition, so any difference comes from the text side. Each image is compared with each class query by cosine similarity. The same scores support two measurements: retrieving images for a class name, and classifying each image by its most similar class.

My main retrieval metric, Recall@1 and Recall@5, was 1.0 for every condition. It asked whether each of the ten class-name queries put a correct image first (Recall@1) or anywhere in the top five (Recall@5), and every query did, however it was worded. A metric that cannot fall cannot compare anything, so the useful evidence came from classifying all 1,000 gallery images instead.

The misleading prompts came out highest, by three images out of 1,000, and the degraded ones lowest, ten images below the baseline. Differences that small are too close to rank. What did change was the pattern underneath.

The notebook stores the top five images for every class under each ensemble: 150 images in all. 147 are the right class. The three misses are cats returned for “dog” under the misleading prompts, and a dog returned for “cat” under the degraded ones.

The same cat–dog and deer–horse confusions appear under every wording, which points at the image side rather than the prompts: at 32 × 32 pixels, small animals are hard to tell apart.

Zero-shot classification needs no labelled examples, only the class names. To see what it gives up, I trained a logistic-regression probe on the same frozen image embeddings: 80 images per class for training and 20 held out, and I scored both methods on those same 200 images.

The probe got nine more of the 200 right, gaining most on frog and deer, while zero-shot did slightly better on dog, horse and ship. Zero-shot's mistakes were also more scattered: its deer errors went to five different classes, the probe's to two.

The stored confusion matrices

I built a small Gradio app in the notebook so that anyone could test the model directly, using the same encoding functions as the experiment. It ran from Colab with a temporary public link and was never deployed.

• Image to text: score an uploaded image against your own comma-separated prompts.
• Text to image: return the closest image in the CIFAR-10 gallery.
• Zero-shot top five: the five most likely CIFAR-10 classes for an uploaded image.
• Ensemble comparison: the class each prompt set (A, B and C) picks for the same image.

CIFAR-10 contains no people, so the study itself raises few direct risks. The point of the report's governance section was what the behaviour means for real systems built on the same model.

• If wording shifts which classes succeed, prompts are part of the system and should be fixed, tested and versioned, not left to each user.
• Some classes are weaker whatever the wording, so a system needs to know where it is unreliable before it is trusted there.
• CLIP was trained on internet-scale data with undocumented biases; anything touching people needs its own evaluation and human review of uncertain results.
• I did not build any safeguards, such as uncertainty estimates or input checks. The report lists them as recommendations.

• Replace hit-at-K with a metric that can move, such as precision at K or mean average precision over all 100 relevant images, and use many more queries than ten.
• Report differences as counts with uncertainty. A gap of 13 images in 1,000 needs repeated samples or a significance test before it means anything.
• Re-normalise each averaged prompt embedding. Without it, classes whose five prompts agree more closely get a slight edge when classifying, which could be as large as the differences I was measuring.
• Add prompts without the class name, to test whether the class word alone carries the result.
• Repeat on higher-resolution images, where the image side limits results less.
• Keep comparisons like for like: one notebook cell sets zero-shot accuracy on all 1,000 images beside the probe's score on 200. The fair comparison, on the same 200, is the one used here.