← back

CLIP

a screenshot of the CLIP Open AI website

CLIP is a neural network trained on image-text pairs. In the context of Stable Diffusion and LAION-5B. CLIP is used to evaluate image descriptions and automatically filter out image-text pairs that don't have a strong relationship from the Common Crawl data. If image-text pairs were rated a score lower than a certain threshold, they were discarded. This allowed the creators of LAION to work with such a large database of images without individually evaluating each pair by hand. To understand how CLIP works and its biases, we should look at the data this model was trained on.

CLIP is trained on Imagenet, a dataset of more than 14 million images (smaller than LAION-5B) scraped from the Internet, labelled by Amazon Mechanical Turk crowdworkers and organised through 21,000 categories. The categories of ImageNet were created making use of WordNet, a lexical dataset of English nouns, verbs, adjectives and adverbs grouped into sets expressing distinct concepts.



Imagenet categories:

Excersize 1:

pick 1 word that seems relatively straightforward as a category and speculate what kinds of images would fall under that category

Now pick a word that seems quite difficult to place as a category and speculate what kinds of images would fall under that category.

Discuss with a partner

Words are wobbly. It can become quite subjective to categorize images under classes like "failure" and "hope"



Excersize 2:

Let's test our caption writing skills with real LAION 5b images

write a 1 sentence caption for each image:


test your captions out in this colab notebook:

Test your captions ↗



← back