Architectures · Vision & diffusion

CLIP

CLIP ViT-B/32 · images and text in one space

compared
CLIP ViT-B/32 vs GPT-2 small
text tower
12 × 512 · causal GPT-2 12 × 768
image tower
ViT-B/32 · 12 × 768 none
output
1 vector of 512 GPT-2 50,257 scores
context
77 tokens GPT-2 1,024
params
151M GPT-2 124M

All steps

  1. Two encoders

    CLIP has two encoders. The text one is shaped like GPT-2 (causal, 12 layers) but outputs one vector, read at the end-of-text token, instead of next-token scores. The image one is a Vision Transformer. Each vector is projected to 512 numbers and scaled to length 1. Click a label to jump.

    image → ViT → 512 · text → Transformer → 512

  2. One shared space

    These are real CLIP embeddings of four drawn images and their captions. On their first principal component, all images sit on one side and all captions on the other (the known “modality gap”); on the second, the heart’s picture and caption sit at one end and the triangle’s at the other, while the square and the star, pictures and captions alike, fall close together in between.

    length-1 vectors · dot product = cosine

  3. The contrastive matrix

    Training takes a batch of image–caption pairs and computes every image against every caption: one matrix product. The loss pushes the diagonal (the true pairs) up and everything else down, along rows and along columns. Real CLIP numbers; hover the cells.

    S = I · Tᵀ · softmax both ways

  4. Zero-shot classification

    Because labels are just text, CLIP classifies without training for the task: write one caption per class, embed them, and pick the most similar. CLIP was never trained on these shapes, yet all eight answers here are right.

    class = argmax cos(image, “a photo of a …”)

  5. Temperature

    The raw cosines are close together (0.2 to 0.4). CLIP multiplies them by a learned scale, the inverse of a temperature, before the softmax. It starts at 14.3 and training pushes it to its cap of 100, which turns small gaps into confident answers.

    softmax(100 · cos)

Code

# two encoders, one shared space
i = F.normalize(visual_projection(vit(images)[:, 0]), dim=-1)          # (N, 512)
t = F.normalize(text_projection(text_tf(ids)[range(N), eot_pos]), dim=-1)  # (N, 512)
# every image against every caption, scaled by the learned temperature
logits = logit_scale.exp() * i @ t.T                    # (N, N), scale capped at 100
labels = torch.arange(N)                                # the diagonal holds the true pairs
loss = (F.cross_entropy(logits, labels) + F.cross_entropy(logits.T, labels)) / 2
# zero-shot: one caption per class, pick the closest
classes = F.normalize(text_projection(text_tf(prompts)), dim=-1)
pred = (i @ classes.T).argmax(dim=-1)

Go deeper