If you are like me, you first came across vectors in high school physics. Questions about a driver with a certain velocity traveling north, and another driver with a different velocity headed south. Vectors in physics are things that have a magnitude (the driver’s speed) and a direction (headed east).

Alice leaves Point A, driving East at 60 kmh...

I know, physics, right? It’s about to get more mathy.

Linear Algebra Vectors

In linear algebra, vectors are one dimensional matrices. If you are a programmer, think array. Each entry has a meaning. We could describe the physics problem above with a 4 entry vector describing Alice’s location (in 2D space) and her speed as:

The first two entries describe her location, and last two describe her motion (speed and direction of travel).

Linear algebra vectors contain data, and can contain a lot of data – there can be a lot of entries in a vector. 

The cool thing about vectors is that when you multiply them together (the dot product), you know the distance between the 2 vectors. By default, the dot product contains the magnitudes of the vectors – and you want the angle between them (cosine similarity), but for this article, dot product will be good enough.

Vectors in Machine Learning: comparing

When a model is trained against millions of objects, vectors are one way that the objects can be described. (Note: It is unclear if ChatGPT works with vectors, but the analogy for their mathematical model is similar.). Because the model can describe any object or sentence as a vector, we can compare the two vectors to get an index

For example, I can ask a LLM model to create a vector of this image:

I can then create text vectors for the terms “dinosaur”, “chicken”, “dog” and “tuna casserole” using the same model.

Each of the vectors created above have 512 entries. This is no longer a dot product for humans to calculate – but easy for a computer. Let’s see what the dot product similarity tells us:

Name	Similarity
dinosaur	30.732
chicken	23.899
dog	27.248
tuna casserole	17.625

Based on the model we used (I am using Corcel’s APIs, and you can use my Jupyter Notebook to test), the image is most likely a dinosaur, and less likely to be tuna casserole. The model works!

Why good prompts matter

In the above example – when the vectors are compared – the image had more aspects related to the string “dinosaur” than to the string “tuna casserole.”

Now, think about your prompt being turned into a vector to find a match. If you don’t have the context you need – you won’t get the accurate result.

My favorite example around prompt context is the term “football.” In the US, football is a sport with pads, helmets and an oval ball. For the rest of the world, it’s what americans call soccer. (For simplicity, I’ll refer to the two sports as football and soccer for the rest of the post.)

Let’s try image creation prompts:

I love football art. Can you create a comic book style image of football players completing a pass?

This is a mess. This player has soccer shorts, but one arm has football pads. He is catching a soccer ball the way a football player might catch a ball.

Lets refine:

Im from scotland and I love football art. Can you create a comic book style image of football players completing a pass?

Much better. Since in Scotland, football == soccer, we have a drawing of a soccer game.

Let’s try a US location:

Im from texas and I love football art. Can you create a comic book style image of football players completing a pass?

This has strong University of Texas vibes. Another generated image has very strong resemblance to the Dallas Cowboys helmet:

The addition of a location to the image gave a much stronger version of the image – first by removing the football/soccer ambiguity, but it is interesting that it was also able to add a regional context to also adding appropriate football jerseys for the region. (replicated below with a Wisconsin location)

The added context in our prompt is added to the vectors, and it allows the AI to generate responses that are even more appropriate.

Conclusion

Vectors are a common way that LLMs and other machine learning libraries describe items. Vector math can be used to find similarities, and similar contexts for the content being created.

In this post, we’ve covered (at a really high level) how the vectors encode information from the text/images, based on the LLM model. We can the use the vectors with the LLM to create new content, or find similar content.

To conclude, I wanted to share this very funny (to me) image.

Generate an image of a soccer ball next to an american football

I love the stars and stripes on the soccer ball, but it isn’t exactly what I meant 🙂