The “1000 Words” project began as a study of the words that we associate with the perception of color. We started with a somewhat outlandish question: could we train a computer to interpret an image? Not just label the image, such as whether or not the picture contains a cat or a dog, but get the computer to express itself with respect to what it saw. Would this process of machines learning also help us as human beings see differently, and think about seeing differently?
They say an image is worth 1,000 words. We wanted to find out which words a computer would help us choose.
To magnify the self-reflexive nature of our undertaking, we decided to focus on Van Gogh’s self-portraits as our first set of visual objects. What would we see when seeing what a machine saw when seeing a painter see himself? Van Gogh famously painted a large number of self-portraits over the course of his life. His work is also particularly known for its vibrant use of color. Van Gogh was influenced by the colour theory of Charles Blanc, especially Blanc’s Grammar of Painting and Engraving, and actively worked on the intersection of colour and language throughout his life’s work.
Next, we needed to decide on a way for a computer to see these paintings. What should it focus on? Machine vision is a fascinating and broad field. We have only scratched the surface in trying to understand it. In our case, we decided to focus on our initial question of color. What colors were distinctive of each of the self-portraits? What made them chromatically unique? After reducing our images down to a standard set of 128 colors using OpenCV, we then ran a statistical test to identify those colors that were distinctive of each self-portrait when compared to the rest. What colors defied expectations when it came to Van Gogh’s self-portraits?
This allowed us to generate versions of the paintings with only these distinctive colors present, producing eerie silhouettes of chromatic focalization, as you can see below. We were surprised to find that the eyes of most portraits stood out against otherwise blank backdrops. For Van Gogh, it seemed, the eyes of the painter were chromatically exceptional, small luminous orbs of painterly significance.
What do these colors say? To attempt an answer to this question we then brought together a collection of over 70,000 poems that were written in the twentieth century and extracted lines in which the colors and objects of Van Gogh’s portraits were mentioned. To move between the quantitative codes that are used by a computer to understand color (HSV or “Hue-Saturation-Value”) and the language of poetry, we used a translation key, pulled from Wikipedia, that assigned color names to HSV values and then created a database of 197,000 lines of poetry where these colors occurred.
We then randomly generated thousands of poems from these lines and trained a machine to choose the ones that we found the most moving. The poems that accompany the self-portraits are thus amalgams of human and machine mentalities. Human portraits seen according to computational procedures whose vision is expressed through human preferences of computationally generated human poetry. Every time you visit the site you will see new poems and thus new ways of seeing, infinitely.
Humans have never existed in the world independently of technology. Our species depends on it, for better and for worse. 1,000 Words is an experiment in our further enmeshment.
Fedor Karmanov and Andrew Piper