NOTE: Check the results of the absurd survey at the bottom of this post!
Plus participate in a new one!
Hello! This is the story surrounding the animated music video, The Idea (see the bottom of the post for the video). This project combines traditional, analog painting with ‘Neural Style Transfer’, an advanced form of machine learning/artificial intelligence.
Thus, the old meets the new.
Tradition meets technology.
Art meets science.
The story begins with sound. I will leave the composing and recording stages of the production for another day, and will start with process of mixing.
Sefi Carmel was my London-based mixing engineer on the album, Music for Scientists. His task on The Idea was to produce a mix that was balanced, articulated and spacious. I supplied Sefi with over 150 separate instrumental and vocal tracks for this song, including dozens of tracks of the 27-piece string orchestra we recorded in Nashville.
That’s a big production, with a lot of moving parts. But Sefi and I had already completed a few of the songs on the album, and I knew of his enormous talent.
As an aside, Sefi immigrated to London as a young man, with the dream of creating an English country garden recording studio — following in the footsteps of one of his greatest influences (and mine), Peter Gabriel. His dream became a reality.
This is one of the mixing rooms in Sefi’s remarkable studio, where the Music for Scientists album was engineered.
When Sefi delivered the final, mastered version of the The Idea, I felt confident it would be the most popular song on the album. I wanted to accompany the song with a music video. I began a long period of visualization: listening to the track, eyes closed, picturing, imagining.
I felt the music video should have a psychedelic vibe. Perhaps because the song deals with problem-solving and those moments where solutions can occur in the early, hypnagogic stages of sleep.
I also felt the video should have an edgy, almost violent quality. We’ve all had problems so challenging, so vexing, often addressed with so much fruitless trial and error, they can feel like an assault on oneself.
Finally, I felt this should contrast with soft, soothing, whispering moments, which are expressed in the song and so many of us have experienced when solutions finally do arrive in sleep.
In my research, I came across the work of Jon Todd, a Canadian visual artist, based near Toronto. Jon’s work struck me as dream-like: jarring, distorted, angular and genuinely original. I reached out to him by email and we began a correspondence.
Jon recommended we contact Jonathon Corbiere, co-founder of Thought Cafe, the Toronto-based animation studio. Jon introduced me to Jonathon (I know, confusing, right?) and the three of us connected on several Zoom meetings. Jonathon soon agreed to direct the video and perform the massive editing function we were proposing.
This was unusual for Jonathon, because as co-founder of multiple successful businesses, he would not normally be involved on this level. But he saw the opportunity for direct artistic expression and chose to get neck deep in a passion project.
We agreed it would be an animated piece, based on the fusion of two radically different approaches. The first approach would involve a camera capturing a massive number of Jon Todd’s sequential brushstrokes across three paintings. This is how traditional, stop-motion animation is done.
Jonathon rigged an overhead camera in Jon’s studio, along with professional lighting. The camera was set to capture 4K images framed on the entire canvas. This would allow for ample zoom when editing later in the process, without concern about pixelizing.
Picture this workflow: The mounted camera was set on a timer to take one photograph per minute during Jon’s painting. Jon was working away. He heard the beep of a forthcoming camera shot. He momentarily stepped back and out of the frame. He waited. The camera took the shot. Jon then returned to the work.
For our video, this cycle occurred a staggering 15,000 times (yes, fifteen thousand).
That is 250 hours (yes, two hundred and fifty hours) of Jon’s active, minute-per-minute artistry, captured.
This is a view from the camera down to the canvas, and an example of one of the thousands of photos assembled into a stop-motion animated work.
On top of this stop-motion animation, we wanted to add fluidity, focus and performance. How to do so in a manner that would reflect Jon Todd’s style? In a way that would be integrated and compatible with the underlying paintings?
Enter Neural Style Transfer (NST) software, the second major approach fused into this video. NST is a form of machine learning and artificial intelligence. The first step of the plan called for me and other actors/dancers to perform on video in a green screen studio.
Selected frames of these videos — about one frame per 48 frames of video (roughly 1/24 of a second, every 2 seconds of play) — were isolated and provided as still photos to Jon Todd. Jon then used digital tools to paint each of these frames in his style, as seen below.
We then fed each of these painted frames into the Neural Style Transfer software, along with the full length green screen performances on video. Over time, the artificial intelligence “learned” how to paint the next 47 frames of video in each two-second interval.
The processing demands of this software were extraordinary. To render an outcome suitable for viewing, each performance took as long as two full days of high-intensity processing. Need to make an edit to a key frame? Another day or two of rendering.
The Neural Style Transfer outputs, once having successfully “painted” a performance, were then married together with the underlying stop-motion animation in yet another specialized video editing software. Jonathon spent hundreds of hours here: colorizing, blending, adjusting timing, executing visual effects, managing transitions.
The editing software allows individual components of the video to be separated into layers, so that each variable can be edited in isolation of the others. This enables incredible control over the total project, but also increases complexity. By the end of the project, Jonathon was editing in hundreds of layers of content.
After six months of effort, we completed the video. I am deeply indebted to all of my collaborators for their amazing contributions.
As subscribers to this project, you were the first to see The Idea. If, by chance, you missed that opportunity, here is the finished video. I hope this background story helps you understand what took place in the production of the video, and allows a better interpretation of what you are seeing.
Thank you for sharing in this with me.
-Patrick
music for scientists
