AI FILMMAKING

Some Brief Observations on AI Filmmaking

It’s early in the game, but what a game!

A Free Udemy Course

In the practical spirit of Bresson’s Notes on Sound, I present below a bullet list of rather economically expressed thoughts about the process of making films — in my case, music videos — with AI.

• It’s difficult to realize a preconceived plan. As soon as you try to get the AI to realize your grand vision, you run up against its rather orthogonal interpretations of your intentions. Be open to changes in the direction of your vision as you compare system output to your textual-intentional input.
• You need a Film School of the Mind — meaning, you need to know your shot types, lighting and lens concepts, the difference between inductive and deductive editing, all the film grammar from D.W. Griffith forward including shaky handheld cellphone vernacular media style and the wide variety of visual effects. And of course, film and television history along with their associated aesthetics. You should understand what a transition is, etc.
• Home and play, repeatedly: to get a feel for where AI can take your video, you need to keep hitting the home key on your keyboard to go back to the start of the sequence, and hit play to see what you’ve got so far. You need to keep doing this with every new clip, so that you can impose your willful coherence onto the sequence as a whole to tame the wild variety that easily results from AI output. You need all these repeat plays as you progress to craft a sequence that will ultimately make sense when finished.
• It can be the simplest editing you’ve ever done, where literally you just drop in clip clip clip as a series of jump cuts, which is how I make all my AI music videos, as shown below in this screenshot from Davinci Resolve, which is just a technically simple sequence of 4-second clips.

Editing Private Worlds

• Forget editing as usual, because the shots you are cutting together will lack the visual coherence to make subtle changes at the clip’s start and end edges pretty much meaningless.
• You may as well learn about all the fine arts of every culture and historical period because who knows, you may need to render your video’s look in egg tempura or make all the people look like Greek statues or Byzantine mosaics some day.
• Generative AI is based on numerical ‘seeds’ so it’s usually sufficient to start with a simple seed idea and try to grow it organically in dialectical dialogue with the AI.
• Free Preview, Free Preview, Free Preview — if you’re using RunwayML, push the heck out of the Free Preview button. You want the most visually impactful images for your masterpiece, of course.
• For racial/ethnic diversity, add in ‘multiracial’ or ‘multiethnic’ as an adjective before ‘people’ otherwise you’ll tend to get mostly caucasians in your imagery. Similarly, you may need to write ‘men and women’ instead of ‘people’ if you get too much of either sex.
• For stylistic continuity across the range of clips, keep the start and end phrases of all the prompts exactly the same and only change out the middle part. For example, you might begin each prompt ‘psychedelic classic cinematic action’ and end them all ‘in the style of….’ (choose your favorite film or television show owned by whatever giant corporation you’d like to be sued by if they ever found out about it ; ) In between these identical start and end sections, change out the middle to vary the content and get the clips that you need.
• Successful prompt writing is just as much about the prompts that work as those that don’t, and often the difference between good and bad prompts is just a few words. Unsuccessful prompts and their unused media are the equivalent of all the footage that ends up on the metaphorical cutting room floor.
• Speckling key words into random and generally ungrammatical places in the prompt (not quite a sentence!) structure works surprisingly well.
• Prompt writing involves a mix of sensibilities related just as much to poetry as to narrative, and to photography as to film and video, because you have to think in terms of juxtaposition and sequence, composition within the frame and the flow between them, denotation and connotation, plot and theme — it’s a complex semiotic and hermeneutic mix.
• Try music videos at first with a tempo of 120 BPM because that fits perfectly RunwayML’s 4-seconds at-a-time clip generating constraint.
• Tell stories that make sense across a population of characters, since at the present moment AI cannot maintain a stable character beyond a single clip, when it comes to appearance, clothes, changes in setting and lighting etc.
• Alternately, describe a single complex scene where interesting things are happening that a large collection of different shots can articulate different perspectives on.
• The AI filmmaking process is highly curatorial and editorial. In essence, you have to regularly exercise judgements in good taste to select the best output for the clip composition you are editing together.
• Embrace the visual weirdness. Generative AI is introducing whole new vocabularies of visual stylization, errors, beautiful mistakes, and funky unprompted transformations, so just go with the glitchy output flow.
• Think in reverse, literally. Meaning: a lot of the clips will work well for your video, if you simply reverse the direction of the clip. Imagine the clip going in both directions and quite a few will work perfectly if you just flip the frame order.
• You need to find something to do that will absorb your attention while generating clips, since 4 minutes of waiting to evaluate 4 seconds of footage is hard on the mind. Reading an ebook, grading student work (if you teach), or practicing your chess app skills are what I gravitate towards.
• Explore absurdity: add prompt details that make no sense at all, just to see what will happen. For example, “…in the style of ancient Greek neo noir,” whatever that means.
• Homage and Easter eggs: to make your creative process as meaningful to you as possible, throw into the prompts random things that only you know about, like a favorite film or filmmaker, an artist or architect you like, or the food you had for lunch that day. Pay tribute to those who inspired your creativity, or just be random and mundane — this adds some personal spice to the clip rendering waiting game.
• Neuro-cinema: get to know how the brain likes to be triggered to release dopamine and give your viewing audience eye candy accordingly, especially by regularly changing the shot, which will happen anyway when composing with short AI clips. But you can also push it further than that, via content switches etc.
• Upscale your video, e.g. from HD to 4K or 4K to 8K. Cannibalize AI with other AI, such as by taking your HD video from Runway and making a 4K video out of it with Pixop.
• The money shot: not to be confused with the use of this term in other contexts, here it means keeping an eye out for the one still frame of video that you will use as your thumbnail image when you present your video online.
• At times it can feel like you’re in the midst of a lucid dreaming session only the dreamer isn’t necessarily you but also the AI. It can be a kind of collaborative lucid dreaming, just go with it.