Intro

​

I'll agree with everyone straight off the bat. This is a scary topic. Even more so once you consider this will be a huge part of our overall future. I think its a good idea to recognize what type of technology we will need to prepare ourselves for. So this is what this article is going to be about. Just plan research, and the pros and cons of AI. 

​

Artificial Intelligence Helps Plan The Next Blockbusters

​

Every wonder why it seems that there is nothing new coming out of Hollywood? Or why certain actors or actresses get used in what seems like every movie in the past year? Or even just why certain movie genres seem to dominate the box office? Well there is a reason why. AI is being used to help predict the next successful movie productions.

​

Certainly, those questions might be answered with different reasons. Those entertainers and genres might be popular in the current time. As well as certain story-lines being more relevant or safer now than in previous years. There are a lot of contributing factors to why copy-and-paste movies, or movies such as in the superhero genre get made over and over again. But this problem might reach its peak with AI.

​

As of 2015, programs have been made to help predict,plan, and make recommendations on what would be a safe movie to make. A company in Belgium called ScriptBook was one of the first to create a software that would predict a movie's success just buy reading the script. Their software can also analyze how likable a character might be, to how much profit could be made from the film.

​

As early as November 2018, 20th Century Fox admitted that they use an AI program to help review trailers before they are released. They described the AI's job as an important one. It scans the trailer for appealing clips and flags them as the parts that the audience will enjoy the most. Therefore, 20th Century Fox can relax knowing which parts of the movie they should market more, and which characters they can create successful merchandise from. 

​

More mainstream film companies are using this technology too. Legendary Entertainment also uses this technology to develop audiences demographics, visual development of films, and trailers.

​

Some other companies these tools are companies like Vault ML. Vault ML develops software to predict revenue intakes on films. Their AI is called 4CAST, and it uses over 40 years of box office data to predict what an opening weekend would look like for a film. It's predictions are right 75% of the time. IBM's Watson was also used to create the movie trailers for the 2016 movie Morgan. In some ways, AI is being considered as the new directors of film making. However, this conclusion is a really far stretch. More on this later.

​

Another company called Cinelytic also focuses on data prediction for movies. They have just partnered with Warner Brothers. Their software is customizable to the client, and allows the client to see predictions based off of different variations for movie castings. For example, if the client is unsure of which actor to cast for a role, he can enter their names into the program and see which actor might attract more viewers to the movie. 

​

AI Use in Visual Effects and Animation

​

Now Let's discuss what is happening: the jobs in visual effects, and the emerging technology. A few of these systems build off each other, so I've tried my best to organize them accordingly. Let's get started.

​

Deep Fakes. You may have seen popular videos of people editing  videos to digitally remove and replace an actors face. Such as the deep fake Back to the Future with Tom Holland and Robert Downy Jr. (HERE) Or the other popular use of deep fakes...adult videos. A deep fake program is an AI that is built from Convolutional Neural Networks(CNN), auto encoders, and Generative Adversarial Networks(GAN). Those systems sound pretty complicated, here is a general description of them. 

​

GAN: This is a system that was first introduced in Montreal in 2014. It is a system designed to tell two other network systems to work against each other to create the best edited visual data set. This system is widely used by Facebook for some of their systems. It is also used in image, video, and voice generation.

​

CNN: This is a system designed to help recognize faces,objects, and traffic signs. It mostly gets implemented in automated self driving cars, and AI vision in robotics. It's been in development since the 1990s and first grew off the theory of image pixel values.

​

Auto-encoder: This program is designed to reconstruct and copy information from pre-existing code by simplifying it and removing any unnecessary data. It can also do this with images.

​

However, deep fakes can also create photo-realistic images based on random imaging data that you a deep fake program. Let's say a client told you on a project that they wanted to have an Apple Store in the background. However, they wanted to avoid copy-write infringement. You could then use them in a deep fake program to generate a similar building, but one the viewer knew represented an Apple Store.

 

Deep fake technology is also huge for film studios in case an actor passes away while filming a movie. Instead of creating a 3D model, animating the actor's face, and then creating the materials, studios can now just use a stand in and map the actor's face over the look-a-like. 

​

CGI Modeling and Animation vs Deep Learning 

​

AI is now improving the modeling and animation parts of characters. Traditionally, most studios create a character by asking a modeler to sculpt a person, then giving it to the rigger to pose it, and then to the animator to make it move. With a Deep Learning Character Program, this process changes a bit.

​

Deep learning doesn't need a model, rig or muscles to move and create a character. The program can be given a set of images or movies of people, and analyze the chosen person's structure, walk , features, and expressions. It uses landmarks on the human face and body to track them, and what are the most definable features of the subject. Then the program creates a generated facial and body texture as well as default poses of the person, and starts animating them. it also mimics the surrounding lighting of the subjects to make the edited character seem more real.

​

A really good example of this program is the Instagram Face Swap tool. It tracks and maps your face, and then maps it to another person's movement. This machine is being programmed to mimic and learn who you are.

​

Because of the lack of modeling, animating , and tracking required for creating these models, companies such as Digital Domain are actively looking at deep learning as a future possibility.

​

Practical effects and make-up artists will also be affected by this technology. If it is easier to face tune and put makeup on in post, then fewer make-up artists might be needed for the filming part of the production.

​

Environment Design

​

This is something that is already being implemented in video games. Most video game terrain is procedurally generated. It's easier to create one rock to scale and scatter about your scene, than to create millions of unique rocks that the player probably won't even notice. Currently, if you were to create one rock to scatter about your scene, you'd still have to model,unwrap it, and texture it. But now there might be an easier way.

​

NVIDIA is creating a machine learning tool called GauGAN. Think of it as a Photoshop tool that generates a 3D terrain from color tones that you can paint on a 2D plane. However, a lot of artists seem to be against this tool as it allows anyone without a creative or technical know-how to look like a senior and skilled artist.

​

Rotoscoping and Rotomation

​

Rotoscoping is the art of extracting individual elements from the background of a shot. People have traditionally used green and blue screens to separate elements from each other, and in early animation days,this was done with different layers of paper and screens. 

​

It is now possible for computer programs to extract characters and objects from a shot without the use of these tools. Some do this by scanning the shot for faces and recognizing what is or is not a person. Then the program traces the person out from the shot. This technology was first developed from government and private security systems. A system developed by OpenFX does this process quite well. It is called Rotobot. 

​

Some other programs are being developed to extract mattes for character's hair movement, and complicated character movements in real time. However, programs like this are expensive to create. It's taken $10 million alone for a VFX startup Arraiy to create tools like this. Autodesk's Flame also contains similar AI uses.

​

Motion Capture and AI

​

You may have heard of motion capture being used widely in the video games industry. This system is designed to capture an actor's movements with the use of tracking markers. It then takes the actors movements and places it on a 3D rigged character. Although, animators are still needed to finalize and tune the animation from the motion capture as it can look un realistic. Take the movie the Polar Bear Express for example. That movie was created from using raw MoCap data, and the characters reside in the Uncanny Valley.

​

However, The University of California is working with a machine learning program to scan random YouTube Videos and sample movements from the videos. But they are developing this algorithm to clean MoCap data, and add further enhancements onto the movements.

​

Push Back, and Arguments Against Using AI

​

There are several arguments for considering the ethical uses of AI before implementing it in a 3D industry or others. One of course being that AI will automatize jobs, and make less available for ready workers. Another is that AI will reduce our motivation to think more creatively, than providing time for us to be creative. 

​

I know from a personal perspective that If a machine could load renders onto the farm for me, take them off, view them, and then send them to my supervisors, I would probably use that free time to watch Netflix. Then that would be my entire day. Sorry Jorge...

​

Considering that the USA and China are leading the AI race right now, they will probably be the first countries to have more automatized industries than others. This will probably have a bigger impact on VFX industries in other countries, because a studio that can make it cheaper and quicker to produce a film, the more work they will receive.

​

This could lead us into lower wages for VFX artists, as we would need fewer hours to on our shots. Instead of working a minimum 44 hours a week, we could be working less. This could be great socially and health wise of artists, but suddenly we won't be making the same amount we did by the end of the week.

​

One of the biggest issues with using AI in the current industry is the price tag. Currently only larger companies can afford to invest in the R&D side of AI. Other studios that can only afford to hire artists, are missing out on developing their in-house technology. Studios might start to compete to do the same amount of work, but fail when the companies with AI do it faster. A big chunk of studios might collapse because clients might move towards cheaper, faster AI driven studios.

​

Another reason not to use AI is that a lot of the programs companies are using to predict the next blockbuster rely on previous data. This is a great way to prevent bad movies from getting green-lighted, but it's also a huge problem when you'd like to introduce a new creative genre into the film industry. Let's take the superhero genre for example. Before the 2000s, superhero movies really didn't appeal to mass audiences. In fact, a lot of these movies were flops. Such as the Captain America 1990s film. If Disney was using AI to decide whether they should have built the MCU, it probably would have never gotten made.

​

Also, introducing new actors in the industry through breakthrough roles would be near impossible. Because these Hollywood programs are built around marketing popular actors and actresses, the AI would be biased towards those people. Therefore anyone who'd like to try at leading a film who didn't have a huge fan base would be denied. 

​

What Does This Mean for the Visual Effects and Animation Industry?

​

In the future, some of our jobs might be completely overtaken by one single computer program. The VFX industry might move away from modeling, tracking and animation jobs, and more towards programming and pipeline work.

​

Programs are being made to streamline and collect data from our creative processes, and our preferences in entertainment. However, right now there is some good news. AI still cannot write a screenplay by itself, and cannot form coherent stores. In 2016, a filmmaker by the name of Oscar Sharp got an AI program to read through a few hundred sci-fi movie scripts, and then generate its own. The result was a huge unexplained mess. The program had characters vomiting up eyeballs randomly, sitting and standing at the same time, and unsure what exact setting they were in. So as of right now, AI still needs us to explain what a story is and think creatively for it.

​

There is also a law in Europe that might help slow down the process of AI in movies. The General Data Protection Regulation is built around the right that everyone has to have and understand the "meaningful information about the logic involved’ in automated decisions. This means European citizens have the right to an explanation concerning an output of an algorithm which directly impacts their way of living. So if an AI generates a movie that has unfair ideologies and hate speech, the public has the right to have access to the information and technology that created that film. And if this film were to have an impact on others beliefs, the public could then go after the creators of the AI.

​

But how can we protect ourselves? 

​

One thing we can do as artists is by continuing to grow our skills.  We can learn how to integrate python, VEX, and other TD skills into our personal workflows. Such as studying computer vision, digital image processing techniques, basic ML stuff/training, classification, regression, and multi layer neural networks. We can also read up on what our competition might be in the future when it comes to AI affecting our jobs. 

 

If you also work as a professor at a university or college, you can teach your students skills that they will need to have an advantage in the industry. You could also add input to your program of how to improve it in the future. 

​

There are a ton of things we can do to improve our chances for ourselves, the studios we work for, and our co-workers. But we need to be smarter than the machines.

​

Let's not allow AI to ruin our future.

​

References

​

Data & AI in Film Creation: https://picturethisconference.com/data-ai-in-film-creation

Can AI help the film industry?: https://www.screendaily.com/news/can-ai-help-the-film-industry/5145193.article

HOLLYWOOD IS QUIETLY USING AI TO HELP DECIDE WHICH MOVIES TO MAKE: https://www.theverge.com/2019/5/28/18637135/hollywood-ai-film-decision-script-analysis-data-machine-learning

Artificial Intelligence: A New Director In Film-Making: https://www.boldbusiness.com/digital/artificial-intelligence-film-making/

'It's a war between technology and a donkey' – how AI is shaking up Hollywood: https://www.theguardian.com/film/2020/jan/16/its-a-war-between-technology-and-a-donkey-how-ai-is-shaking-up-hollywood

​

A comprehensive guide to the state-of-art in how AI is transforming the visual effects (VFX) industry: https://rossdawson.com/futurist/implications-of-ai/comprehensive-guide-ai-artificial-intelligence-visual-effects-vfx/

​

How AI is reinventing visual effects: https://www.ibc.org/create-and-produce/how-ai-is-reinventing-visual-effects/4060.article

​

AI in VFX: https://www.reddit.com/r/vfx/comments/aj0zd5/ai_in_vfx/

​

THE NEW ARTIFICIAL INTELLIGENCE FRONTIER OF VFX: https://www.vfxvoice.com/the-new-artificial-intelligence-frontier-of-vfx/

​

AI for VFX and whats it mean for you.: https://www.onsetfacilities.com/on-set-ai-for-the-vfx-industry/

​

‘A.I.’ for VFX at SIGGRAPH Part 1: https://www.fxguide.com/fxfeatured/a-i-for-vfx-at-siggraph-part-1/

​

Lights, camera, artificial action as AI takes on the special effects industry: https://www.fanaticalfuturist.com/2019/10/lights-camera-artificial-action-as-ai-takes-on-the-special-effects-industry/

 

Does Artificial Intelligence Pose a Threat to Creativity and Animation?: https://www.jumbla.com/news/does-artificial-intelligence-pose-a-threat-to-creativity-and-animation

​

State of AI in Animation: https://theblog.adobe.com/state-of-ai-in-animation/

 

Artificial Intelligence- The Future of Animation: https://www.twistedframe.ca/artificial-intelligence-future-of-animation/

 

Animation and Artificial Intelligence: https://www.britishcouncil.org.tr/en/programmes/education/cubed/animation-and-artificial-intelligence

 

Please Leave AI Out of the Movie Business: https://www.popularmechanics.com/culture/movies/a30455274/artificial-intelligence-movies/

​

Netflix AI will eat VFX: https://www.linkedin.com/pulse/netflix-ai-eat-vfx-dexter-davey/?fbclid=IwAR1yt7-WhL-bpQyxBO56iC4DnKXnAs9Kwxncfwb9Yi42OX8eOiyRWQJ1Lj8

​

An Intuitive Explanation of Convolutional Neural Networks: https://ujjwalkarn.me/2016/08/11/intuitive-explanation-convnets/

​

A Beginner's Guide to Generative Adversarial Networks (GANs): https://pathmind.com/wiki/generative-adversarial-network-gan

​

AI and the Auteur: Implications of Using Artificial Intelligence in Film Studio Decision-Making: https://www.jurist.org/commentary/2020/01/kelsey-farish-ai-and-the-auteur/