Creating an AI Tool with Animators in Mind
My interview with Andrew Carr, co-founder of Cartwheel
Last week I wrote a post about how AI tools that could control animation software like Maya or Unreal would have a much better chance at disrupting the animation industry than generative video tools such as Sora or Runway.
A few days after I hit publish, a new tool called Cartwheel was released to the public that appears to do just that. Instead of using AI to generate final frames of animation, it uses text prompts to have the AI control rigs, and the data can be sent to a 3D program of your choice for further refinement. You can read about some of the news coverage of the launch here and here.
The tool is still in the early days, but since this is exactly the kind of thing that might have a big impact on the future of animation and VFX (for better or worse) I wanted to check it out.
I contacted the company, and co-founder Andrew Carr agreed to chat with me about their new tool, its launch, and how it might affect artists in our industry. Andrew (formally of OpenAI) is a self described math and science guy, but he told me his interest in classical piano from a young age created a love for the arts that he hopes to merge with math, science, and AI.
Below is our conversation, lightly edited for length.
LIFE IN THE MACHINE: Thanks so much for meeting with me. What made you decide to get into this space? How would you describe the company’s origin story?
ANDREW CARR: I was a scientist at OpenAI working on program synthesis - It's a jargon term - it essentially means code that writes code. So, I was on a team doing this and I was looking around at other tools that were up and coming. DALL-E, the first image generator from OpenAI, was on the horizon and we were testing this tool internally. But I felt extremely frustrated every time I used it because the output was pixels, and I couldn't modify it after the fact.
So, I had this program synthesis tool we were building, and I was frustrated with the image generator and I thought, “I wonder if I can put this ‘AI that writes code’ into Blender.” So I hooked it all up and I asked Blender -- Hey, make this lighting set up, make these things do that. And it was able to do it! Most importantly, after the fact I could go into the 3D viewport and change whatever I wanted.
Light bulbs were going off for me. I saw the ability to move quickly and then have complete creative control. So, I ended up leaving OpenAI to pursue this idea. I knew there was something here.
A few months later, OpenAI emailed me and said, “We tried to hire this guy and he pitched us exactly what you went to go build. You should probably talk to him.” That's my co-founder, Jonathan Jarvis. He's a brilliant animator and product designer. He did animation for Google, ran his own studio. Brilliant man. We met up and connected over this idea of editability at the core.
One day over breakfast we were talking and I said, “What's the hardest thing that we could try and solve in animation?” Immediately, no hesitation, he said, “3D characters. It's super tough but people love it. There's almost infinite demand for like 3D character animated content.”
That was the genesis for Cartwheel. This was in 2022.
LM: You seem to be taking an interesting approach here because as a director when I look at video models like Sora or Runway they can do all these impressive things, but the minute you want to go in and change something, it falls apart.
AC: Right. I always call it the AI slot machine. You pull the lever and you hope it gives you what you want. It’s going to give you something fantastic, but the chances of it matching your vision seems very low. And that's wrong. We should give animators the tools to tell the stories that they want to tell, not the stories that the AI guesses.
LM: Do you see Cartwheel as being mostly a professional product, something more consumer-facing, or something in-between?
AC: Our initial target group of customers is people who are paid to do this today - 3D professionals. We’ve noticed that the industry is buckling under the load of demand and the studio green light process is kind of broken, but the tools haven't kept up with that. Individual animators cannot make animated films. Even small teams can't really do it. [Oscar winning animated film] Flow is amazing, but it took quite some time.
We’re targeting this professional group of people who want to do both smaller and bigger projects. However, I think a side effect of making it easier is that the adjacent folks who are creative but don't do animation today can use it. It doesn't require the two years of schooling to pick it up like you would if you're doing Unreal or Maya, but if you do know Unreal and Maya, we slot right into that workflow and just supercharge it.
Eventually I would love for everyone to be an animator in the same way that we're kind of all photographers with our smartphones. Kind of, right? I would love for the world to exist where animation is everywhere, and everyone uses it for communication. But then there's still, of course, the top-of-the-top experts who work in this field.
LM: Currently with Cartwheel you can work exclusively in the web app, or you can export what you do and use it in Maya, Blender, or whatever. It seems to me that second way is how the tool could potentially be used in production.
AC: Yes.
LM: Do you always see it working in that intermediary kind of way? Do you see a world in which you would work directly inside a program like Maya?
AC: Yeah, so we are building out an API, which would allow you to plug motion and pipe it directly into your experiences. We would love to integrate with all these tools. The easiest way today to see if we're building in the right direction is download the motion, see if it's useful.
LM: Got it. Can you tell me a bit about the process of training the model for Cartwheel?
AC: Great question. If you look across the AI landscape you have different AI models that do different things. So ChatGPT is like mostly a text model, although there's some images now. Midjourney is an image model. Runway does video There's a new data type, which is 3D motion. This data is different than text and different than images.
So, you look around the internet and you say, where can I find this data? I call it jokingly, ethically sourced. We've purchased and licensed motion capture libraries. We've paid animators to animate these things. We've done mocap ourselves. I've even got in the suit and danced around myself. So today it is mostly trained by motion capture data.
Critically, one thing we've done, however, is clean that up so the splines that come out of the system are not mocap splines because that would not be useful. And for those who are unaware, mocap splines would be a key on every frame. And once you try to edit that curve, it just falls apart. So we've taken great care to not output a key on every frame. We output beautiful curves that have great tangents that you can manipulate in your system.
LM: Did you have animators go in and clean up that mocap data or is that something that you're automating?
AC: We do both. Our Pixar animators on the team informed the process and they've looked at thousands of hours of automated cleanup splines to dial this thing in. But when you download generated motion, it's cleaned up automatically by our system.
LM: I actually wanted to ask about the former Pixar artists you’ve hired. I’m curious how they've pushed against the technology and how you've adjusted based on their feedback.
AC: It's amazing. We’ve hired Cat Hicks and Neil Helm, 15-year directing animators, brilliant folks, very kind. The short version is they wanted to build the tools they wish always existed. And we're taking that to heart. They have put the Pixar tools through the paces over the years and they know what good tools are. One of the things that we take into account is how would a professional use this tool?
For example, we have built a full IK/FK rig with a line of follow spaces and all the good things that goes onto any custom character on any motion that you can download. That was one thing that they said we need and cannot launch without. And me, as a scientist, I'm like, “You just get the rotations and it's all forward kinematics and you're fine.” And they said, “No, chance. No one's going to use this.”
So, it's little things like that that elevate the experience from a fun toy to a powerful professional tool being able to actually slot into production.
One thing that we're going to be starting really soon for folks who are interested in the Cartwheel community is Zoom chats with the animators on our team show you how to use the tool and answer your questions. We're going to be trying to make sure the communication is strong. Because that matters here. A lot.
LM: It feels important to connect with animators and what they actually need in the day-to-day work of creating shots.
AC: Yeah, that's right.
LM: One thing I noticed playing around with the tool is that the motion library seems to be core [the ability to combine pre-animated motions together into a shot]. Was the animation in the library created generatively with AI, or was it animated by your team?
AC: It's a mixture of both. We've generated a lot of that motion and a lot of it is motion capture that we've cleaned up. One thing we've observed is animators don't actually care if the motion is AI generated. The scientist in me cares, but animators want great motion wherever it comes from. We're trying to serve that desire.
Another thing that we're launching very soon is video reference upload. So you can record yourself or upload a reference video. It’s almost like rotoscope, but we'll extract the 3D motion from there. So yes, we try to have this three-pronged approach. You can type it in and generate, you can search for it, and very soon you can upload reference to get the motion that you care about out of our tool.
LM: Right. And obviously the generative stuff can get a bit wonky at times.
AC: It’s hallucination at its purest form.
LM: Which is totally understandable. Is there a process where that improves over time? How do you get stuff to work more consistently?
AC: The motion generation system is in its infancy and there's lots of crazy [results]. There are many ways we could imagine improving this over time. One thing we're not going to do is train on uploaded motions. That's not part of our company culture. We're not going to train on your data. But you could easily imagine once we launch this reference video upload system that we ourselves go out and find reference video, label it, extract the 3D motion and use that to continuously improve the model.
We have an amazing suite of artist data labelers who help us. They look at the motion, they watch it, they describe it. That process is ongoing giving us beautiful, dense captions that are descriptive of the motion. Once we get more and more of those, then the model better understands the intent of the users when they ask questions.
LM: A lot of criticism of AI generated art and video is that there's a certain sameness - a sanding off the edges that can make it feel generic. What are you doing to make sure that your animation doesn't just end up being the average of everything, but instead that you can get something interesting and unique?
AC: It's a little bit more of a science question, so without getting too much into the jargon, there’s two steps to training these models. The first is called pre-training where you train on all this beautiful motion capture and all this hand animated stuff. If you start there and finish there, then you get this sameness and this sort of averaging of all the big pile of data. There's another step that's called post-training where you inject the life of the motion. It’s just simply a very small hand curated set of beautiful stylized motion with lots of anticipation, lots of secondary motions and so on. You really force the model at the end to respect that quality. This is an open research question that is not yet solved by anyone. We're not there yet. We have a long way to go, but there are promising first results.
LM: You mentioned Flow earlier which is a very different style of animation from Across the Spider-Verse, which is very different from a Pixar movie. Can you see a world in which you could tune your model to the style of the show that you're making?
AC: I love that. The short answer is emphatically yes!
There's two ways to do it. If you are a studio and you have a large history of motion that you've done for different shows, you can work with us to get a custom model. We don't advertise that anywhere, but we'd love to work with anyone who wants a custom model. That involves additional training, access to the model and stuff. It's a little bit more complicated.
And there's a second option, which is right around the corner, is simply where you upload three or four or five references - a walk cycle, an idling cycle, maybe some performance motion. Because of how the model itself is structured you can prompt with examples in motion and then all the generations thereafter will be closer to the style of this motion. Does not require additional training.
I can imagine even a 2D animated film that's trying to make the transition into 3D, they have all this beautiful 2D motion. Put it through our video reference uploader train a custom style model and then suddenly you can generate 3D versions of this beautiful 2D style that's existed forever. I am very excited for that future.
LM: As you know, a lot of artists working in animation are really nervous that tools like yours are going to automate their jobs away. In the past you have mentioned that Cartwheel could make the work 100 times faster. Jeffrey Katzenberg, who is one of your investors, has said he thinks animated movies could be made with just 10% of the current workforce. So it's obviously pretty scary for people working in animation. What's your message to people who are worried right now?
AC: I believe that the demand for animation is so much higher than the supply that it's unreal. We don't see that demand because it's just too expensive today, but if it were slightly easier then I think animation is going to be everywhere. We're going to see it in all of our communication and all of our ads.
If you think back to the early days of design, you used to have an entire design firm to make a poster. You'd have someone to do the typeset and you'd have someone to do the drawing and so on. With the advent of Photoshop now, every team has a designer and there's so much more design being done. Yes, we don't have a huge design firm in the same way we used to, but there's so much more work for designers. And these designers can accomplish so much more - a single person can make a great print ad.
I want to live in a world where the individual is more empowered. Today we live in a world where the studio holds a lot of the power and they green light and hire and fire sort of at will and the animators are unfortunately not treated as well as they should be. I want to flip that script - empower the animators and give them the tools they need so that they can create more.
I don't know what's going to happen in the future. I can't pretend to predict what's going to come, but we're building the tools for the animator to be empowered to make more things. That’s our goal here at Cartwheel.
LM: So you see a world with smaller teams, more independent artists, and these tools are empowering individual animators to do that?
AC: Both ends of this - I think smaller teams can do more and I think larger teams can do more. We're currently stuck in this middle ground of animation where you have to be of a certain size to do anything and I would love to spread that out to more people.
LM: Thanks for taking the time to talk with me. Is there something I didn't ask that you want to mention?
AC: Just one thing briefly. I've said this implicitly, but I love to say it explicitly. We're very open to feedback. Please use the tool and tell us what you like. Tell us what you don't like. We have a Discord, we have emails, you can tweet at me, whatever you need. I have a vision in my head for what this tool needs to be, but I'm very open to being wrong. So please, please, please reach out. Tell me what you like, what you don't like.
LM: Thanks again, Andrew.
AC: Thank you!
I hope you enjoyed my conversation with Andrew Carr. Let me know if there are other people you’d like me to feature here. And if you’ve had a chance to take it for a spin, let me know what you think of Cartwheel. Does it deliver on what the company claims? Is it a threat to animation jobs?
Until next time,
Matt Ferg.
Very interested in seeing some shorts based on this toolset.
Great Article Matt. Cartwheel seems to be a company with integrity. interested to see what they do next.