Runway Chronicles: Melding Art and AI in a Transient Tech Terrain

There is almost nothing less stable than the landscape of Generative AI (GenAI). The last week's changes include a sudden $0.3 billion drop by Jasper. It’s not a surprise in a world where apps are built on someone else’s model/technology. But we will be talking about something ‘stable’ today: the Stable Diffusion model and Runway – whose researchers are behind the model development. It’s interesting that Stable Diffusion has also become a bedrock for another unicorn from our chart – Stability AI. But today we concentrate on Runway, number 7 in the unicorn family (so far).

1. The starting point of Runway

2. Financial situation

3. Open-source and copyright tensions

4. Mission of Runway

5. Founder’s thoughts about AGI

6. Way to Runway and Stable Diffusion success (linked to the important research papers)

7. Flagship product (and other interesting projects)

8. Tech behind Runway’s latest Gen 2 model

9. How does Runway make money?

10. Little-known facts about Runway

11. Bonus: All important links about the founders

The starting point of Runway

Four years before Stable Diffusion sparked the text-to-image excitement wave in the GenAI world, the idea of melding art and tech was brewing in the minds of three young visionaries in New York University’s Interactive Telecommunications Program (ITP), a grad program at the intersection of technology and the arts. Meet Cristóbal Valenzuela, Anastasis Germanidis, and Alejandro Matamala-Ortiz.

The idea of the name for Runway traces back to Valenzuela’s thesis project at NYU’s ITP where he collaborated with Dan Shiffman (Board of Directors at ITP) and met his future co-founders while understanding the applications of machine learning (ML) in the creative domains. In the early stages of research, Valenzuela sought a concise name to discuss the project with his advisors. During his brainstorming sessions, he recognized that the concept of "a platform for models" already had a fitting moniker: a runway.

In August 2018, while TikTok was gaining prominence followed by its merger with Musical.ly, the trio decided to establish Runway – a startup dedicated to developing tools for video creators supercharged by ML; while also crafting the tech behind these tools.

Initially launched in 2019 as a model directory, Runway lets users deploy and run open-source models for various artistic use cases. Over the years as the user base expanded, the startup pivoted towards developing ML-based video editing tools. It was in the founders’ initial concept to be open-minded about how people might be using their platform.

Today, Runway has tapped into multi-billion dollar software markets.

Financial and legal situation

Total funds raised soar to $237 million, tripling its valuation to a whopping $1.5 billion in 2023. But there are some tensions in the GenAI world…

Open-source tension

While the company has managed to keep its legal slate clean during all these years, last year, the company got in a tussle with Stability AI due to which the companies have turned from friends to foes.

The discord between the companies arose from the release of the Stable Diffusion model, a collaborative initiative led by Patrick Esser from Runway and Robin Rombach from LMU Munich (also, Stability AI). The model's code was open-sourced under the CreativeML Open RAIL M License, thanks to a trilateral effort involving Runway, Stability AI, and LMU Munich's CompVis group. However, tensions flared when Stability AI lodged a takedown request against Runway citing an IP leak with Runway ML's SD 1.5 model, an extended version of Stable Diffusion. Runway's CEO, Cris, refuted the IP breach claim in a Hugging Face discussion, appreciating Stability AI for a compute donation aiding the retraining of the original model. The takedown request was later withdrawn by Stability AI, suggesting a resolution to the disagreement.

Such hiccups in collaborative AI projects highlight the importance of balance promoting an open-source ethos for communal growth, and upholding IP rights to ensure fair recognition and protect innovations.

Copyright tension

Furthermore, shortly, it is highly unlikely that the company akin to Midjourney.Inc and Stability will be able to dodge legal issues arising from the unconsented use of copyrighted material in their model training processes. In May, Runway appointed its first general counsel, hinting towards the fact that the startup is gearing up for the incoming matters of law. 

In a recent discussion with The Information, Valenzuela recollected how bootleg DVDs were common, peddled by street vendors when he was a child. He noted that eventually, norms and regulations were established to curb such practices, pointing toward the current situation surrounding AI and copyrighted content. “No one ever thought about, like, restricting the technology or the progress of cameras,” he said.

Mission of Runway

Hailing from Chile (Valenzuela and Matamala-Ortiz) and Greece (Germanidis), these immigrant entrepreneurs wanted to democratize generative AI, making capabilities ranging from crafting realistic copy to generating art accessible to people without coding backgrounds, akin to Adobe Creative Suite's approach.

For him, the primary mission isn’t catering to pros; it’s expanding the borders of the AI art world, Valenzuela stated in 2019.

Unlike the tech goliaths, Runway has built its model with a customer-centric approach in mind. “This is one of the first models to be developed really closely with a community of video makers,” the CEO had told MIT Technology Review. “It comes with years of insight about how filmmakers and VFX editors actually work on post-production.”

Founder’s thoughts about AGI

Valenzuela is certainly from the anti-AI doom camp. “First of all, not everyone in the industry is trying to build God,” he told The Information. “We’re trying to build tools. The keywords are not replacement, but augmentation, right? Enhancement.”

To illustrate his point, he drew parallels between the current AI discourse and the early days of cinema when the first film was screened. “Legend has it that when the film was first screened, the audience panicked and fled the theater, convinced that the train in the film was going to burst through the screen and run them over. AI is not going to burst through the screen and run us over.”

Way to Runway and Stable Diffusion success (linked to the important research papers)

Flagship product (and other interesting projects)

The narrative further evolved in 2021 with the introduction of Sequel, an ML-enabled video editor, signaling a move from a freewheeling open-source hub to a more structured layout reminiscent of Adobe's Creative Suite.

Building upon this diffusion architecture, Runway recently unveiled Gen-1 and Gen-2 video generation tools, expanding the horizons of artistic expression by synthesizing videos from text, images, and other videos. The Gen 2 model, in particular, empowers filmmakers and editors to craft productions from scratch, which can be further enriched by AI-generated content.

In October 2023, they partnered with Canva, “to bring the power of Runway and AI video generation to Canva users all around the world.”

As of October 2023, Runway has metamorphosed into a full-stack platform offering a spectrum of applications from model research to end-user tools like text-to-image, video-to-video, and text-to-video generation.

Tech behind Runway’s latest Gen 2 model

Patrick Esser who is a Principal Research Scientist at Runway co-authored the foundation model behind Stable Diffusion with Robin Rombach, a LMU Munich and Stability AI research scientist. “We got really promising results on our smaller scale. It showed us that it’s really effective for synthesizing images for different tasks. Then it was like - can we scale this up? And that led us to Stable Diffusion. It is really the same model, with slight changes but not too essential. Just on a bigger scale in terms of our resources,” Esser mentioned in an interview explaining the origins of SD.

Tech Spec

How does Runway make money?

The company has revamped its pricing model several times since its debut. While predominantly attracting “prosumers” (professional consumers), Runway is also witnessing enterprise traction as today the billion dollar worth of its solutions have already been used by New Balance, CBS, Ogilvy, VaynerMedia, and Publicis.

Its software is free to download and comes with 125 credits. The company offers four options depending on the usage amount for individual users to enterprises which is a source of revenue.

Its annualized revenue is estimated to be around $1 million, and its annual recurring revenue (ARR) is now "under $5 million". This revenue is generated through the sale of its products and services.

Some users have reported that Runway has affiliate programs, where they can earn money by promoting the company's products and services. However, the extent of the company's reliance on affiliate programs as a revenue source is unclear.

Little-known facts about Runway

• Oscar-winning films ‘Everything Everywhere All at Once’ and ‘The Late Show with Stephen Colbert,’ have used Runway’s software to create visual effects and scenes.

• The famous video animator group Corridor Digital has used and promoted RunwayML’s tools in several instances in recent years.

• American hip-hop band Brockhampton’s 2021 album has a dozen videos made with Runway.

• In 2023, Runway made it to TIME’s list of 100 most influential companies.

• Co-founder Germanidis spent time as an ML researcher at IBM Research.

• The third co-founder Matamala-Ortiz has been associated with Ediciones DAGA, an independent art book publisher focused on South American visual artists.

Bonus: All important links about the founders

Based in New York, Runway is an “applied AI research company shaping the next era of art, entertainment and human creativity”, founded in 2018.

Cristóbal Valenzuela, Co-Founder and CEO

Twitter / LinkedIn / Personal website

Anastasis Germanidis, Co-Founder & CTO

Twitter / LinkedIn / Personal website / Publications / GitHub

Alejandro Matamala-Ortiz, Co-Founder, Product Designer and Front End Developer

Twitter / LinkedIn / Personal website / GitHub

To be continued…

If you like us and want to receive a complimentary Premium subscription, or you just want to make us happy, invite at least three of your friends and colleagues. They might enjoy it as well.

Thank you for reading.