Sora Pushing the Boundaries of Realistic Video Generation in Artificial Intelligence

29.04.2024

OpenAI’s video generation tool, Sora, made a surprising entrance into the AI community in February, showcasing fluid, lifelike videos that appeared to surpass its competitors. However, the initial unveiling, carefully orchestrated as it was, left out crucial details—details that have now been elucidated by a filmmaker granted early access to Sora for creating a short film.

Shy Kids, a digital production team based in Toronto, was selected by OpenAI to produce short films primarily for promotional purposes, although they were granted considerable creative freedom in crafting their pieces, such as “air head.” In an interview with the visual effects news outlet fxguide, post-production artist Patrick Cederberg shared insights into his experience of “actually using Sora” as part of his creative process.

One significant takeaway from Cederberg’s account is that while OpenAI’s presentation of the shorts may suggest they were generated fully formed by Sora, the reality is that they were professional productions involving extensive storyboarding, editing, color correction, and post-production work like rotoscoping and visual effects. Similar to Apple’s “shot on iPhone” campaign, which highlights the end product without revealing the behind-the-scenes process, the Sora post emphasized what the tool can achieve rather than how it achieves it.

Cederberg’s interview sheds light on the challenges faced in using Sora, particularly regarding control and consistency. Achieving precise control over elements like character wardrobe or movements proves elusive, requiring detailed prompts and manual adjustments in post-production. Unwanted elements, such as unexpected faces on balloons or extraneous strings, often had to be removed manually, adding to the time-consuming nature of the process.

Moreover, precise timing and movements of characters or the camera are difficult to achieve with Sora, leading to approximation rather than exactness in animations. Common filmmaking terms like “panning right” or “tracking shot” were found to be inconsistent, highlighting a disconnect between the tool’s capabilities and filmmakers’ expectations.

Despite these challenges, Sora remains a powerful and valuable tool in filmmaking, albeit not yet capable of creating entire films independently. Its refusal to generate content resembling copyrighted material raises questions about its training data and recognition capabilities, which OpenAI keeps closely guarded.

In conclusion, while Sora represents a significant advancement in video generation technology, its current limitations necessitate a collaborative approach with human filmmakers, rather than entirely autonomous creation. As advancements continue, however, its role in filmmaking may evolve, as hinted by the famous adage: “that comes later.”