Sponsored by Adobe
At last week’s Adobe MAX event in Miami, Adobe announced over 100 new updates and innovations across its portfolio of applications. Only last month, Adobe announced a series of significant upgrades to Premiere Pro, including automated colour management, faster performance for file imports and exports, a more elegant and intuitive UI and more refined toolsets for dialogue editing, voice isolation, dubbing and lip syncing.
Adobe had seeded the story of Firefly integration with Adobe Premiere Pro over the past year and shared sneak previews at trade shows and other events. What was announced at Adobe MAX was more than just fulfilment on this promise with three key AI developments that deliver extraordinary video content and are commercially safe to use.
Generative Extend
Many video editors will tell you that they commonly don’t have enough footage to cover dialogue, work with the soundtrack or simply to fulfil the narrative from the recorded clips they have available within the edit. In turn editors employed workarounds, like slowing down the footage, which can feel awkward or unnatural, cutting in alternative footage that doesn’t quite work in context or losing narrative or audio they would rather leave in. This challenge is most commonly linked to dialogue length or the soundtrack within the audio timeline or might simply be a contextual judgement call.
With Adobe’s new Generative Extend feature within Premiere Pro editors can now add those extra frames to cover gaps in footage or to smooth out transitions. Generative Extend also allows the editor to hold on to both A and B roll shots for longer so the narrative can all be included or so that the soundtrack cuts where it feels more natural. Generative Extend empowers the editor to make better timed edits to better serve the story without compromising visual flow. The results look natural even with foreground movement within a detailed environment.
This revolutionary new feature is powered by the Firefly Video Model. Adobe showed working examples of Generative Extend where the extra frames were only apparent when pointed out, demonstrating both the simplicity and immediacy of this unique feature.
Generative Extend is directly integrated into Premiere Pro and clips can be extended and rendered in moments on the timeline, offering not just better results but considerable time savings.
You can see the amazing results within the thumbnail video at the top of this story.
The Adobe Firefly Video Model
Text Prompt to Video
Adobe also seeded their Firefly Video Model over the last six months which has now launched in beta.
Adobe Firefly can interpret detailed text prompts with exceptionally nuanced photorealist results and crucially without the IP issues that have to date undermined the viability of earlier text to video models.
The key difference is that the video is only trained on content that Adobe has the rights to use.
A still from Text-to-Video in Firefly. Image: Adobe
You can use highly detailed long prompts such as, [Prompt] Cinematic closeup and detailed portrait of an elderly man in the middle of a street at night. the lighting is moody and dramatic. The colour grade is blue shadows and orange highlights. the man has extremely realistic detailed skin texture and visible pores. movement is subtle and soft. the camera doesn’t move. film grain. vintage anamorphic lens.
See this lovable robot animated here
Prompt: cinematic realistic detailed shot of a cute robot holding up a red glowing heart. The lighting is gorgeous and sun-kissed, with dappled lighting on the robot’s face and a strong sunny backlight. realistic details and textures. The color grade dreamy, sunny, warm tones, shallow depth of field. Film grain. Shot on kodak 35mm film. Sunlight filters gently through the window, creating a delicate and ethereal atmosphere. Out-of-focus green plants in the foreground. Slow-motion, gentle motion. Camera is static and locked-off.
You can see further examples within a sizzle reel here
Image and Illustration to Video
Adobe has applied the same learning model to Adobe owned or your own images or illustrations and transforming them into beautifully realised 2D and 3D animations.
See this fire lizard here
Directing the new AI content
The editor or designer additionally has the ability to iterate on or direct the new sequences with rich camera controls including angle, motion, and zoom to create the perfect perspective while delivering showstopping video quality at a key stroke.
Adobe’s training model has been engineered to encompass a world understanding of motion and light movement to create more believable and realistic images. In turn this more detailed understanding allows for further iterations on AI generated content.
Generative AI: Accountability, Responsibility and Transparency
At the start of the year, the industry was shocked to see detailed, often photorealistic, AI generated clips. However, the conversation quickly turned to questioning two fundamental questions:
- What was the provenance of the source material? Had the content been created by scraping the web for third-party images and therefore had probably involved appropriating third-party unlicensed IP? Many agreed this infringement was not just an ethical dilemma but also held future legal implications
- A second consideration was that having created the content, it was impossible to further iterate on the clip, that is to adjust the clip to what was required
With Adobe’s Firefly Video Model, these primary issues are resolved. Commercial usability is no longer an issue as all the assets used to build the video clip or animation are sourced from within Adobe’s own fully licensed content library with millions of high-quality images and clips to draw upon or from stock that is within the public domain.
Adobe’s unique Firefly Video Model does not draw on customer data or any data trawled from the internet. Without the possibility of infringing on IP, the user is crucially offered full indemnification. Adobe has taken this one step further again to include content credentials and so fully reassure enterprise customers that Firefly text to video is safe for commercial use.
“We see generative AI as a tool for, not a replacement of, human creativity,” said Adobe CEO Shantanu Narayen in his opening remarks at Adobe MAX. “We only train when we have permission. We compensate creators for training. We do not train on customer content. We do not scrape the internet.”
See what all the excitement was about at Adobe MAX and visit Firefly.Adobe.com.
Pippa Considine
Share this story
