What is Luma AI’s Dream Machine? Why fans are flocking to video generator

Luma Labs, the AI company known for creating the Genie generative 3D model, has now ventured into AI-powered video technology with its new offering, Dream Machine — and social media users have been raving about it. Among several new AI video makers on the scene, it is in direct competition with OpenAI’s upcoming Sora tool.

What is Luma AI?

Luma AI is a platform where users can create 3D models using text, images, or video inputs. The tool helps professionals such as those in architecture, product design, content creation, and research and development, dream up abstract concepts. The company streamlines the process of exporting work across various platforms.

The 3D models are also advancing meta-activities such as gaming, virtual reality, and e-commerce by simplifying the creation of 3D online spaces.

Developed by LumaLabs in San Francisco, California, Luma AI uses knowledge of machine learning, augmented reality, and 3D photography to merge AI with the field of 3D modeling.

Using Neural Radiance Fields (NeRF) technology to transform images and videos captured by smartphones into highly realistic 3D models, NeRF alters ordinary scenes and objects by adding an element of realism to them.

Co-founder and CEO Amit Jain, who was previously part of the Apple Vision Pro and Apple’s computer vision team, is currently leading the charge.

Among the various gadgets, Dream Machine is the company’s new contribution, an AI model designed to produce high-quality, realistic videos from text and images. So far, it generates accurate, consistent, and dynamic visuals. The tool moves toward creating a universal imagination engine and is now accessible to all.

In a statement on their blog page, Luma Labs said: “We have built Dream Machine on a scalable, efficient, and multimodal transformer architecture and trained it directly on videos.”

They described the tool as an “imagination engine,” as “we find joy and purpose in making things. At Luma we are building general AI systems that will help people make beautiful, powerful, creative things that are inaccessible or simply impossible.”

How have people reacted to Dream Machine?

The launch of Dream Machine saw such a surge in demand that Luma Labs had to implement a queuing system after their servers were overwhelmed.

Introducing Dream Machine – a next generation video model for creating high quality, realistic shots from text instructions and images using AI. It’s available to everyone today! Try for free here https://t.co/rBVWU50kTc #LumaDreamMachine pic.twitter.com/Ypmacd8E9z

— Luma AI (@LumaLabsAI) June 12, 2024

She's alive 🤯 pic.twitter.com/dnr3nPC1Aw

— fofr (@fofrAI) June 13, 2024

Beepee-Oh and Bah-bah – My kids' cherished stuffed animals brought to life tonight by @LumaLabsAI and a little elbow grease from dad. 🐦‍⬛ I get to show them tomorrow morning pic.twitter.com/ELm4bNtokW

— Stormcrow (@Stormcrow_JE) June 13, 2024

Early videos from users with prior access to Dream Machine posted on social media were so polished, they seemed almost too good to be true as if only the best results were chosen.

First try with #Luna AI and the result was amazing, happy to switched my career from cinema and animation to development some years ago so that I can survive for a couple of more years pic.twitter.com/Yn6JegVA8P

— Amir | Ξ𝕏𝕏ΞO (@Amir_EXXEO) June 13, 2024

The text-to-video generation field is intensely competitive, with key players such as OpenAI’s Sora and Lightricks’ LTX Studio, both of which provide decent videos.

Other startups doing something similar include Pika Labs Inc. and Runway Inc., both of which focus on generating video content from text.

How to use Luma AI’s Dream Machine

To make the best use of the tool, users have to type extremely detailed prompts, more so than when using tools like ChatGPT. For example, writing “grazing cows move slowly across an idyllic meadow, the camera tracking alongside them in a smooth side-angle motion” could make a highly realistic five-second video clip showing exactly that scenario, according to the company.

Dream Machine should be able to create such a video within just two minutes, it said.

ReadWrite tested it, and it provided some interesting results. We used the prompt: “Dream sequence of a woman walking up a mountain turning into an eagle and flying away”

While it starts off fairly accurate, it does something strange by creating a clone of itself showing two different women – one standing and the other flying away.

The second prompt had even less success: “Sequence of a sports motorbike riding on Route 66 in the US, transforming into an eagle”

Instead of the motorcyclist changing into an eagle, he appears to speed off to the side, having a mini accident while a bird randomly flies over. What we found is that the text prompts have to be rather specific for the best results.

With the final test, we used the prompt suggested by Dream Machine itself: “An old lady laughing underwater, wearing a scuba diving suit, her expression denotes calm and happiness”

Its own recommendation worked rather well. Although we can’t zoom in to spot the usual wonky eyes as seen with many AI images, it appears to look very realistic.

Is Luma AI’s Dream Machine free?

As mentioned, anyone interested can use it, but you’ll need to register for an account using your email. Once registered, you’re entitled to 30 generations per month on the free plan. More costly plans offer up to 2,000 generations per month for $499.

Currently, the service is in high demand, which may limit accessibility. The Dream Machine website displays a caution stating, “Generations take 120 seconds. However, due to extremely high demand, your request will be queued.” Being able to create 30 videos for free is useful, especially since most people may not require more than that for personal purposes.

One of the biggest benefits of Dream Machine is its open-source framework, courtesy of Luma AI. In contrast, OpenAI’s Sora, though impressive, is only available to a select test audience, with hints from the company that its top features will be exclusive to paying subscribers, similar to its ChatGPT model.

Dream Machine, on the other hand, is currently accessible to anyone interested in exploring its capabilities. Looking ahead, Luma AI plans to release various APIs and plugins that will allow integration with creative software tools, including Adobe.

Featured image: Luma Labs / Canva