Reminder: It’s only been 1 YEAR since the Will Smith eating spaghetti videoOpenAI Sora: https://openai.com/sora#researchThumbnail character: BasedAFMKBHD Mer...
I think the idea of using natural language to generate video is flawed for the vast majority of applications we want. Imagine you could give a script to one of these models and have it output a TV show episode. While we can make these models deterministic it seems like the vast majority of generative content with some amount of quality requires the addition of random noise through the process. Should we want TV episodes whose visual quality and little details shift from model to model? Why not store a plain text description infered by some model and store the video component in a medium less prone to misinterpretation? We may use deep learning compression for videos and audio in the future if there are significant advancements but I doubt the compression will be to English.
I think the idea of using natural language to generate video is flawed for the vast majority of applications we want. Imagine you could give a script to one of these models and have it output a TV show episode. While we can make these models deterministic it seems like the vast majority of generative content with some amount of quality requires the addition of random noise through the process. Should we want TV episodes whose visual quality and little details shift from model to model? Why not store a plain text description infered by some model and store the video component in a medium less prone to misinterpretation? We may use deep learning compression for videos and audio in the future if there are significant advancements but I doubt the compression will be to English.