About the conference

In its 52nd year, the SIGGRAPH conference remains home to a global community of computer graphics and interactive techniques enthusiasts whose collaboration and storytelling shape our trajectory toward a better future. Our content, whether submitted or invited, is selected by experts whose goal is to create inspiring experiences and to showcase excellence. We invite you to be part of this community of outstanding innovators — where engineers, artists, designers, computer scientists, educators, academics, and industry leaders seamlessly mingle to create new technology and shape the human story.

Link to the conference

Our talk

August 12, Tuesday 11:30 - Mainstream Movies video eval toolkit: How we evaluate cutting-edge VideoGen models

As video generation technology rapidly advances, the need for robust evaluation frameworks becomes critical. While automated metrics are mostly sufficient for traditional AI modalities, video generation models require human experts to assess temporal coherence, context, motion realism, visual quality, and aesthetics in ways that automated methods cannot easily replicate yet. Additionally, current benchmarks for visual content are not always actionable and understandable for the AI developers, being too subjective and lacking formal criteria. At Toloka, together with industry professionals we have developed the Mainstream Movies video evaluation toolkit, a comprehensive evaluation framework for cutting-edge VideoGen models to close the gap between creative users and engineers. Our approach combines domain expertise with systematic and detailed evaluation protocols to deliver actionable insights for AI developers. We will demonstrate how our toolkit addresses the unique challenges of video generation evaluation, showcase evaluation results from leading VideoGen models, and discuss the methodology that enables scalable, professional-grade video assessment.