Leveraging AI and game-engine technology to transport music festival attendees to a virtual recording studio.
Tool worked with the ambitious team at Coke Studio and our great collaborators at Momentum Worldwide to create a traveling experience that will be activated at a dozen or more music festivals throughout the summer and fall of 2023. The experience was conceived of and designed to target a young demographic of festival attendees who are fanatics for music and love to capture themselves and share on social.
Leveraging AI
In order to excite this audience, we leveraged Tool’s custom AI pipeline (ChatGPT API + Stable Diffusion + Custom trained models + Deforum Animation) and added a few layers of game engine and virtual production, to transport guests to a virtual Coke Studio.
We asked guests a few simple questions about themselves such as where they were from, their favorite genre of music as well as some questions about their visual aesthetic and their vibe/personality. We took those user data points and leveraging a few layers of AI, generated some dynamic assets to make the experience amazing.
The Activation Space
The installation was built with smart container construction, allowing it to be a modular transportable build. It even included a second level. The main spaces included:
Queuing / User Onboarding
VIP Rooftop
Recording Studio
Merch Giveaway
The User Experience
As people passed by the activation space, they walked up to the onboarding bar where a brand ambassador would explain the experience and give them an iPad to interact with. A few quick, fun questions allowed us to get their email address and gather some additional user info that would power the generative assets. In real time we returned a few AI generated band names, along with AI generated album art, with the user selecting their favorite options.
Their band’s recording session was then automatically added into the queue and they were led up to the VIP rooftop where they could enjoy the view, have some selfie moments and wait for their number to be called before making their way to the studio. (In the background, our system also began to render a 35 second animated AI video unique to the user.)
The AI Recording Studio
The moment for Magic has arrived. Upon entering the studio, the guests see a green screen performance space and a 75” touchscreen which intuitively guides them through their experience.
Upon tapping ‘Start’, a music visualizer animates on the screen and the music that was generated for them would blast in the studio for them to sample. After this quick preview of their music a countdown appears “3, 2, 1” and their performance begins.
Seeing a virtual camera boom through an interesting Coke Studio, the crew would quickly see a live preview of themselves on the screen in front of them, their green screen video keyed out, with them performing on the stage. The studio includes a large LED wall behind them, displaying the animated AI visuals that were generated for them. Each performance was also enhanced with game engine atmospherics including fog, pyrotechnic effects and a laser show. As the virtual camera moves from a wide to a close up, their music plays throughout the space, bringing the virtual to physical and inspiring them to dance. After the 35 second performance concludes, they had the opportunity to save their performance or re-record and encore. Upon tapping ‘Save’ their performance (already rendered in full HD) as uploaded to the cloud and a unique QR code was displayed for them to save their video and share on social.
The Takeaway
Each user was immediately delivered a 00:35 second video of their performance, within 3 seconds of completing the experience. They could scan a QR code in the booth which would launch a landing page where their video is hosted, or they could find the link in their email immediately. The videos were captured and rendered at 4k and were output in portrait orientation to maximize their native share ability on TikTok and IG.
