OPERAcraft

If you are a gamer and/or have kids, there is practically zero chance you’ve never heard of Minecraft. This truly unique sandbox/game hybrid has taken the world by storm, becoming an integral part of our 21st century digital culture, and spawning an ever-growing army of clones. In 2013, when my colleague Ariana Wyatt told me she was  looking for an exciting way to introduce kids to the opera genre, I suggested to her producing a virtual opera that could be viewed simultaneously by thousands of viewers all across the globe using a gaming engine. Given its unprecedented popularity among the target population, I suggested modding Minecraft and making it into a comprehensive real-time machinima production environment that goes beyond Minecraft’s default affordances. That same year, the research team received a $25,000 ICAT seed grant to pursue this idea.

OPERAcraft is a Minecraft mod designed to interface with pd-l2ork, so as to allow for real-time control of various aspects of opera production. Children engaged in the project were given an opportunity to craft their own story and libretto, make their own costumes and build an entire world with predetermined locations where the action would take place, as well as control various aspects of the real-time theatrical production. My colleague and project collaborator Tracy Cowden focused on retrofitting famous Mozart arias with the newfound libretto and composing connecting passages to smooth transitions between scenes.

OPERAcraft in performance

Considering opera called for a real-time production, there was a need for a number of features that were unavailable in Minecraft and, to the best of my knowledge to this day remain absent in any of the machinima-friendly gaming engines. These include:

  • A reasonably accurate real-time mouth animation matching that of the speaking human, or in this case opera singer;
  • A reasonably flexible body language, in this case independent arm movement and postures commonly associated with opera acting;
  • Ability to cross-fade between scenes and instantly teleport actors to predetermined locations, so as to allow for uninterrupted scene changes;
  • Ability to use different camera angles and seamlessly transition between them, akin to that of a movie production, that are visible to actors and appropriately marked to reflect the current active camera but invisible to the audience;
  • Ability to communicate among actors and production staff without making such communication visible to audience, and
  • Subtitles accompanying opera singer to make drama easier to follow.

Below is a diagram of how we went about making these possible. Pd-l2ork served as our audio capture mechanism that used a simple algorithm designed to monitor for melismas and transients from singer’s audio feeds. These were broadcast to a custom socket inside Minecraft that controlled “helmets” on players heads, each helmet offering a different mouth position. Pd-l2ork was also used to broadcast production staff cues, subtitles, and control scene and camera changes. To make Minecraft capable of handling these new features, my undergraduate student researcher Cody Cahoon and I set out to decompile, deobfuscate, and modify Minecraft code base, which at the time was version 1.5.2, using tools like Modloader, Forge, and Mod Coder Pack. The newfound mod offered all the aforesaid features, plus a reasonably simple configuration, while also leaving room for future expansion where we could have audience members as spectators inside the game experiencing the drama from their preferred vantage point, while remaining invisible ghosts unable to affect the drama or the world (so, no, you could not blow up the key scene with a TNT block spam).

OPERAcraft technical setup diagram

In the final production, there were five soloists accompanied by piano (performed by Tracy Cowden). There were five characters and two camera operators, all of whom were middle- and high-school students. Additional professional staff assisted with various technical aspects of the production. The premiere took place in December 2014 in Virginia Tech Center for the Arts‘ Cube in front of a standing room only audience. Perhaps an even more exciting aspect of the premiere were over 1,500 viewers who experienced OPERAcraft via a live stream with an unexpectedly high viewership from Germany (it was approximately 230am German time when the performance took place). Due to demand, the performance was repeated several days later and over the next month an additional 30,000 visitors viewed the archived video stream that has since been mirrored on YouTube. The 20-minute opera was seen as a resounding success and we were literally showered with requests for a sequel.

Although there was no way of anticipating this at the time, the supporting tools we used for decompiling and deobfuscating Minecraft code have since been deprecated in favor of better and more portable approaches, which unfortunately made our changes forward incompatible. Nonetheless, in December 2015, OPERAcraft is scheduled to present its long awaited sequel as part of Center for the Arts’ 2015-6 season engaging a whole new generation of kids. As I look forward to reworking the mod in a more forward-compatible fashion, this is something that is unlikely to make the December deadline. I hope kids won’t mind creating an opera that lacks cats and ponies.

Research team:
Ivica Ico Bukvic (Co-PI)
Tracy Cowden (VT)
Katie Dredger (JMU)
Kelly Parkes (VT)
Ariana Wyatt (VT, PI)

Research Publications and Published Interviews:

Software:

OPERAcraft website