News
2022.08.11 · Maintenance release with additional calibration and visualization options, version 2.5.5 (CHANGELOG).
2022.04.12 · Bug-fix release with improvements, version 2.5.4.
2022.03.17 · Bug-fix release with minor improvements, version 2.5.3.
2022.02.15 · Bug-fix release with minor improvements, version 2.5.2.
2022.02.09 · Bug-fix release with minor improvements, version 2.5.1.
2022.01.25 · Bug-fix release with new builds of the external, version 2.5.0.
2022.01.17 · Minor release with documentation and demo patch improvements, version 2.4.1.
2021.06.08 · Major release with the introduction of the binaural mesh, bug-fixes, documentation, version 2.4.0.
2019.09.17 · Minor improvements and bug-fixes, version 2.3.1.
2019.08.22 · Major release with a native Max external, bunch of additions, improvements, and bug-fixes, version 2.3.0.
2018.09.10 · Minor improvements and bug-fixes, version 2.2.2.
2018.08.06 · Minor improvements to the modular output example, version 2.2.1.
2018.08.03 · Enabled applying mask to custom parameters and modular output abstractions, included Moogfest 2017 demo, version 2.2.0.
2017.05.19 · Minor version bump for the Moogfest 2017, version 2.1.2.
2016.09.14 · Minor version bump, version 2.1.1.
2016.09.11 · Upcoming workshop at the ICMC 2016 (Sept. 15th @ 130pm in RASA Studio). Special pricing available for the workshop participants.
2016.08.15 · Version 2.1.0 now available with new features, including the D4.timeline.editor, and a series of optimizations.
2016.07.07 · ICAD 2016 paper now available online.
2016.07.05 · ICAD 2016 workshop, released version 2.0.6.
2016.06.20 · First public release, version 2.0.0.
Welcome to the Future of Audio Spatialization
Originally introduced in 2014 as a framework for a tornado simulation in the newfound Virginia Tech’s $30M holodeck-like ICAT Cube, D4 (2016) is a comprehensive audio spatialization library for the MaxMSP (version 7) visual programming language. It consists of two components:
- An innovative Layer Based Amplitude Panning (LBAP) algorithm first publicly introduced at the ICAD 2016 conference in Canberra, Australia, and
- A multimodal library of MaxMSP patches and abstractions designed to optimize the use of the LBAP algorithm and complement it with first-of-a-kind cutting-edge features, such as the Binaural Mesh, Motion Blur, and the Spatial Mask.
Yet another spatialization algorithm?
From an artistic perspective, whereas the 20th century can be seen as the emancipation of timbre in music, 21st century is shaping into the long overdue emancipation of spatial sound. Sound spatialization is indeed the last remaining frontier in the musical expression that is yet to attain structural importance matching that of pitch, loudness, timbre, and their derivatives, such as rhythm, harmony, and polyphony.
The lack of the spatial revolution may be in part indebted to the slow and often costly adoption of High Density Loudspeaker Arrays (HDLAs) which has in turn limited its extensive research and integration as a core element that guides musical form and structure. The second reason may be the lack of tools that make time-based spatialization a breeze, particularly in a growing number of unique HDLA venues. Even though massively multichannel electronic music works already point towards the aforesaid spatial emancipation, they tend to be regarded as experimental and are usually difficult to transport between various venues. Similarly, the mainstream media seldom goes beyond the conventional stereo sound diffusion with Dolby Surround entertainment systems rarely utilizing additional channels in a compelling way.
In respect to research in the area of audification and sonification, spatial sound remains largely an unknown quantity. Other emerging areas include immersion, Virtual and Augmented Reality. The lack of an accessible and easily transportable/reproducible translation of data sources into spatial sound is seen as a major obstacle. Consequently, the review of the existing spatialization tools, their features, and limitations has identified core traits necessary for a tool to optimally address the said challenges:
- Support for irregular High Density Loudspeaker Arrays (HDLAs), with particular focus on perimeter-based 3D spatialization
- Focus on the ground truth and immersion with minimal amount of idiosyncrasies
- Leverage vantage point to promote data comprehension
- Optimized, lean, scalable, and accessible, and
- Ease of use through supporting rapid-prototyping time-based tools
D4 aims to address these challenges by:
- Introducing a new LBAP algorithm that gracefully scales to a large number of speakers supporting virtually any speaker configuration (e.g. in the ICAT Cube it has been utilized with up to 128 speakers with sub-20ms latency), while also providing ground truth with minimal amount of idiosyncrasies;
- Integrating the the LBAP algorithm into a comprehensive library of time-based editing tools, offering different levels of granularity from audio spatialization beginners to experts, thereby shifting focus away from the algorithm and low-level knowledge of spatialization to content spatialization itself, and
- Offering a Mostly Open Source Software (MOSS) approach where the proprietary components are lean and leverage maximum possible amount of built-in objects and tools whose open source model allows the ensuing community to easily and collaboratively build upon the core spatialization model and fine-tune system to their specific needs, including additional filtering and other techniques that further enhance its core functionality.
Unique Opportunities
What arguably sets D4 apart from other spatialization approaches is its accessibility and malleability. Alongside the usual azimuth and elevation parameters that enable it to serve as a drop-in replacement for the existing spatialization approaches, D4 also offers an array of interactive and time-based tools for manipulating spatial audio, making it suitable for hobbyists and professionals alike. These include options like Radius, a high precision version of VBAP‘s source spread that enables growing a sound source akin to a spotlight, and, more importantly, the Motion Blur, Spatial Mask, and the Binaural Mesh.
The Spatial Mask mimics its visual counterpart by allowing users to specify predetermined amplitude limits onto each physical source or loudspeaker. Each sound source or an audio bus can have its own mask. Where the Spatial Mask is 0 (zero), no sound will come out no matter the source position or amplitude, whereas where it is 1 (one) the mask will have no effect on the sound source as it traverses target loudspeaker. Therefore, as the sound source traverses the HDLA, its output will be limited by the Spatial Mask. Consequently, D4 leverages MaxMSP‘s Jitter library to empower its users for the first time ever to paint spatial sound. The built-in Mask Editor offers visual rendering of each loudspeaker’s Spatial Mask, enabling complex spatial mapping that goes beyond a simple point and spherically expanding source. More importantly, the mask can read from any grayscale multidimensional array and can be imported from just about any data source that is readable by the Jitter library. From video (e.g. with its RGB channels split into three masks and therefore three sources) to abstract multidimensional data sets, Spatial Mask can render such data by leveraging to the fullest extent human ability to perceive audio spatially. This makes it particularly fit for pursuing research in the area of spatial data sonification, artistic production, as well as the movie industry.
Spatial Mask is further coupled by other abstractions, including time-based mask reader and renderer. This allows for both real-time and non-real-time rendering of mask data and the ensuing sound. Each of these elements are carefully crafted as optional complementing modules that can be invoked and operate in perfect harmony with other components of the D4 ecosystem and beyond. Another exciting feature introduced by the D4 library is ability to customize per-channel audio envelope up and down ramps. As the audio increases in loudness, its increase can be different than its decrease. The ensuing envelope manipulation serves as the foundation for an aural version of a spatial Motion Blur.
As of version 2.4.0, D4 has introduced another first-of-a-kind feature–the Binaural Mesh. Whereas traditional approaches to virtualizing sound using headphone-based binauralization of the audio source utilizes one binaural pathway or plugin per source, the Binaural Mesh creates an array of binaural pathways or plugins for each virtualized loudspeaker. This approach offers a number of unique advantages over its traditional counterparts. First, it allows for a more accurate rendering of a venue it aims to simulate, including the way Radius propagates across various loudspeakers, any unique coloring of various loudspeakers, a finer control over special loudspeakers such as subwoofers, avoidance of potentially inconsistent coloring of the sound as it transitions from one position to another due to limitations of the parametric and/or convolution-based mapping, as well as simulation of other potential idiosyncrasies of the simulated space and the underlying spatialization algorithm, such as phasing between adjacent loudspeakers rendering the same sound. The same also allows for the use of D4‘s unique affordances, such as the Spatial Mask, as well as a battery of its supporting tools and widgets. It offers seamless transition from physical to virtual, requiring only minimal adjustments to your code. Each such virtual loudspeaker further enables per-loudspeaker enhancements, e.g. per-loudspeaker modulation of the audio signal that goes beyond the usual parametric options offered by the common binaural tools and plugins, something that may prove particularly useful in the area of immersive sonification research.
The D4‘s approach to binaural reproduction of spatial audio sources is also uniquely modular. By default, it relies on a free Sennheiser’s Ambeo Orbit plugin. With minor adaptation of the provided example abstraction, this design choice allows users to experiment with different plugins and their approaches (e.g. parametric versus convolution-based) to binaural rendering of sources.
Lastly, D4‘s binaural mesh has a unique ability to scale. It instantiates all the outputs once to be used by as many of audio sources as the users desires. Once the outputs are instantiated, additional sources generate minimal additional computational overhead. Note that depending on the number of virtual loudspeakers, rendering even a single source may prove significantly more computationally demanding than a traditional approach to binaural rendering of the same that utilizes only one binaural plugin. However, as the number of virtual sources increases, a live rendering of binaural sources using traditional one-plugin-per-source approach eventually catches up and exceeds the computational overhead of D4’s Binaural Mesh implementation. As of 2021, D4 library has been able to render up to 32 virtual and parallelly processed loudspeakers using Ambeo Orbit’s HRTF-driven binauralization.
D4‘s focus is on optimization. It dynamically creates only minimal amount of necessary objects to offer the desired functionality. As such, numbers of outputs are dynamically generated using supporting scripts at runtime. The library implementation also utilizes parallel processing to the extent afforded by the MaxMSP (version 7). As part of its original launch in 2014 D4 has been utilized in the Virginia Tech ICAT Cube for the inaugural tornado simulation with 128 channels of audio at 48KHz 24-bit using Dante interface with single audio buffer size of 256 bytes (or 5.33ms) with 21 simultaneous audio streams, 5 of which were in constant motion, for a total of 1,018 software audio streams being mixed down to 128 physical channels. No audio dropouts were observed in the said scenario. In 2019, with new optimizations introduced in the version 2.3.0, MaxMSP and the library were further pushed as part of a new composition titled Traces to utilize 130 (124.4.2) channels of audio, while increasing the number of software audio streams to 7,955, or almost 800% over the Tornado implementation, with the number of function calls increasing 500% from 5,103 in the tornado simulation to 25,337. D4has also served as the foundation for a NSF-funded research and comes with an example of sonified spatial mask generated from the rich 3D geospatial data. Today, library is used worldwide among professionals and universities.
Best of all, D4 is designed to leverage maximum amount of builtin MaxMSP resources. With the exception of the two proprietary externals responsible for the computation of the LBAP algorithm and its rendition, all of the abstractions, editors, and supporting tools are free and open source, published under the Creative Commons Attribution 3.0 United States License designed to encourage customization and derivative work. Unlike other spatialization algorithms whose implementations tend to be exclusively black box solutions, this hybrid approach leaves plenty of room for users and researchers to extend its functionality, embed such functionality in their own projects, and share the ensuing findings and improvements with the community at large.
Supporting tools include 3D visualization of the space and speaker levels using built-in 3D visualizer, as well as a collection of helper abstractions for advanced audio motion, like the sound rotation around the perimeter at an angle, so that the sound’s rotation appears to wobble or be off-center. The system is comes with a built-in set of level monitors for ICAT and Virginia Tech spaces (e.g. D4 ICAT Cube Monitor showcased above), plus a template monitor for a 7.1 space. Additional visualizations are easily created using a collection of built-in abstractions specially built for this purpose.
D4 allows for easy reconfiguration of all of the spatialization objects through a simple message. Its impact is near instantaneous and allows for seamless transportability of spatialized audio to a broad array of loudspeaker configurations, from monaural, to as many as your computer can handle. The LBAP algorithm can handle just about any loudspeaker arrangement and offers an ability to define independent layers (e.g. a subwoofer array) whose spatialization is calculated in parallel to the main loudspeaker system. More so, its implementation treats each audio stream as a separate bus that can be individually manipulated and monitored.
D4 in Action
TL;DR Version
We all lead busy lives. Below are key highlights why you may want to consider investing in D4 for all your audio spatialization needs:
- Innovative LBAP algorithm that can handle just about any loudspeaker arrangement with particular focus on High Density Loudspeaker Arrays (HDLAs)
- Innovative implementation of high precision audio source Radius and Motion Blur, and first-of-a-kind aural Spatial Mask and, most recently, first-of-a-kind Binaural Mesh that allows for seamless virtualization of spatialized content and the library’s advanced features
- Revolutionary visual spatialization editing tools for hobbyists and professionals alike, including spatial source painting
- Ability to import and spatialize a broad array of multidimensional data by leveraging the Jitter library
- Multilayered system with independent layers and focus on perimeter-based 3D immersion
- 3D visualization of spatial audio and hooks for immersive scenarios, including Virtual Reality (VR) and Augmented Reality (AR)
- A comprehensive collection of time-based editing tools
- A Mostly Open Source Software (MOSS) approach empowering users to tinker and improve the system
- Lean and optimized implementation leveraging maximum amount of built-in MaxMSP objects, including dynamic creation and parallel processing
- A drop-in replacement for the existing spatialization options (e.g. using azimuth and elevation)
- Ideal for researchers, production engineers, and artists alike
Download D4
If you are here to check out the timed demo, please read the EULA and once you click on the EULA agreement button below, you will be given the download link. If you have decided to invest in D4, thank you for supporting the ongoing development of the D4 library! You will be able to download the latest demo version below and authorize it with purchased authorization key.
Please note D4 offers the following pricing packages (all prices are in US Dollars):
- Students $75
- Professionals $150
- Organizations (REQUEST A QUOTE)
Students can purchase the software at a discount, while professionals pay the full price. Organizations can either receive software at a professional price per station or may want to consider paying for a site license which may also include a design of a custom solution for their specific audio infrastructure, as well as special licensing arrangements that offer greater flexibility in terms of sharing derivative approaches to spatialization. Given the potential complexity of such an endeavor, quotes are provided on individual project basis based upon the project scope and size.
IMPORTANT! All transactions are managed through the Paypal service and are subject to service terms. Each license is authorized per computer for up to two computers per license. Once you run any of the included examples containing the proprietary externals for the first time, you will be given a challenge token that will be printed out in the Max Console. Please email that challenge to the address provided in the console or click HERE. Students should also attach a scan of their student ID as a proof of their student status. Once received, you will be emailed an authorization key. Simply drop the authorization key into the folder with the D4 objects and you should be all set. All authorization requests will be processed 9am-5pm Monday-Friday Eastern Time.
Please review the End User License Agreement (EULA) prior to purchasing the software and/or downloading the demo.
Frequently Asked Questions
What operating systems does D4 support?
D4 is a library that runs inside MaxMSP. Therefore it currently supports Windows and Mac.
I would like to try D4 before buying it? Is there a demo?
Yes! Similar to MaxMSP, D4 by default is a timed demo downloadable from the “get” section above. Once D4 library is authorized, the timed demo limitation will be disabled. See FAQ below for more info on how to authorize D4.
What are the software requirements for running D4?
D4 is a software that runs inside MaxMSP, a commercial visual programming environment you will need to obtain separately. In addition its proprietary components require Java 1.6 or newer. While many computers come preinstalled with Java, you may need to manually install it yourself. Java is free to download and use.
Does support both 32-bit and 64-bit versions of MaxMSP?
Yes.
How do I get started?
The best way to start is to explore the D4.calc~-example file. From there you can investigate D4.mask.editor, D4.mask.renderer, D4.meter.monitor.* objects (cube~, disis~, and 7.1~), D4.mask.3D.visualizer, and D4.mask.player.
How do I authorize D4?
Open the D4.calc~-example and you will be issued a challenge token on the Max console. Email the challenge token as per instructions provided there and you will receive an authorization key. Simply place the authorization key in the folder where the D4 library is located and you should be all set. You are more than welcome to have multiple copies of the D4 library on the same computer. Simply copy the same authorization key into all of the project folders containing the D4 library.
Can I have multiple installs of D4?
Yes. A single license allows you to install D4 on up to two computers. Both should be owned by the same user and should not be used simultaneously.
Can I install D4 on multiple locations on the same computer?
Yes, you can have as many D4 installs as you like (e.g. for multiple projects) on the same computer. Just make sure that each location has the same authorization key and you should be all set.
Is the authorization key usable on multiple computers?
No. Each computer needs to be issued a unique authorization key.
I have a Bootcamp computer with both Mac and Windows operating systems installed. Do I need to authorize twice?
No. The D4 authorization is hardware-specific. Therefore, the same authorization should work on both Windows and Mac running on the same hardware, as is the case with Apple’s Bootcamp.
Once I purchase a license, am I eligible for free software updates?
All minor software updates are included in your license for free. For instance for owners of version 1.0, updated versions 1.1 or 1.7 will be free. For the same owners, however, the version 2.0 may require them to upgrade for a fee. Such upgrades will be announced with the next major release.
When I run all the different editors, my frame rate slows down significantly. What can I do about it?
D4 comes with a collection of tools designed to provide complex visualization of dynamic audio data, such as D4.mask.editor and D4.mask.3D.visualizer. Their calculations when using a large number of speakers (e.g. >100) can be rather computationally intensive. In order to allow for the system to scale, the library also includes lean versions of objects that can pre-render dynamic changes to the Spatial Mask into coll files and/or calculate such data on the fly from an array of grayscale Jitter matrices, like the D4.mask.renderer. The same can be also played back using a lean D4.mask.player. Doing so will vastly increase performance.
I am on a Mac and when using the Cube visualizer, the meter refresh rate becomes visibly slower. What can I do about it?
If your primary focus is on editing, you can disable “Audio Interrupt” option found inside Max’s “Audio Status…” window. The side-effect of the ensuing speed-up may be that things are not as real-time as they ought to be. This is why in real-time tests and performances you should enable “Audio Interrupt” which will optimize latency at the expense of visual updates. Even if meters may not be updating fast enough, your audio output should remain unaffected.
Are there plans to implement D4 in other formats (e.g. VST)?
Yes. Their implementation will be driven primarily by the user demand. The goal is to also integrate a version inside the Ableton Live as a Max4Live plugin.
When using Max 8.1.11 Binaural Mesh examples seem unstable on Windows. How can I fix this?
As of 2.4.0 release (June 8th, 2021), it appears there may be a bug or a regression on Windows that is causing this instability whenever Max’s Signal Vector Size is anything other than 64. This bug has been reported to the Cycling74 devs and has been fixed for the upcoming 8.2.0 release. In the meantime, if you are looking to work with the Binaural Mesh patches on a Windows machine, until the bug is fixed you will want to stay with the Max version 8.0.8.
Have another question that has not been covered here?
Please feel free to contact me.
Support
Please do not hesitate to contact me should you encounter a problem or an unanswered question regarding installing or using D4.