PERSONAL PROJECT — MAY 2025

Inside the Quartet: Prototyping an Immersive Spatial Audio Experience for Vision Pro

TIMELINE

A few days across April and May 2025

KEY SKILLS

Swift Prototyping
AI Pair Programming
↳ Link to Github ↗

Sole Designer and Co-Founder — Product Thinking, End-to-End Product Design, Brand and Visual Design

TEAM

Just me!

TOOLS

Swift in Xcode
Reality Composer Pro
Dia Browser

Note: I left the bow from the original models next to the violins (and had to scale up the violin to pretend to be a viola because I couldn't find a good enough model). I apologize to all of the string musicians in advance 🙏

THE RESULT

The Final Experience

FIG. 0

NOTE THAT THE AUDIO HERE IS STEREO — TO GET THE FULL EXPERIENCE, YOU NEED A HEADSET!

THE CONTEXT

The Brief: Create an XR experience that breaks a barrier (cultural, physical, etc.)

I had two main goals:

Utilize the spatial capabilities of Extended Reality (XR)

Take advantage of the 3D medium rather than creating something easily transferrable to mobile or desktop

Share an authentic experience that most haven't had before

Utilize my personal experiences to create something that gives people a gateway into another world

the idea

What if you could sit in a musician's seat and hear performances from their POV?

THE INSIGHT

Playing music in a group setting is a spatial experience that most people haven't tried before.

FIG. 1

my pov performing at the concertgebouw in amsterdam!

Live music = spatial sound

Each musician produces a range of sounds with their instrument from different positions on the stage

Bringing the concert to you

This is an opportunity to make live music a tangible and accessible experience beyond audio on a screen

THe dream

I originally wanted to create a full orchestra experience… with 100+ seats.

To do this, I started by figuring out what music content to include. I had to:

Choose a piece that highlights every instrument and is easy to listen to

Find 3D assets for music stands, chairs, and 20+ individual instruments

Study the stage layout and compose the scene in 3D modeling software

FIG. 2

trying to map out the placement of every musician in holst's the planets

ThE REALITY

I realized that making the 3D scene would be a huge task… and besides the point.

the pivot

I pivoted to creating a string quartet experience so I could focus on creating a quality MVP prototype. This was my simple 3D model!

FIG. 3

3d model of a string quartet + INTERACTION DESIGN

THE PROCESS

Learnings from using Apple’s spatial pipeline and pair programming with AI

I used Apple's native tech stack over Unity because:

I could utilize visionOS elements like buttons and take advantage of built-in eye tracking interactions like hovering

This was an opportunity to dive deeper into vibe coding and Swift prototyping

I wanted to re-familiarize myself with Xcode for my upcoming internship!

FIG. 4

tools & technologies i used

I used Dia as a pair programmer and gave it clear instructions and referenced online documentation for best results. I had it:

Answer my questions when reading documentation or fixing errors

Set up simple data structures and custom classes with the APIs

Write logic to extract data from your 3D model, like positions and names

Create visual markers to debug 3D positioning and write vector math

FIG. 5

ONE OF MANY QUESTIONS I ASKED DIA WHEN MAKING THIS PROJECT

Modifying the positions to allow the user to "swap seats" was the most difficult part of the process. I leaned on terminal outputs and visual reference nodes.

Positioning Limitation

Due to privacy concerns, you can't access the user's camera position to move it on the Vision Pro. Instead you need to reposition the 3D model to the user's viewpoint.

Bringing in CG principles

I was taking Computer Graphics this semester and got to apply those concepts! This is a World → Camera Space transformation and is achieved through vector math.

FIG. 6
THE MODEL WAS MOVING… BUT TO PRETTY FAR AWAY IN SPACE.
FIG. 7
USING VISUAL NODES AT KNOWN LOCATIONS TO TEST POSITIONS
FIG. 8
GETTING CLOSEr — the scale is better, but positions are still off
FIG. 9
POSITIONS ARE ACCURATE, but some angles are off
FIG. 6
THE MODEL WAS MOVING… BUT TO PRETTY FAR AWAY IN SPACE.
FIG. 7
USING VISUAL NODES AT KNOWN LOCATIONS TO TEST POSITIONS
FIG. 8
GETTING CLOSEr — the scale is better, but positions are still off
FIG. 9
POSITIONS ARE ACCURATE, but some angles are off
FIG. 6
THE MODEL WAS MOVING… BUT TO PRETTY FAR AWAY IN SPACE.
FIG. 7
USING VISUAL NODES AT KNOWN LOCATIONS TO TEST POSITIONS
FIG. 8
GETTING CLOSEr — the scale is better, but positions are still off
FIG. 9
POSITIONS ARE ACCURATE, but some angles are off
FIG. 6
THE MODEL WAS MOVING… BUT TO PRETTY FAR AWAY IN SPACE.
FIG. 7
USING VISUAL NODES AT KNOWN LOCATIONS TO TEST POSITIONS
FIG. 8
GETTING CLOSEr — the scale is better, but positions are still off
FIG. 9
POSITIONS ARE ACCURATE, but some angles are off

Finalize your 3D model before coding if possible — it's hard to know if any modifications will update in your project, or if re-importing it entirely will break your project.

FIG. 10

reworking the 3d model later… but some updates didn't go through.

REFLECTIONS

I've always wanted to utilize music background for a design project. This is the first time it clicked!

FIG. 11

PRESENTING IN CLASS

Experience is the moat, and other key takeaways

Experience is the moat

Especially now that ideas can be just a few hours of designing, coding, and prompting away from reality, it is more important than ever to leverage your unique life experiences in your creative process.

Stay solo if you have a clear personal vision

Having help is great, but the extra friction of communicating a vision and staying aligned can sometimes dilute the joy of working on something you deeply understand and care about.

Use passion projects to learn new tools

If you're working with a tech stack or technology that you're not as familiar, find a project that you are deeply motivated to work on and tackle it with AI on your side.

What's next?

Recreate the full orchestra experience

Now that I don't have a deadline for the project, I have the space to expand the scope again. I just need to get my hands on a Vision Pro headset!

Partner with music ensembles to archive their performances in Spatial Audio

I would love to partner with orchestras and other groups to help record and recreate these recordings in 3D space so that people can experience them again!