In this class students develop interactive music projects: pieces of music that are not linear, but offer multiple dimensions for listeners to explore. Applications include generative music installations, novel instruments, participative performances, museum exhibitions, musical games, and tools for producing and teaching music.

The first half of the semester, the class format is a combination of lectures, exercises on musical design and coding, and reading discussions, culminating on a midterm project. On the second half of the semester, the class shifts to a more self-directed project development studio format.

For their final projects, students will take a project from concept to execution over several iterations, applying interaction design, creative coding, and music production tools and techniques. The project development process will include gathering aural references, creating user paths and developing focused studies to explore interaction with specific musical elements. This work will inform the design and implementation of a functional prototype which students will evaluate and refine to produce their final piece. Professional practitioners will come in to share their work in the field and provide feedback to students.

While in-class examples and tutorials will be done in Ableton Live, Max/MSP and Max for Live, students will be free to use other languages and frameworks for their final projects. Supporting examples and resources will be made available for web-based projects (p5.js + Tone.js), and physical projects (Arduino-compatible micro-controllers).

Coming in with a specific project to develop is welcome; conceiving a project during the class is encouraged, too. Some experience in making or producing music will be useful, but is not required. ICM and Physical Computing or equivalent experience are required.


Spring 2021



Luisa Pereira |


3:20 - 5:50pm

Office Hours

Syllabus Overview

Jan 29
Feb 19
IxRhythm: Making and Breaking Virtual Time Grids
Feb 26
Mar 5
IxMelody: Hopping Around the Pitch Spiral
Mar 12
Mar 19
Spring Break: no class
Apr 9
Concept Presentations
Guest Talk: Valentina Camacho
Apr 16
Aural Mood Boards + User Paths
Guest Talk: Johann Diedrick
Apr 23
Focused Interaction Prototypes
Assessment Criteria Workshop
Apr 30
Integrated Prototype
May 7
Final Project Presentations

Class Notes and Assignments

Assignments are due the day before class. Please post all assignments using this form. You can find your classmates' posts on this spreadsheet.

This section will be updated weekly with references, code examples and assignment details.


Class Meeting

Introductions + music pieces (you can find them here)

Class approach and syllabus overview

Activity + Discussion: application examples and possibilities in the field.

Activity: in pairs, analyze a music piece and create a concept for an interactive experience based on it.


  1. With your partner, finish your interactive experience concept poster: 1 title, 1 paragraph, 1 sketch. Post it here.
  2. Download the Ableton Live Demo, go to Help/Help View, and follow these tutorials:
    • A Tour of Live
    • Recording Audio
    • Creating Beats
    • Playing Software Instruments

A Case Study + IxD Framework + Technology Demo

Class Meeting

Presentations: Interactive Experience Concepts

Case Study: Circumstance Synthesis | writeup | video

Lecture + Activity: IxD framework by Bill Verplank

Technology Demo: Creating a generative ambient music with Ableton Live and Max for Live


Create a first prototype of your experience concept (or a different, new idea, starting from any of Verplank's 'concerns'):

  • Compose/sketch the music using Ableton Live
  • Prototype user actions (for example, 'user moves faster, then slower'), by interacting with Ableton's interface (for example, a synth filter opens up, a certain track's volume goes down). If you have one, you can also use a MIDI controller to model user input.
  • Document your prototype by making a screen capture video and labeling user actions. Post it here.

Live Control: Tools Overview + Intro to Max for Live

Class Meeting

Assignment sharing

Live Control: Overview of tools

Demos: Ableton + MIDI Controller, + Arduino via Connection Kit

Tutorial: Intro to Max for Live. Videos: Volume Control, Clip Launcher


  • Take your experiment from last time, and control it via MIDI, OSC, or Max for Live
  • Read Chapter 14: New Notations, from Sonic Writing by Thor Magnusson (available to read online via NYU Bobst Library)


Digital Audio Workstation: Ableton Live +

  • Control via Firmata (using Ableton's Connection Kit)
  • Control via MIDI
    • Commercial physical controllers (ask for the Launchpad XL or the Oxygen at the ER)
    • Custom physical controllers with Arduino (and compatible boards):
      • Software Serial - Arduino Uno, other boards with only one serial port
      • Serial - Boards with a second hardware serial
      • MIDIUSB (see example MIDIUSB/write) - ARM boards like Arduino MKRZERO (and the MKR series), Nano 33 IoT, Due, any of the M0-based derivatives; Teensy micro-controller
      • Bluetooth LE MIDI - MKR Series, Nano Series, Uno WiFi, Zero. See this tuturial
      See this tutorial for a discussion of the techniques above and code examples
    • Custom digital controllers:
  • Control via OSC
    • Most creative coding frameworks have OSC libraries:
    • You will need an intermediary between the system sending OSC messages and Live. This can be your own (for example a Max patch, a Processing or OF app), or some standard hardware/software controller like TouchOSC or LeapMotion, which the Connection Kit has devices for (in that case your sender system will need to follow their spec for constructing messages)
    • Useful tools: Live Connection Kit, which includes an OSC Monitor device, and an OSC Send device which receives MIDI notes and sends OSC messages
  • Control via Live API

Programming Languages, Libraries, Environments:

  • Visual Programming Languages
  • Text-based Programming Languages, Frameworks, Environments:
    • Processing + Minim library
    • ChuCK and SuperCollider - real-time synthesis, composition and performance.
    • Javascript + Web Audio - Tone.js library. Documentation, Guides
    • Live-coding: Gibber, TidalCycles, more.

Rhythm + Interaction: making and breaking virtual time grids

Class Meetings

Elements of rhythm:

  • Definitions + listening examples
  • Representation and manipulation in Max for Live: Shifting accents, Syncopation, Cross-rhythms. Code from class here.

Inspiration piece: Clapping Music by Steve Reich

Discussion: Fluid formats, music as invention (notes and prompts here)

Interfaces: interactivity, visualization, probability, learning.

  • Beat Box. Math-based sequencers, by Lalalab.
    Implementation: Javascript
  • Iannix (desktop app + OSC). A graphical sequencer inspired by the work of Iannis Xenakis. Documentation here.
    Implementation: C++ (Qt)
  • Rhythmic Probability. Sequencer.
    Implementation: Max for Live
  • Magenta studio: Continue, Interpolate, Groove, Drumify.
    Implementation: Max for Live with Node.js


Design several interactions that focus on exploring rhythm / time grids. Prototype one of them. Post documentation here.


Melody + Interaction: hopping around the pitch spiral

Class Meeting

Elements of melody:

  • Definitions + listening examples
  • Representation and manipulation in Max: frequencies and pitches, pitch ranges, sets of pitches, deriving the pentatonic scale and the equal tempered chromatic scale. Patches from class here.

Inspiration pieces:



Listen + Ideate:

  • Pick a piece of music with an engaging melody and draw it. Based on it, sketch an interactive melodic experience.

Class Meeting

Rhythm explorations presentations

Representation + Manipulation, continued:

  • Max examples: dividing the octave in different numbers of steps; scales in MIDI.
  • Organization and presentation: sub patches, presentation mode, MIDI mappable elements
  • Max for Live examples:
    • Play random MIDI pitch; random within range; random within scale. Use a keyboard slider.
    • Play a random clip from a track.

Patches, M4L devices and Ableton Session from class are here.


Design and prototype an interaction that focuses on exploring melody - pitches, scales, intervals. Post documentation here.

Timbre + Interaction: mapping synthesis

Class Meeting

Elements of sound: pitch, timbre, volume

Inspiration pieces:

Synthesis Techniques: Additive, Subtractive, and FM synthesis

  • Step by step examples in Max/MSP (find them here)
  • Equivalent Tone.js exampels can be found here
  • Ableton implementations: Analog (Subtractive) and Operator (a hybrid of Additive, Subtractive, FM synthesis)


Read: The Importance of Parameter Mapping in Electronic Instrument Design by Marcelo Wanderley et al.

Option 1. Design and prototype an interaction that focuses on exploring timbre, eiter via synthesis or sampling.

In case you could use some direction/inspiration, here is a suggested exercise, based on the reading on Mapping:

  • Pick a synthesizer, and a type of interface, physical or digital (for example: buttons+sliders, different kinds of sensors, keys+trackpad)
  • Design four interfaces to interact with the synthesizer, using only
    • One button
    • One slider
    • Two buttons and two sliders
    • N buttons and N sliders
      (any combination you choose, different from the previous ones)
  • Implement two of these interfaces, and compare the user experience
  • Note: above, 'button' stands for any control with on/off states; 'slider' stands for any continuous control (it could be the X position of the mouse, or the position of a hand, as measured by a proximity sensor)

If you are more interested in acoustic sounds or sampling, you can focus on samples/recordings instead of synths.

Option 2. Pick one of your previous explorations and develop it further.

Post documentation here.

Next week, everyone will present a project in class - either their timbre exploration, or an extension of a previous one.

Project Presentations

Class Meeting


Define the concept for your final project, and prepare to present it to the class. You can use this template.