This course explores music through the lenses of computation and interactivity.

The first part of the semester consists of a structured exploration of rhythm, melody, timbre, and harmony, from the perspectives of code, design, and music theory. For each musical element, we will hold listening sessions, represent and manipulate the element in code, and create an interactive study around it.

The second half of the semester we will explore algorithmic composition techniques such as Markov Chains, Neural Networks and L-systems. The last three weeks of class will be dedicated to the development of a final project ––an interactive/generative musical piece.

In-class coding and assignments will be done in p5.js and Tone.js. Introduction to Computational Media (ICM) or equivalent programming experience is required.

This class is a good fit for students who are interested in:

  • Creating interactive music pieces and digital instruments
  • Deepening their understanding of how music works
  • Continuing to develop coding skills they acquired in the Creative Coding course


Spring 2021

Interactive Media Arts - NYU Tisch


Luisa Pereira |


Yona Ngo |


Mondays and Wednesdays
3:30 - 5pm EST

Office Hours

Luisa: Sign up here
Yona: Sign up here


Assignments will range from readings to design and programming exercises. They are required and you should be prepared to show/talk about them in class. You will be expected to post them to your blog the night before they are due.

Midterm and Final Projects

Prompt: Create an interactive/generative musical piece or instrument.

  • Option 1. Extend or refine one of your previous studies.
  • Option 2. Create a new piece based on what you have learned.

Prepare a short presentation describing your musical, design, and programmatic decisions and process.

Syllabus Overview

Mon, Feb 1
Wed, Feb 3
Mon, Feb 8
Wed, Feb 10
President's Day
Mon, Feb 15
Wed, Feb 17
Thurs, Feb 18
Mon, Feb 22
Wed, Feb 24
Mon, Mar 1
Wed, Mar 3
Mon, Mar 8
Wed, Mar 10
Mon, Mar 15
Wed, Mar 17
Mon, Mar 22
  Workshop + Meet
Wed, Mar 24
Mon, Mar 29
Wed, Mar 31
Mon, Apr 5
Wed, Apr 7
First Project
Mon, Apr 12
Wed, Apr 14
Spring Break
Mon, Apr 19
Wed, Apr 21
Mon, Apr 26
Wed, Apr 28
Mon, May 3
Wed, May 5
Final Project
Mon, May 10

Class Notes and Assignments

Assignments are due the day before class. Please post all assignments using this form. You can find your classmates' posts on this spreadsheet.

This section will be updated weekly with references, code examples and assignment details.



Introductions + music pieces (you can find all pieces here)

Class approach and syllabus overview

Activity: Inspiration Projects - Interact + Discuss.


1. In pairs: get ready to share the project you experienced + discussion highlights with the class on Wednesday.

2. If you don't have one already, set up a blog or web page where you can post your assignments for this class ––see resources below. If you need any help, please contact Yona (see office hours and contact info on the top right of this page).

3. If you are a beginner programmer or are not yet familiar with p5.js, take some time to do a personal Creative Coding review this week. Make sure to review the following concepts in p5.js/Javascript (see resources below):

  • Variables
  • If Statements
  • For loops
  • Arrays
  • Functions
  • Callback functions

Audio-Visual Instrument

Present + Discuss

Inspiration Projects


Audio-visual instruments / compositions


W1.1. Audio-visual instrument

  • In pairs, design and implement an audio-visual instrument, following these guidelines (you will receive an email from Yona after class with the assigned pairs)
  • Improvise/compose a 30-second piece with your instrument, and make a video recording of it (a screen capture with audio, or a phone recording)
  • Create a blog post with documentation, including:
    • Screenshots of your plan drawing
    • A link to your sketch on the p5 editor (if you are using a different editor, include both a link to the running sketch, and a link to your code
    • A short description of your process: what was challenging? did any questions come up? did you learn anything new?

Submit your blog post by Sunday night

Interactive Composition

Present + Discuss

AV Instruments


Activity: catalog the elements of the musical piece you brought in


Activity: based on the catalog, sketch an interactive experience


W2.1. Post your catalog and experience concept to your blog



W2.2. Keeping in mind the examples from class, reflect on the possibilities that open up by representing and manipulating musical tracks and loops in code. Then:

  1. Sketch (as in: draw) two ideas for interactive compositions. What musical material are they based on? How is this material arranged / manipulated by the program? How does the listener interact with the composition?
  2. Sketch (as in: code) an interactive composition. You can implement one of your ideas, prototype one aspect of them, or do something else. When you are done, improvise a 30'' piece, and record a video of it. Include both running code and video on your blog post.

  3. A few suggestions:

    • Consider your musical material: start from found sounds; compose and record the stems yourself
    • Consider user input: keys, mouse position, speed of motion, sensors
    • Consider speed: try playing tracks extremely slowly, or extremely fast. Have one track overlap with itself, at different playback speeds
    • Explore using audio effects like delays, reverbs, filters, panners
    • Pick random sets of tracks to play, depending on user actions or input data. Constrain the randomness using probability.
    • Consider automation: can the sliders move up and down following a sine wave? Other functions?
    • Consider visualization

W2.3. Find a song / musical piece you find rhythmically interesting. Submit it via our form.

All assignments are due on Tuesday night (Feb 16). They can be done individually or in pairs.


Tone.js Quick Start | API Reference | Examples

ffmpeg, a command-line tool you can use to convert wav files to mp3 in batches. Here is the command I used:

 $ for f in *.wav ; do ffmpeg -i "$f" -acodec libmp3lame -q:a 2 "${f%.*}.mp3"; done



Interact: Four web interfaces

Listen: Elements and Examples

Discuss: What possibilities does representing these elements in code open up? Consider interactivity: input gestures and control, visual/tactile representation of each element, mapping, and feedback modes.

Design Activity: in pairs, sketch two ideas for interactives to explore rhythm


W3.1. Finish your concept sketches and post them to your blog


Tone.js classes: Transport, Time

Drum Machine Tutorial

Tutorial: Create a Drum Machine, Step by Step

  1. Make a drum pad (click to trigger kick).
    Include Tone.js, import a sound, play it back when the mouse is pressed. Try making a whole drum kit: have different sounds play when you click on different areas of the screen.
  2. Make a beat (four on the floor)
    Create a loop to play the kick drum twice per second. Try making it faster and slower.
  3. Make a drum pattern
    Add more sounds and play them back on specific steps using modulo. Try creating different patterns.
  4. Make a drum machine
    Replace steps by modulo with pattern arrays. Try different patterns by editing the matrix.
  5. Draw your drum machine
    Draw the drum pattern. Try making it a variable size. Group beats in bars. Add colors and margins. Try your own palette, shapes, images, positions, animations. Make a circular sequencer.
  6. Make it interactive. Click to turn steps on and off.
  7. Add a full drum kit and interface. Add margins.
    Add sliders and buttons to change the tempo, number of measures, time signature, accents, swing, and any other parameter you find relevant. These can be DOM elements or physical controllers.


Present + Discuss: students' rhythm interactive sketches

Code: make the drum machine code your own (suggestions in prompts above)


W3.2. Interactive Rhythm Study. Design and implement an interactive exploration of rhythm. Once you're done, record a 20-second composition/improvisation using it. On your blog post, you can include some text (your inspiration/motivation), sketches, a brief description of your sound, visual, and interactive design decisions, as well as any challenges and questions that came up.

W3.3. Find a song / musical piece you find melodically interesting. Submit it via our form.



Interact: Four web interfaces

Listen: Elements and examples


Activity: Listen to the melodically engaging piece you posted. Draw its melody, using your own visual language


W4.0. Finish your melody drawing, and post it to your blog, along with a link to the piece itself.

W4.1. Sketch two ideas for interactives to explore melody.


Video: Learn music theory in half an hour (a short introduction to chord progressions and melodies, recommended by a student)

Tone.js classes: Frequency, Synth


Code: From drum machine to melody sequencer

  • Make a melody sequencer. Replace the drum samples with note samples, and create an 8 x 8 grid: 8 steps, one octave. Things to try:
    • Different numbers of steps and tracks
    • Different sounds (a cat meowing? a person talking?)
    • Different pitches: which ones are you including? Are they from a specific scale or mode? Are they a custom / random set of frequencies? How large are the intervals from note to note? What's the smallest / largest interval possible in your sequencer?
  • Make your own visuals. Add colors. Try your own palette, shapes, images, positions, animations. How will you represent different notes, pitches and scales? Make a circular sequencer.
  • Extend the grid. Group beats in bars, add octaves, and draw dividers.
  • Add controls. Add a DOM button to toggle play/stop, and a slider to change the tempo. Try adding controls for the scale, range of notes, and any other variable you find relevant.


Activity: Share your interactive melody ideas with a partner. Together, pick one for them to share with the rest of the class. Switch roles.

Activity: Starting from the first Sequencer sketch above, pick one or two "things to try" suggestions and implement them. Once you are done, move on to the next sketch and do the same.


From class: Finish your sequencer exercise (or exercises) and post it to this document.

W4.2 Melody Study. Design and implement an interactive exploration of melody. Record a 20-second composition/improvisation using it. Post the running sketch, code, and documentation to your blog.

W4.3 Listen: find a piece of music you find interesting in terms of timbre, and post it using our assignment form.


Quantization in Tone.js


Listen + Interact


  • Pitched sounds vs percussive sounds
  • Fundamental frequency and overtones
  • Volume

Code: Synthesis


  • Optional: Read this interactive tutorial: Synthesizers. Using the synth at the end of the synthesizer tutorial, design three different sounds. You can share them by creating screen capture videos.

  • 5.1. Sketch three different interfaces to interact with timbre
  • 5.2. Timbre Study. Design and implement an interactive exploration of timbre (using sampling, synthesis, or both). Record a 20-second composition/improvisation using it. Post the running sketch, code, and documentation to your blog.
  • 5.3. Listen: find a piece of music you find interesting in terms of harmony, and post it using our assignment form.


The computer music tutorial, by Curtis Road. Part II: Synthesis (available at NYU library); Chapter 4: Sampling Synthesis.


Code: Sampling

Code: Mic input + analysis and visualization

Listen + Interact


Finish assignments 5.1., 5.2, 5.3. The sketches and interactive study can focus either on Synthesis or on Sampling



Elements and examples


  • W6.0. Pick one of the interfaces above, spend some time exploring it, then create a 1-min improvisation/composition using it and screen-record it. Post the video to your bog, along with a paragraph describing your experience with the interface: was it interface easy to learn? Hard to learn? Was it expressive? Limiting? How?
  • W6.1. In pairs, sketch three ideas for interactives to explore harmony.


Class Meetings


Wednesday Workshop: worksheet exercises


1. Finish at least three of the exercises in the worksheet from class. Post them by linking to your worksheet.


  • Option 1. W6.2. Harmony Study. Design and implement an interactive exploration of harmony. Record a 20-second composition/improvisation using it. Post the running sketch, code, and documentation to your blog.
  • Option 2. W7. First project. Pick one of your previous studies, and develop it further - you can expand it, add some features, improve the interaction design, the sound design, the graphic design, the code organization; combine it with another study.


First Project


Student Projects

Code Review

Demo: Refactoring Code


Review and iterate on your project's code, or swap with a classmate. Start by adding comments, then look for repeated code, variable and function names that could be clearer, code that can be organized better (by creating functions, for example), and so on. The goal is that the code is better organized / more succinct, and the sketch does exactly what it did before.

Post documentation to your blog, including a link to the original code, a link to the 'refactored' code, and a description (can be bullet points) of the changes you did.

Algorithms: Markov Chains


Musical systems - a brief history.
Randomness, Probability, Algorithms, Interactivity.

Markov Chains, musical applications



  • Option 1: Create a markov-chain based melody / drum pattern / chord proggression.
  • Option 2: work on your final project.


  • Option 1. Markov Study. Design and implement a generative/interactive piece based on Markov chains. Record a 20-second composition/improvisation using it. Post the running sketch, code, and documentation to your blog..
  • Option 2. Make progress on your final project, and post documentation of it to your blog.


The Computer Music Tutorial, by Curtis Roads (available at NYU library)

  • Ch.18. Algorithmic Composition Systems
  • Ch.19. Representations and Strategies for Algorithmic Composition


Algorithms: L-Systems


Grammars and L-Systems, musical applications


Due Monday, May 3:

  • Option 1. L System Study. Design and implement a generative/interactive piece based on L Systems. Record a 20-second composition/improvisation using it. Post the running sketch, code, and documentation to your blog.
  • Option 2. Make progress on your final project, and post documentation of it to your blog.

Algorithms: Neural Networks


Neural Networks, musical applications

OS libraries + models

Magenta Project


Continue sequences, generate from scratch

Web interfaces:

More Models

  • MusicVAE - Interpolate and sample musical sequences. Includes models for melodies, drum loops, chord-conditioned multi-track sequences, drum performance "humanizations". Tutorial with p5.js + Tone.js here.
  • MIDI me - Personalize MusicVAE model with in-browser MIDI input
  • Piano Genie - Map 8-button input to full 88-key piano in real time, with VQ-VAE.
  • Onsets and Frames - Automatically transcribe polyphonic audio to MIDI, with CNNs + LSTMs
  • GAN Synth - Generate hi-fi audio with Generative Adversarial Networks

See demos here (all have code and documentation).


Due Monday, May 10:

  • Option 1. Neural Networks Study. Design and implement a generative/interactive piece based on Neural Networks. Record a 20-second composition/improvisation using it. Post the running sketch, code, and documentation to your blog.
  • Option 2. Finish your final project.




Finish your final project (either a Neural Networks study or an extension of a previous project). When you are done, record a 20-second composition using it. Come prepared to do a 5-minute presentation in class. You can include:

  • Inspiration and concept
  • A live demo (have the 20' video as a backup)
  • A description of your main musical and design decisions
  • An overview of how it works
  • Challenges + future work

The live demo is the main part of your presentation: the other sections provide context, but are secondary and can be brief. In preparing, keep in mind you only have 5 minutes to present.

Post your code, video and documentation to your blog (you can follow the same outline as your class presentation); submit using our usual form.



You will need a modern laptop (4 years old or younger is a good rule of thumb). Limited numbers are available for checkout from the department. Commercial software required in the course will be made available on the laptops available for checkout. Other required software is freely available.

Online course support

Extensive online resources will be available on this course website

Grading and attendance

Grades for the course will follow the standard A through F letter grading system and will be determined by the following breakdown:

  • Assignments and Midterm Project: 50%
  • Participation and Attendance: 40%
  • Final Project: 10%

At most two (2) unexcused absences will be tolerated without effect to your grade. Any more than two (2) unexcused absences will result a lowering of your final grade by one whole grade for each unexcused absence. For example, three (3) unexcused absences will result in your highest possible grade being a B instead of an A. Four (4) unexcused absences will result in your highest possible grade being a C and so on. Six (6) unexcused absences would result in an automatic F for the course. Two (2) late arrivals will count for one (1) absence.


This class will be highly participatory. You are expected to contribute to discussions, engage in group work, give feedback to your peers, and otherwise fully participate in class.

Phone, Tablet, Laptop and other electronics use

Recreational use of phones, music players, video game systems and other devices are forbidden. Laptops and tablets are allowed for note taking, in class work, as well as relevant research only. Activities not related to the class, such as recreational use of the internet, including all social media websites, email and instant messaging, game playing, and work for other classes, will not be permitted. Such activities are disrespectful to the instructor and distracting to others. Your laptop should always be closed whenever a fellow student is presenting.

Teaching Style

Classes will be a mixture of lecture, discussion, hands-on tutorials, homework review, presentations, and group work. You will need to come to class prepared with a laptop and any other supplies specified for that class.

Course Schedule

The course will be two (2) times per week for one hour and thirty minutes (1:30) for a total of 14 weeks.


There will be regular assignments that are relevant the class material. These assignments must be documented (written description, photos, screenshots, screen recording, code, and video all qualify based on the assignment) on the class website. Each assignment is due by the next class unless stated otherwise.

It is expected that you will spend 6 to 8 hours a week on the class outside of class itself. This will include reviewing material, reading, watching video, completing assignments and so on. Please budget your time accordingly.

Each assignment will be marked as complete (full credit), partially complete (half credit), or incomplete (no credit). To be complete an assignment should meet the criteria specified in the syllabus including documentation. If significant portions are simply not attempted or the assignment is turned in late (up to 1 week) then it may be marked partially complete. If it is more than a week late, not turned in, or an attempt isn’t made to meet the criteria specified it will be marked incomplete.

Responses to reading and other written assignments are also due in class one week after they are assigned and must also be submitted via the class website. Written assignments are expected to be 200 to 500 words in length unless otherwise specified. Grading will follow the same guidelines as above; on time and meeting the criteria specified will be marked as complete. Late (up to 1 week) or partially completed work will be given half credit. Work that is more than a week late, not turned in, or fails to meet the criteria specified will be given no credit.

Statement of Academic Integrity

Plagiarism is presenting someone else’s work as though it were your own. More specifically, plagiarism is to present as your own: A sequence of words quoted without quotation marks from another writer or a paraphrased passage from another writer’s work or facts, ideas or images composed by someone else. More information can be found on Tisch’s page regarding Academic Integrity.


Academic accommodations are available for students with documented disabilities. Please contact the Moses Center for Students with Disabilities at 212 998-4980 for further information.


Your health and safety are a priority at NYU. If you experience any health or mental health issues during this course, we encourage you to utilize the support services of the 24/7 NYUWellness Exchange 212-443-9999.