MPATE-GE 2623: Music Information Retrieval

Instructor: Assist. Prof. Magdalena Fuentes, PhD
Email: mfuentes@nyu.edu
Fall 2023,  3 credits
Class meetings: 12:00 PM – 1:40 PM W
Classroom: 35 West 4th Street, EDUC building, room 610
Office hours: https://bit.ly/mf_office_hours 

 

Course description

This course gives a comprehensive overview of research in Music Information Retrieval (MIR). MIR is a multidisciplinary field which uses knowledge from areas as diverse as signal processing, machine learning, cognition and music theory. The course will explore how this knowledge is applied to different problems within MIR, with an emphasis on common research methodologies.


Course objectives

The goal of this course is to provide students with:

  • Broad knowledge of common topics in MIR, including their definitions, challenges, and evaluation metrics.
  • Awareness of the current state-of-the art and remaining challenges for each topic.
  • The ability to critically analyze the behavior of MIR systems.
  • Practical skills for working with open source MIR resources in Python.

Student Learning Outcomes

By the end of the course, students will be able to:

  • Explain and describe common tasks in MIR, including definitions, challenges and evaluation metrics.
  • Code MIR pipelines using open source Python libraries.
  • Assess and analyze the behavior of MIR systems.
  • Explain current trends in MIR research.

Recommended Skills

Basic knowledge of Python programming and Digital Signal Theory are required.

Required Course Materials

A computer with Internet access.

Course Requirements

Class participation

This course will be highly participatory and interactive. We will have conversational lectures and group readings. It is expected and essential that students participate actively in class. For the class readings, we are going to follow the role-playing paper reading format of Jacobson and Raffel. In this format, students read the same paper, but each student takes one specific role for that week. We will work with three different roles, to be determined at the beginning of the course. Each student will participate by choosing one role in the semester. Papers will be assigned to students during the first week of classes. See a short guide on how to read a paper.

Homework assignments

There will be 4 homework assignments. Each assignment will consist of Python code snippets and written components to be completed by you. These assignments are individual. You will have two weeks to complete each assignment. For the homework assignments, your lowest score will be dropped. If you can’t finish one assignment for any reason, you should make the time to cover the material, but no need to panic about late assignments.

Final Project

The final project can be done in groups of 2-3, and will have a code and presentation component. Projects will be presented during the last class. A project can cover the assessment of an existing MIR system in new data, or a  new system proposed by the students. All projects should include a baseline system for comparison. The project will have three assignments through the semester as follows:

Topic, group definition and data report. Each group will propose a project’s topic and composition and will submit a short report (max. 2 pages) describing the scope of the work and data resources they will use. The project’s final scope and composition will be determined jointly with the instructor.  Examples of feasible projects will be provided by the instructor in advance to guide the students in the definition of their project. The groups will submit a short report describing the nature and origin of the data they are planning to use for their project, along with some examples. E.g. If the students are using an existing dataset, they should explain why they chose that dataset, include a description of the dataset, and explain how they are planning to use it. If the students are planning to collect data or create a new dataset, they should detail their plan to do so and describe the resulting dataset as above. This assignment is required but not part of the project grade.

Documented code, and data. Students will submit their working (and well documented!) code along with the data they investigated. The code should be contained in a GitHub repository, should run out-of-the-box, and should have instructions on how to run it (e.g. where should the data be placed?). The code should be well organized and well documented, explaining the decisions made and presenting a rigorous analysis of the data. Students are encouraged to follow the structure of the homework as a guide.

Final Presentation. All students from each group should participate in a final presentation. Each group should prepare slides as part of the presentation and a demo. Presentations should last between 5-10 minutes, and will be followed by a short Q&A from the class. 

 

Your grade for the class will be determined by the following break-down:

  • 40% assignments (individual)
  • 40% final project (groups)
  • 20% class participation (individual)

Classes will be in person weekly unless indicated by the instructor, and materials will be available on NYU Brightspace.

 

Recommended Readings

Other recommended readings are suggested in the weekly schedule.

Class Environment

Please do your part by seeking to promote the success of others, and by treating each other in ways that respect and celebrate the diversity of talent. Here are a few specific things that you should know about my policies on creating an inclusive and equitable class environment (both in the classroom and on the course website/forum):

  • Preparation: Students come to this class from a wide range of backgrounds, and greatly varying previous exposure to mathematics, media and programming. I want to assure students who may feel out of place here that you are indeed prepared to succeed in this class! If you feel that there are gaps in your knowledge, please speak to me and I will help you find additional materials as needed.
  • Classroom environment: The classroom is an open forum for discussion, and I encourage all students to feel free to ask questions in class. Please do not be afraid to ask any question, no matter how basic it may seem. What is basic to some of the class may be new to the rest.
  • Student accessibility: If you have any accessibility requirements, please present a letter from the Moses Center (telephone: 212-998-4980 / website: www.nyu.edu/csa / email: mosescsa@nyu.edu) to me at your earliest convenience, so that I can ensure that materials comply with your needs. I am always willing to do what it takes to support you.
  • Mental Health & Wellness: If you are experiencing undue personal and/or academic stress during the semester that may be interfering with your ability to perform academically, the NYU Wellness Exchange (212-443-9999) offers a range of services to assist and support you. I am available to speak with you about stresses related to your work in my course, and I can assist you in connecting with the Wellness Exchange. The Wellness Exchange offers drop-in services on campus on a regular basis. Additionally, if you anticipate any challenges with completing the assignments, readings, exams and other work required in this course, I encourage you to register with the Moses Center (212-998-4980 or mosescsd@nyu.edu) in advance so that you may be granted the proper academic accommodations.
  • Preferred name and pronouns: You are always welcome to write your preferred name on all class assignments, exams, etc. If you have a name and/or pronoun that doesn’t match the class roster delivered from the registrar, please let me know and I will ensure that you are addressed correctly in our class.
  • Class expenses (textbook, devices, etc.): If obtaining any material for use in our class presents a financial hardship for you, please let me know and I will do my best to arrange for loaner materials. 
  • Feedback: I will solicit (anonymous) feedback from students throughout the course, but if you have pressing or specific issues, please do not hesitate to let me know if any aspect of our course or class community can be improved.
  • Academic Integrity and Honesty: All students are expected to do their own work on homework. Students may discuss homework with each other, as well as with the course instructor. Each student must turn in their own write-up of the homework. Excessive collaboration (i.e., beyond discussing problem set questions) can result in honor code violations. Questions regarding acceptable collaboration should be directed to the class instructor prior to the collaboration. It is a violation of the honor code to copy or derive homework solutions from other students (or anyone at all), textbooks, previous instances of this course, or other courses covering the same topics. Finally, a good point to keep in mind is that you must be able to explain and/or re-derive anything that you submit. Please also refer to the general NYU academic integrity statement.
  • Anti-Racism, inclusion and equity: NYU values an inclusive and equitable environment for all students. I hope to foster a sense of community in this class and consider it a place where individuals of all backgrounds, beliefs, ethnicities, national origins, gender identities, sexual orientations, religious and political affiliations, and abilities will be treated with respect. It is my intent that all students’ learning needs be addressed both in and out of class, and that the diversity that students bring to this class be viewed as a resource, strength and benefit. If this standard is not being upheld, please feel free to speak with me.

Resources at NYU

Access your course materials: NYU Brightspace  (brightspace.nyu.edu)
Databases, journal articles, and more: Bobst Library (library.nyu.edu)
Assistance with strengthening your writing: NYU Writing Center (nyu.mywconline.com)
Obtain 24/7 technology assistance: IT Service Desk (nyu.edu/it/servicedesk)
University Policy on Academic Integrity
Moses Center for Students with Disabilities

 

Weekly Schedule

Week 0: Supplemental materials 

If you want to get some practice before we start with the homework assignments, take a look at:

Week 1, 09/06: Course introduction & Music representations

  • Overview of MIR field
  • Course structure and schedule
  • GitHub classroom
  • Music Representations 
    • Sheet music
    • Piano-roll
    • Midi
    • Audio
  • PRACTICE: GitHub classroom
  • ASSIGN: paper readings for the semester

Week 2, 09/13: Time-Frequency Representations 

  • Short Time Fourier Transform
  • Log-Frequency Spectrogram
  • Constant-Q Transform 
  • Recommended reading: McFee’s Digital Signals Theory Chapters 5 and 9.
  • Role-playing paper reading 1

Week 3, 09/20: Machine Learning and MIR 

  • Supervised learning 
  • Features
  • Training/validation/test data
  • Classifiers
  • Metrics
  • Data augmentation
  • Recommended reading: Murphy Chapter 1 or Shalev-Shwartz Chapters 1 and 2
  • Role-playing paper reading 2
  • ASSIGN: Homework 1

Week 4, 09/27: Sound Classification  

  • Instrument Identification 
  • Mel frequency cepstral coefficients 
  • Confusion matrices
  • Metrics: Precision/recall/f-measure
  • Musically valid audio and label transformations/deformations: 
    • Pitch shifting 
    • Time stretching
    • Resampling 
    • Background noise
    • Dynamic range compression
  • Recommended reading: MFCC Blog, FMP chapter 4.5.1
  • Role-playing paper reading 3

Week 5, 10/04: Beats and Downbeats 

    • Beat and downbeat tracking 
    • Onsets and novelty curves 
    • Metrics
    • Recommended reading: FMP chapters  6.1.1, 6.3 and ISMIR tutorial
  • Role-playing paper reading 4
  • DUE: Homework 1
  • ASSIGN: Homework 2

Week 6, 10/11: Music Similarity

    • What is similarity
    • Embeddings
    • Fingerprinting 
    • Version identification
    • Music recommendation
    • Recommended reading: FMP chapter 3.2, 7.2, 7.3
  • Role-playing paper reading 5

Week 7, 10/18: Chords [Class ONLINE]

  • Chord estimation
  • Chroma
  • Metrics
  • Recommended reading: FMP chapters 3.1.2, 5
  • Role-playing paper reading 6
  • DUE: Homework 2
  • ASSIGN: Homework 3

Week 8, 10/25: Open Week

Week 9, 11/01: Pitch

  • Pitch tracking
  • Melody estimation
  • Salience representations
  • Metrics
  • Recommended reading: FMP chapters 1.3.2, 1.3.3, 8.2
  • Role-playing paper reading 7
  • DUE: Homework 3
  • ASSIGN: Homework 4

Week 10, 11/08: Transcription

  • Transcription 
  • Non-negative matrix factorization 
  • Metrics
  • Recommended reading: FMP chapter 8.3
  • Role-playing paper reading 8
  • ASSIGN: project topic, data and baseline.

Week 11, 11/15: Source Separation

  • Harmonic-percussive separation
  • Instrument separation
  • Stems 
  • Metrics
  • Recommended reading: FMP chapter 8.1 and ISMIR tutorial
  • DUE: project topic, data and baseline.
  • DUE: Homework 4
  • ASSIGN: project code and data.

Week 12, 11/22: NO CLASS (Thanksgiving)

Week 13, 11/29: Multimodal MIR

  • Overview and applications of multimodal machine learning in MIR
  • Building multimodal models
  • Embedding space
  • Contrastive Loss

Week 14, 12/06: Music Generation

  • Overview and applications of multimodal machine learning in MIR
  • Stable diffusion models
  • Variational models
  • DUE: project code and data.
  • ASSIGN: project presentation.

Week 15, 12/13: Final Presentations

  • Overview and applications of multimodal machine learning in MIR
  • DUE: project presentation.