(version 2022_02_06)
This tutorial introduces the setup and functionality of a Jupyter notebook and shows how CAMAT (Computer-Assisted Music Analysis Tool) can be used for the visualization and analysis of sheet music data.
Based on this, the tutorial Part 2 presents options for simple statistical queries.
A Jupyter notebook is a file that can be opened and edited within the browser. Each Jupyter notebook consists of a sequence of text blocks (Markdown Cells) and blocks of programming code in the Python programming language (Code Cells). By pressing the Enter key within a block, text or code can be changed, added or deleted. Own, changed versions of the notebook can be saved under a new name (Menu: File -> Save as...).
IMPORTANT: The commands of each code cell are executed with the Run button. During execution (which may take some seconds for some commands), an hourglass icon is displayed at the top of the browser tab. Afterwards, results (or error messages) are displayed, if any, and the cursor jumps to the next box. Please go through the tutorials code cell by code cell, activate the respective code with the Run button and wait and see what happens!
In the code cells, all lines following a hash sign (#) are not executed; this provides the opportunity to note either explanations of the code or additional optional commands that are executed by deleting the hash (and then running).
# Of course, you can also calculate with the commands:
# Please press 'Run'!
21 * 7 + 1
# Now change the calculation numbers.
# Then press 'Run' again!
148
A new cell is created with the + button or the menu item Insert and can then be defined in the menu as markdown or code cell. Here you can add your own comments or commands. (And please don't forget to save the notebook under a new name beforehand!).
Parallel to the notebook a command window is running, which you do not have to consider further - but don’t close it! Please close a notebook with the menu command 'File' -> 'Close and Halt', then log out and close the command window at the very end.
Every session with a Jupyter Notebook starts with the import of the required Python libraries:
import sys
import os
sys.path.append(os.getcwd().replace(os.path.join('music_xml_parser', 'ipynb'), ''))
import music_xml_parser as mp
from music21 import *
import csv
from IPython.display import HTML, display
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
# These commands are used to parse the core of CAMAT, the 'music_xml_parser',
# and load the python libraries 'numpy' and 'pandas' for static evaluations,
# 'matplotlib' for graphical representations,
# csv' loaded for table export
# and ' music21 '.
# The following command allows the download of URLs from the Internet:
environment.set('autoDownload', 'allow')
from IPython.core.display import display, HTML
display(HTML("<style>.container { width:100% !important; }</style>"))
# optimizes the display in the browser
The next step is to load a note file in MusicXML format (xml for short) and give it the name 'xml_file'. You can also choose any other name, but then you have to change the corresponding commands in the code cells accordingly!
There are two options when loading sheet music files:
In the following example we load the first movement (Adagio) from the String Quartet No. 11 in E-flat major - Adagio, Köchel catalog 171, by Wolfgang Amadeus Mozart from the project database (cf. https://analyse.hfm-weimar.de/doku.php?id=en:mowo resp. https://analyse.hfm-weimar.de/doku.php?id=en:notenauswahl#mozart_wolfgang_amadeus_1756-1791), in which we transfer the link to the xml file with a right click and "Copy link address".
Subsequently, a so-called Pandas data frame (see section 3) of the xml file is generated, which serves as a kind of internal database for all further evaluations.
xml_file = 'https://analyse.hfm-weimar.de/database/03/MoWo_K171_COM_1-4_StringQuar_003_00867.xml'
# The command loads a MusicXML file from the Internet
# and gives it the internal variable name 'xml_file'.
# You can of course load other files here,
# by copying their URL (or path).
# And you can assign other internal names for the files.
# The following command creates a pandas dataframe (named 'm_df') from this:
m_df = mp.parse.with_xml_file(file=xml_file,
do_save=True,
save_file_name='mozart.csv')
Downloading file:MoWo_K171_COM_1-4_StringQuar_003_00867.xml >>> MoWo_K171_COM_1-4_StringQuar_003_00867.xml downloaded! File at: ../music_xml_parser/data/xmls_to_parse/hfm_database/MoWo_K171_COM_1-4_StringQuar_003_00867.xml
The command allows you to specify two parameters (green fields):
do_save=True -> the Pandas dataframe file will be saved in the 'Exports' folder. If you change the value to 'None', the saving will be waived.
save_file_name = 'None' -> the Pandas dataframe file will be saved under the same file name as a .csv file; csv means: 'comma separate variable'. Instead of True you can enter any file name with the extension .csv, e.g. change 'mozart.csv' (please use single quotes!) Then you can open the file in the exports folder.
The xml-Parseris a Python module developed as part of the Fellowship project for processing a MusicXML file and is used, among other things, for visualization and statistical analysis of the information contained in the file. The basic concept of the module is to store all the information present in the note file in the form of a Pandas-Dataframes, which allows very efficient calculations and data management of the data read in.
The so-called xml parser has an internally defined data structure, which can already be found on your computer after the zip file has been extracted: There are three folders in the music_xml_parser folder:
core contains all program relevant Python scripts. Please do not change!!
data contains the folders
• xmls_to_parse. This folder contains all xml files that can be read in directly (i.e. without specifying a path). You can put your own xml files into the folder. Alternatively, you can read in files directly from the Internet. • exports: Here are all the files you have generated in the course of your work with the Jupyter notebook, e.g. tables (.csv) or graphics (.png).
After you have created a dataframe 'm_df' of your note file by the last command, you can view the data by opening the file m_df.csv saved there in the exports folder (with Excel or a text editor) - provided you have selected 'do_save=True'.
You can also output the data as a table in the browser. To do this, please activate the following command ‘m_df’ by deleting the hash (#) in front of it and then pressing 'Run'.
IMPORTANT: Even with smaller note files, this representation requires a lot of memory; therefore, the command should preferably be skipped!
# m_df
# If you delete the hash (#) and press Run,
# the command 'm_df' is executed and a table
# with first and last rows of the pandas dataframe is displayed.
# For a complete view use the following command:
# mp.utils.print_full_df(m_df)
# IMPORTANT: This can be very computationally intensive!
In the following code cell you activate the piano roll visualization of the sheet music file.
IMPORTANT: Unfortunately, large sheet music files cannot be displayed completely as piano rolls; the display is aborted and an error message is returned ('OverflowError: sleep length is too large'). Then it is better to choose the external display in a popup window (see below). Or use the filter function (see section 5) to select those sections that are of interest to you!
As an alternative to the piano roll representation in the browser you can choose the representation with the Sonic Visualiser. You can find out how to proceed here: https://analyse.hfm-weimar.de/doku.php?id=en:sv-pianoroll
m_df = mp.parse.with_xml_file(file=xml_file,
plot_pianoroll=True,
plot_inline_ipynb=True)
File at: ../music_xml_parser/data/xmls_to_parse/hfm_database/MoWo_K171_COM_1-4_StringQuar_003_00867.xml
Each voice is represented with its own color. If two voices play in unison (i.e. the same notes), the two colors mix.
Now open the sheet music file: First save the file on your hard disk (in the sheet music database: right click on the link -> 'Save target as'), open your score editor (e.g. MuseScore) and load the file.
Compare the sheet music representation with the piano roll representation. Where do the two representations differ? What can you see on the piano roll graphic? What are the advantages of the piano roll compared with the sheet music? What are the disadvantages?
Here is an alternative way of displaying by selecting external/internal display: By activating the command '%matplotlib' the display is changed to an external pop-up window. All following graphics are also affected by this!
# %matplotlib
# This command (activation by deleting #).
# selects an external representation.
m_df = mp.parse.with_xml_file(file=xml_file,
plot_pianoroll=True,
plot_inline_ipynb=True)
Using matplotlib backend: Qt5Agg File at: ../music_xml_parser/data/xmls_to_parse/hfm_database/MoWo_K171_COM_1-4_StringQuar_003_00867.xml
If you want to switch back to the browser display, you must execute the following command:
%matplotlib inline
In the following code cell some parameters are explained, with which the command above can be further modified. In each case the default settings are given, which apply as long as the parameter is not called explicitly, and it is noted in each case whether a string (string: 'str' or None), a number or a value ('bool', usually 'True', 'False' or 'None') has to be entered.
m_df = mp.parse.with_xml_file(xml_file,
#file= str, (string value) # Possibility for other variable names under which you have loaded sheet music files.
# Default: xml_file
#plot_pianoroll= bool # show piano roll; default: False
#plot_inline_ipynb= bool # overwrites Matplotlib
#do_save= bool # by do_save=True the dataframe will be saved; default: False
#save_at= str # this sets the file path; default: None
#save_file_name= str # the file name is specified here; default: None
#x_axis_res= Zahl # Resolution of the X-axis as a divisor of the quarter note; default: 2 (=0.5 quarter note)
#ignore_upbeat= bool # Ignore preamble correction; default: False
#ignore_ties= bool # Ignore slur calculation; default: False
)
File at: ../music_xml_parser/data/xmls_to_parse/hfm_database/MoWo_K171_COM_1-4_StringQuar_003_00867.xml
The following command allows to flexibly select voices and measures - for graphical representations or statistical queries. In the following example, the first 5 measures ('Measure':'1-5') of the first two voices ('PartID':'1-2') are selected:
%matplotlib inline
# First, a filter named 'filter_dict_t' is defined.
filter_dict_t = {'PartID':'1-2', 'Measure':'1-5'}
# Then the filter is added as another parameter to the piano roll command (see above):
# filter_dict=filter_dict_t
m_df_2 = mp.parse.with_xml_file(file=xml_file,
plot_pianoroll=True,
plot_inline_ipynb=True,
filter_dict=filter_dict_t)
File at: ../music_xml_parser/data/xmls_to_parse/hfm_database/MoWo_K171_COM_1-4_StringQuar_003_00867.xml
In this excerpt, you can see how the two violins play in unison at the beginning (measures 1 and 2, brown), then their own voices (measure 3/4), and then in unison again in measure 5.
You can change the selection as you wish by changing the voice and bar specifications (red in the code cell). You can also define other filters with other filter names, but then you must insert the new filter in the last line: filter_dict= [filter name].
Select different sections of the composition based on the score, write appropriate filters and visualize the sections with piano roll representations. Compare the progression of the four voices by generating a separate piano roll representation for each voice.
Generate piano roll representations from other compositions and get familiar with the Jupyter notebook this way.