Why sheet music is harder for a computer to read than text
Point a phone at a printed page of text and it will read it back almost perfectly. Optical character recognition, or OCR, has been good enough for everyday use for years. Point the same phone at a page of printed music and the results are much less reliable. Doing that well is a decades old research field of its own, called optical music recognition, or OMR (1).
At first that seems odd. Printed music looks tidier than printed text: a small set of symbols on ruled lines, placed by strict engraving rules. But it shouldn't surprise anyone who has learned to read music, which takes most people far longer than learning to read words. The reasons it is hard for a person are the reasons it is hard for a program. Text is a line, read in one direction. Music is several grids laid on top of each other, and every note sits where they overlap.
Text is a line, music is overlapping grids
A line of text has one reading order. Letters go left to right, spaces split the words, and each letter means the same thing wherever it appears. A reader that gets most letters right gets most words right, and a dictionary can repair many of the rest.
Music runs in two directions at once. Left to right is time. Bottom to top is pitch. A chord is several notes stacked at the same moment, and a piano part is two staves read together. One staff can even carry two independent melodies, which engravers call voices, one with stems up and one with stems down. These layers overlap on the page. Two voices share the same five lines. Lyrics, dynamics and chord symbols sit in the gaps between staves, and a beam or slur can cross from one staff to the next.
The meaning of a mark is mostly set by other marks. A filled notehead on the second line from the bottom is a G when a treble clef starts the line, and a B when it is a bass clef. A sharp in the key signature at the far left changes every F on the line. The time signature says how many beats each bar holds, and the beams (the thick bars joining note stems) say how long each note lasts. To name one note, a reader has to look up, down, and back to the start of the line.
One mark, several meanings
OCR has a few look-alikes: the letter l and the digit 1, O and 0. Music has many more, and the clue that tells them apart is often somewhere else on the page.
- A dot right after a notehead makes the note half as long again. The same dot just above or below the notehead means play it short (staccato). A few pixels decide which.
- A curve between two notes of the same pitch is a tie: hold the first note through the second. The same curve between different pitches is a slur: play the notes smoothly. To tell them apart, you first have to read the pitch at both ends.
- Ledger lines are short extra lines above or below a staff for very high or very low notes. On a crowded page a column of them can line up with the edge of a staff and look like five evenly spaced lines. We have seen a reader take such a stack for a staff of its own. The tell is length. Real staff lines run the full width of the system (one row of staves played together), while ledger lines cover a notehead or two.
A trained musician sorts these out without noticing, because they read the whole page with a sense of what should be there. That sense is the hard part to give a program.
Small slips travel far
In text, one misread letter spoils one word. In music, one misread mark can spoil everything after it. Miss the time signature and every later bar is measured against the wrong count of beats. Miss a clef change and every note after it lands in the wrong register until the next clef. Miss one sharp in a key signature and every F on the page is wrong.
Notation also leaves things out on purpose. The key signature is not repeated on every note. An accidental (a sharp or flat written beside a note) lasts to the end of its bar and then expires without a mark. When a second voice drops out for a while, its rests are often left out. A musician fills these gaps by habit. A program has to carry them as state through the whole score.
The result also has to add up as music. A bar in 3/4 must hold three beats in every voice. A tie must join two notes of the same pitch. A reader that gets most symbols right can still write bars that don't add up, and a notation program shows those as plainly broken. That is one reason OMR accuracy figures are hard to compare: counting correct symbols and counting correct bars give very different numbers.
It takes a stack of techniques
No single method reads a page of music. Systems in the field, ours among them, combine several, each aimed at one part of the problem.
- Finding the staves. Most readers start by locating the five-line staves, because every pitch is measured against them. Staff lines run through almost every symbol, so many systems then subtract the lines to free what sits on them. That step alone has its own studies and benchmarks (2).
- Detecting small symbols. A neural network object detector, the same kind of model that finds faces in photos, scans the page for noteheads, rests, clefs, accidentals and dots. Music is a hard case for detectors: the objects are tiny, crowded, and there can be thousands on one page (3).
- Reading a staff as a sequence. Another approach skips separate detection and reads a whole staff the way speech recognition reads audio, with a network that writes out the symbols in order. It works well on single-line music and gets harder once voices and chords stack up (4).
- Working out what connects to what. Finding a notehead, a stem and a beam is not the same as knowing they form one eighth note. Notation assembly builds a graph (the symbols, with links between the ones that belong together), and that step can be learned too (5).
- Checking against the rules of music. A bar in 3/4 must add up to three beats, a tie must join equal pitches, an accidental lasts to the barline. A constraint search, which tries candidate readings and keeps the ones that break no rule, can choose between several plausible readings of a crowded bar.
- Cleaning the image first. Photos and scans arrive tilted, curved near the binding, or unevenly lit, so lines that should be straight are not. Straightening and evening out the page comes before everything above.
Each technique solves part of the problem and brings its own ways to fail. Much of the craft is in how they are combined and how each one checks the others.
What this means if you convert a score
Errors in converted music are rarely random. They gather around the marks that carry the most context: clefs, key and time signatures, places where one staff splits into two voices, and symbols packed tight against each other. If a whole line comes out shifted by the same interval, suspect the clef or the key signature before the notes. If a bar looks badly wrong, it is often one misread mark rather than ten.
Reading music is less about recognizing shapes than about working out how they relate. Most of the difficulty sits there, and so does most of our work.