What finite state automata and grammars have in common is that they describe languages — which are finite or infinite sets of strings composed of symbols taken from some alphabet, such as letters, or notes in our case.
Finite state automata is useful when you want to determine if a string belongs to a language, but they don’t explicitly provide a way to generate a string belonging to that language. Loosely speaking, they are made up of a collection of states linked together by transitions — a transition gets you from one state to another when a certain symbol is read the string you feed it. You start at a unique initial state, and each symbol consumed takes you to a new state, some of which may be final states, in which case the automaton indicates the string belongs to the language.
The caveat is that finite state automaton can only recognize a small subset of all possible languages, as they have very low computing power. A grammar allows you to extend the span of languages you can reach, as they are more computationally powerful. They also provide a direct way of deriving strings belonging to their language, and have a more intuitively understandable structure, which is why they seem fitter for music pattern generation.
They are made up of a set of production rules, where each production rule is identified by a variable, and where each one leads to a combination of other variables or terminals, which are just elements of the alphabet. For example, if we want a very simple definition of a sentence, we could have this (anything beginning with a capital letter is a variable, while lowercase is a terminal, with the character | signifying or):
Sentence -> NounPhrase VerbPhrase
NounPhrase -> Determiner Noun
VerbPhrase -> Verb NounPhrase
Determiner -> the | a
Noun -> cat | dog
Verb -> saw | chased
Here, «the dog chased the cat» is a derivation of the language described by the example grammar.
Derivation Tree
Sentence
/ \
/ \
/ \
/ \
/ \
/ \
V V
NounPhrase VerbPhrase
/ \ / \
/ \ / \
V V V V
Determiner Noun Verb NounPhrase
| | | / \
| | | / \
V V V V V
the dog chased Determiner Noun
| |
| |
V V
the cat
Furthermore, you can extend grammars to support more features, such as probabilistic rules (which allow the introduction of Markov chains), at the cost of it becoming much harder to parse it — reversing the creation process of a string belonging to the language to uncover the way it was derived from the grammar, which we don’t care too much about if we just want to generate music.
Now, backing off from this field — abstract algebra seems to be a fascinating area of mathematics which I am not familiar with. I’d be intrigued to see how they can be applied to music.
And about set theory — or actually not set theory but the representation of notes it uses — modulo 12 arithmetic notation seems like a convenient way to notate music as it makes intervals obvious. However, it seems like one of it’s drawbacks is that it discards information about the absolute pitch value and it’s overall height relative to other notes, which is an important characteristic of melodies, and also has a large effect on how we perceive chords. If you build a complex chord by chaining different intervals, you can get a pleasant, harmonious result if you spread the voicing effectively across octaves, while it is very likely to sound like horribly dissonant if condensed within an octave. Our sense of harmony also fades away as we reach higher frequencies. But still, modulo 12 arithmetic is much easier to work with as a human.