Agreed. Music uses time as an initial vehicle, but definitely latches into our emotional core (/memory, if that is separable) for the longer journey. Brün has some stuff about this:
The smallest group of motifs which expresses a Thought is called a sentence. If the motifs in this group are of equal meter, a metrical sentence results, as in a poem, a folkdance, almost all music up to Stravinsky, etc. If each motif in the group is of a different meter, the result is a prose sentence: of a tale, a movement, some new music since Schoenberg, etc.
Motives and sentences, quite independent of motif’s meter, become larger units in Time. Thus Time becomes divided again into accented and non-accented groups of motifs and of sentences. The inner division of these larger units and the time-relations between them display for us the Form and Forms of Time: Rhythm. All sentences, metrical or prose, can be presented in all kinds of rhythm, regular or irregular. There never is “no rhythm”.
When one can choose rhythm, the Artist must choose a rhythm. Only so will the nature of one’s Thought become known in Time. To be able to choose, the Artist must know all the possibilities. One must know the nature of Time and Rhythm, and how it serves the interpersonal requirements of one’s work. The Artist must beware of the damage which Time and Rhythm will do when used to serve the personal requirements of the Artist’s own nature, following the Artist’s laws instead of theirs, that is, from the Artist’s inside out, that is: superficially. The Artist must know that two sentences, each sound and strong in rhythm, may appear nevertheless weak and inexpressive when following one another, and why this is so, and how it can be corrected. One will have to revise one’s choice of Rhythm in each sentence for the benefit of both.
And so on, until the whole work stands and moves and speaks as one rhythmic unity, in which the varieties of motif and meter, sentence and rhythm, can be understood, remembered and related to the Thought, Idea, Feeling which the work proposes to convey.