When Milton Babbitt died in January of this year, I speculated in my Gramophone obituary that if only Babbitt-the-man had kept mum about his theories, Babbitt-the-composer might have had an easier time of it.

Whether Babbitt wanted an easy time, or even whether his music deserves wider recognition, are imponderables that are guaranteed to run and run. But I never expected to hear my friend Tom, admittedly a man who talks as much about Ray Davies and Bob Dylan as about his academic interest in atonal music, fess up shortly after Babbitt’s death 'Phil, you know, I just don’t like listening to atonal music these days.' I’ve spent many hours talking to Tom about Babbitt over many years. But have we been deluding ourselves? Is Babbitt’s music more a talking, and reading, point than a canon of work worth listening to?

Doubts dogged Babbitt throughout his long career, not helped by an article he wrote for High Fidelity magazine in 1958, which he called 'The Composer As Specialist', but the magazine provocatively ran as 'Who Cares If You Listen?' The very name ‘Babbitt’ came to encapsulate everything that was allegedly wrong with contemporary music – a scene dominated by fusty academic composer types who held audiences in distain, who relished being photographed against a backdrop of tone rows chalked onto blackboards, who wrote pieces so damn impractical that only five people could play them which, actually, was fine because only three people wanted to hear them. As Elliott Carter morphed into the Queen Mother of Atonality, Babbitt became a dumping ground for frustration and despair at the state of modern composition – scapegoat or villain, depending on your point of view.

Most criticism of Babbitt was/is predicated on the notion that he misread music history – that his particular view of atonal music severed any link to the ‘grand tradition’ which, for all his faults, Schoenberg managed to retain; Babbitt’s music was self-contained and had abdicated its responsibility to communicate. Anybody whose working assumption is indeed that Babbitt was a geeky boffin lacking depth in the history department, could do worse than read Words about Music, his book culled from a 1983 lecture series hosted by the University of Wisconsin, which gives the most penetrating insight I’ve read into the functionality of atonal music; and, guess what, context is everything, just like in tonal music.

Schoenberg, naturally, is Babbitt’s point of departure, but he is not uncritical. He picks holes in the inconsistencies of Schoenberg’s early atonal technique, asserting that Die Jakobsleiter remained unfinished because he (Schoenberg) failed to recognise his system could generate aggregate note choices; so he ran out of juice. But this critical edge lets his advocacy of a later masterwork like the String Quartet No 4, in Babbitt-speak a central text, resonate all the more eloquently. With a canny turn-of-phrase, he demolishes the misnomer that writing serial music was your fail-safe passport to success and relevance: 'Schoenberg wasn’t saying that if you do this (i.e write atonal music), you will therefore create great or coherent or marvellous music. But he was saying that if you don’t do this, your music will be worthless.'

And throughout Words about Music, Babbitt shows no mercy towards those who would muddle the technical nuts-and-bolts of 12 tone theory with music itself. He mocks the common misunderstanding that a 12 tone row is a 'kind of theme' that to produce music 'you do funny things to…(without) skill, continuity, structure, imitation'. Lampooning the idea of, as he puts it, 'one note, one vote' – 12-tone music as somehow managing to defy harmonic gravity, making each note ‘equal’ – he asks 'how can any 12 notes be all equal? One’s going to be longer, one’s going to be higher, one’s going to be first, one’s going to be last.'

Moreover, as he meticulously illuminates the ins-and-outs (and the retrogrades and inversions) of atonal theory, he invokes base tonal alchemy obsessively, like an experimental scientist who must always have his/her control. Discussing modulation, Babbitt explains that the Eroica Symphony is in E flat, and its second theme is in B flat – but 'is the B flat of that second theme the same B flat of the first theme of the Fourth Symphony of Beethoven?' 'Obviously not' comes his quick reply.

It’s the context, stupid – the B flatness of a B flat only has an audible function when heard against, say, the C-ness of a C, and that context is governed by whether the C is a fleeting appoggiatura, or a sustained bass note, or something in between. Later, Babbitt muses: 'I have never written a piece within the past twenty years that I didn’t get letters about in which people say "I’m sorry but there are 4892 mistakes in your piece"…Maybe one or two of them are right; there are misprints or miscopying or something. But mainly they incorrectly assume if the piece starts a certain way, it has to go that way, that’s all. Part of the notion of the piece may have been that certain things change their dependencies, their contingencies, and their relationships in the course of the piece. Things change!'

But for me Babbitt’s killer line is this: 'I don’t have to say more than once that 12 is not important, the effect of the 12 is.' And his book traces that cause and effect: how each individual note is effected by the actual, or implied, presence of the other 11; how those aggregate combinations missing from Die Jakobsleiter can accumulate material, layers of meaning too, that reach far beyond the basic mathematical permutations of the row; how composers need to deploy their taste, imagination and individual voice to mould this flow of material.

So there’s the theory. But by saying precisely nothing about how this backstory makes the music sound, have I just confirmed everyone’s worst fears that there is more to be said about the theory than the music? One thing is for certain: the intensity of Babbitt’s engagement with not only theory, but with the philosophy of ‘why theory’, means his music, for better or worse, never sounds like anybody else’s. But faced with an extended, discursive score like his 1985 Piano Concerto, how to listen? If you don’t spot those aggregate tone rows in action is it game over?

Certainly not! In fact, anybody trying to train-spot the tone rows in Babbitt will be left very disappointed. But who listens to the Eroica expecting to flag up every E flat en route? The E flatness of the Eroica’s opening bars is only significant anyway because, in bar 7, Beethoven aims to undermine it: he contradicts E flat with a tonally alien C sharp that seeds harmonic tension on which the rest of the symphony will feed. Likewise, Babbitt’s Piano Concerto (recorded by Alan Feinberg and the American Composers Orchestra under Charles Wuorinen on New World Records - buy from Amazon) challenges us to gauge any present moment against what has gone before; how ‘now’ fits with, or disrupts, your perception of the gradually evolving whole. That understanding might feel worryingly superficial on a first listen – nothing grander than realising one sound is higher than another; of an awareness of shifts in orchestration; that the flow of material is suddenly accelerating. But that’s when you need to dig your heels in; to take a leap of faith, to listen harder: not attempting to zone into reconfigurations of individual tone rows that nobody this side of Pierre Boulez can hear, but to the shapes, gestures, characterisations, tensions, timbres and interrelationships Babbitt has harvested from his material.

I’m not saying that’s an easy thing to do. There are some user-friendly hooks in the concerto, like stark percussion jabs that punctuate the solidly interlaced music they surround; or that magical sense of the music dissolving and thinning as the end approaches. But the harder you listen, the more there is to hear and the harder you want to listen. And if the Piano Concerto really is a stretch too far, Swan Song No 1 (performed by the Cygnus Ensemble on the Bridge compilation 'The Music of Milton Babbitt' - buy from Amazon) is representative of Babbitt’s more elegant side – the play of plucked mandolin, guitar and strings against brittle woodwind; the cryptic way he sets up expectations of countable time that he purposefully upsets. But the mood is kept whimsical and playful, like serial light music.

Babbitt died aged 94, still the source of controversy. He was born into a generation of composers who believed music needed to sting: but there’s no longer a buzz around E flat crashing into C sharp because we’ve absorbed Beethoven into our culture; the shock has been cushioned. Babbitt embedded designer difficulties into his music as an insurance policy against his work similarly being sucked into the mainstream. The intellectual beauty of his music is invigorating and inspiring, and contrary to perceived wisdom, nobody would be interested in his theories if his music wasn’t so potent and unique. So who cares if you listen? Ultimately, no one. But in a modern music scene where Anna Nicole Smith is suddenly fair game for opera, where there’s post-minimalism and pseudo-minimalism – and post-modernism and neo-modernism – and downright rubbish minimalism and modernism – Babbitt’s determination to remind us of the materiality of music is well worth caring about.