When Conlon Nancarrow – the Arkansas-born composer who, in 1940, relocated to Mexico to escape anti-Communist censor – was (re)discovered during the late 1970s/early 1980s no one seemed sure how to deal with his unlikely genius.

Disillusioned by the continuing epic failure of living, breathing musicians to make his interlocking ratios of mathematically irrational rhythms add up, Nancarrow decided to devote his life to composing pieces for player piano, the mechanical instrument originally designed as a functional gizmo to reproduce music in the home. Aaron Copland, John Cage and Elliott Carter had all taken a keen interest in Nancarrow before his self-imposed Mexican exile. Copland had reviewed his music enthusiastically; Carter was trying to hook him up with a commission to write a ballet score. But Nancarrow was motivated more by creating the music he wanted to create than by having a 'career'. When Frank Zappa, György Ligeti and James Tenney started discussing his work in interviews, he was, whether he liked it not, winched out of obscurity. Ligeti praised 'the best music of any composer living today' and Nancarrow toured European concert halls with recordings of his Studies for Player Piano on tape.

This was of course a mammoth compromise, and as last weekend’s festival devoted to his music at the Southbank demonstrated – knowingly subtitled Impossible Brilliance – live Nancarrow remains a tricky issue, mainly because live performance didn’t interest him.

Onstage at the Purcell Room was a mock up of his workshop, complete with comfy sofa and louche cocktail cabinet. The one time Nancarrow organised a concert of player piano studies in Mexico he concluded afterwards that the best – indeed the only – place to experience his music was in his own workspace, preferably accompanied with a bottle or two of something 'warming'. Only friends turned up to his gig – so why go to the expense of hiring a hall again? And, anyway, how suitable is mechanical music in a space designed for living musicians?

Nancarrow understood that you can't wish a formal space into a state of informal being any more than you can 'decide' to have a sense of humour, and on Saturday and Sunday laboured, wordy introductions imposed a layer of unintended stiffness to proceedings. But the music didn't care. Watching a player piano from the back of the Purcell Room was unsettling. What's the protocol? As pieces finished who/what were we applauding – the mechanics, or the composer’s memory? Structural chickens come home to roost in each Nancarrow study as the line your ear happens to have latched on to crashes into a spaghetti junction of counterpoint cruising the other way. A sonic pile up ensues. And this complex overlay is tantalising because Nancarrow’s starting point is usually instantly recognisable. A boogie-woogie vamp, or a melodic line with Baroque precision and direction. 'Real' music diced up; injected with speed.

This reality-twist clearly played well with Zappa and Ligeti whose public statements were largely responsible for Nancarrow's outing. But with celebrity came potential compromise and the more Nancarrow was turned into the 'classical' composer he never really was, the greater the problems. A concluding concert by the Arditti Quartet included the structurally lumpen Third Quartet he wrote for the group in 1987, which sounded tame and uncertain against a pairing with Ligeti’s über-composerly Second Quartet; and a London Sinfonietta concert of transcriptions raised issues that were even more uncomfortable.

The desire to contain, to rationalise Nancarrow’s music – to back-transcribe it for human performers – is understandable. But his studies rely on deadpan timbral neutrality – the idea that this piano-like droid might, at any point, stop behaving like a piano, while still sounding vaguely pianistic. When carefully intentioned choices are made about which line to give to the piccolo, the bass clarinet or the bassoon, that uniformity is upset and the implausible becomes plausible, but only via a technicality. Why would anyone want to rain on Nancarrow’s parade of impossible brilliance, torpedoing that which made him endlessly intriguing and unique in the first place?