What does the name 'Brahms' conjure up in your mind? For most people the image that automatically springs to mind is of the composer in middle age, corpulently rotund, lavishly and greyly bearded and reassuringly avuncular in appearance. Blame the photographs because he belongs to the first generation of composers who were caught on film (and images of Brahms 'at leisure' are greater in number than for most composers). That image attaches to his music, too — from whatever time in his life. For his detractors it's the very confirmation of stodgy North German-ness: solid and dependable, but backward-looking. For those who adore his music, it's an image that throws up contradictions for his is music of such passion, of such intensity that image and imagining never quite tally.

But there was a time when he was slim, blond and, according to many reports, almost girlish in appearance. That was the Brahms who left his native Hamburg in the early summer of 1853, aged just 20, returning later that year, a doted-upon composer, known to Liszt, Wagner, Berlioz and, of course, Clara and Robert Schumann, who declared him a genius. He had achieved fame and presented in his first few works a solid foundation from which that celebrity would grow. Very few people who crossed his path were unaffected by his obvious and abundant talent.

When he died, in April 1897, aged 63, he was the German-speaking world's foremost composer; indeed he was probably the most famous composer in the entire world. His music was played across the USA as his disciples took up posts in the musical capitals of the New World. He had amassed a considerable fortune almost entirely from the royalties to his music with occasional supplements from performing. He was a composer who did not rely on financial patronage but who truly lived off his talent and was beholden to no one. Today, Brahms's music — particularly his orchestral music — has a grip on the repertoire like few others, his god Beethoven apart. No other composer offers a symphonic legacy that is so well balanced, so concentrated and at such an equally high level: four works, like the four sides of a square, that comprise a unit of great strength and integrity. It's no surprise that barely a week (hardly even a day) goes by without a performance of one of his symphonies somewhere in the world; in the US during the 2004-05 season Brahms, according to the American Symphony Orchestra League, will be the fourth most-performed composer. On disc, Brahms symphony cycles have continued even when record companies have hesitated over Beethoven's. This month sees the conclusion of one such cycle, from the London Symphony Orchestra under Bernard Haitink, and the start of another, from the London Philharmonic Orchestra under Marin Alsop.

I caught up with both conductors just before Christmas to talk Brahms, a subject they each address with considerable passion. Alsop was passing through London, from a concert the previous night with the Residentie Orchestra in The Hague which had included Brahms's First Symphony, en route to her home in Denver, where she started 2005 with a Brahms cycle with the Colorado Symphony. Bernard Haitink was concluding a three-concert series with the LSO. There was no Brahms on the bill, but it made a glorious close to a season that has provided musical gold aplenty for London concert-goers in his 75th-birthday year with a handful of Europe's greatest orchestras.

Those of us who love Brahms's music love it with a passion that wanders through his entire output and alighting on specific pockets, like the choral works. Marin Alsop remembers her first encounter with Brahms's music. 'I heard the B flat Sextet when I was 11 years old — and that was the very first time I was moved by music. I had this experience of an emotional reaction to music — of course my hormones were probably racing, too — but I remember being just gripped in a way I never had been before and I didn't know that music could do that. Of course I raced off and bought the record — it was the Amadeus Quartet's — and by the time I was 16 years old the record was so thin because I'd played it so many times! There was something very 'connecting' about Brahms's music.

'Then of course I got more engaged. It's like with an author — you read everything they wrote, until you OD on it. And I did that with Brahms — analysing the way he approached his works, particularly the symphonies which he comes to so late in life, and what psychologically led to them and how Beethoven's influence played a role in them and so on. How one incorporates that interpretively becomes a real Sherlock Holmes experience, which I love.' For Alsop, the 2004-05 season has turned into something of a Brahmsfest. 'It wasn't terribly conscious but I knew I'd be in the midst of these recordings. It was interesting because a lot of orchestras asked for Brahms from me so it seemed to work out well. It's fantastic because orchestras who play Brahms a lot don't do the Third very often so you have the challenge of working on a piece together.'

For Haitink, who is just concluding his third Brahms cycle on disc, the composer has been with him 'right from the beginning. But I have learned by trial and error. I've discovered that concert organisers, particularly on the Continent, give Brahms symphonies to young conductors — "Let him do a Brahms symphony!" — but it's not that easy. You need — maybe now that I'm older I would say this — but you need more experience to bring off Brahms well. It's difficult to say why. I think when we think of Brahms we all conjure up a huge, inflated sound and that is totally wrong because Brahms wrote so much wonderful chamber music which is a fine indication of the textures he was after. If you look at the scores of the symphonies it's amazing how often he writes piano dolce or pianissimo dolce. Dolcecomes up so often, even more than with Beethoven, and Beethoven uses that "warning" a lot. Take the beginning of Brahms's Fourth Symphony: it's very often played far too loud, but then you can't build it any more. When he uses piano, he creates a beautiful sound and when you start at that level you can make a huge build-up, otherwise, with that movement in particular, it gets a little stale. That is one of the secrets of playing Brahms: you have to be very acute about how you pace the dynamics.'

One of the intriguing things about the composer is how extraordinarily well he covered his tracks, both personal and musical. He left virtually no drafts, few first thoughts or even first versions. The published work we have is virtually all there is. For Alsop, Brahms's witnessing at close hand Robert Schumann's mental decline, and the public interest in it, is key. 'I think Brahms is very difficult to get a handle on. From seeing Schumann in the limelight and watching how the paparazzi of the time worked, he was extremely aware and conscious of not leaving much of a trail. So he rarely documented anything, he rarely kept notebooks and sketches, so most of what we know is anecdotal or conjecture. I think that was all very intentional.'

Naturally this presents the double-edged situation where almost every piece of his is in its finished text. 'Before I started work on the symphonies I spoke with the Brahms expert Robert Pascall. We went through the scores and what you find are mostly dynamic issues, the kinds of minor things that you get when you transfer from a manuscript to a print where the publisher or editor would make some changes, often unintentionally — but they're really tiny alongside, say, Mahler's. Maybe that's a nice element not to have to agonise over. But one agonises in different ways, but then that's great, too. You can imbue his music with a lot of your own associations.'

For Haitink, the question of Brahms the man and Brahms's music — and indeed any composer and his music — presents the ultimate conundrum. 'How far the character of an artist has influence on his actual creative process one never knows. I always try to read the biography and read around the composer. It's always good to do but as a performer one doesn't learn anything from it strangely enough, but to be confronted again by the man and the artist is important. It is an enormous riddle to me. How does the personal life of an artist influence his music? Look at Mozart: during his most miserable last years he wrote extremely sad pieces but always balanced by humour or a slight optimism. Maybe not the G minor Symphony [No 40] which is a very dark brooding work, but K543 [No 39] and the Jupiter are written by a man who was broke, people ignored him, perhaps he gambled all his money and his credibility away. He was maybe a very lonely man and he writes this fantastic C major music. It's amazing!'

Over the years, Haitink admits, his attitude to the music has changed. 'I'm much more touched by it, much more intrigued by it. Brahms was not an easy man and I think you can hear that in his music. One of the famous things he said when he left an evening gathering was "If I haven't offended someone here, I apologise!" He was difficult. People always think of this man with the beard, relaxing with his cigar, but there was so much more to him that that. I was in the Musikverein in Vienna a while back — where he left his library — and it's amazing what is there. He was an incredibly well-read man, very educated. But I think he was a very lonely man, too.'

Read the booklet-note or programme-note to any Brahms symphony and you'll learn that the composer waited until middle age to unveil his first work in the genre — well after he'd grown that splendid beard and taken up the mantle of upholder of German music. He spoke often of the towering example of Beethoven's remarkable nine symphonies. But scale alone did not deter him — one of Brahms's earliest works is the Piano Sonata in F minor, Op 5, a five-movement work of titanic proportions and reach. Perhaps a need to develop an orchestral language delayed his first attempts, but by the time he started work on his First Symphony, some time in the mid-1860s, he'd already written two serenades (admittedly for small ensemble) and the huge (in every respect) First Piano Concerto. When he did grapple with the symphony he published two in relatively short succession (1877 and 1878) and then, before a decade was through, another two, again almost as a pair (1884 and 1886). So when we reach Brahms's First Symphony, we are not encountering the work of a tyro composer: it is the mature embracing of a form that he always viewed with dread. 'You don't know what it is like,' he once told the conductor Hermann Levi, 'always to hear that giant marching along behind me'.

How many other composers have started their first symphony in so imposing — and confident — a way? The steady tread pounded out as if to acknowledge that marching giant but defiantly refusing to be cowed by it. For Alsop, the key to interpreting the First is not to allow indulgence. 'You cannot wallow. There's an organic tempo in each symphony. For me, I feel there is a very particular tempo but I've certainly heard some dramatic extremes. I think it just depends on what one's take on this pulse at the beginning is. If it's a heartbeat then for me it has to be manageable — I'm not having to run a six-minute mile. But if it's not, and is just a general kind of life-pulse, then I can understand speeding it up. One has to be very committed to it.

'The music is so strong. More than with any other composer the timpanist shapes Brahms for me. It's absolutely critical to have someone who understands not only pacing and timing, but also the colour of sound and the weight of sound. One of the things I adore about Brahms is his ability to have one foot firmly anchored in the tradition of music. He was famed for his adoration of Beethoven, but also of Bach as well — and that leads to his beautiful bass-line construction — but he was also completely aware of where music was heading. And to be able to straddle those two worlds successfully is his real genius. Take his choice of instrumentation — in the First Symphony he saves the trombones for the last movement just so they were still something special. Or to use the tuba in only one symphony — to make these choices, to use horns as if they were still natural instruments even though he didn't need to. He could really play tribute to the past but look ahead to the future so strongly.'

Haitink is in agreement. 'The opening of the First Symphony is marked Poco sostenuto. It should not be too slow. He can tempt indulgence. It sounds a bit odd for a conductor to say this but one should leave it alone. It plays itself in many ways as long as one takes care of the sound, of the tempo, of the dynamics — that's already a lot to ask! — but don't fool around with ritardandi that he hasn't written. When he writes poco ritenuto, don't make it molto. It's a habit that when people want to impress, they think they impress by pulling it around. But they don't. Going back to that opening of No 1, it's passionate but there's also the danger that performers want itfortissimo, when he writes forte. Later on they will have the chance for fortissimo in the resolution. It's very important to get a wonderful sound for that beginning: passionate but not fortissimo.' 

Symphonies Nos 1 and 2 were written just a couple of years apart, so does that encourage thinking of them as a pair? Alsop: 'One of the great rewards about conducting these symphonies is that they're all different planets but they belong to the same solar system. With Brahms there's a completely unique world with each work: it's a life unto itself, and a very different cohesive life. That's the joy of it. But I really do see them as complements of each other so No 1 and No 2 feel related. I feel so strongly in Nos 1 and 2 the influence of Beethoven. For me, No 2 is highly influenced by the Eroica so I don't feel the term that is often applied to it — Pastoral — is quite accurate because it has that inner strength.

'If you look at the openings of 1 and 2 they couldn't be more diametrically opposed. No 1 is all about downbeats and No 2's all about upbeats.' A very Bernstein-y comment? 'The one thing I remember Bernstein talking to me about in Brahms — and yes, he loved to make those epic statements — was when he was working on the Fourth Symphony. He was explaining to the orchestra that "there's always a moment in Brahms when there's an amazing silence" — actually there are many moments — but he would try and prioritise what the biggest silent moment in a Brahms symphony would be. Those things are so great to think about, those big ideas.'

Haitink, too, finds Pastoral for No 2 a misnomer: 'I think that's a bit simplistic because it could invite a very relaxed attitude. Of course the first movement is relaxed, but it should not be too slow in tempo: it should be very transparent and it has some wonderful textures. One can't sit back and enjoy the countryside all the time! The second movement has no pastoral atmosphere at all for me. It's a very strange movement, I love it, but it's so dark and brooding. The third movement is lovely, a real scherzo in mood and the finale is very festive. One could almost think of Haydn there. But it's so dangerous to use these labels. In comparison with the other symphonies, yes there's a pastoral feel but ...'

The Third and Fourth Symphonies followed in the mid 1880s and, in their very special way, present an even more intriguing pairing than the first two symphonies: No 3 is concise and concentrated — and fiendishly difficult to bring off — the equivalent perhaps of Beethoven's similarly proportioned Eighth Symphony, a powerful little work before the epic splendours of the Choral Symphony. The Third is traditionally the most neglected of the four. Marin Alsop laughs and admits to the vapidity of the conducting profession who avoid it, it's often suggested, because of its gentle ending, hardly a way for a conductor to leave the stage in a blaze of sonorous glory. 'I think the Third is probably the most challenging to conduct but I don't think it is obscure in any way. The opening is crucial. I think it's really a technical dilemma which must stem from an interpretive question on the part of conductors because it's quite tricky to find the right tempo that propels it without pushing it too much. That is crucial and that is the key to Brahms — to give it the space without making it sound slow. And I think great orchestras can really do that. They can fill in the time so you push the envelope of time without really expanding it. Just like filling it up as much as you can before it changes shape. I think that in the Third it's really important to feel the water filling in all those chords at the beginning.'

Haitink agrees. 'It's always said that conductors don't want to end with it because of its quiet close. No 3 is a very complex piece and it's not for nothing that's it's not performed as often as the other symphonies. But I would always place it last. When I do it later this year in Berlin the programme goes Haydn symphony, Bartók Dance Suite — interval — then Brahms Third. The first movement is incredibly symphonic and technically quite difficult for a conductor to keep together in 6/4. It's very difficult to launch: there was this old tradition of those two chords, major and minor in the brass and that conductors made a crescendo towards the third bar. For me they're two pillars that shouldn't be messed around with: here's one and there's one. It's very passionate, but then it breaks down at the end of the first movement already to a piano finish. Then this wonderful second movement, which is pure chamber music, and the third movement is so introverted, but such an extremely touching piece. And the last movement starts very foggy, rather North German fog but then the sun cuts through and there is an enormous, heroic outburst but at the end it breaks down again and ends maybe again in resignation.'

And so to the great Fourth Symphony. Every time I hear it I have a feeling that something has been happening before the music starts. Marin Alsop points out that in its original form — and it's one of the few examples of an early thought that survives — there was a slow chordal opening. 'I can see why he abandoned the original opening of the Fourth which was four bars of sustained chords because the Third opens with those chords, I think the idea of having these pillars again just doesn't appeal. With the Fourth you sort of open the door and the conversation is already happening. That was a brilliant thing.' For Haitink, 'something already's going on and the curtain rises. And there it is...'

And again, the label that often gets attached to the Fourth is Tragic. Haitink: 'It reminds me always of autumn, and like that season there's always a little resignation with life. Summer has gone... but again we're putting stickers on it, and that's so dangerous!' Alsop, too, doesn't see anything as strong as a tragedy here. 'I think there's an element of longing, perhaps a sense of loss but not "spoken-out-loud" loss, more like "I wish I had said what I meant to that person but didn't" — that kind of thing. To me the slow movement of the Fourth really is one of the most sublime movements in music. I think it's spectacular. The way it opens with this repetitive figure — it's almost like you're all by yourself in the mountains, you're singing something out that keeps echoing back and forth. The audience has no real feel for where the beat is and Brahms puts us off centre so often. He moves bar lines by displacing the downbeat and that's just a brilliant compositional technique. People who aren't musicians don't understand why, but they feel a little bit uncomfortable: that's fantastic. But that slow movement in particular I find very poignant and gorgeous.'

Brahms was, famously, anointed by Schumann as Beethoven's natural successor, a man who from his earliest compositions was the creator of 'veiled symphonies'. He was seen as the torch-bearer for the great German tradition that seemed to jump from peak to peak from Bach to Beethoven to Brahms. In the opposite camp were Liszt — whose music Brahms loathed — and Wagner — whose music, especially Meistersinger, he rather admired. But was Brahms the loyal, traditionalist, more keen on carrying a tradition, or was he forward-looking, into the future. Arnold Schoenberg, a man who altered the course of music as no other, was a great Brahmsian, the man who spoke in a lecture given in February 1933 on the centenary of Brahms's birth, of 'Brahms the Progressive'.

Marin Alsop in is no doubt as to Brahms's musical direction. 'I do think he's a real progressive. And I think that often great progressives are the ones who pay homage to the past whilst looking into the future. One of my favourite things is to do the Schoenberg orchestration of the First Piano Quartet. I'd love to do it as part of this cycle. It's fun and actually very Brahmsian.' And let's leave the last words to Bernard Haitink. 'I have friends, many of whom really do know and love music, but for them Brahms is too Teutonic. How can I answer that? I think it's probably the fault of bad interpretation, when the sound gets clogged, it's a problem. That can happen easily — it's the same a bit with the music of Elgar who occupies a similar sort of sound world. I always get irritated when people say he is a traditionalist and a reactionary. Schoenberg, who was hardly a fool, recognised that! When you listen to Brahms's music, you encounter his imagination, his orchestration, his incredible structural strength. For me, there is nothing backward-looking about Brahms.'