2003/04 proved to be one of the most remarkable seasons in the club's history. Up until the New Year Walsall were flying. West Bromwich Albion and Nottingham Forest were both destroyed 4-1, as new-signing, the ex-England and Arsenal star, Paul Merson seemed to be repeating some of the heroics that had led Portsmouth to promotion the previous season. Following a Boxing Day victory at Cardiff City, the club sat just 4 points off a place in the play-offs. It wasn't to continue though.

Walsall suffered demotion on 22 April 2006 to league two after losing 3-1 to Huddersfield Town. Rotherham United's 1-1 draw with Scunthorpe United saw an unassailable gap of seven points formed. Broadhurst was sacked the next day. Player coach Mark Kinsella was put in charge for the final two matches of the season, with Walsall reverting to duties with the reserve and youth teams. On May 3, the team appointed their fifth manager of the season in ex Scunthorpe manager Richard Money.