I wouldn't necessarily agree with Bob that the TD04 is "gutless" - but that doesn't mean he's wrong either. Largely down to what you're comparing with. The power level your two previous Imprezas were running, and whether the one you're buying is bog standard or not will all influence whether a bugeye WRX would "feel slow" to you or not.
If your previous cars were standard-ish 260/280bhp, then a well sorted TD04-equipped classic would compare very favourably, because, as you say, the smaller turbo spools earlier and would deliver superior throttle response to a car running the same power via a larger one. Andy's hybrid TD04 is an excellent turbo and makes a better option than an old VF28/29/etc nowadays, but whether this would be enough to make a newage feel as quick as a well sorted TD05-equipped classic is good question.
The problem you have to be aware of with any newage is that spec for spec they're a good 150kg or so heavier (more when you figure in the weight of the 6 speed box on the STis). This makes them slower, and added to the improved cabin insulation they feel less responsive than a classic running the same power level.
So, if you came out of a classic with a tuned engine, then Bob's probably right, a standard WRX won't feel that quick in comparison, but ultimately the only way to decide whether it's quick enough for you is to try one.