True, but more often that not, driving slow (i.e. below the limit) appears to be a symptom of unskilled/unconfident drivers. Having a few of them driving around you keeps you exposed to a consistent risk of an accident... drivers that speed past you (especially when it's only 10-20kms above the limit), and hence away from you, are more of a risk to themselves if anything, than those they speed past (i.e. you), and the time frame in which they are a threat to your safety is far shorter. Speeding and recklessness, while definitely not mutually exclusive, aren't always the same thing. Not only that, not all speed limits for given roads can be unarguably justified. No one set of rules, or system of thought can be infallible.
Still, I'd say the risk of an accident is incalculable and pretty much the same for every second you are on the road, regardless of the legal speed limit, or the speed of the drivers around you. Like Damo said... the risk starts when you turn the key, not floor the accelerator. I myself have had more accidents involving other motorists when I haven't even been in a car.