I'd rather look this: you have the right to drive so long as you aren't endangering other people. Too bad safe driving is subjective.
Unfortunately because people cannot be trusted to judge whether they are driving safe or not, it must be regulated. And by "driving safe" I don't just mean being able to control your car at a certain speed on a certain road and weather condition, it's also realizing that the drivers around you may contain people that have misjudged their ability to drive in a safe manner. So someone slaloming their car through traffic moving 40 kph slower than them is not a safe driver, regardless of their driving ability.

