I think the biggest hurdle is having enough available cars that you can call one up and have it at your door within 10 to 15 minutes. I don't think anyone living out in the suburbs would wait over an hour for a ride to do their daily tasks especially when the weather is lousy.
I think at scale when the thing becomes ubiquitous you would be able to get a car in a few minutes, and you would also be able to schedule your daily ride.
At a macro level today we have a car per family, maybe more, and they spend most of the time just parked someplace doing nothing. We could have half as many self driving cars on the road all the time and it would be far cheaper. With that many cars in circulation you wouldn't wait more than two or three minutes.
Of course between today and that world there will indeed be an intermediate world where it's more like calling a cab and people in the burbs will wait. It's going to happen in phases.
Today is DRIVERS that limit the availability of cabs. A self driving capability is ultimately going to be built into every car, so why would your car just sit someplace parked when it could be out earning fares while your aren't using it? So long as the income exceeds the depreciation it's not going to make sense to leave a car parked.
One of the first will be jitney services. Minivans that can carry five or six people that run around picking people up and dropping them off. Takes you to work but also picks up somebody else who is on the way and going near where you are.
Those jitneys have a real shot at eliminating public transit, go trains, etc, as they will be cheaper, more convenient (door to door), and potentially more economical.
Things like building a new subway line or light rail may be mistakes in light of the coming self driving car revolution. People may abandon those things for cheaper and more convenient self driving transit systems.