I agree with the people who say that downtown is probably not a great place to raise a family. In some ways that's too bad--I think it's a self-fulfilling prophesy. Developers convince the city planning department that only single people buy condos and use that as an excuse not to build schools and parks, and to build units with layouts that aren't well suited to a family with children. The people looking for a place to live then say, gee I don't want to live downtown, there's no school for my kid and no park to play in. It's self defeating and cheapens the city.
It is true NOW, but it doesn't have to be--there ARE a few places downtown where you can reasonably raise a family, but they're few. The south side of Queens Quay is one such place--it's zoned such that you can send your kids to school at the Toronto Island School, which is one of the best in the city--and they're safe, in a giant park all day with fresher air than the burbs and lots of big open spaces. However most places in downtown don't offer anything like that and I'd agree for the most part it's not a great place to raise a family.
All that said downtown is a great place to live if you DON'T have kids, and you save a hell of a lot of money living here, assuming you also work here.
I live close enough to where I work that I can walk there in 20-25min. I don't own a car, don't pay for parking, I don't even pay for a TTC monthly pass. That's hundreds of dollars, maybe even $1000/month that I was able to plow into my mortage, enabling me to afford a more expensive place than I could have bought in the burbs, and have it paid off years sooner.
I rent cars when I need them, but even that's a freedom--quite often when I leave the city it's by airplane, and I rent the car at my destination.