Downtown means different areas to different folks, and when you get south of St Clair or Bloor on the central city, it's such a unique driving experience that I wouldn't think it would make a very useful test, if those streets were all you drove on. In olden times when Toronto stopped at Eglinton, it might have been politically impossible not to have even one test centre in the provincial capital, but now Toronto stretches from Mrs Sauga to Pickering and all the way t Steeles. I suspect one or both reasons accounts for why there's no longer a downtown centre.
That and selling the testing business off to a private—British I seem to recall—profit-making company, instead of running it as a service for taxpayers. My son who lives in Leslieville found the Scarboro location was the best bet.