If you're taking Greyhound, I believe they'll drop you off at Yonge/Dundas Terminal. I'm not sure if they can drop you off before the stop but if they do, you'll be much closer.
http://goo.gl/maps/ZVjGR
It's going to be 50 minutes walk (a very fun walk), so have fun sight seeing!