What qualifies as Downtown Toronto is often in relation to one's distance away from it. Roughly speaking, the eastern border could be considered the Don Valley, St. Clair Avenue to the north, but that too is a stretch. Most Toronto Maps show a few blocks to the north of Bloor Street, including the Yorkville neighbourhood as the upper limits of Downtown.
Bathurst Street to the west more or less is agreed upon as an unofficial official limit to Downtown T.O
Those who live and work within the above limits, often refer to a downtown with downtown as the Downtown Core or simply The Core, which more or less includes The Eaton Centre, the Financial District and all that falls inside the loop of the Yonge-University-Spadina Subway line.
See the Neighbourhoods page for more.