Maximum Number of Listings in Directory Theme

Is there a maximum number of listings that you would recommend for the Directory theme?

I have 300 categories and about 24,000 listings..

I recently imported 23,000 local business listings and immediately started experiencing a huge slow-down on page load times. Caching has helped some, but wondering what a practical limit on the number of listings would be on a shared hosting environment?