Some of Drupal's biggest strengths are its flexibility and massive library of user-contributed modules. Unfortunately this can also lead to "heavy" pages that require many database queries to generate.
One of the best secrets to boosting Drupal performance is to switch to Pressflow, a performance-optimized fork of the core Drupal code. Assuming you're using MySQL as database backend (and that you haven't hacked core), Pressflow is usually just a drop-in replacement that offers several immediate benefits.
Pressflow gives several immediate benefits right out of the box, but the real fun comes when you pair it with a reverse proxy like Varnish.
For Drupal 5-based sites, I've got my own modified version of Pressflow 5 that includes a rewrite of the session/cookie handling code. Unlike the current version of Pressflow 5, my fork doesn't require modules that uses sessions to make any changes, it only creates a session when it's needed, and it removes sessions as soon as they're no longer needed. Tight session management is important because requests that include a session cookie typically can't be cached by Varnish. My fork is available here: https://code.launchpad.net/~eli-elidickinson/pressflow/pf5-session-rewrite and I'm hoping it will eventually be merged into the next official branch.