The best quality depends on two factors:
- Capture size vs Output size; and
- Compression method
For best quality, always capture 1:1. Set your capture area (what you want to capture) and then choose the Same as capture size option from the gear dropdown just above Output video in the toolbar.
Now you've ensured that iShowU HD won't scale the video.
Secondly, select Apple Animation as a compression method (drop down the same gear icon you did before, above Output video and choose "Apple Animation")
This is lossless and provides excellent quality and speed. AA can be somewhat disk intensive however, so if your framerate drops choose Apple Intermediate.
Finally, compress the movie with Stomp. The AA and AIC codes provide excellent visual results but are massive. You'll need to compress to H264 if you want to put the movie onto the web.