Summary (aka: can't be bothered reading below, give me some suggestions)
- Try using Prores or Jpeg.
- Try reducing the output size of the movie.
- Turn on Variable frame durations, and turn off Low CPU mode. (PRO)
- Turn on Low CPU mode (compression will be performed after capture) (PRO)
Those last two options - why would they help?
iShowU HD (and HD Pro) are both compatible with Final Cut. What this means in practice is that they write video frames at exactly the right points in time, and guarantee to do so. However in some cases this can yield bad performance (e.g: capturing a heavy scene in Second Life).
You can tell because the movie itself says 30fps but playback appears to be only 10fps.
The real solution is to relax the frame duplication constraint and allow iShowU HD full freedom to skip frames when it needs to. This by the way is what iShowU classic does. With variable frame durations enabled, HD can drop a frame and simply expand the duration of the next frame to suit. In this way total CPU and disk load is reduced, giving a smoother capture.
The other option is to enable Low CPU mode, so that you turn off compression entirely and let HD write out temporary frames to disk. When you're done recording, HD will compress the frames. This is a tradeoff between slowing the computer down while recording or slowing it down at the end. By turning off the compressor while recording we free up the computer while recording, thus resulting in faster, smoother captures.
- The output size is the movie. That is, what size (width x height) are you creating?
- The compression method are you using (H264, ProRes, Jpeg)
- The frames per second of the capture (and whether variable length frame durations are enabled)
- Speed of machine
The first two questions are the most important: What is the output size and compression? Let me provide an example.
If I have a MacBook Pro, and try to capture full screen (1440x900) using H264 at 30fps, the MacBook Pro isn't going to be able to do it. H264 is simply too resource (CPU) intensive. As a result, you'll get frame skip and the movie may look jerky.
The solution is record using ProRes or Jpeg and then convert the movie to H264 later using either Stomp or QT Pro. Both of these use far less in the way of CPU resources, and allow for faster (and thus smoother) captures.
About capture size / output size
Note that the capture size itself doesn't make much difference. It's the final movie size that matters. The larger the size, the more load is put onto the CPU. Capturing full screen to 640 x 480 is pretty easy for almost any modern machine. Capturing the same full screen to 1440x900 is about four times the amount of work.