Summary (aka: can't be bothered reading below, give me some suggestions) 

1) Try using Apple Intermediate for video. You must have iLife installed for this to appear. 

2) Try reducing the size of the movie.

What matters? 
Capture "smoothness" can be tricky and depends on a number of factors:

1) The size is the movie. That is, what size is it (width x height) that you're creating? 
2) The compression method in use (H264, Apple Intermediate)
3) The frames per second of the capture
4) Speed of machine

The first two questions are the most important: What is the output size and compression? Let me provide an example.

If I have a MacBook Pro, and try to capture full screen (1440x900) using H264 at 30fps, the MacBook Pro isn't going to be able to do it. H264 is simply too resource (CPU) intensive. As a result, you'll get frame skip and the movie may (will!) look jerky.

The solution is record using Apple Intermediate (AIC) or Apple Animation (AA) and then convert the movie to H264 later using either Stomp or QT Pro. Both of these use far less in the way of CPU resources, and allow for faster (and thus smoother) captures.

An alternative is to lower the capture size. Often you'll need to do both if you've a slower machine (such as a G4 or early G5).  In the case above I might lower the movie size to 1024x768 (or even 800x600) and also use Apple Intermediate.   H264's CPU hungry nature and resulting frame skip (jerkiness) is particularly obvious on G4 and G5 machines.