What happens when you run it outside DTb (i.e., from the command-line)? Same result? If so, post the mencoder output.
Is anything else running at the time?
Just to clarify - I meant run mencoder.exe, not run DTb, from the command-line. See here:
http://babgvant.com/forums/thread/3815.aspx