Problem solved! If anyone read this thread and was scratching their head over it, it turns out my problem wasn't with vertical detail frequency or squeeze mode at all. I'd just accidentally ticked the 16:9 box instead of "native dimensions" when exporting my QTref, and for some reason this had Procoder convinced that my footage was only 640 pixels across; the problem was entirely with my MPEG-2...
Whew.
Whew.