Linux (Ubuntu Gutsy) - Ripping DVD to h264 (using mencoder and x264)

For a while I was ripping all my content to avi files using the xvid codec. I liked the codec well enough, and the encode times were really good. The downside was the end result was not great. If I had a DVD where I was really concerned with quality I just kept it in it's original form. If I was just trying to archive a season of something I compressed it. Some of my favorite shows just don't require any kind of quality (Penn & Tellers Bullshit, Law & Order, etc...)

A friend told me how great h264 was, so I thought I'd give it a try. For the most part I was really pleased with it. If I just want a low quality copy, but still enjoyable, I go for about a 1k bitrate. If I want the best quality/size ratio I go for about 1.8k. If it's something I really want quality on I either leave it as a DVD or give it a 2.5k+ bitrate. The real downside is encode times. Holy spaghetti monster does h264 take time to encode.

After spending weeks browsing blogs and man pages I am prepared to pass on all that I have learned. The process itself is not very well documented, so if you can give me any information or corrections please feel free to do so. This is what I know and how I do it using the knowledge I have so far.

During this process I added several libraries to Ubuntu from the repositories. I don't remember them all, so if we come across a command you don't have search the ubuntu pages for it. I didn't do anything special as far as that goes. If I am using it here, it is in the Ubuntu repositories.

I also want to write my own encoder program. So everything I am doing has an eye on me being able to do this process programmatic in the future. Next time I'll talk about my python program I'm writing to automate all of this. So be warned everything is command-line.

Step 1: Get DVD Info

So you pop your DVD in the drive and now you need to figure out what is on it. The best tool I've found for this is "lsdvd".

$lsdvd

Disc Title: THE_DISC_TITLE
Title: 01, Length: 00:29:51.153 Chapters: 05, Cells: 05, Audio streams: 03, Subpictures: 00

Title: 02, Length: 00:29:56.043 Chapters: 05, Cells: 05, Audio streams: 03, Subpictures: 00

Title: 03, Length: 00:28:04.110 Chapters: 05, Cells: 05, Audio streams: 03, Subpictures: 00

Title: 04, Length: 00:29:51.043 Chapters: 05, Cells: 05, Audio streams: 03, Subpictures: 00

Longest track: 02

So this shows us the tracks on the dvd. If you include the -a switch ("lsdvd -a") it will also show you the audio for each track. Now we know what is on the DVD, and we decide what to rip.

Step 2: Rip Your Tracks

Let's say we decided we want Track 1. Now we need to rip it to the hd. Again it's very easy:

$mplayer dvd://1 -v -dumpstream -dumpfile myfile.vob

You can replace the "1" with whatever track you want. So if you want track 2 "mplayer dvd://2...". Replace "myfile.vob" with whatever you want to call the file.

Step 3: Cropping

Most DVD's are not at the actual 720x480 resolution of a DVD. We can save file size by cutting out the extra black borders. Yet again mplayer can help us. It has a tool for detecting the best crop rate.

$mplayer -vo null -ao null -vf cropdetect myfile.vob

[CROP] Crop area: X: 0..719 Y: 0..479 (-vf crop=720:480:0:0).0 0

The output will spit out a lot of those [CROP] lines. One frame for each frame as the vob file plays. I've found it's best to skip around the movie until you find a spot where it settles down. This tool does weird things at the beginning (as most movies and shows start black). So let it run a bit and once it settles down to the same "crop=" line, then you have mplayers recommended crop settings. In the case above it is "720:480:0:0". Remember this, we will need it later.

Step 4: FPS and DVD weirdness

For this part we need a quick understanding of what is going on in these DVD's. From what I gather basically nothing is produced in the 29.97 frames per second (fps) that a DVD plays at. There are 3 main ways that the content is transferred to DVD's. They are Progressive (Star Wars 3), Interlaced (Penn & Tellers Bullshit), and Telecine (History Channel: The Universe).

Progressive (Star Wars 3):

In movies like this they are normally shot at 23.976 fps. Since a DVD plays 29.97 fps, the extra frames have to come from somewhere. Progressive means that entire extra frames where inserted per second to make up the difference. When playing the file you shouldn't see any jagged lines appearing. Encoding is as simple as stripping these extra frames away, and the fps is 23.976.

Telecine (History Channel: The Universe)

In this case the original is also at 23.976 fps, but the missing frames were made differently. Just playing one of these files using mplayer should show jagged lines. I gather that what is going on here is that the frames that were produced to fill up the space are actually combinations of different frames around it, instead of full duplicate frames. A good way to tell if you have a telecine video is to detelecine it and see if that fixes the jagged lines:

$mplayer myfile.vob -ao null -vf pullup,softskip

The "pullup" filter understands what is going on with the duplicate frames and removes them. If this command shows you your video without the jagged lines, then you have telecine content. Basically the same as above. We just need to drop these extra frames, and the output is 23.976 fps.

Interlaced (Penn & Tellers Bullshit)

In this case the footage was originally shot in 60 fps. You can't just halve the fps for a video like this. The end result comes out with choppy motion as the whole motion was recorded in 60 fps not 30 fps. So what they do is make a series of 30 frames where each is 1/2 of the original 2 (ie this frame never existed in the original video, it is a combination of the 2 frames it is replacing). You basically have 2 options:

1) re-create the extra frames returning it to 60 fps

2) deinterlace

Both essentially do the same thing. You cannot actually get the content back to it's original glory. Half the data is simply gone. In my tests returning things to 60 fps made the picture look no better while simultaneously adding to encode time, upping the decode requirements, and making the file bigger. None of that is selling it to me. We can just deinterlace the content and leave it at 29.97 fps. That's about the best we can do. You can check to make sure this fixes the picture:

$mplayer myfile.vob -ao null -vf yadif

If the lines are gone, then you have an interlaced picture and an output of 29.97 fps.

**yadif can fix telecine material. The output isn't as good though. So be sure to check for telecine first, don't just assume that since yadif works it must be interlaced.

By knowing which of the 3 above you have, we can now move on to encoding.

**I believe I have found a way to programmatic determine which of the above you have. That will be for next time though, when we try to make a python program to automate this process.**

Step 5: Audio

Ultimately I want to write a program to do this so I had to settle on an audio spec. I want everything to be standardized. If I go shopping for any multimedia device I just have to worry about 1 video and audio codec. Since mp3's can't do 5.1 sound (at least to my knowledge) I decided to go with aac. The quality is good, encoding is fast, file size is good, and it does stereo and/or 5.1 surround sound. I just think all that makes this the best for audio of this nature. We can even rip the audio directly from the vob file in 1 step.

When using "lsdvd -a" you will see the audio channels for each track listed. The "Stream id:" value is the one you want. It's reported as something like 0x80 or 0x81, etc. I don't know what or why (I haven't bothered to look that up yet) but those are the audio channels. We need them in integer form so (0x80=128, 0x81=129, ...).

$mkfifo audiodump.pcm

2 Channel Stereo like this:

$faac -P -R 48000 -q 100 -C 2 -X --mpeg-vers 4 -o myfile.aac audiodump.pcm & mplayer myfile.vob -aid 128 -vc dummy -vo null -ao pcm:nowaveheader:fast -channels 2

6 Channel 5.1 like this:

$faac -q 100 -I 1,6 -P -R 48000 -C 6 -X audiodump.pcm -o myfile.mp4 & mplayer myfile.vob -aid 128 -vc dummy -vo null -ao pcm:nowaveheader:fast -channels 6

As you can see they are very similar. In mplayer "-aid 128" will select the audio track 128. Change as needed. The difference is the "-C" flag for "faac" and the "-channels" flag for mplayer. "-C 2" and "-channels 2" tells them both to do 2 channels. For whatever reason the 6 channels will not map correctly by themselves, so we have to remap them (otherwise the channels are messed up; ie center will come out some other speaker, and left and right will be jumbled). That's what "-I 1,6" does in "faac". With "-I" you can specify the channel that has the center speaker and the one for the sub-woofer. In my tests those are channels 1 for center, and 6 for sub. So "-I 1,6" fixes the audio.

Step 6: Encoding

Now we're ready to encode, here are the commands I use. If you want more details, then go look them up.

Progressive:

mencoder myfile.vob -o /dev/null -ofps 24000/1001 -vf crop=720:480:0:0,scale,harddup -ovc x264 -x264encopts bitrate=1800:frameref=6:analyse=all:me=umh:subme=7:trellis=2:bframes=1:subq=7:brdo:mixed_refs:weight_b:bime:no_fast_pskip:direct_pred=auto:mixed_refs:nr=200:threads=auto:turbo=2:pass=1 -oac copy -of rawvideo

mencoder myfile.vob -o myfile.264 -ofps 24000/1001 -vf crop=720:480:0:0,scale,harddup -ovc x264 -x264encopts bitrate=1800:frameref=6:analyse=all:me=umh:subme=7:trellis=2:bframes=1:subq=7:brdo:mixed_refs:weight_b:bime:no_fast_pskip:direct_pred=auto:mixed_refs:nr=200:threads=auto:pass=2 -oac copy -of rawvideo

Telecine:

mencoder myfile.vob -o /dev/null -ofps 24000/1001 -vf pullup,crop=720:480:0:0,softskip,harddup -ovc x264 -x264encopts bitrate=1800:frameref=6:analyse=all:me=umh:subme=7:trellis=2:bframes=1:subq=7:brdo:mixed_refs:weight_b:bime:no_fast_pskip:direct_pred=auto:mixed_refs:nr=200:threads=auto:turbo=2:pass=1 -oac copy -of rawvideo

mencoder myfile.vob -o myfile.264 -ofps 24000/1001 -vf pullup,crop=720:480:0:0,softskip,harddup -ovc x264 -x264encopts bitrate=1800:frameref=6:analyse=all:me=umh:subme=7:trellis=2:bframes=1:subq=7:brdo:mixed_refs:weight_b:bime:no_fast_pskip:direct_pred=auto:mixed_refs:nr=200:threads=auto:pass=2 -oac copy -of rawvideo

Interlaced:

mencoder myfile.vob -o /dev/null -ofps 30000/1001 -vf crop=720:480:0:0,yadif=3,framestep=2,harddup -ovc x264 -x264encopts bitrate=1800:frameref=6:analyse=all:me=umh:subme=7:trellis=2:bframes=1:subq=7:brdo:mixed_refs:weight_b:bime:no_fast_pskip:direct_pred=auto:mixed_refs:nr=200:threads=auto:turbo=2:pass=1 -oac copy -of rawvideo

mencoder myfile.vob -o myfile.264 -ofps 30000/1001 -vf crop=720:480:0:0,yadif=3,mcdeint,framestep=2,harddup -ovc x264 -x264encopts bitrate=1800:frameref=6:analyse=all:me=umh:subme=7:trellis=2:bframes=1:subq=7:brdo:mixed_refs:weight_b:bime:no_fast_pskip:direct_pred=auto:mixed_refs:nr=200:threads=auto:pass=2 -oac copy -of rawvideo

Step 7: Wrapping up

Once this is all done all we have left is to combine the 2 files together into the new mp4 file. I do this with "MP4Box". All you need is the name of the video file, audio file, and the fps to use.

$MP4Box -add myfile.264 -add myfile.aac -fps 23.976 myfile.mp4

Change "myfile.aac" to "myfile.mp4" if you made 6 channel audio. Also change "-fps 23.976" to "-fps 29.97" if you deinterlaced. The result should be a reasonably good h264 compression of a dvd.

Now to make a python program to automate this whole mess.