I bought an Nvidia GT 240 just so I could use CoreAVC with CUDA to stream to my PS3. It went pretty smoothly, except for one trick, and that is... Windows 7 preferred filter tweaker! As you might guess, this is only really tricky with Windows 7, due to the way the filter priorities are set. If you have Vista or XP, you shouldn't have any trouble with that.
So anyway, what do you need to get hardware accelerated streaming to your PS3?
- PS3 Media Server (I linked to the beta build)
- AviSynth - Make sure to get 2.57, if you get 2.58 you'll get chipmunk audio!
- Combined Community Codec Pack
- Win7 Preferred Filter Tweaker
- Install everything. When installing CoreAVC, don't install the Haali Media Splitter.
- Go into the CoreAVC settings - Programs -> CoreCodec -> CoreAVC -> Configure CoreAVC. Make sure CUDA is checked and the tray icon is also enabled.
- Go into the ffdshow audio decoder settings - Programs -> Combined Community Codec Pack -> Filters -> ffdshow audio decoder configuration. Ensure that Mixer is checked, and in the mixer panel, make sure you have the speaker configuration set to 3/0/2 - 5 channels, and LFE is checked.
- Run the Win7 preferred filter tweaker. For H.264, make sure CoreAVC is checked. Hit Apply and then Exit.
- Run PS3 media server. Configure your share points, then head to the transcoding tab. Make sure Avisynth/Mencoder is enabled and has highest priority. Under "Common transcode settings", make sure everything on the page is unchecked (particularly the ones that say "avisynth not supported".
- Check to make sure it works! When you play a video, you should see the CoreCodec icon in the system tray colored green. If it's blue, it's not using CUDA.
After doing all that, you should be able to stream just about anything to your PS3! On my computer with a 2.7ghz Pentium dual-core (E2160), I can stream and reencode 1080p video with about 70% CPU usage. Before my new nvidia card, my computer could only handle 720p files.
Some notes: when I used avisynth/ffmpeg as noted in the PS3 media server readme, I had some occasional audio stuttering that went away using avisynth/mencoder.
If anything is unclear, leave a note in the comments and I'll try to address it.