RIPBOT264 Distributed Encoding Yewww

Discussion in 'Overclocking & Hardware' started by Deus_Ex_Machina, Feb 16, 2020.

  1. Deus_Ex_Machina

    Deus_Ex_Machina Member

    Joined:
    Jun 6, 2007
    Messages:
    438
    Location:
    Geelong VIC
    So this is the first time I have actually had 2X decent CPU's on the same network and thought I'd give distributed X264 encoding a go has anyone tried? Encoding on the Ryzen 2600 and have the old 2600K helping out. Super simple to set up and yikes did the encode times just come down. Sweet.
     
  2. Agg

    Agg Lord of the Pings

    Joined:
    Jun 16, 2001
    Messages:
    32,001
    Location:
    A Reported Post Near You
    Sounds interesting.. if I can get my various home PCs to help convert files for the media player that'll reduce an annoying bottleneck. I'll get googling, but did you follow any particular guide?
     
  3. OP
    OP
    Deus_Ex_Machina

    Deus_Ex_Machina Member

    Joined:
    Jun 6, 2007
    Messages:
    438
    Location:
    Geelong VIC
    You need to download Ripbot264 and unzip the contents into a folder on each machine (when you run the first time it may prompt for another freeware tool or two to be installed). The slave machines need to run the "encoding server" .exe in the folder and the main machine runs the full client/GUI/Program which works much like Handbrake if you have used that. Once the encode starts you add the IP addresses of the other machines on your network.

    This is a really quick video guide once you actually have the software on each machine.



    I think you would find the benefit greater the more machines you have (obviously), there is some down time in encoding as packets get sent back and forth and chunks encoded on each machine which offsets some of the gross power advantage you have. I have actually ended up just running both machines encoding stuff separately as I'm doing some bulk work but if you had 3-4 machines on a network it should give your single machine a bump depending on their CPU's.
     
  4. ae00711

    ae00711 Member

    Joined:
    Apr 9, 2013
    Messages:
    1,576
    don't transcode / encode as much as I used to... but still want.... for Linux please
     
  5. Aetherone

    Aetherone Member

    Joined:
    Jan 15, 2002
    Messages:
    8,723
    Location:
    Adelaide, SA
    Now that's interesting. Especially since it can farm out x265 encodes too. My power bill is going to kill.
     
  6. Butcher9_9

    Butcher9_9 Member

    Joined:
    Aug 5, 2006
    Messages:
    2,192
    Location:
    Perth , East Vic Park
    Does it handle multiple core well?

    I have to run 2X Handbrake instance to get 90-100% utilization for 720/1080p x265 encoding.
     
  7. elvis

    elvis Old school old fool

    Joined:
    Jun 27, 2001
    Messages:
    40,423
    Location:
    Brisbane
    ffmpeg (the library behind Handbrake) scaling works up to about 16 cores well, then tapers off after that. There's not much you can do there, as there's certain parts of the quantisation pipeline that just don't scale well past a certain level due to how the maths works.

    Where I worked previously they had a render farm - somewhere in the order of 100+ machines running 54 - 72 cores (dual socket Xeons) per machine, with 256-512GB RAM per machine. We found running transcode jobs on the farm most efficient when we capped ffmpeg to 16 threads and then ran multiple concurrent jobs.

    Not useful if you don't have a lot of transcodes to run in parallel of course (we had *thousands* of transcodes to do in parallel for big customers like the ABC and others). But if you're clever, maybe chopping longer videos into chunks, transcoding in parallel and then gluing them together (although that in itself can be a bit tricky).

    With all of that said, the cost effectiveness today of CPU transcoding is pretty low. GPUs do a much better job for the dollar cost if your requirements are getting single videos out the door quickly.
     
  8. Aetherone

    Aetherone Member

    Joined:
    Jan 15, 2002
    Messages:
    8,723
    Location:
    Adelaide, SA
    Has anyone made a (not million dollar) GPU accellerated encoder that delivers as good a result as CPU? Last time I looked at CPU vs GPU with HB and Avidemux the GPU quality to compression ratio was garbage. Fast, yes. Good, no.
     
  9. elvis

    elvis Old school old fool

    Joined:
    Jun 27, 2001
    Messages:
    40,423
    Location:
    Brisbane
    What settings are you choosing?

    ffmpeg offers both CPU and GPU encoding with a variety of settings and levels. If you're always choosing realtime or low latency modes designed for streaming, they're going to look like shit. Try the higher quality settings instead.

    I'm not a huge fan of Handbrake. It obscures too many settings for my liking. ffmpeg is closer to the source, and easier to automate for what I need to do.
     
  10. OP
    OP
    Deus_Ex_Machina

    Deus_Ex_Machina Member

    Joined:
    Jun 6, 2007
    Messages:
    438
    Location:
    Geelong VIC
    Sounds like Elvis is across it more than me. I'm only encoding X264, my Ryzen 2600 (6 core) is going flat knacker 100% when transcoding as s my old 2600K (4 core) secondary machine.
     
  11. Aetherone

    Aetherone Member

    Joined:
    Jan 15, 2002
    Messages:
    8,723
    Location:
    Adelaide, SA
    Typical 4k encode is 10bit x265, constant framerate, 2pass "slow" encodes with an average bitrate of 10-15mbps.

    There's my tradeoff - using CPU, I can squeeze the video from 60-70-80Mbps UHD blueray to 10Mbps x265 and it still looks pretty damn good, even with high motion scenes. NVENC doesn't do anywhere near as well unless the bitrates are much higher. Blazing fast, yes. Good quality, no.
    So far it feels like swings and roundabouts - do I spend money on the electricity to compress vids, or on spinning rust to store it?

    I'm struggling to get my head around all the possible options and settings. Gonna have to have another go when I've got the time to sit down and digest.
     
  12. elvis

    elvis Old school old fool

    Joined:
    Jun 27, 2001
    Messages:
    40,423
    Location:
    Brisbane
    Worth checking out the other settings too. CRF has a lot to do with it. And again, make sure it's not choosing a low latency profile.
     
  13. Butcher9_9

    Butcher9_9 Member

    Joined:
    Aug 5, 2006
    Messages:
    2,192
    Location:
    Perth , East Vic Park
    I find it depends on Resolution , 720P is only get 40% on my 12 core, 1080p is about 50-60% and 4K is 100%.

    What is your preferred transcoder? I just use handbrake as that is what I have always used and it works.

    I don't like the auto Cropping as its broken half the time which pretty much precludes automation (Always wants to crop a couple of pixles here and there and not just black bars).
     
  14. elvis

    elvis Old school old fool

    Joined:
    Jun 27, 2001
    Messages:
    40,423
    Location:
    Brisbane
    Handbrake is not a transcoder. It's a front end to the ffmpeg transcoding suite.
     
  15. Aetherone

    Aetherone Member

    Joined:
    Jan 15, 2002
    Messages:
    8,723
    Location:
    Adelaide, SA
    Yeah, it's been a good 9 months and many many nightlies since I last checked it out. So with nothing better to do while WIMs capture I'm going around the mulberry bush again tonight :D
    CRF and dual-pass encoding appear to be mutually exclusive. In face, NVENC doesn't appear to support it either.

    So far, NVENC wins on speed - for my 6 minute test clip, SLOW churns through in 1:41 (m:ss) and "HIGH QUALITY" takes an eternity at 3:17. Shame both of them look terrible.

    A proper dual-pass CPU "SLOW" encode is more like 88:18 at the same bitrate, delivering a fractionally smaller file but I'm struggling to pick the difference between the 5mbit encode and the 30.5mbit source.
    Dual-pass CPU "FAST" is much quicker @ 25:30 but is discernibly lower quality. MEDIUM actually looks pretty good to my eyes, plus it only took 40:44.

    Now re-running FAST & MED @ 10mbit just to see how they go.
     
    elvis likes this.
  16. mikezilla2

    mikezilla2 Member

    Joined:
    Jan 24, 2009
    Messages:
    244
    this is relvent to me on some level

    one day.
     
  17. Aetherone

    Aetherone Member

    Joined:
    Jan 15, 2002
    Messages:
    8,723
    Location:
    Adelaide, SA
    More updates: at 10mbit, there's precious little difference between FAST and MED CPU renders. Oddly, the MED render was quickest @ 43:30 with the FAST clocking in 57:33. Might have to rerun these two to double check.
    As expected, file sizes pretty much doubled.
    NVENC redone at 10mbit (1:50) still looks obviously less good. Re-re-doing it at 20mbit (2:10) still doesn't look as good as CPU @ 5mbit. Doubling that yet again to 40mbit (3:07) still delivers an output that's worse than the 30mbit original but pretty close to the 5mbit SLOW CPU (like I can't tell them apart without checking the file name).

    Conclusion: crap render engines are crap at any reasonable bitrate. So NVENC is great if you're time poor and severely visually impaired.
    If you want actual quality in your render, the slow way is the only way.
     
  18. Butcher9_9

    Butcher9_9 Member

    Joined:
    Aug 5, 2006
    Messages:
    2,192
    Location:
    Perth , East Vic Park
    If only they were good, soo fast.

    Semantics , question still applies
     
  19. Aetherone

    Aetherone Member

    Joined:
    Jan 15, 2002
    Messages:
    8,723
    Location:
    Adelaide, SA
    It's a pity nobody has gone "why don't we take 10 minutes instead of three and do the job properly?" :sick:
    Or they just wanna sell more 20TB hard drives.
     
  20. Butcher9_9

    Butcher9_9 Member

    Joined:
    Aug 5, 2006
    Messages:
    2,192
    Location:
    Perth , East Vic Park
    Yeah, would still be many times faster.
     

Share This Page

Advertisement: