Overclockers Australia Forums

OCAU News - Wiki - QuickLinks - Pix - Sponsors  

Go Back   Overclockers Australia Forums > Specific Hardware Topics > Storage & Backup

Notices


Sign up for a free OCAU account and this ad will go away!
Search our forums with Google:
Reply
 
Thread Tools
Old 14th December 2016, 6:50 PM   #16
ae00711
Member
 
ae00711's Avatar
 
Join Date: Apr 2013
Posts: 963
Default

Quote:
Originally Posted by elvis View Post
Solomon reed error correction is all CPU calc. Data blocks are quite small. This is similar to how RAID5/6 works, which is often done on controllers with very small amounts of RAM. Likewise QR codes use it, which need to be processed on embedded systems with very low RAM.

If you're looking to speed it up, GPU might be an option. Just reading the Multipar website, it says it can do it on GPU, so that might be an option.

But if you're doing it multi threaded on lots of data, then yeah, it's going to consume all of your CPU resources to do this as quickly as it can. Use your standard process priority management tools to lower the priority of this to "idle" if you plan to create PAR files for terabytes of data while you use your system for other things.
shortly after making that post I fired up AIDA64 to check CPU temps........BSOD. First BSOD I've had in aaaaaaages. Not happy.

Any idea which GPU it needs? (NV/AMD?)
__________________
I LOVE CHING LIU
SMOKING IS UN-AUSTRALIAN
I prefer email to PM!

ae00711 is offline   Reply With Quote

Join OCAU to remove this ad!
Old 14th December 2016, 7:22 PM   #17
ae00711
Member
 
ae00711's Avatar
 
Join Date: Apr 2013
Posts: 963
Default

Quote:
Originally Posted by ae00711 View Post
Any idea which GPU it needs? (NV/AMD?)

to answer my own question:

AMD is definitely used - a) I can enable GPU use in app b) running the app + GPU-Z shows GPU load @ 99%
__________________
I LOVE CHING LIU
SMOKING IS UN-AUSTRALIAN
I prefer email to PM!

ae00711 is offline   Reply With Quote
Old 14th December 2016, 8:32 PM   #18
elvis
Old school old fool
 
elvis's Avatar
 
Join Date: Jun 2001
Location: Brisbane
Posts: 27,897
Default

Quote:
Originally Posted by ae00711 View Post
GPU-Z shows GPU load @ 99%
I'm not sure why you're reacting the way you are to something that is calculation bound using up all of your resources.

This would be the same if you were encoding video, 3D rendering, or calculating some large/infinite mathematical series (Pi calcs, etc). All of those would use 100% of your resources for hours on end too.

Fairly standard stuff.
elvis is online now   Reply With Quote
Old 14th December 2016, 9:41 PM   #19
demiurge3141 Thread Starter
Member
 
Join Date: Aug 2005
Location: Melbourne 3073
Posts: 762
Default

Quote:
Originally Posted by ae00711 View Post
to answer my own question:

AMD is definitely used - a) I can enable GPU use in app b) running the app + GPU-Z shows GPU load @ 99%
That's what you want, right?
demiurge3141 is online now   Reply With Quote
Old 14th December 2016, 9:47 PM   #20
ae00711
Member
 
ae00711's Avatar
 
Join Date: Apr 2013
Posts: 963
Default

Quote:
Originally Posted by elvis View Post
I'm not sure why you're reacting the way you are to something that is calculation bound using up all of your resources.

This would be the same if you were encoding video, 3D rendering, or calculating some large/infinite mathematical series (Pi calcs, etc). All of those would use 100% of your resources for hours on end too.

Fairly standard stuff.
didn't know anything about the app!

Quote:
Originally Posted by demiurge3141 View Post
That's what you want, right?
ya!
__________________
I LOVE CHING LIU
SMOKING IS UN-AUSTRALIAN
I prefer email to PM!

ae00711 is offline   Reply With Quote
Old 15th December 2016, 6:52 AM   #21
elvis
Old school old fool
 
elvis's Avatar
 
Join Date: Jun 2001
Location: Brisbane
Posts: 27,897
Default

Quote:
Originally Posted by ae00711 View Post
didn't know anything about the app!
Have you never done anything intensive on your machine before? Or did you buy a beasty computer just to use 20% of it?

Sometimes I do wonder what the heck OCAU members do with all of their collective computing power.
elvis is online now   Reply With Quote
Old 15th December 2016, 10:05 AM   #22
cvidler
Member
 
cvidler's Avatar
 
Join Date: Jun 2001
Location: Canberra
Posts: 10,286
Default

Quote:
Originally Posted by elvis View Post
Have you never done anything intensive on your machine before? Or did you buy a beasty computer just to use 20% of it?

Sometimes I do wonder what the heck OCAU members do with all of their collective computing power.
benchmarks for ewang measuring.

or games, where you can't see the utilisation graph while you're playing.

It is a pretty fair statement though, CPU horspeower is so great these days, regular tasks don't stress anything modern at all. (I say this having had to use a PC at a customer all week that's stressed just running win10 and opening an email or webapge - it's got asset stickers going back to at least 2008).
__________________
We might eviscerate your arguments, but we won't hurt you. Honest! - Lucifers Mentor
⠠⠵
[#]
cvidler is online now   Reply With Quote
Old 15th December 2016, 11:22 AM   #23
elvis
Old school old fool
 
elvis's Avatar
 
Join Date: Jun 2001
Location: Brisbane
Posts: 27,897
Default

Reminds me somewhat of when SuperCache came to Windows, and folks started freaking out about "all their free RAM vanishing". As if having hardware lying around idle was a good thing, instead of putting it to use doing something more complex than opening a web browser.

We've got more computing power than ever before in human history, yet we seem to be doing fewer interesting/useful things with it than ever.

</misanthrope>
elvis is online now   Reply With Quote
Old 15th December 2016, 2:16 PM   #24
m0n4g3
Member
 
m0n4g3's Avatar
 
Join Date: Aug 2009
Location: Perth, WA
Posts: 3,164
Default

Quote:
Originally Posted by elvis View Post
Have you never done anything intensive on your machine before? Or did you buy a beasty computer just to use 20% of it?

Sometimes I do wonder what the heck OCAU members do with all of their collective computing power.
Some of us buy the power for it to be used. 4k Gaming punishes any rig! :P
__________________
i7-5930k @ 4.0ghz with EK Supreme-HF | Asus X99 Sabertooth TUF | 2x 980TI SLI with EK CU Blocks | 16gb Corsair Vengeance DDR4-3000 | 256gb Samsung 840pro | Xigmatek Elysium | Danger Den Monsoon D5 Dual Res bay, Koolance PMP-450 Pump | 2x XSPC RX360 Rads with 3xScythe Slipstream 1900rpm each | 3/8" Danger Den Black UV tubing | Red Monsoon Compression fittings
m0n4g3 is online now   Reply With Quote
Old 15th December 2016, 2:53 PM   #25
ae00711
Member
 
ae00711's Avatar
 
Join Date: Apr 2013
Posts: 963
Default

Quote:
Originally Posted by elvis View Post
Have you never done anything intensive on your machine before? Or did you buy a beasty computer just to use 20% of it?

Sometimes I do wonder what the heck OCAU members do with all of their collective computing power.
ofc I have! video encoding/transcoding
__________________
I LOVE CHING LIU
SMOKING IS UN-AUSTRALIAN
I prefer email to PM!

ae00711 is offline   Reply With Quote
Old 15th December 2016, 2:57 PM   #26
elvis
Old school old fool
 
elvis's Avatar
 
Join Date: Jun 2001
Location: Brisbane
Posts: 27,897
Default

Quote:
Originally Posted by m0n4g3 View Post
Some of us buy the power for it to be used. 4k Gaming punishes any rig! :P
Yes and no. Realtime 3D is an interesting problem, but try doing workloads that take CPUs and GPUs months at full load to generate a few seconds (or even milliseconds) worth of visual data, and you're in a whole different world.

Parity calc on terabytes of data is in the same realm as that sort of problem. Saying "it brings my rig to its knees" is somewhat missing the point of what's going on if you're used to realtime data that frequently sits well shy of 100% load to generate an adequate amount of information.

For folks who only know PCs to be gaming rigs, consider how your computers would react if you were running 24x7 gaming benchmarks in parallel on them for months on end. That's the world of computing I live in, comparatively speaking.

There are much bigger, much more complex, and much more demanding tasks out there for computers to work on than just video games (and I'm not trying to diminish video games either in their fun nor commercial value, as both are high). And as banal as Reed Solomon error correction or Erasure Coding tasks sound, they're a real world problem that are indeed CPU bound, particularly when you're not doing these tasks inline per I/O operation.

Which, if you've been following this thread and others like it, is why we storage folks constantly recommend file systems that actually do these tasks inline, like ZFS, BtrFS and ReFS. And also, that's invariably why people jump on them thinking they're "free", and wonder why lots of disk reads and writes chew up their CPU more than before, which annoys folks who just want to load games faster off disk, and not worry about long term critical data storage and data integrity.

Quote:
Originally Posted by ae00711 View Post
ofc I have! video encoding/transcoding
We recently did a project for the ABC at work where we had to transcode 30 years worth of TV shows and footage the ABC owned. Most of it had to be brought in from tape (both digital and analogue), captured, and transcoded to 3 different qualities for web and broadcast.

If you think transcoding a 1.5 hour movie is demanding on your hardware, try 30 years worth of the stuff.

Last edited by elvis; 15th December 2016 at 2:59 PM.
elvis is online now   Reply With Quote
Old 15th December 2016, 3:13 PM   #27
ae00711
Member
 
ae00711's Avatar
 
Join Date: Apr 2013
Posts: 963
Default

Quote:
Originally Posted by elvis View Post
We recently did a project for the ABC at work where we had to transcode 30 years worth of TV shows and footage the ABC owned. Most of it had to be brought in from tape (both digital and analogue), captured, and transcoded to 3 different qualities for web and broadcast.

If you think transcoding a 1.5 hour movie is demanding on your hardware, try 30 years worth of the stuff.
lulz, delicious!
what kinda hardware did you throw at that project?
__________________
I LOVE CHING LIU
SMOKING IS UN-AUSTRALIAN
I prefer email to PM!

ae00711 is offline   Reply With Quote
Old 15th December 2016, 4:09 PM   #28
m0n4g3
Member
 
m0n4g3's Avatar
 
Join Date: Aug 2009
Location: Perth, WA
Posts: 3,164
Default

Quote:
Originally Posted by elvis View Post
Yes and no. Realtime 3D is an interesting problem, but try doing workloads that take CPUs and GPUs months at full load to generate a few seconds (or even milliseconds) worth of visual data, and you're in a whole different world.

Parity calc on terabytes of data is in the same realm as that sort of problem. Saying "it brings my rig to its knees" is somewhat missing the point of what's going on if you're used to realtime data that frequently sits well shy of 100% load to generate an adequate amount of information.

For folks who only know PCs to be gaming rigs, consider how your computers would react if you were running 24x7 gaming benchmarks in parallel on them for months on end. That's the world of computing I live in, comparatively speaking.

There are much bigger, much more complex, and much more demanding tasks out there for computers to work on than just video games (and I'm not trying to diminish video games either in their fun nor commercial value, as both are high). And as banal as Reed Solomon error correction or Erasure Coding tasks sound, they're a real world problem that are indeed CPU bound, particularly when you're not doing these tasks inline per I/O operation.

Which, if you've been following this thread and others like it, is why we storage folks constantly recommend file systems that actually do these tasks inline, like ZFS, BtrFS and ReFS. And also, that's invariably why people jump on them thinking they're "free", and wonder why lots of disk reads and writes chew up their CPU more than before, which annoys folks who just want to load games faster off disk, and not worry about long term critical data storage and data integrity.
Don't get me wrong, I understand that it'd be completely different workloads, and completely appreciate the severity of huge workloads.

For me, it's more around throwing the correct solution at the problem. If you want faster game loads off disk, you wouldn't necessarily need data integrity, thus you are better off putting that on say NTFS/ext*/xfs/Fat32 really, and coupled with an SSD. The same with Ram, rather than throwing 64gb of ram @ the game, bring it down to 16gb but throw faster specced ram. Totally appreciate that kind of thinking.

I was more commenting that some of us do buy "big hardware" for a purpose. I see 90% load on cpu + 100% load on gpu's @ 4k resolutions... Sure it's not the same workload as banal tasks such as high intensity compute workloads, or error checking/integrity checks on file systems, but the hardware is specced out for the workload
__________________
i7-5930k @ 4.0ghz with EK Supreme-HF | Asus X99 Sabertooth TUF | 2x 980TI SLI with EK CU Blocks | 16gb Corsair Vengeance DDR4-3000 | 256gb Samsung 840pro | Xigmatek Elysium | Danger Den Monsoon D5 Dual Res bay, Koolance PMP-450 Pump | 2x XSPC RX360 Rads with 3xScythe Slipstream 1900rpm each | 3/8" Danger Den Black UV tubing | Red Monsoon Compression fittings
m0n4g3 is online now   Reply With Quote
Old 15th December 2016, 6:08 PM   #29
ae00711
Member
 
ae00711's Avatar
 
Join Date: Apr 2013
Posts: 963
Default

I'll be testing if multipar runs on nvidia shortly..

I did mostly want to run this app on my file server, so I installed a GTX460(1GB) to offload from CPUs (2x 6C/12T) - CPUs are a little low on clock-speed unfortunately (2.66).
__________________
I LOVE CHING LIU
SMOKING IS UN-AUSTRALIAN
I prefer email to PM!

ae00711 is offline   Reply With Quote
Old 15th December 2016, 6:20 PM   #30
elvis
Old school old fool
 
elvis's Avatar
 
Join Date: Jun 2001
Location: Brisbane
Posts: 27,897
Default

Quote:
Originally Posted by ae00711 View Post
lulz, delicious!
what kinda hardware did you throw at that project?
* 40x 56 thread, 128GB render nodes
* 300GB of GlusterFS clustered storage (temporary while transcoding - final deliverable was on LTO6 tape)
* Linux and ffmpeg
elvis is online now   Reply With Quote
Reply

Bookmarks

Sign up for a free OCAU account and this ad will go away!

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +10. The time now is 11:24 AM.


Powered by vBulletin® Version 3.8.9
Copyright ©2000 - 2017, vBulletin Solutions, Inc.
OCAU is not responsible for the content of individual messages posted by others.
Other content copyright Overclockers Australia.
OCAU is hosted by Micron21!