Overclockers Australia Forums

OCAU News - Wiki - QuickLinks - Pix - Sponsors  

Go Back   Overclockers Australia Forums > Specific Hardware Topics > Storage & Backup

Notices


Sign up for a free OCAU account and this ad will go away!
Search our forums with Google:
Reply
 
Thread Tools
Old 15th December 2016, 7:27 PM   #31
ae00711
Member
 
ae00711's Avatar
 
Join Date: Apr 2013
Posts: 1,032
Default

Quote:
Originally Posted by elvis View Post
* 40x 56 thread, 128GB render nodes
* 300GB of GlusterFS clustered storage (temporary while transcoding - final deliverable was on LTO6 tape)
* Linux and ffmpeg
tsk tsk tsk... shoulda dedicated a single Core 2 Duo rig & charged by the hour! trollololol
__________________
I LOVE CHING LIU
SMOKING IS UN-AUSTRALIAN
I prefer email to PM!

ae00711 is offline   Reply With Quote

Join OCAU to remove this ad!
Old 15th December 2016, 7:32 PM   #32
ae00711
Member
 
ae00711's Avatar
 
Join Date: Apr 2013
Posts: 1,032
Default

ok, running the app now..

why is it that the NV card is only being loaded up to ~63% and the AMD 99%?

I did some googling around yesterday, only concrete GPGPU mention re; Multipar was 'OpenCL'... I believe AMD are/were ahead in that area(?) or at least the in comparison for the two (generation) of cards I have.

I have a spare 7950... tempted to chuck that in instead now..
__________________
I LOVE CHING LIU
SMOKING IS UN-AUSTRALIAN
I prefer email to PM!

ae00711 is offline   Reply With Quote
Old 15th December 2016, 11:12 PM   #33
ae00711
Member
 
ae00711's Avatar
 
Join Date: Apr 2013
Posts: 1,032
Default

http://forums.overclockers.com.au/sh...1#post17403481

__________________
I LOVE CHING LIU
SMOKING IS UN-AUSTRALIAN
I prefer email to PM!

ae00711 is offline   Reply With Quote
Old 10th January 2017, 9:58 AM   #34
ae00711
Member
 
ae00711's Avatar
 
Join Date: Apr 2013
Posts: 1,032
Default

could anyone teach me how to use this MultiPar?

I must be dialling in the settings wrong - the most I've done is move the 'redundancy' slider, tried @ 5% & 100%. At either 5% or 100%, the app makes 9 or 10 files.. @ 5% those files use up about as much space as the files that it is hashing - what's the point?? Might as well back up to another HDD etc! @ 100% the resulting hash(?) files take up.. something like 40GB or there-abouts - absolutely self-defeating, given the original folder is ~4GB.

Clearly I'm a nub!
__________________
I LOVE CHING LIU
SMOKING IS UN-AUSTRALIAN
I prefer email to PM!

ae00711 is offline   Reply With Quote
Old 10th January 2017, 10:13 AM   #35
elvis
Old school old fool
 
elvis's Avatar
 
Join Date: Jun 2001
Location: Brisbane
Posts: 29,361
Default

Tried to download MultiPar now to test it out, but both the .exe and .zip downloads reported that they had malware inside by Windows Defender.
__________________
Play old games with me!
elvis is online now   Reply With Quote
Old 10th January 2017, 10:21 AM   #36
ae00711
Member
 
ae00711's Avatar
 
Join Date: Apr 2013
Posts: 1,032
Default

Quote:
Originally Posted by elvis View Post
Tried to download MultiPar now to test it out, but both the .exe and .zip downloads reported that they had malware inside by Windows Defender.
his domain has been hijacked apparently

get the real deal from here:

http://www.vector.co.jp/soft/dl/winn.../se460801.html

I obtained the above link from here:

https://www.livebusinesschat.com/smf...p?topic=6106.0
__________________
I LOVE CHING LIU
SMOKING IS UN-AUSTRALIAN
I prefer email to PM!

ae00711 is offline   Reply With Quote
Old 10th January 2017, 10:40 AM   #37
elvis
Old school old fool
 
elvis's Avatar
 
Join Date: Jun 2001
Location: Brisbane
Posts: 29,361
Default

Quote:
Originally Posted by ae00711 View Post
his domain has been hijacked apparently

get the real deal from here:

http://www.vector.co.jp/soft/dl/winn.../se460801.html

I obtained the above link from here:

https://www.livebusinesschat.com/smf...p?topic=6106.0
Cheers. Not booted into Windows any more, so I'll have to test later.
__________________
Play old games with me!
elvis is online now   Reply With Quote
Old 10th January 2017, 12:39 PM   #38
Gibbon
grumpy old man
 
Gibbon's Avatar
 
Join Date: Jun 2001
Location: NSW
Posts: 5,535
Default

Seems fairly neat and easy to use.

Quote:
Originally Posted by ae00711 View Post
could anyone teach me how to use this MultiPar?

I must be dialling in the settings wrong - the most I've done is move the 'redundancy' slider, tried @ 5% & 100%. At either 5% or 100%, the app makes 9 or 10 files.. @ 5% those files use up about as much space as the files that it is hashing - what's the point?? Might as well back up to another HDD etc! @ 100% the resulting hash(?) files take up.. something like 40GB or there-abouts - absolutely self-defeating, given the original folder is ~4GB.

Clearly I'm a nub!
Definitely doing something wrong. Here is a test with the redundancy slider set at 10% and 100%. 10% should use 10% of the total size of the files. 100% should equal the total size of the files (you'd never do this).


Click to view full size!


Also, there is a slider in the options to limit CPU usage if you so wish.
With it set at LOW, and GPU acceleration turned off, my CPU usage peaked at less than 20%. If you don't like seeing your CPUs doing what you bought them for, this could be an option


Click to view full size!

Last edited by Gibbon; 10th January 2017 at 12:53 PM.
Gibbon is offline   Reply With Quote
Old 11th January 2017, 5:11 PM   #39
ae00711
Member
 
ae00711's Avatar
 
Join Date: Apr 2013
Posts: 1,032
Default

Quote:
Originally Posted by Gibbon View Post
Seems fairly neat and easy to use.



Definitely doing something wrong. Here is a test with the redundancy slider set at 10% and 100%. 10% should use 10% of the total size of the files. 100% should equal the total size of the files (you'd never do this).

image
Click to view full size!

Also, there is a slider in the options to limit CPU usage if you so wish.
With it set at LOW, and GPU acceleration turned off, my CPU usage peaked at less than 20%. If you don't like seeing your CPUs doing what you bought them for, this could be an option

image
Click to view full size!
you only changed the slider, no other settings?
__________________
I LOVE CHING LIU
SMOKING IS UN-AUSTRALIAN
I prefer email to PM!

ae00711 is offline   Reply With Quote
Old 11th January 2017, 5:31 PM   #40
Gibbon
grumpy old man
 
Gibbon's Avatar
 
Join Date: Jun 2001
Location: NSW
Posts: 5,535
Default

Yes, that's right.
Gibbon is offline   Reply With Quote
Old 17th January 2017, 10:24 AM   #41
Aetherone
Member
 
Aetherone's Avatar
 
Join Date: Jan 2002
Location: Adelaide, SA
Posts: 8,432
Default

Quote:
Originally Posted by elvis View Post
but try doing workloads that take CPUs and GPUs months at full load
b... bu... but it's Prime stable for 58 minutes and 14 seconds ... as long as I don't wiggle the mouse or walk in the room
Aetherone is online now   Reply With Quote
Old 23rd July 2017, 9:12 PM   #42
ae00711
Member
 
ae00711's Avatar
 
Join Date: Apr 2013
Posts: 1,032
Default

so let me get this straight...

what ZFS does with file verification, just automatically, in the background... would be the same as if one could get multipar and a checksum app (like this: http://corz.org/windows/software/checksum/) to run and do their things on auto also?
__________________
I LOVE CHING LIU
SMOKING IS UN-AUSTRALIAN
I prefer email to PM!

ae00711 is offline   Reply With Quote
Old 23rd July 2017, 9:58 PM   #43
rowan194
Member
 
Join Date: Jan 2009
Posts: 2,004
Default

Quote:
Originally Posted by ae00711 View Post
so let me get this straight...

what ZFS does with file verification, just automatically, in the background... would be the same as if one could get multipar and a checksum app (like this: http://corz.org/windows/software/checksum/) to run and do their things on auto also?
The basic idea with ZFS is that it's self healing.

It uses checksumming to validate each block that it reads. If it detects corruption, AND there's an alternate source of that data block (not necessarily the entire file) which validates correctly, it will overwrite the corrupt block with the valid data.

So as well as validating data on read, it also repairs corruption. If it's unable to repair, it will signal an error, rather than returning the corrupt data.

Alternate sources of blocks include drive mirroring, RAID-like striped parity arrays (RAIDZ1, RAIDZ2 etc), and storing multiple copies of the block within the same file system.

In your case, it sounds like you need an application that runs in the background, regularly "patrolling" to look for changes in stored hashes of files...
rowan194 is offline   Reply With Quote
Old 23rd July 2017, 10:34 PM   #44
HobartTas
Member
 
Join Date: Jun 2006
Posts: 529
Default

Quote:
Originally Posted by ae00711 View Post
so let me get this straight...

what ZFS does with file verification, just automatically, in the background... would be the same as if one could get multipar and a checksum app (like this: http://corz.org/windows/software/checksum/) to run and do their things on auto also?
Yes and no, it actually checksums every block from which files are composed from. The default setting is "copies=N" where N defaults to 1 and this means one copy of data and N+1(=2) for metadata and what happens if you get a bad block of data the checksum of the data won't match the expected checksum stored in the metadata and so it reports it as bad, also because there are two copies of metadata you are not likely to need anything like CHKDSK to fix filesystem problems like lost clusters, lost files etc, etc, however, you do occasionally have to do a scrub and if there are any dodgy blocks it will tell you which files are "damaged".

If you use some kind of redundancy which is the main reason to use ZFS like mirrors for high IOPS or Raid-Z (raid 5) or Raid-Z2 (raid 6) for maximising storage at the least cost (I use 10 drive Raid-Z2's) then because you have checksumming and Raid working together it can automatically repair and replace damaged blocks once it becomes aware of this on either a read or a scrub and all of this is automatic.

With all the other "checksum checker programs" they are just that basically programs that you have to manually run occasionally and then you have to delve in and delete bad copies and replace with good ones and all this requires time and effort that you don't do with ZFS. If you want to know more then have a read of this 44 page pdf.

The other reason I use ZFS is that I can snapshot the entire filesystem with one command that locks the data down and hence it can't be affected by ransomware because all that does is write fresh (scrambled) copies of the data while the original data is read only and can't be altered.

If your considering using it I strongly recommend you do so because even if you don't have redundancy you at least have automatic block checksumming and a far more robust filesystem. If your comparing NTFS to ZFS its like comparing VHS tape to DVD disk.

Cheers
HobartTas is online now   Reply With Quote
Old 23rd July 2017, 11:30 PM   #45
demiurge3141 Thread Starter
Member
 
Join Date: Aug 2005
Location: Melbourne 3073
Posts: 864
Default

Quote:
Originally Posted by ae00711 View Post
so let me get this straight...

what ZFS does with file verification, just automatically, in the background... would be the same as if one could get multipar and a checksum app (like this: http://corz.org/windows/software/checksum/) to run and do their things on auto also?
ZFS is much deeper and more thorough than that. Problem is portability, you can't get ZFS on a usb drive and plug it into any any system.
demiurge3141 is offline   Reply With Quote
Reply

Bookmarks

Sign up for a free OCAU account and this ad will go away!

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +10. The time now is 12:29 PM.


Powered by vBulletin® Version 3.8.9
Copyright ©2000 - 2017, vBulletin Solutions, Inc.
OCAU is not responsible for the content of individual messages posted by others.
Other content copyright Overclockers Australia.
OCAU is hosted by Micron21!