1. Check out OCAU's review of the SpaceX Starlink satellite internet service!
    Dismiss Notice

Can 7zip really make 2gb into 97mb?

Discussion in 'General Software' started by peteed1985, Dec 29, 2020.

  1. peteed1985

    peteed1985 Member

    Joined:
    Feb 23, 2009
    Messages:
    889
    So I kinda don't want to lose my save games for ark survival evolved so I tried zipping the 2gb save folder up and the result was a 97440kb 7z file which means 2gb became under 100mb which I didn't think was possible. Tbh I have 0 experience with zipping stuff up so wanted to ask people if this is really my 2gb of saves or if it's missing stuff if it's that small?
     
  2. ipv6ready

    ipv6ready Member

    Joined:
    Feb 10, 2014
    Messages:
    2,831
    Location:
    North Sydney
    Have you opened the zip file and see the files are there and usable?

    Having said the above compression from 2gB to 100mB seems highly unlikely ratio, unless all your files are all simple text and or spreadsheets.

    If just 15% of your files are movies, pictures or music, it is unlikely you will end up goingfrom 2gB to 100Mb
     
  3. rthy

    rthy Member

    Joined:
    Dec 6, 2012
    Messages:
    1,412
    Location:
    Melbourne
    2gb saved game?
     
  4. OP
    OP
    peteed1985

    peteed1985 Member

    Joined:
    Feb 23, 2009
    Messages:
    889
    Despite having 7zip installed on the PC the file is not associated with it when I double click on it xD I normally right click to compress and uncompress stuff. It's as I said in my post the folder is the save files for ark survival evolved which is an unreal engine made video game about taming dinosaurs and riding them and the like.

    When I was zipping it it mentioned something on the progress page about compression ratio 4% but idk what that means.

    It seems to make it's own backups named to include date and time of backup on top of having it's current save as well as a bunch of ini files and mod setting files. Each save file or backup seems to be between 30 and 55mb. Ark Survival Evolved the game itself is a 295GB game folder though some of that might be the mods downloaded from steam workshop but I know the game itself is massive.
     
    Last edited: Dec 30, 2020
  5. theSeekerr

    theSeekerr Member

    Joined:
    Jan 19, 2010
    Messages:
    3,687
    Location:
    Broadview SA
    Completely plausible if the saves are inefficient or repetitive.

    At work I often find myself needing to transfer multi-gigabyte log files - because the content of logs is so self-similar even ye olde ZIP generally collapses them to 10% of their original size, and newer algorithms (like the LZMA algorithm used by 7zip) can do much better.
     
    vladtepes likes this.
  6. DarkYendor

    DarkYendor Member

    Joined:
    Feb 25, 2008
    Messages:
    3,212
    Location:
    Perth
    If you have a lot of near-identical files, or files with repetitive content, it's pretty believable that you could shrink them at 25:1.

    If you want to check, you could decompress the files and then calculate and compare checksums for them. (If you're any good with powershell, you can write a script to automate this?).
     
    BurningFeetMan likes this.
  7. EclipZ

    EclipZ Member

    Joined:
    Mar 11, 2002
    Messages:
    116
    Location:
    bne.qld.au
    yes 7zip can get 4% of the original size if the data being compressed is suitable.

    Right click context menu only is normal behaviour on default install, the "Test archive" option should tell you if the .7z archive is okay or if it contains errors.
     
  8. Optimus.

    Optimus. Member

    Joined:
    Jun 27, 2002
    Messages:
    6,688
    If it's mostly text-based files that aren't already compressed (eg. gzipped by the game engine) then yes, this should be about right.
     
  9. mtma

    mtma Member

    Joined:
    Aug 12, 2009
    Messages:
    5,906
    Just a little extension to this, 7zip has the test mechanism however it has no mechanism to attempt any level of data recovery if an archive does encounter any data corruption. Something to watch out for when storing mega-archives as merely a minor issue will ruin everything.

    Of course historically it's a rare issue on magnetic media and probably only an annoying inconvenience for save games but it's a possibility.
     
  10. Myne_h

    Myne_h Member

    Joined:
    Feb 27, 2002
    Messages:
    11,433
    Yeah, if you want it to survive minor corruption, RAR + recovery record is the way to go.

    Might be 150mb but you can lose a few bits and still rebuild it.
     
  11. elvis

    elvis Old school old fool

    Joined:
    Jun 27, 2001
    Messages:
    45,033
    Location:
    Brisbane
    While true, this is no different from corrupted files that are uncompressed too.

    Best not to use RAR. It is grossly outdated, and desktop "WinRAR" tools have had plenty of vulnerabilities that took enormous amounts of time to patch.

    7zip is based on the LZMA family of compressors, both of which are open source. If you want further data integrity on top of that, I recommend the PAR2 algorithm, again which is open source. There are plenty of tools that support it, such as QuickPar.

    Even then, 7zip is getting long in the tooth. ZStandard (aka zstd) is a more modern format that offers far better compression at the high end, and far better speed at moderate compression rates, as well as multi-threading support that can scale past 2 threads (unlike LZMA1, and LZMA2 isn't much better capping out at 4 normally).

    I do long term data preservation for both work and hobbies. If you're interested, more in depth talk about it here:
    https://stickfreaks.com/misc/digital-data-preservation
     
    Luke212, flain and theSeekerr like this.
  12. theSeekerr

    theSeekerr Member

    Joined:
    Jan 19, 2010
    Messages:
    3,687
    Location:
    Broadview SA
    Incidentally, 7-Zip (the application rather than the format) supports adding compression formats by plugin (they call them codecs, a usage I'm a little uncomfortable with).

    zstd support can be added by 7-zip-ztd or Modern7z

    Obviously 7z files with contents packed with these formats will be incompatible with other 7Zip users, but the two implementations should be compatible with each other.

    Both also support LIZARD/LZ5, a format optimised for decompression. At high levels it compresses about as well as low levels of zstd (i.e. similar to moderate levels of LZMA) but decompresses about around twice as fast at comparable compression ratios, or around 1000-1500 MB/s on modern hardware. Low levels are even faster and probably limited by your disk even on a good SSD.
     
    elvis and flain like this.
  13. elvis

    elvis Old school old fool

    Joined:
    Jun 27, 2001
    Messages:
    45,033
    Location:
    Brisbane
    With multi threading zstd happily reaches 2.5GB/s or more at compression level 3, which is roughly comparable zlib/gzip/PKZIP level compression.

    https://github.com/facebook/zstd/blob/dev/README.md

    I use real time zstd compression on my NAS filesystem (BtrFS), again at level 3, and it's much faster than my disks, so it actually increases the perceived data transfer speed.

    I leave most files uncompressed, as the inline zstd compression and deduplication reduce the data nicely. If I need something long term archived, I use zstd at level 19 compression, which beats LZMA1/LZMA2/xz for both compression rates and speed.
     
    theSeekerr likes this.
  14. vladtepes

    vladtepes Member

    Joined:
    Sep 30, 2015
    Messages:
    2,925
    Location:
    Brisbane, Qld
    So as a TL;DR for general readers -

    what's the recommended zip program for a 'typical' non-technical user nowadays ?
     
  15. JSmithDTV

    JSmithDTV Member

    Joined:
    Jun 13, 2018
    Messages:
    9,018
    Location:
    Algol, Perseus
    Well WinZip is already built into Windows, but I and many prefer 7z;

    https://www.7-zip.org/download.html



    JSmith
     
  16. Luke212

    Luke212 Member

    Joined:
    Feb 26, 2003
    Messages:
    10,211
    Location:
    Sydney
    whst is recovery record? Never seen that b4. There was something like that in the old torrent days maybe something like that?
     
  17. Optimus.

    Optimus. Member

    Joined:
    Jun 27, 2002
    Messages:
    6,688
    7zip, so many supported formats + free.
     
  18. theSeekerr

    theSeekerr Member

    Joined:
    Jan 19, 2010
    Messages:
    3,687
    Location:
    Broadview SA
    7-zip is good and free.
    If I recall correctly it doesn't associate itself with any file formats out of the box so you might want to set that up. Personally I recommend leaving .zip with the default Windows Explorer handling because it's a bit less clunky than having a seperate tool open, but giving everything else to 7-zip - it can open pretty much any other archive or container you might run across (cab, rar, tar, gzip, various disk image formats like ISO, DMG, WIM, VHD etc)
     
    vladtepes likes this.
  19. vladtepes

    vladtepes Member

    Joined:
    Sep 30, 2015
    Messages:
    2,925
    Location:
    Brisbane, Qld
    Confirms my assumption. Thanks :)
     
  20. rireland

    rireland Member

    Joined:
    Jul 11, 2003
    Messages:
    259
    Bittorrent does not need a recovery file as inherent error checking and repair is one of the protocols selling points. Any such files would of been intended for other distribution methods.
     

Share This Page

Advertisement: