Cache server software - Cache apt & wget downloads

Discussion in 'Other Operating Systems' started by gibbz, Feb 11, 2017.

  1. gibbz

    gibbz Member

    Joined:
    Jan 20, 2004
    Messages:
    1,473
    Location:
    Adelaide, SA
    Hi,

    Ive transitioned all my home computers over to some form of ubuntu. And I regularly test out new ones.
    I have a bunch of scripts the install via apt, snap or wget to setup my applications automagically.

    The problem I face is that most apt servers are sluggish, and my apps are big! I need to cache these on my server, then updates through the other machines in the house should check there for their files.

    What are my options?
    Ive found:

    http://www.squid-cache.org/ (web cache)
    https://help.ubuntu.com/community/Apt-Cacher-Server (apt cacher - dead?)
    http://www.tecmint.com/apt-cache-server-in-ubuntu/ (apt-cache-ng)
    https://help.ubuntu.com/community/AptProxy (apt proxy)
    https://apps.ubuntu.com/cat/applications/squid-deb-proxy-client/ (squid deb, looks very easy?)
    http://askubuntu.com/questions/3503/best-way-to-cache-apt-downloads-on-a-lan (various)
    http://xmodulo.com/apt-caching-server-ubuntu-debian.html (simple apt-cache-ng)
     
  2. Primüs

    Primüs Member

    Joined:
    Apr 1, 2003
    Messages:
    3,418
    Location:
    CFS
    wget is primarily a HTTP downloader - a web-proxy would work.

    If you dont want all your traffic to go through a proxy though you could just maintain a mirror of what you want and when you wget you just give the local site instead of remote.

    aptitude mirrors are easily set up and you can change the sources list on your client machines to check your local mirror.
     
  3. elvis

    elvis Old school old fool

    Joined:
    Jun 27, 2001
    Messages:
    43,141
    Location:
    Brisbane
    At home I use apt-cacher-ng, because I don't have the space to store an entire mirror. I simply insert my server and port infront of the regular sources.list entries, and it caches them silently for other hosts to pick up.

    https://www.unix-ag.uni-kl.de/~bloch/acng/

    On top of that, I run a cron job from the server doing the caching that calls this every night:

    Code:
    /usr/bin/curl 'http://127.0.0.1:9999/acng-report.html?abortOnErrors=aOe&doDownload=dd&asNeeded=an&doMirror=Start+Mirroring'
    
    That triggers a refresh of any packages already downloaded, grabbing the latest version of them. Should a client then ask for the latest package again within the next 24 hours, chances are it's already pre-cached.

    Squid is OK, but expires cache too quickly, and isn't mirror aware. apt-cacher-ng has a list of mirrors, and is smart enough to cache multiple mirrors to one set of cache files (good if you have different machines point to different mirrors of the same distro).

    At work I run a combination of rsync and apt-mirror:
    http://apt-mirror.github.io/

    Apt-mirror mirrors an entire distro, with config files similar to your regular sources.list. This can get pretty large, but it's worth it for us because we have an enormous volume of systems, including many that provision automatically during the day for different reasons. We previously had apt-cacher-ng in place, but the downside was if the cache ever corrupted it meant scrubbing it and starting again (much easier than trying to get it to a consistent state again), which meant the next provisioned system was slow (Internet speed, instead of LAN speed).

    Hopefully rsync is obvious, and some mirrors (aarnet, internode, and others) allow you to rsync directly from them, which allows very fine-grained control over includes/excludes, and allows us compression over the wire too (not a huge deal, seeing as most packages are compressed already anyway).

    Rsync is great where supported, but apt-mirror will allow mirroring of things like PPAs and other virtual or http only sources.

    As a couple of space examples, we currently mirror the three LTS releases (12.04 Precise, 14.04 Trusty, 16.04 Xenial) with all sub sections (main, restricted, multiverse, universe, updates, backports) and that clocks in at 760GB of data. LibreOffice PPA clocks in at 40GB. WINE clocks in at 1.5GB. There's other stuff there too (CentOS mirrors, oVirt mirrors, MacOSX updates via "reposado", and other open source stuff we host that all clock in together at around 1.4TB, which is worth it for the bandwidth savings across so many hosts).
     
    Last edited: Feb 12, 2017
  4. [MadHatteR]

    [MadHatteR] Member

    Joined:
    Jun 12, 2006
    Messages:
    125
    Location:
    Melbourne
    I used to run 'apt-cacher-ng' for my home network (about a dozen headless Ubuntu 14.04 servers), and have recently switched to apt-mirror. I found apt-mirror to be nearly instant for client machines, since they are only communicating with the mirror machine. The entire index update for a client machine takes about 3 seconds, vs 30-40 seconds, for around 70 PPAs in sources.list

    With apt-cacher-ng, the clients still need to fetch the indexes from the PPAs, and also the first time a package is requested, so the speed advantage only occurs on subsequent downloads. But it is easier to set up, just drop a 02proxy config into apt.conf.d/, no edits to sources.list required, it's all self-contained as well. I found total space required on NAS was about 15-20GB.

    The downsides to apt-mirror are that you need to run a web server and also edit the sources.list to point to the mirror machine. There could also be a delay if a critical security update drops, which can be reduced by setting mirror to update more often (say twice a day). Or you can also get around this by specifying only the ubuntu security update server to connect directly to internet. I found total space required on NAS is about 160GB. If you can spare the space, I would recommend this solution.

    Squid is a web proxy, I think you could theoretically set it up to cache apt downloads, but better to use one of the above tools. There is also another tool called apt-fast which splits apt downloads into multiple threads which has it's uses.
     
    Last edited: Feb 13, 2017
  5. OP
    OP
    gibbz

    gibbz Member

    Joined:
    Jan 20, 2004
    Messages:
    1,473
    Location:
    Adelaide, SA
    ok thanks guys.
    I think apt-chacher-ng will suit my needs. I wont bother with wget downloads as theyre fairly small anyway.

    A full mirror is a bit over the top :Paranoid:
     
  6. elvis

    elvis Old school old fool

    Joined:
    Jun 27, 2001
    Messages:
    43,141
    Location:
    Brisbane
    FWIW, we switched from apt-cacher-ng once we hit around 30 systems per site that needed regular provisioning and updates. That seemed to be the magic point where apt-cacher-ng couldn't keep up, and it was worth our time to put a full local mirror in (especially as we provision from PXE boot).

    For a small business or home setup, apt-cacher-ng deals with the hit rate just fine, especially if you're building systems from removable media installers.
     
  7. HyRax1

    HyRax1 ¡Viva la Resolutión!

    Joined:
    Jun 28, 2001
    Messages:
    7,925
    Location:
    At a desk
    Another vote for apt-mirror here, at home and at work.

    It's also really useful to be able to take an offline mirror when you need it too.
     
  8. mesaoz

    mesaoz Member

    Joined:
    Jan 15, 2015
    Messages:
    7,451
    Location:
    South East QLD
    Have used squid a few times in the past, did it's job without much fuss.
     

Share This Page

Advertisement: