How Does Your Company Manage Microsoft Patching?

Discussion in 'Business & Enterprise Computing' started by RavenKittie, Jun 13, 2009.

  1. RavenKittie

    RavenKittie Member

    Joined:
    Dec 12, 2002
    Messages:
    1,602
    Location:
    Sydney
    Hi Guys,

    I'm curious to see how other corporate environments handle their Windows patching. Basically I am looking at ways to do it better due to a bit of an issue I had with one of the KB Articles (at 2AM in the morning, still onsite :sick::tired::thumbdn:).

    Currently we use WSUS and the patches are only approved after we've at least done some reading on them, what they change and any known issues. We have 3 non critical servers in a test group ,as well as some client machines. Things get released to this group first, then if no major issues arise they are pushed out to the WSUS Servers group and then the Workstations group.

    The problem is someone still has to sit there and install the patches, reboot, then test everything is still working. We have outgrown this, as to do 34 production servers it took 6 hours, on top of the problems I had with Exchange it added up to about an 8 hour job.

    Surely there is a better way to manage patching! I can't imagine any organisation with this amount of servers wasting 8 hours of man hour to install security updates surely.

    So, how do you guys manage your updates? Do you even bother updating certain servers or do you only update your public accessible boxes?

    Do you use any sort of third party software in conjunction with WSUS?

    I have spoken to some other geeks in the field and they find if the PC's and servers were set for automatic installs (prompt for reboot) they had no problems. I'm guessing because if you do it this way you are obviously installing the patches in the same order that MS are releasing them.

    I guess I'm just curious because I can't imagine how organisations with 100+ production servers go about doing patching and the subsequent reboots outside of working hours. There has to be a better way to do it!
     
  2. Fitzi

    Fitzi Member

    Joined:
    Jun 27, 2001
    Messages:
    519
    Location:
    Central Coast, NSW
    Most of our client are on our MPLS network and their DC's point to our WSUS server when looking for updates, saving bandwidth (free data over MPLS).

    We usually just setup the servers to automatically update and prompt for reboot (default 3am) most of the backups are complete by this time so there's usually no issue with this. Of course this is not on mission critical servers which require 24/7 uptime, these are of course updated and the restart window scheduled to minimise downtime.

    As for reading up on each individual patches, we have a lot more faith these days in M$ they seem to have gotten their act together with patching, apart from the odd bug here and there they have come a long way over the last few years (havent had a server spontaneously combust due to a patch for some time). In saying that we dont run many properiarty apps and if we did I guess we would look a bit harder at individual updates, security fixes however are a necessity that cant be overlooked.

    As for public facing vs internal servers, unless there are specific needs I dont think there should be a division between the two, just because a server is on the internal network this dosent make it immune to security breaches, your only as strong as your edge device I guess.
     
  3. Mac

    Mac Member

    Joined:
    Aug 1, 2001
    Messages:
    762
    Patch first, ask qustions later.

    As cowboy as it seems we really havent any issues. Though to minimise risk I manually approve all patches and updates in WSUS one week after they are released. So my Patch Tuesday occurs the Tuesday after Microsofts.

    It leaves me 'unprotected' for week but zero day patch exploits and patches that break stuff on a wide scale are pretty widely reported on in the IT media circles (like OCAU front page) so if anything super urgent comes up I can revise my strategy and bring my patching forward (or not patch at all).
     
  4. Falkor

    Falkor Member

    Joined:
    Jun 27, 2001
    Messages:
    4,045
    Location:
    Sydney
    Personally I do all servers manually.

    I don't trust anything automatic when it comes to servers, at least if I patch it and it doesn't come back up Its usually easier and I can not continue on with any other servers then.

    Users PCs we approve pretty much everything bar big updates such as service packs etc, which we tend to test on a bunch first and then rollout to the rest.
     
  5. NSanity

    NSanity Member

    Joined:
    Mar 11, 2002
    Messages:
    17,716
    Location:
    Canberra
    Patches via WSUS.

    For workstations/laptops this is my method. Our laptop fleet picks up the patches typically within 3 days (and we chase them down if they are older than 14 days).


    Servers are updated manually, with daily reading of any critical fixes that apply to servers (various news sites and packetstorm/secuna.
     
  6. lavi

    lavi Member

    Joined:
    Dec 20, 2002
    Messages:
    4,004
    Location:
    Brisbane
    i snapshot the vm's then patch if issues revert simple as pie
     
  7. AzzKikr

    AzzKikr Member

    Joined:
    Aug 25, 2002
    Messages:
    1,078
    Location:
    .au
    Patches deployed via Shavlik, first to a test group of servers and to production a few days later.

    -A.
     
  8. gbh

    gbh Member

    Joined:
    Aug 29, 2001
    Messages:
    2,175
    Location:
    Sydney
    what you said
     
  9. gbh

    gbh Member

    Joined:
    Aug 29, 2001
    Messages:
    2,175
    Location:
    Sydney
    wsus to the whole domain, manual approval on all since one day a few years ago a patch to sbs 2000 killed my exchange machine.

    so snap, manual approve via wsus, install

    since that one patch ages ago that did screw things up, no other patches have done anything detrimental/caused a roll back...but you never know.......
     
  10. BMan2

    BMan2 Member

    Joined:
    Feb 17, 2002
    Messages:
    822
    what is this "patching" thing you talk of? Our company doesn't seem to know about it :p
     
  11. Spingo

    Spingo Member

    Joined:
    Jun 28, 2001
    Messages:
    1,052
    I look after what you'd consider server-heavy infrastructure (200+ servers, 20 full time staff, 40 workstations). Many servers are 24x7 jobs, and many depend on others to be running in order for their functions to be operational .(such as the transcode management server, which our developers have never got around to, and probably never get around to setting up as a fault tolerant system regardless of how often I persist that this should happen and how many photos I email them of the server that's sitting there waiting for them to write the code, but I digress)

    Anyway...

    WSUS. Updates for Office, Active X Killbits, Windows Defender Updates, Windows XP or Windows Vista (with some exceptions) are automatically approved. All other patches go through a review process unless I deem them to be emergency security patches (where a zero-day exploit has been discovered).

    The patch review process involves me creating an issue in our issue tracking system, having another senior member of the IT team approve the update as well as myself and then passing it on to the development team for review. One of our developers reviews the update, occasionally installs it on a couple of staging VMs in order to ensure that the update doesn't break production code. Once the developer approves the update, it's approved for all office system and staging systems. In general, only the updates that are likely to break production system functionality are tested thoroughly (Windows Media Encoder updates being one example of a type that's always tested).

    We work on a 6-week development cycle, and updates are manually installed on staging systems when the testing for a particular release commences. Once testing for that release has been completed, updates for production are manually approved, and installed when the production release is rolled out.

    A similar process is followed for HP Firmware and PSP updates.

    For internal systems, patches are force installed via group policy at 3:00am on a Friday night (or first thing Monday morning if people do the proper thing and shut down their systems). Updates are always installed manually on production systems. This can be extremely time consuming, but can usually always be done in the background while production relleases are being installed on each system.

    In all, I can get all Sydney production servers (~50 servers) updated with a new release of our software and all patches in around 4 hours. Singapore productions servers (~70 servers) are updated within the space of around 6 hours. Other remote servers in remote sites take a matter of minutes, depending on what updates are being installed. The rest of our servers are internal systems are internal, testing or staging systems and are patched previously.
     
  12. OP
    OP
    RavenKittie

    RavenKittie Member

    Joined:
    Dec 12, 2002
    Messages:
    1,602
    Location:
    Sydney
    Thank you very much for the replies guys!

    I'm going to move the patching to a weekend, it gives us a full day if all hell breaks loose. I think the problem is how WSUS has been treated in the past and how many updates were missed. I would do it all automatically but I think it's too late now. We are going to be virtualising alot of our servers in the coming months which should help alot in terms of backups and DR.

    We also purchased some SOE deployment software which has a component that manages Windows updates and ties in with WSUS and apparently makes the removal much easier. Will have to look into it ASAP.

    Spingo - 50 Servers in 4 hours is quite impressive. Do you have automatic reboot set or are you doing it manually in a night / weekend?

    Actually another question :)

    Do you guys do Windows Updates remotely / via VPN from home etc, or do you do them in the office?

    I have recently taken over the process from another sysadmin, and the boss sounded surprised when I insisted I do it onsite. It was lucky I did because two boxes needed console access as they wouldn't respond to ping after I attempted a restart. They were two very old servers, running SQL 2000 / Windows 200 Server standard.
     
  13. bcann

    bcann Member

    Joined:
    Feb 26, 2006
    Messages:
    5,714
    Location:
    NSW
    Use some kind of patch management tools (wsus, altiris, kaseya, take your pick which one)

    select which servers to patch, click go, grab coffee.
     
  14. joyufat

    joyufat Member

    Joined:
    Jun 27, 2001
    Messages:
    1,014
    Location:
    Moral High Ground
    VPN from home, that way you can do it late when there is least impact. No way I'm staying in the office at midnight. Go get console for these situations - you can get an IP KVM, or iLO, DRAC etc
     
  15. Mac

    Mac Member

    Joined:
    Aug 1, 2001
    Messages:
    762
    How many patches in the 3/6/12/24 months have caused problems with your production servers? Of those patches that have caused problems are you able to establish why they caused problems?
     
  16. GiantGuineaPig

    GiantGuineaPig Member

    Joined:
    Oct 23, 2006
    Messages:
    4,026
    Location:
    Adelaide
    WSUS, with automatic staggered weekly reboots for servers. Anything of importance is usually a day behind, plus we have backups.

    We do manually approve items on WSUS first though.
     
  17. bsbozzy

    bsbozzy Member

    Joined:
    Nov 11, 2003
    Messages:
    3,925
    Location:
    Sydney
    We approve pretty much straight away, only ever had a handful of issues, e.g. Update for acive x killbits messed with the sql report server web print controls, a simple update to that and all is well.

    Saying that, hotfixes only, not service packs/drivers/other stuff.
     
  18. vladdy76

    vladdy76 Member

    Joined:
    Feb 19, 2003
    Messages:
    3
    Location:
    Brisbane
    This is actually a very interesting topic of discussion...

    There are the two questions of - do we rollout all patches or not - then the question of how?

    There are risks involved with both rolling out patches and not rolling out until sufficient testing has been done. IMO the better option is to patch, as I would rather tell a customer their server broke to a patch rollout rather than an vulnerability due to the server not being patched.

    Manually patching is the lowest risk method at the cost of time. This simply becomes unfeasible as soon as your server farm grows.... you'll spend more time patching than it takes before the next patch releases are available.

    Automatic patch/reboot is the only scalable method. To reduce risk group servers into test (non-production), pilot (small sample of production) and production (the rest). Allow at least 5 days before each rollout, which should be enough time to uncover any major flaws. You'll never be able to reduce risk to nothing, as every server installation/configuration will likely be unique in some way.

    The next question is how... unless your company has the $$ to spend on third party apps like kaseya or kbox, you'll be limited to WSUS. The only downside to WSUS is flexibility (there is a nice add-on reasonably priced from emanentware worth looking at). If your server farm is big enough you won't want to be rebooting all your servers at once, so you'll need to find a way to group and manage WSUS policies...

    I've had a fair bit of experience with MOM 2005 and have written a custom script to manage registry settings. I can create as many computer groups and rule groups I want, then I just add the servers to whichever computer group is desired. In total I have about 30 unique policies (computer groups). From group membership I can query the MOM database directly to get a list of computers and their patch time, so another script I use to marry up computers with customers, and provide automated notifications. The membership from MOM also allows me to automatically put the servers in maintenance mode for an hour during patch time. Any problems with the patch on a server should be flagged in monitoring as soon as the maintenance mode period has elapsed.

    I'd be interested to hear how other people/companies perform patching on a large scale, in particular using Kaseya (something we will be evaluating).
     
  19. Skitza

    Skitza Member

    Joined:
    Jun 28, 2001
    Messages:
    3,753
    Location:
    In your street
    Easy free method - WSUS :)
    Complex package not free method - LANDesk :)
     
  20. Jarwedy

    Jarwedy Member

    Joined:
    Nov 22, 2003
    Messages:
    1,007
    Location:
    Rockhampton, QLD
    Same as others, manual approval of server patches. Auto approval of general patches and hotfixes for XP clients, drivers left out as are service packs. Manual approval of Office patches.

    Server reboots are set for weekends. No auto reboots for clients.
     

Share This Page

Advertisement: