1. OCAU Merchandise is available! Check out our 20th Anniversary Mugs, Classic Logo Shirts and much more! Discussion in this thread.
    Dismiss Notice

Should we still test patches?

Discussion in 'Business & Enterprise Computing' started by elvis, Sep 8, 2011.

  1. elvis

    elvis OCAU's most famous and arrogant know-it-all

    Joined:
    Jun 27, 2001
    Messages:
    46,819
    Location:
    Brisbane
    Interesting question which I've had lengthy discussions about in numerous businesses:

    http://isc.sans.edu/diary.html?storyid=11527

    Points of view:

    Security guy: patch early, patch often. It's better to have a 1 day per year outage from a bad patch than to ever have a compromised server or defaced site, especially when policy is to assume everything in the same subnet is also compromised and needs to be cleaned with fire and holy water. A compromised site also means risk to our reputation, which is worse than not meeting an SLA 1 day out of 365.

    Business guy / customer rep: patches risk unplanned outages, which means we risk not meeting SLA. Customer-facing SLAs are our bread and butter, and must not be messed with. They must be tested in non-prod environments like any other software rollout.

    Change approval committee: 1 week per SDLC environment, because we're ITIL. That means 5 weeks from patch release to production. We don't understand what you mean by "zero day threat".

    Sysadmin: whether it's hacked or breaks from bad patches, I get my arse kicked. And my arse is pretty sore already.

    Marketing: What's a patch? I ate my crayon.
     
  2. NSanity

    NSanity Member

    Joined:
    Mar 11, 2002
    Messages:
    18,664
    Location:
    Brisbane
    Doesn't the USAF patch on day 0?

    How often does their crap break?

    I've been Sysadmin/Security Guy at the same time in most roles.
     
  3. OP
    OP
    elvis

    elvis OCAU's most famous and arrogant know-it-all

    Joined:
    Jun 27, 2001
    Messages:
    46,819
    Location:
    Brisbane
    Either them or US Navy. I forget which one. But yes, definitely one of them.

    Unsure. However obviously infrequently enough for them to be OK with zero-day patching. Speaking entirely for myself, it's been literally years since I've had an OS level patch break something.

    Ditto. However when pitching the USAF example to executives, I'm told "but we're not military, so the example doesn't hold". Yet when I ask them how much money they'd lose from a week's downtime due to a security breech (and subsequent environment rebuilds and audits), they scoff at me like it will never happen.

    The whole Sony/Anonymous/LulzSec thing seems to have been entirely forgotten also.
     
  4. Fishmaster

    Fishmaster Member

    Joined:
    Oct 2, 2006
    Messages:
    1,345
    Location:
    Singapore
    Our company used to have a bloody long turn around on patch testing (3-6 months) and even then we only ever did this for security patches. Feature patches or programs which for our use didn't require patching weren't even touched.

    Then in about March or April 2009 we got hit with the Conficker worm which took down most of our infrastructure in a European country. It initially entered through USB from auto run on an admin account (stupid admin at the site left the feature on...) then spread like wild fire through the MS08-067 exploit for which a hotfix had been released in October the year before.

    My vain attempts at trying to warn that we would get hit were ignored with the comment of "the patch is still being certified, just have to wait. What’s the worst that could happen?”. I still proudly bring up that original warning email when the boss questions security.

    End result of the whole situation though we now have WSUS rolling out updates in each site around the world within a week of release for non critical and same day for critical.
     
    Last edited: Sep 8, 2011
  5. Gecko

    Gecko Member

    Joined:
    Jul 3, 2004
    Messages:
    2,715
    Location:
    Sydney
    If its something that is being actively exploited, we will patch immediately. Usually we like to wait a few days to let someone else iron out the bugs if it is not screamingly urgent though.

    There is one app in particular (a certain accounting package that I have mentioned here a few times) that we wait at least 2 weeks between them releasing a patch and applying it after past experiences... but that is a rare case.
     
  6. youngpro

    youngpro Member

    Joined:
    Jun 18, 2006
    Messages:
    1,451
    hm im always of the thought that not patching could be so much more destructive than the result of not patching, but i am a security minded guy..

    most patches we test, but live test, so we have our little group of test users and if nothing breaks within a day or depending on the urgency perhaps a few days, then we roll that out to the environment..

    something really nasty is a no brainer, just send it out ASAP and deal with the repercussions if it breaks something...

    depending on your environment it can be alot more complex i guess, depending on number of applications, dependencies, how critical the specific systems are you are patching.. hard to answer this one with a generic yes or no
     
  7. 4wardtristan

    4wardtristan Member

    Joined:
    Apr 9, 2008
    Messages:
    1,181
    Location:
    brisbane
    all imo:

    should we still TEST patches, as the title of this thread? absolutely. snapshots make things alot easier, but no way im rolling out a xenapp rollup/feature pack without testing.

    should you patch often? As long as you can split the saturday night/early sunday morning around your engineering team, to save burning-out, then once a month is good :thumbup:

    case by case scenario though. as gecko said, if something is wreaking havoc during prod hours and there is a fix out there, its getting implemented regardless.

    edit: beaten by a minute or so :)
     
  8. cvidler

    cvidler Member

    Joined:
    Jun 29, 2001
    Messages:
    16,363
    Location:
    Canberra
    Which is where you need to be specific.

    Sure they may patch their internet facing web servers and windoze desktops quickly. But in the scheme of things, they're not important, downtime is not a problem compared to a defacement/breach.

    No start talking mission critical systems (incl. weapons), and you can bet patches do not occur without thorough testing. They're also not internet connected. In this case, downtime, breach or failures are worse than SLA's

    All of the views (barr marketing) from the OP are valid, the problem is determining, which wins on a system by system basis. There's no one 'right' answer.
     
  9. Taceo Corpus

    Taceo Corpus Member

    Joined:
    Sep 19, 2005
    Messages:
    3,253
    Location:
    Tokyo
    Am I running obscure, business-critical software that was developed a number of years ago and utilises the software being patched?

    Am I managing a system that has been put together with best practices in mind, or is it a tower of cards held together by sellotape and unsecure RPCs?

    Am I rolling out a desktop update for Microsoft Office, or patching our primary domain controller?

    I think a lot of it comes down to context. Something like a desktop patch that fixes a Windows 7 vulnerability or an Exchange patch that fixes a security exploit when recieving emails with attachments of between 100kb and 102.5kb in size doesn't really require testing any more. If I'm patching an RPC issue on a dodgy old magic box of a server built by El Dodge Software Co. that uses some unknown interface to accept data, I'd probably test it more thoroughly.
     
  10. bubblegoose

    bubblegoose Member

    Joined:
    May 18, 2007
    Messages:
    4,508
    Location:
    Molesworth - Tasmania
    Whats this patch testing you speak of? :lol:

    I haven't been in the game long enough to have worked in any environment where WSUS wasn't a big player in keeping things up to date. :thumbup:
     
  11. Fresh79

    Fresh79 Member

    Joined:
    May 29, 2006
    Messages:
    7,310
    Location:
    Gold Coast
    How many of you guys run full dev/test environments purely for patch testing? How does your business management team feel towards the costs associated with running such environments? My org is so backwards when it comes to this stuff.
     
  12. cvidler

    cvidler Member

    Joined:
    Jun 29, 2001
    Messages:
    16,363
    Location:
    Canberra
    Again that requirement depends on the system (and at times the service provider*).

    If it's the system that generates all your moneys, then hell yeah, test it. as downtime = $$$.


    *if they're held to SLA's, they it's in their interest to make sure any patching is controlled to prevent downtime.
     
  13. bsbozzy

    bsbozzy Member

    Joined:
    Nov 11, 2003
    Messages:
    3,925
    Location:
    Sydney
    Patch testing for our CRM systems are done in a lab and verified by the end users.

    Patch testing for Operating Systems - What is that?
     
  14. OP
    OP
    elvis

    elvis OCAU's most famous and arrogant know-it-all

    Joined:
    Jun 27, 2001
    Messages:
    46,819
    Location:
    Brisbane
    How do you know if something is being active exploited? (Particularly a zero day)? Does your security team tell you? Do you subscribe to various services to tell you (AV-vendor/SANS/AusCERT/Exploit-DB/?). Do you just get it from the media?

    I've worked in places before where the quoted text was their informal policy of sorts, but when I asked how exactly they objectively that exploits were occurring, they couldn't tell me. Do you have a formal method for determining the exposure risk?

    I'd be pretty afraid if military weapons ran off Windows for starters.

    The last three places I worked at all did their own internal development. I guess it was convenient for me as a sysadmin to piggyback off that process. I'd alternate with developers between their major releases to push patches up through their SDLC, so that they were tested various times (as little as 2 for one company, as many as 5 for another) before they hit production systems.

    If you don't have that luxury, staggering patches across servers could give you some sleep at night. Patching low use servers first, and rolling up to high use systems. Of course, that doesn't help if you've only got one exchange box or something, and you've got no choice but to patch it blind.

    As someone else mentioned already, virtualisation is taking a lot of pressure off sysadmins. Snapshotting things pre-patch is a godsend. Likewise, the long term plan for RedHat Enterprise Linux is to include a new filesystem that does a system snapshot every time the package manager is used (patch, add or remove software, and you get an automatic snapshot which you can roll back to live should things go tits up).

    I dream of a day when that technology is standard everywhere across all software, but for now it's all still quite a while off yet.
     
  15. cvidler

    cvidler Member

    Joined:
    Jun 29, 2001
    Messages:
    16,363
    Location:
    Canberra
    Why the pointless stab at windows?

    Linux/all Unixes, and any other operating system need and get patches.

    Then there's the applications, again all get bugs fixed.
     
  16. NSanity

    NSanity Member

    Joined:
    Mar 11, 2002
    Messages:
    18,664
    Location:
    Brisbane
    My understanding is that there is some really basic and obscure OS that most critical weapons systems are run off.

    Basic as in its very easy to review the source.

    My Google-Fu is failing me this morning, but i'm sure i've read about this at some point, although its probably 10 years ago.
     
  17. cvidler

    cvidler Member

    Joined:
    Jun 29, 2001
    Messages:
    16,363
    Location:
    Canberra
    A wide variety of code is used, from machine code, through real time OS's, to even unix/linux and yes even Windows OSs.

    IIRC the Collins subs use Windows (and not a recent one) for their radar/sonar systems.
     
  18. OP
    OP
    elvis

    elvis OCAU's most famous and arrogant know-it-all

    Joined:
    Jun 27, 2001
    Messages:
    46,819
    Location:
    Brisbane
    I'd be slightly less scared if Linux was used, but still extremely concerned. Decrease level of concern once again for UNIX. There are proper, purpose made operating systems for these things which should be used in place of commodity operating systems.

    Those subs did use WinNT for ages, but in the last few years were all upgraded to a proprietary custom OS, purpose written for the task (the way it should be, under full end-to-end development review).

    Commodity OSes belong on commodity systems, not on weapons. End of story. I might be a raging open source hippy evangelist, but I draw the line at weapons and the military. There are obvious reasons why code that runs on such things should be secret and reviewed in an entirely different way to standard desktop or server code.

    Banking, finance, medicine, voting and public sector systems - Linux for all of them! Weapons and subs - no way.
     
    Last edited: Sep 9, 2011
  19. s4mmy

    s4mmy Member

    Joined:
    May 20, 2004
    Messages:
    2,221
    Location:
    Melbourne
    So intercontinental balistic missles on WindowsME is a no no? :Paranoid:
     
  20. OP
    OP
    elvis

    elvis OCAU's most famous and arrogant know-it-all

    Joined:
    Jun 27, 2001
    Messages:
    46,819
    Location:
    Brisbane
    That shit would be blowing up inside silos all over the US! :lol:
     

Share This Page

Advertisement: