What's your on-call process and what do you like about it?

Discussion in 'Business & Enterprise Computing' started by oh_noes, Jun 19, 2012.

  1. oh_noes

    oh_noes Member

    Joined:
    Oct 29, 2006
    Messages:
    991
    Location:
    Australia
    As a sysadmin/engineer, how does your current employer do on-call, and what do you like the most about the processes in place? Is there any part of your on-call process that you particular like the most especially in comparison to your past employers?

    Is there something that is so flawed and you hate so much that you can't help but mention it here?

    Additional to this, what on-call support tools do yo have at your arsenal? Who's still passing around a bulky laptop and a team blackberry? What's the 2012 version of this?

    I think it would be really interesting to here some feedback on how others do it.
     
  2. ECHO

    ECHO Member

    Joined:
    Jun 17, 2002
    Messages:
    632
    Location:
    Canberra
    I guess this thread hasn't been visited since 2010 I suppose!

    We're a small business (10-15 staff) and the boss still has the phones almost all year around except for when he goes on holidays or other reasons when he doesn't want them!

    This is a snippet from our internal policies:

    * If you are on call between 5.30pm Sunday and 5.30pm Friday, you are paid $75 per night.
    * If you are on call From 5.30pm on Friday to 5.30pm on Saturday and/or from 5.30pm on Saturday to 5.30pm Sunday, you are compensated $150 for the full day and night.

    We CHARGE our customers higher rates if they proceed with going to our after hours service. This drastically reduces the amount of calls we receive.

    It will obviously be a different scenario if you're the main sysadmin for a company...
     
  3. s3kemo

    s3kemo Member

    Joined:
    May 13, 2003
    Messages:
    5,889
    Location:
    in a house
    Great rostering and enough guys right through the line (from helpdesk right up the chain) to cover when the guy/guys need the sleep-in after a long night. Good sharing of knowledge before/after any planned changes so no one is left in the dark come cutover day.

    My laptop + IM client + webex + home VPN connection is all I need.
     
  4. m0n4g3

    m0n4g3 Member

    Joined:
    Aug 5, 2009
    Messages:
    3,491
    Location:
    Perth, WA
    3 staff taking care of nearly 1k users, on-call 24/7 and they make sure to use it...

    Don't get paid shit for it.... although i suppose the pay is reasonably good. We rotate the on-call phone once a week between the 3 of us, but generally it's only 2 of us that does any of the work as the other guy is useless and we can't rely on him for anything. :(

    Get calls at 3am fairly often, when the power gets cut off and i have to come in to do a systems check on our equipment after the generator kicks in...

    SO pissed off during those nights... :(
     
  5. scrantic

    scrantic Member

    Joined:
    Apr 8, 2002
    Messages:
    1,656
    Location:
    Melbourne
    This has just been raised in our organisation. There are only two full time IT staff but there is the potential for the expectation of "official" on-call arrangements to be put in place, rather than a if you call after hours someone may/may not be available.

    We operate some of the non national public holidays so need to cover those; We do where possible but have issues of people expecting someone to be sitting by your phone from 8-6 which can make life a bit tedious.

    Will be interesting to see what pans out as a on-call roster between two guys will be tough.
     
  6. DonutKing

    DonutKing Member

    Joined:
    Mar 21, 2004
    Messages:
    1,092
    Location:
    Tweed/Gold Coast
    I'm the sysadmin for a company with about 100 staff, I don't work weekends but the nature of our business means someone needs to remote in and check a few things every day including public holidays and weekends.

    There's no on-call process. They just call me on my personal phone. If I happen to be 100km away with no laptop they crack the shits, even though I'm not on call.

    I got woken up at 7am last christmas eve when I was a week into 3 weeks worth of leave over christmas. Had a word to the boss after this and still no process has been in place but I believe the offending staff member got asked politely not to call when I'm on leave.

    Oh well, job interview on friday (with a different company), here's hoping :)
     
  7. Diode

    Diode Member

    Joined:
    Jun 17, 2011
    Messages:
    1,710
    Location:
    Melbourne
    We pass around a support iphone rotated weekly. There is a laptop, but it's so old and crap that no one bothers to take it, we need to be reasonably close to work or home where we can VPN or be onsite within 20-30 minutes of a call. We get paid a little extra for being on call, it's not heaps but it's enough considering it hardly ever rings. It works out to around $200 a week extra. So far we have 3 people in our team it rotates around, so I'm on call once every 3 weeks. Soon we will have a 4th and maybe a 5th guy to share the love with. :)
     
    Last edited: Jun 20, 2012
  8. Smokin Whale

    Smokin Whale Member

    Joined:
    Nov 29, 2006
    Messages:
    5,174
    Location:
    Pacific Ocean off SC
    That sounds like a pretty good system, minus the crap laptop. Most of the time people just call me whenever they feel :tired:
     
  9. kenks

    kenks Member

    Joined:
    Aug 30, 2010
    Messages:
    17
    Sysadmin for a large company. On Call in a rotating roster 1 week (between 7 people). We each have our own i7 laptops with SSD, VPN, plus iPhones with VPN etc. We get paid a flat rate for the week which is pretty good, then if we do get called its time and a half for time spent. We don't have any dramas with how it works, and we utilize SCOM, BMC and Director for monitoring.
     
  10. Diode

    Diode Member

    Joined:
    Jun 17, 2011
    Messages:
    1,710
    Location:
    Melbourne
    Now this sounds like best so far. :thumbup:
     
  11. eazy-e

    eazy-e Member

    Joined:
    Jun 27, 2005
    Messages:
    37
    I work as a network engineer for a large national (with some international) retailer. There are a few hundred stores and 10000+ users. With an organisation this size there is a helpdesk who field all the calls.

    We have a team of 4 with 3 of us passing the on call duties each week. Calls are filtered by the helpdesk before getting to us, and we only get calls (in theory) when something is broken.

    Pay is 12 hours just for the pain of carrying the phone and laptop around. If you get a call its 2 hours as soon as you pick up. The pay is pretty good, but nobody really likes the pain of potentially interrupting your night/weekend.
     
  12. elvis

    elvis Old school old fool

    Joined:
    Jun 27, 2001
    Messages:
    30,823
    Location:
    Brisbane
    Previously, I spent the last 11 years on call. Pretty much 24x7 either directly, or indirectly as an "expert" in some system or another. It sucked, no matter what the configuration, number of backup engineers, or extra pay. Obviously solo is worse, but there was never a "good" setup.

    As of two weeks ago I started an R&D role with no on call. Sleeping with your phone off is the best feeling ever. :thumbup:
     
  13. s3kemo

    s3kemo Member

    Joined:
    May 13, 2003
    Messages:
    5,889
    Location:
    in a house
    I've had the last 6 months with no on call - it's awesome knowing you won't get a call in the night --- even with the phone still on! :Pirate:

    That's about to change though, but my new on-call rotation is going to be pretty well organised so not really fussed about it.
     
  14. SirNemesis

    SirNemesis Member

    Joined:
    Sep 22, 2002
    Messages:
    1,432
    Location:
    Geelong
    Ours is weekly rotation, wednesday to wednesday. Support is 24/7, but chargable to the customer outside of the normal 7:30am-5:30pm window. Pay is decent, but we are really lacking any kind of filtering between the customers and on-call engineers.

    Out of the 10 or so phonecalls received every night, 9 of which become non-urgent after the customer is advised they will be charged for the time....

    Until we have some kind of "Press 1 to indicate you agree to the charges" option before the mobile rings, it is a massive pain in the ass. Supporting of hotels means the majority of after hours calls come through between 1:30-3:30am.

    We are still carrying around the team blackberry and bulky laptop....
     
    Last edited: Jun 20, 2012
  15. iNCOGNiTO

    iNCOGNiTO Member

    Joined:
    Jun 27, 2001
    Messages:
    41
    Location:
    Wantirna, Vic
    Hi all,

    Do any of you have any self service support solution in place? Also, I've not noticed any comments about educating the end user to avoid the call in the first place. I would think that the remuneration and conditions and feeling around the worth of that reward would also be affected by the business focus on reducing the calls as well?
     
  16. BLeR

    BLeR Member

    Joined:
    Jun 27, 2001
    Messages:
    444
    Location:
    Gold Coast, QLD
    My team consists of 4 people (Wintel Sys admins mainly). We do 1 week in 4. On-call is incorporated into the yearly salary.

    Use our own phone (sharing a phone wtf??) and then SSL VPN in. Helpdesk is 24x7 so they filter most of the junk. Aside from that we are pretty darn awesome, our stuff is too scared to break and interrupt our sleep.:leet: High Five.

    Every week we have a release night where new projects get released to Prod. Sometimes that can be the only work we do...other times we get bitch slapped with outages etc. Occasionally we get single user issues escalated, but they tend to be those at the CEO type level. Anyone else can GF :)
     
  17. KeEnA

    KeEnA Member

    Joined:
    Aug 21, 2001
    Messages:
    118
    Location:
    VIC 3977
    Government on-call.

    Rotates between 4 staff (1 week each). If recalled it's a min of 6 hours pay. Nightly stand-by rate is low (~$15 per day, double for fri, sat and sun). Sometimes 5 recalls a week, sometimes 1 or 2. Must be within 10-15 mins of primary site at all times.
    No official laptop provided, everyone uses their own PC should then need to VPN to fix the issue.
     
  18. link1896

    link1896 Member

    Joined:
    Jul 28, 2005
    Messages:
    366
    Location:
    Melbourne
    We support approx 40 sites in Victoria and Tassie with resolution guidelines promising 80% of cases are closed in four hours. I have four guys doing one week each on call. $200 a week to be available, time and a half for a min of three hours, 4 on Sundays the moment they answer the phone. Customer runs their own two man help desk who field the " turn it on stupid" calls. Hours on call are 7am till midnight. GM is so out of touch it's taken him 9 months to approve the capex for MacBooks. It grows tiring buying netbooks when on sale under the $300 capex threshold, makes it hard for my guys trying to remote in via a netbook.

    I take the initial phone call whilst my guys are still gaining experience, and normally take a gander at the logs. Part and parcel of a salary package where you sell your soul Bart Simpson style sadly.
     
    Last edited: Jun 22, 2012
  19. meth05

    meth05 Member

    Joined:
    Apr 11, 2005
    Messages:
    556
    Location:
    BENDIGO
    Currently 3 Techs on rotating roster supporting 3500 staff

    On-call Duration: From Friday 5pm through to Friday the following week 8.30am

    On-call Retainer: $163.70 (for the on-call week)

    Phone Support: Billed in 1 hour block x1.5 your hourly pay rate.
    Even if the call takes 5 minutes, this includes remote accessing back to work

    Recall: Billed in 3 hour block x2 for the first hour and 1.5 for the next two.
    Even if it takes minutes, this is being recalled on-site

    - Home Internet is paid every month
    - Self password re-set and account unlock portal about to go live
    - Laptop & 4g modem avail if needed, most times just access via vpn from home PC
     
    Last edited: Jun 22, 2012
  20. evilasdeath

    evilasdeath Member

    Joined:
    Jul 24, 2004
    Messages:
    4,676
    15k extra per annum paid quarterly

    24/7/365 on call

    phone provided (acceptable use policy exists for personal calls, but I only use this phone and never been questioned)
    home internet subsidised $50 per month
    most work 99% can be complete remotely via VPN I have other hands and feet to do field work.
    no official TIL policy however there is understanding with the managers so you work the previous night you aren't expected to be in on time, and its roughly 1.5>2 hours per hour of work.


    most calls are fairly short and few and far between but when they come they can be ball breaking.

    My last major call was 11pm on a Friday night and I didn't finish until 3pm the following day after doing a full days work on Friday.

    Fair chunk of my day work is to prevent things getting to the level they break that bad thou, so events like that are very rare.
     

Share This Page