Business IT COVID-19 response

Discussion in 'Business & Enterprise Computing' started by elvis, Mar 25, 2020.

  1. elvis

    elvis Old school old fool

    Joined:
    Jun 27, 2001
    Messages:
    40,426
    Location:
    Brisbane
    Let's start taking notes. What are we seeing out there in the wild? What's working well? What's not working well? Are we keeping staff working at high capacity, or are we failing to get them connected? Are we doing it cleverly and securely, or are we taking huge risks?

    Document your wins and losses here. In a couple of years, let's look back at this crazy time and see what we've learned, and what we've forgotten.
     
  2. looktall

    looktall Working Class Doughnut

    Joined:
    Sep 17, 2001
    Messages:
    25,389
    We've gone from face to face appointments with public to conducting them via phone.
    The problem is for many of those meetings we need to confirm who we're talking to.
    This is most easily managed by a visual observation. Ie some sort of video call.

    We haven't provided any of our staff with a way to do this.
    At the moment it's a haphazard mix of apps (mostly whatsapp) on mobiles that the users are downloading and installing themselves.
    This despite the face we have a perfectly functional MDM that we could use to push apps to the phones.

    There's no way at all to do this from our desk phones.

    Edit: we also haven't been proactive in supplying headsets for our users who have gone from using their phones maybe 10-15 times per day to essentially being on it all day.
     
  3. QuakeDude

    QuakeDude ooooh weeee ooooh

    Joined:
    Aug 4, 2004
    Messages:
    8,479
    Location:
    Melbourne
    Microsoft Teams, people. Speak to Microsoft, they're providing it free for 6 months to businesses, and its a great way to not only video conference but IM functionality.

    We're fairly lucky (in head office) as we're on Skype for Business for the phone system, so people can go home, log in and their phones effectively follow them. I've spent the last two weeks getting everyone in head office onto a laptop so that they're portable. I guess our biggest learning was BCP - everyone tends to assume you need to plan for a site being offline, but no one plans for something like this where you have to separate all your staff, so we're putting anyone critical on laptops going forward.

    In our stores - right now its business as usual. We have to supply the sparky's that support critical infrastructure, so we're not closing stores unless absolutely everything stops.
     
  4. NSanity

    NSanity Member

    Joined:
    Mar 11, 2002
    Messages:
    17,865
    Location:
    Canberra
    My work has been WFH wherever possibly - globally - since the start of march, with a view that it will be that way till May 31 at least.

    We have had restrictions on travel in place since early Jan. Updating almost daily at this point - currently even domestic travel requires senior exec vp approval - this includes Mass Transit (Buses, Trams, Trains) and Ubers/Taxi's. To work onsite at a customer is almost impossible.

    Strong focus on work from home (we have this policy anyway for vast swathes of our workforce), stay out of customers/offices if you feel unwell, paid time if you need quarantine - as well as tons of crisis support if you require. All sites have had hand sanitiser on building entry, every lift and multiple points throughout floors.

    In terms of technology - all remote users get laptops (outside of our call centers - however they are working to enable call center work from home), dual screens for home, external kb/mouse, dock, headset (basic). This has been ramped up to a wider audience in response to Covid-19.

    Tech stack is Internal S4B, Zoom, Sharepoint and a number of SaaS providers for a lot of things. We do leverage VPN for a number of web-based "legacy" systems (i.e not SaaS) - which hasn't struggled in response to a larger user base hitting it.

    We had an internal call for ANZ recently, highlighting that our productivity hasn't really dropped since moving to this model, and that leadership is somewhat impressed at the ease of which we've transitioned to this approach of work.

    *edit* one thing i should add - is that this is the first time in over a decade that i haven't been personally able to impact/interact w/ how the business delivers IT internally.

    I'm just a user :)
     
    Last edited: Mar 26, 2020
  5. bcann

    bcann Member

    Joined:
    Feb 26, 2006
    Messages:
    5,861
    Location:
    NSW
    Like most IT folk, i've been telling the company given our regional location we could literally work from anywhere, even the proverbial beach on Hawaii, with a laptop + Mobile phone.

    We are fairly secure, i didn't relent on making things easy for lazy folk, but stood my ground. Everything lives behind the VPN, and we run a software PBX. Our only real issues is that the laptop upgrade program got killed about 18 months ago, so were a bit tight on laptops, but we have about 60% of folk working from home other then those in critical roles, who are office bound.

    Our only real issue, is that we have such a low IT skill base (Because we insist on spoon feeding idiots) that we could literally be 100% work from home with our phone system and laptops, but its "Too Hard" and one person had a "Bad experience" using 3cx on their mobile phone who is part of the upper part of management, but unsurprisingly enough they can barely operate a mobile phone as well.

    Look at the end of the day, if we Had a higher IT skill rate, and more laptops, we wouldn't even need office space. Unsurprisingly enough when people were forced to actually plug their laptops in themselves at their own home and connect to their own wireless, they could without needing their hands held, when they were told, either do it, or take paid leave.... surprising what having to lose your own paid leave does for your ability to pull your finger out and do things as opposed to the way it is normally done, where you can blame IT, after having done absolutely nothing...

    Like i believe has been said in the IT crap thread, if you can configure your own phone, customise the background and put in custom ringtones and sign up for facebook, you can damn well plug your fucken laptop in and use our vpn software and RDP... Hell now we just need to force them to use the soft phones and we'd be set.

    The only real issues we've had have been 3rd party provider issues, but these were foreseen as the country went stupid, and conducted 1000 unnecessary meeting in the first few days of this week, which ground our conference provider, and we have had a few mobile phone contact issues as mobiles were then used to make all calls, but again this was expected. i suspect for the most part, by mid next week when people settle into a groove most things will jyst work without issues again.

    Probably one of the lucky things we have going for us is our location. Its all FIBRE baby.
     
    Last edited: Mar 26, 2020
  6. power

    power Member

    Joined:
    Apr 20, 2002
    Messages:
    62,166
    Location:
    brisbane
    we're currently almost completely business as usual. if i didn't read the news i'd barely know it's a thing.

    and i'm grateful for this.

    I work for a metals supplier for those interested, so we supply smaller trades and other commercial businesses predominantly.

    The funny thing is that apart from phone and face to face sales/retail desks we could all work from home, but we aren't.

    I literally have a laptop at home all the time that has a dedicated 4G modem and automatic VPN connection that I could be working on.

    We have many such machines, most are already used but for our external sales reps who are almost all being turned away by the customers they normally visit
    so these machines most likely are sitting in our retail stores on desks being used by people who could just as easily be using the machine that's already on the desk.

    The IT office can also divert any line to any other phone at will as we centrally manage all PABX here.
     
    bcann likes this.
  7. GumbyNoTalent

    GumbyNoTalent Member

    Joined:
    Jan 8, 2003
    Messages:
    8,311
    Location:
    Briz Vegas
  8. miicah

    miicah Member

    Joined:
    Jun 3, 2010
    Messages:
    6,797
    Location:
    Brisbane, QLD
    We have zero policy on how to reset passwords remotely. It's a big issue with kids/parents/not with student issues. The only recourse I have is to check their name and that they are listed as an allowed contact.

    WFH is impossible as a tech in QLD public schools.
     
    icewind likes this.
  9. OP
    OP
    elvis

    elvis Old school old fool

    Joined:
    Jun 27, 2001
    Messages:
    40,426
    Location:
    Brisbane
    Context: I've joined my wife's company as of July last year, consulting about the place. Seeing a wide variety of stuff going on. Some snippets:

    -------------------

    Previous workplace (film, VFX, advertising, multimedia, on-set, audio production, etc) who I still catch up with - we put extensive efforts in about 5 years back to completely overhaul the network. Turned an organically growing mess into a properly planned out, neatly VLANed and subnetted arrangement, and matched that across numerous sites (Brisbane/Sydney/Tokyo for main offices, dozens of other sites for smaller location setups).

    One of the big changes was WiFi - we disconnected that from production, and made an Internet-access-only VLAN for it. Getting from WiFi to production was only possible by VPN (OpenVPN). This achieved three things:

    1) WiFi was more secure than alternative methods. Users stopped sharing AD/RADIUS passwords with guests to give them WiFi (despite there being guest account details on the wall of every office!), and OpenVNP is a step up from WPA2.

    2) VPN could be tested on site without a 4G dongle. Made it easier to train users on site and send them home with fewer issues remotely

    3) Almost 100% coverage of OpenVPN on laptops, almost 100% coverage of admin staff with laptops, which meant all admin staff pretty much could work from home in a pinch, and frequently did once they found out it was possible.

    For production users on beefy HPE workstations, they're currently using HP RGS:
    https://www8.hp.com/au/en/campaigns/workstations/remote-graphics-software.html
    https://en.wikipedia.org/wiki/Remote_Graphics_Software

    Appears to offer them excellent latency and speed (single 4K or dual 1080p at 60FPS on WAN links, and 30FPS comfortably on most home links). Their site bandwidth is good but not unlimited, so they're prioritising critical work first. Works on both Linux and Windows (either client or server). Perfect for them, as they have numerous production Linux workstations running higher end versions of thing like DaVinci Resolve (the toy version runs on Windows and Mac, the big grunty version on Linux only).

    Most of the industry has also been investigating Teradici for an age, as it works very well with high speed video and 3D. Downside is its quite expensive, versus HP RGS which is "free" (if you have a HPE workstation as the server side):
    https://www.teradici.com/

    And we were for a period investigating Teradici to virtual workstations on Google. It didn't work for them financially, but I'm hearing of quite a few "pop up" VFX studios here in Australia now doing this in production:
    https://www.teradici.com/google/

    All their Email, video chat, groupware and documents are GSuite (I converted the site from a utter shithouse Exchange setup 7 years back), so that box ticks itself. Zero issues for them. Finance moved to Xero a few year back, so again that's easy enough now that a few financial years have passed and legacy finance systems are sitting in read-only mode and rarely used.

    -----------------------------

    Another media client, this one all editorial (no 3D or high end VFX). I've been slowly taking them through a bunch of industry-specific security and compliance stuff. That meant dumping their (quite antagonistic) MSP and putting in our own kit, including a few HyperV nodes and AD (correct, no centralised auth at a prior to my work, complete AD greenfield), a pfSense firewall and OpenVPN. Great timing, as the OpenVPN part went live this week tested under fire (pfSense/AD has been going for 6 months).

    Users are using a combination of RDP (Windows) and Mac remote desktop (basically VNC) to keep working. Not great, but it's the zero-dollar-budget solution they've got right now without much planning. This will be an evolving challenge.

    Again, all email/chat/groupware/documents on GSuite, which was in effect before I helped them out. That's working as normal. No idea what their finance team uses, as it's not part of my brief (yet).

    --------------------------------

    Large government site. I'm told 40K users all up, looked after by a centralised IT group who I have nothing to do with. From what I can piece together, they migrated entirely to O365 (including Azure hosted AD) about 6 months before I rocked up.

    Upon landing, I found about 25% of users were O365 savvy, and the other 75% were still doing everything "the old way" (legacy systems hung around, legacy interfaces to new systems stayed to "ease the transition", which honestly only prolongs the pain). I can't blame the IT provider at all here - from what I can tell they put out extensive documentation, including hiring a full time employee to do nothing but email out very well presented how-to emails, portals, training platforms and the like (I even learned a few things myself!). So I can only file that under user ignorance from the small window of perspective I have.

    I'm working directly with about 300-400 people, focusing closely on 50-ish who I consider the power users. My job is to bring their clustered compute system out of the 20th century and into the 21st, which is going well except for the excessive security requirements placed on everything. Word as it once upon a time this network sat within spitting distance of another that had sensitive information on it, so they locked everything down under the same rules. I've mostly been clawing rights back to open things up a peek and allow users to actually get work done on site (not even remotely at that point). Literally 30+ million bucks worth of compute lying around unused and zero load average because people just couldn't get on and do things. So that's been going well, and people were using it more and more. COVID19 hit and they all went home. It's taken me about a week to find the right channels to beg and grovel to, and we've gotten a few more secure remote protocols open (all via official VPN tunnels already in place mind you). So as of yesterday we've got users on the system from home, and they're trucking along again. Big win!

    I'm refusing to talk to anyone over any channel other than O365 (whether cloud shared documents, email or Teams). We do have a handful of users who, for some reason I can't explain, use time-limited Zoom meetings (times out in 15 minutes or so, everyone dials back in, over and over - why???). Again I think it comes down to ignorance rather than a specific choice they've made ("Teams has video? Wow!"). This week has been a considerable learning experience for all, and Teams usage across the org has exploded in a positive way. There's even a "water cooler chat" channel for everyone to shitpost in (which can be muted if you need to work), which is keeping a few folk sane.

    There were phone tree tests that went on last wee, but I've had zero phone calls. Almost everything is now Teams based, and even the biggest whingers and nay-sayers are finally breaking down, using the tools, and their complaining has stopped.
     
  10. QuakeDude

    QuakeDude ooooh weeee ooooh

    Joined:
    Aug 4, 2004
    Messages:
    8,479
    Location:
    Melbourne
    elvis likes this.
  11. cvidler

    cvidler Member

    Joined:
    Jun 29, 2001
    Messages:
    13,091
    Location:
    Canberra
    I work for a software vendor, I see lots of customers across sate/federal Gov, and private.

    Common thing I've seen is while most already had a well supported remote access system (and policy/procedures) in place, none were scaled to cope with everyone working remote, and not able to scale up quickly enough (for various reasons, hardware, licensing, connectivity).

    I only deal with IT people, so don't see the user impact (thank fuck). Hard to believe people can be so resistant to change/dumb.


    I now have a bag and phone full of 2FA tokens. One plus out of all of this, everyone has 2FA on their VPN/remote access.
     
  12. waltermitty

    waltermitty Member

    Joined:
    Feb 19, 2016
    Messages:
    1,176
    Location:
    BRISBANE
    software dev, still use JIRA, DevOps, TFS, all the cloudy shit from home. Teams and internal S4B for meetings.

    only shit thing is no linux S4B client (or even a web one) so I've got to remote into my laptop to do Skype meetings (with headset paired)

    i'd say I'm just as productive as usual
     
  13. Bradzac

    Bradzac Member

    Joined:
    Aug 17, 2003
    Messages:
    1,740
    I work for a Software Vendor as a Principle Software Engineer in their AI/ML team. The company is 100% working from home as of today (doesn't affect me, I've been 100% WFH for a number of years). Offices are having access restricted from COB tomorrow. This is world wide, so 13k+ staff.

    Since our team is remote first and most of us already WFH we've provided a little bit of guidance to some of our subsidiaries during the transition, that started a few weeks ago.
     
    Last edited: Mar 26, 2020
  14. Fred Nurk

    Fred Nurk Member

    Joined:
    Apr 5, 2002
    Messages:
    2,227
    Location:
    Cairns QLD
    We'd rolled out a document management solution a couple of years ago, and one of the feature it had was ability to set up additional security and access around its desktop clients. I'd rolled out a deployment about 18 months ago for laptops, but when the directors enquired about people taking desktops home I redeployed the laptop settings to the company desktops. End result, anyone with a company PC can remotely access the Document Management System from anywhere, complete with preshared key setup that blocks any non-company desktop from connecting.

    For various reasons though, there is still discussion around using RDP or other fuckery to access shared folders. IT is not my job though, so that aspect is out of focus, and someone else's problem for a while.
     
  15. QuakeDude

    QuakeDude ooooh weeee ooooh

    Joined:
    Aug 4, 2004
    Messages:
    8,479
    Location:
    Melbourne
    I'm interested to know whether anyone has considered the impact on their business given that alot of companies are going to be freezing any spending. I know for us, we're halting any non-critical projects for 12 months in order to stockpile cash to keep the business afloat for as long as possible. This will undoubtedly have a flow on effect for our suppliers / vendors.
     
  16. DivHunter

    DivHunter Member

    Joined:
    May 31, 2005
    Messages:
    3,158
    Location:
    Melbourne
    Software dev firm so everyone could already work from home.

    In some areas we have been picking up new clients because of corona.
     
  17. OP
    OP
    elvis

    elvis Old school old fool

    Joined:
    Jun 27, 2001
    Messages:
    40,426
    Location:
    Brisbane
    I've been dealing a lot with HPE for the last 2 or so years, and their desperation is very evident. Cloud is hurting their bottom line greatly, and I dare say that they're going to be suffering because of exactly what you mention here.

    It's too late to join the cloud race now. If you're not Microsoft/Amazon/Google (or smaller specialist players like Zoom), you're dead in the water now if you've been relying on slinging tin. Anyone in the software development space who hasn't made remote/mobile/web accessibility a priority of their tools is also in for a hard time. Conversely, anyone who was ahead of the curve here is in for a boom year.

    I mean, all of this was coming anyway. But these circumstances have been one hell of an accelerent, turning a small campfire into a forest blaze overnight.
     
  18. waltermitty

    waltermitty Member

    Joined:
    Feb 19, 2016
    Messages:
    1,176
    Location:
    BRISBANE
    we were fucked before with not a lot of jobs in the pipeline and now we're probably extra screwed, i'm 100% billable on a remote project in another office but not sure for how much longer
     
  19. Raptor_Eye

    Raptor_Eye Member

    Joined:
    Sep 19, 2002
    Messages:
    2,512
    Location:
    Bathurst NSW
    I'm on the Business Applications side however my Infrastructure Colleagues had fun buying and deploying 14 off the shelf laptops.

    They are considered throw away's and as such are a security risk as they did not have the time to Encrypt them with such a rush to get them deployed.

    Can see this coming back to bite us in the future once all the dust settles.

    Especially as auditors have zero sympathy.

    Oh well just glad i'm not responsible.
     
  20. GumbyNoTalent

    GumbyNoTalent Member

    Joined:
    Jan 8, 2003
    Messages:
    8,311
    Location:
    Briz Vegas
    Yep, main income stream is being put on ice end of next week, till undetermined point in time. They weren't too impressed when I told them there is no guarantee I will be available at their beckon call sometime in the future, as the past 2 years they paid monthly for the privilege, and got shitty when I explained its just business which was their explanation for putting the project on ice.

    I expect a lean 6 months, which in my bidness planning was always to have 12-24 operational capital for down turns in the economy, so 3 years into my bidness plan I can cover 24 months. :)
     

Share This Page

Advertisement: