1. OCAU Merchandise now available! Check out our 20th Anniversary Mugs, Classic Logo Shirts and much more! Discussion here.
    Dismiss Notice

Sustainted High IOPS 40+ Minutes

Discussion in 'Business & Enterprise Computing' started by bcann, Jan 4, 2016.

  1. bcann

    bcann Member

    Joined:
    Feb 26, 2006
    Messages:
    5,883
    Location:
    NSW
    Gday All,

    I'm finding in the last month there is something that is causing a sustained high IO on my file server which is a 7 disk Raid 6 10K SAS hdd system for 500+ IOPS for 40 minutes and then it dies down to nothing. can anyone point me as to how to find the process or whatever is causing this sustained IO for 40+ minutes on server 2012.

    Thanks
     
  2. PabloEscobar

    PabloEscobar Member

    Joined:
    Jan 28, 2008
    Messages:
    13,924
    Anti Virus :)
     
  3. NSanity

    NSanity Member

    Joined:
    Mar 11, 2002
    Messages:
    17,912
    Location:
    Canberra
    that would be my guess.

    Also

    Perfmon.
     
  4. Daemon

    Daemon Member

    Joined:
    Jun 27, 2001
    Messages:
    5,470
    Location:
    qld.au
    Resource Monitor -> Disk tab.
     
  5. NSanity

    NSanity Member

    Joined:
    Mar 11, 2002
    Messages:
    17,912
    Location:
    Canberra
    I use this *so* damned much now.

    It showed up in what? Vista? but its pretty much the answer to working out why a pc/server is slow these days.
     
  6. OP
    OP
    bcann

    bcann Member

    Joined:
    Feb 26, 2006
    Messages:
    5,883
    Location:
    NSW
    ok so it looks to be multiple excel spreadsheets with tens of thousands of data links in them that pick data from other excel spreadsheets with again thousands of links in them.
     
  7. NSanity

    NSanity Member

    Joined:
    Mar 11, 2002
    Messages:
    17,912
    Location:
    Canberra
  8. FiShy

    FiShy Member

    Joined:
    Aug 15, 2001
    Messages:
    9,682
    bahaha


    Where is that excel cluster when you need it.


    [​IMG]
     
    Last edited: Jan 4, 2016
  9. OP
    OP
    bcann

    bcann Member

    Joined:
    Feb 26, 2006
    Messages:
    5,883
    Location:
    NSW
    ok,

    so out of 10 problem spreadsheets I picked one which was of medium size for us here that they QA guys use.

    wait for it.....

    wait for it....

    wait for it....

    156,000 links in just ONE FREAKEN SHEET.

    All up 250,000 Links in one god damn spreadsheet........
     
  10. Iceman

    Iceman Member

    Joined:
    Jun 27, 2001
    Messages:
    6,647
    Location:
    Brisbane (nth), Australia
    I've found in IT, you either provide people a database to do the task they require.. or they'll ape one built out of shit and excel (sometimes access, if your MOE mistakenly installed it).

    Your choice ;)
     
  11. Daemon

    Daemon Member

    Joined:
    Jun 27, 2001
    Messages:
    5,470
    Location:
    qld.au
    Elvis rant in 3...2...1... :D
     
  12. NSanity

    NSanity Member

    Joined:
    Mar 11, 2002
    Messages:
    17,912
    Location:
    Canberra
    if we say his name 3 times does he come?
     
  13. ^catalyst

    ^catalyst Member

    Joined:
    Jun 27, 2001
    Messages:
    11,894
    Location:
    melbourne
    Lol quick introduce them to Access.
     
  14. OP
    OP
    bcann

    bcann Member

    Joined:
    Feb 26, 2006
    Messages:
    5,883
    Location:
    NSW
    Dear god almighty no we already have enough half arsed access DB's floating around.
    This is what happens when contractors rule the place for 20 years.
     
  15. elvis

    elvis Old school old fool

    Joined:
    Jun 27, 2001
    Messages:
    41,068
    Location:
    Brisbane
    [​IMG]


    [​IMG]
     
  16. s4mmy

    s4mmy Member

    Joined:
    May 20, 2004
    Messages:
    2,196
    Location:
    Melbourne
  17. FerrisXB9R

    FerrisXB9R Member

    Joined:
    Jan 18, 2005
    Messages:
    3,147
    Location:
    AB, CAN
    Bwahaha ya fucker Elvis.
     
  18. OP
    OP
    bcann

    bcann Member

    Joined:
    Feb 26, 2006
    Messages:
    5,883
    Location:
    NSW
    Ok,

    SO final proper root cause found.

    Offline File Sync storms.... found that about 6 users of my base had offline file stores containing between 10-60GB of files, one of whom had over a million files set to sync. So come 8:30-9:15am in the morning when everyone comes in and does their sync, our disk array was getting spanked hard for the better part of 40 minutes @ 1000+ IOPS (7 Disk Raid 6 10K SAS).

    Given head office controls our Offline File sync Via GP, I've put in a request that by the end of the month we enforce a 1GB limit on offline files to give folk time to move their 15+ years of family photos and WORK documents to a more suitable location.

    And yes the 160,000 odd link excel spreadsheets didn't help but were not really the real root cause.
     
  19. PabloEscobar

    PabloEscobar Member

    Joined:
    Jan 28, 2008
    Messages:
    13,924
    Nice Job, Did Resource Monitor point you in the right Direction?

    Also, props for posting back with what the root cause was :).
     
  20. OP
    OP
    bcann

    bcann Member

    Joined:
    Feb 26, 2006
    Messages:
    5,883
    Location:
    NSW
    I used Veeam one to plot the times and Disk IOPS and then cross compared that to Resource Monitor, specifically the disk section to see what was flying over the disks which showed bulk file reads happening in the usershare folder(Not shown in this screenshot below, as Disk IOPS is now normalised at this moment, but will still be a bit bumpy as I force people to relocate their files for the next day or so).

    Our average usage in Disk IOPS is generally 50-100 with the usual expected peaks and troughs.

    as per below:

    Click to view full size!
     
    Last edited: Jan 5, 2016

Share This Page

Advertisement: