Consolidated Business & Enterprise Computing Rant Thread

Warning: this is a rant. Turn away now if you are easily offended.

The last three years have left me scratching my head. I'm at a complete loss as how to explain the complete lack of quality and professionalism in an industry I used to be proud of.

I make a living as a sysadmin. What does that mean, to be a sysadmin? Well, where I come from it means knowing a lot. It means knowing how to config routers and networking equipment, it means advanced firewalling, DNAT, SNAT, it means knowing how to do traffic sniffing and deciphering packet-level information, it means knowing how to build and configure common services like SMTP/IMAP/POP/mail via a dozen different pieces of software on three different families of operating systems, it means knowing how to build clusters for high availability and high performance, it means knowing when to use CIFS, NFS, SMB, GFS and when not to and what the difference is between them all, it means knowing hwo to configure iSCSI, fibre channel, SANs, direct and non-direct storage, it means knowing SQL and getting information out of databases, it means knowing how to program in a dozen different languages and how to script and automate events in any OS to make life easier, it means understanding authentication and security settings, how to configure any directory service from LDAP to AD to NIS, it means understanding DNS is more than just a optional addon to look up system names occasionally, it means understanding encryption, knowing what terms like Diffie Hellman, AES, SHA1 and others mean, and what parts of the encryption process they apply to, it means being able to make everything you do completely redundant and fault tolerant, right down to you own job, and it means so much more.

Why is it then, that over the last three years I've seen fewer and fewer people who call themselves sysadmins understand these things? Why is it that I've been surrounded by "IT professionals" from junior sysadmins to CTOs who don't have a goddamn clue about one tenth of the above? Why is it that in three years I've met ONE person in professional IT who I would consider worthy of sitting down and having a conversation with?

Why is it that professional IT services today consist of service reps who tell you the things you are doing are untested, dangerous, unsupported, different, not usual, or a host of other words meaning they are scared shitless and unwilling to learn something new? Why is it that I spend my time building things people tell me for 6 months during build and test "will never work", only to have them go into production and work ten times faster for one tenth the cost of the old system? Why is it that IT professionals today choose brand labels over intelligence, and post-justify it by hiding behind "board confidence" when providing a solid, working, profitable system is the best thing to boost confidence from the board?

I tried switching industries. I've done IT in engineering, architecture, film and TV, retail, medical, finance and superannuation. Some of the places I've worked for have been fortune 500s. Some have been Fortune 5s. Did the quality of the IT staff go up? No. Was I met with people who were open minded and willing to learn new things to better the workplaces they were in? No. Was I met with fear, close-mindedness, and nothing but people who rattle off marketing bullshit as an excuse for not knowing technical information? Yes.

And every time I leave, I hear the same things. Some new guy comes in to replace me. Within days/weeks he's broken something necessary for production, lost terabytes of data, destroyed the backup/DR/recovery systems, spent hundreds of thousands replacing something that met the businesses' every need with some proprietary/generic piece of rubbish that performs half as well when there were dozens of other things that could have been improved instead. And all because they didn't take the time to understand the business, it's needs, and the solutions currently in place.

My latest job is no different. I've walked into a place that holds and controls financial data for over 6 million Australians, and around 50 million Americans. A place where data integrity is paramount, and system stability and performance is of the highest regard.

The hardware is provided by a tier 1, namebrand hardware provider (number 2 worldwide in server sales, I hear). The support guys who come on site are paid absolute buckets of cash and are supposedly the best of the best. These guys come out and utterly bollocks up installs. They constantly tell you things are impossible to achieve, only to stare slack-jawed in amazement three weeks later when they are achieved and working faster than their setups were supposed to provide. They rant and spit when I build things for zero-dollar licensing cost that their multi-hundreds-of-thousands-of-dollar hardware is supposed to be the only stuff that can do the job (my latest GFS/CLVM cluster outperforms their SAN snapshotting, and is free of charge compared to their pay-a-license-per-snapshot "solution"). And of course, their golden trump card is to say "well that's fine, but we don't support it" when you offend them. Watch the CIOs scramble when their hardware vendors threaten to not offer support! Yet ask them when they last called on the "professional" support (other than simple break/fix/replace stuff), and most can't answer.

I'm disgusted. I'm pissed off. Quite frankly, I'm over IT. I don't consider myself smarter than the average bear, and I don't consider that I have higher expectations than is realistic. I expect that "professional" IT people are professional. I expect that they have a desire to learn, a technical competence to achieve the tasks they set out to do, and a constant need to push the envelope of what's achievable and always move forwards. I expect everyone to have a goal to leave something in a better/faster/more efficient way then they found it. Yet it seems that the last few years have shown people in IT are by and large the complete and polar opposite. Don't get me wrong - I'm not some gung ho cowboy. I've met plenty of those (and sacked a few along the way). Being conservative with sensitive material is always a smart option. But letting it deteriorate is utter ignorance.

Many moons ago, I used to have a mentor. A man who quite frankly I considered genius level. I don't throw around words like "genius" frequently. In my life I've met three people who would rightly qualify as geniuses. Only one I've had the pleasure to work with, and more importantly learn from. In the small amount of time I worked with the man my rate of learning tripled. He had the right amount of sage advice coupled with the sense to let you make your own mistakes from time to time. Sadly the company in question got bought out, and the new owners were typical of all of my criticisms above. Within three months 50% of the IT staff left (myself and my mentor were two of the first). Within 9 months they'd spent 10 times our annual budget on a variety of completely unnecessary infrastructure, and completely ignorant and underqualified consultants (all of whom got the work via personal ties to the new owners, of course), and the company was brought to the brink of destruction. From what I hear this week, they'll be liquidated by the end of the year.

So when did this happen? When did "the IT guy" turn from the person who was cross trained with the breadth and depth of knowledge across a wide variety of systems and procedures turn into a drivelling half-wit who sees more value in a commercial certification than actually learning and building things, and who decides to be "the Microsoft guy" or "the UNIX guy" or "the Cisco guy" and learns nothing but one brand-name item to the ignorance of all others, and often poorly because they can't separate concepts and ideas from brand names and marketing acronyms?

When the hell did professional IT people stop being professional?

I've had a gut full. Something must come of this. The industry as a whole is in for a rude shock if it keeps going the way it does. We keep packing IT departments full of more people who know less. Things break constantly because unqualified people manage them, and departments stop communicating because the connecting technologies are always "somebody else's problem". The industry gets flooded with cowboys who have no concept of system and data integrity, who don't take care with the systems they are put in charge of, who don't bother securing things in a proper fashion so that data doesn't leak everywhere. It's almost a daily event to hear of some horrendously scary security breech that affects millions of innocent people who put their trust in these idiots.

Will there be a crash? Will there be a bottom to this rapidly declining curve? Will it get to the point where IT is just so shithouse that the people relying on it start to demand a certain level of competence? I look at other professional industries - lawyers, doctors, engineers, architects. All of these people need to maintain a certain level of education and prove a certain minimum of competence to practice their arts. My guess is if the same thing happened in IT, 2/3 of the workers would be out the door, and rightly so before they destroy something important.

Maybe I just need to get the hell out of this industry. I keep getting offered roles that are "bigger and better", but they just lead to paperwork hell. I want to stay technical and hands-on, but at the same time am just sick to death of the ignorance and stupidity I see every waking hour in this profession. Were it not for mortgage and children, I would have been out half a decade ago. But maybe it's seriously time to begin working on an exit strategy.

And of course, it doesn't help at all that I've been reading "Atlas Shrugged" these last few weeks. If ever there was a book to convince you to say "fuck the world", that's it.

If you've read this far, good for you.

If you think I'm some sort of uppity, pompous, self-righteous know it all, then you really don't know me. But flame away all the same. Like everyone else, you'll give me a list of reasons why I expect too much, why not everyone is good enough to know everything, blah blah blah. A long list of excuses as to why mediocrity is acceptable and nobody should strive for anything other than average. Your comments will be filed away with the rest of them.

End rant.

 

Mate, IT isnt the ONLY industry that all the above happens in.

 

I.T Is dead baby.

*But* i do have a plan. a company i work at does thousands of transactions a day, and theres all these remainders... like, 1/1000th of a cent.. and after a while they all add up.. what i'm going to do is open a bank account and put the money in there, are you in?


But err, seriously........

A while ago i dropped out of a dual degree electrical & comp sys engineering after a couple of years and went to TAFE.(start of 2005) out of a class of 20.. perhaps 4 or 5 i'd consider hiring

I.T is not worth it these days. shit job, shit conditions, most job advertisements are fake and/or the hiring person does not understand I.T technical roles, much better things to do..

If it was not for the rate i'm getting paid at the moment i'd be fucking off right about now

 

Granted. But it doesn't make me any happier about the situation, nor does it excuse the behaviour of most of the people in the industry right now.

 

haha sooo fcuking true.

When I was in high school I was actually TEACHING my IT teacher (who just happens to be the schools Network admin). The system was always down because he knew nothing about anything.

They booted him after I finished last year cause he couldnt hold it together himself.

 

I expect my comments to be taken with a grain of salt, but they are food for thought.

I used to be in the IT industry, and I used to be very good at all of the things you mentioned above. It meant no life though. It meant long hours, little to no appreciation (as people didn't know what you were doing half the time as they wouldn't understand it), it meant working in a dark room for days...

(I started coding when I was 6, I was elbow deep in the IT industry in my early 20s when the bust was on, I'm now 28 - you can do the math on when all of this happened)

The thing that kept me sane with all the idiots around me was the motorbike I had at the time. (weekends down on the GOR - nothing like it)

That is not who I am, that is not who I wish to be. I love being hands on, but not at that cost.

There are a lot of idiots out there, and over the last 10 years more and more people have wanted to become "IT professionals" without really knowing the ins and outs, or wanting to spend the time learning how to do the basic things.

I can't count the number of times I've read "RTFM" while I asked questions trying to learn *nix for instance, so what did I do? spend hours and hours RTFM and with sheer force of will, I actually got somewhere with it. I'd argue that a lot of wanabes don't have the get up and go and do that.

Perhaps, that drive is what keeps people away from doing all of that. Perhaps it is the taking ownership part.

The ownership problem is a big problem in just about every industry, nobody wants to take ownership of the problems they face because they could get fired over it, with a disposable workforce like the current one, simple wanabes are easy to hire and fire.

So when you look at the problem, the combination of not wanting to take ownership, because of lack of job stability if you do that, combined with not being loyal enough due to the lack of job stability - so they don't want to RTFM and do their own pushups thus they don't take ownership etc .. etc etc.. it's a cyclical problem.

What would it take to change the scenario? - A complete change of dynamics in terms of thinking when it comes to IT and management. IT workers should be paid for the hours they work, and the hours they study out of work for work or at least get tax cuts for it like some professions get.

This gives them the incentive to study. Now I'm not saying all study these days is good, cause a lot of the MSCE crap is just that, crap, there I said it. I think that there needs to be some unification and some serious thought put towards IT education (in a *nix environment). That said compared to say, commerce, business or law - IT is in its infancy.

Would this work? yes. But it would take a savvy group of people combining with the major players in the industry to put in place the infrastructure and they'd have to have the credibility and the IT contacts to do it - And not many IT people are very good networkers (and I'm not talking about the TCP/IP style, I'm talking people skills).

Few companies would shoulder a risk like this, but this is what it would take for the situation to improve.

 

Just to pick out something in your rant (well done by the way!). Arent mentors great! I had a mentor who I worked for in the UK for 5 years, me and him worked our proverbial "asses" off for our company - definitely after the first year he was not just my boss but a very trusted friend.

As an example, as is the culture in England, our weekly meetings were always held at the pub next door - very informal, where everything was freely discussed. All manner of strategy, ideas, problems, fixes, company politics... you name it, it was placed on the table for the two of us to nut out for the coming week/month. In the end the two of us became 10... and I was eventually rewarded by him and sent back to the colony, from whence I came, to head up the Australian IT department. I still chat to him regularly, to the point where I just got my hands on a nice video conferencing kit

In the 5 years I worked over there I learnt a damn site more than I ever could of thru uni/books, to me it was like an apprenticeship! I did not get paid a great deal but boy did I learn. We need more of this in IT.... what do you guys think???

 

I personally feel this is the crux of the issue.

What's the expected term of employment in IT these days? 2 years max?

The other side of the coin is, we're relying on IT more and more to do absolutely everything for everyone these days - Things are becoming mission critical in the smallest of businesses and no-one wants to spend a dime.

To get staff that are skilled in such a broad spectrum of areas in IT these days is getting more and more expensive - everyone is specializing.

This isn't exactly a bad thing on its own, training for the high-end MSFT stuff, RHCP's, Cisco/Juniper/Nortel stuff are getting more stringent in terms of testing/passing - its just the entry point and basic certs aren't the stuff that used to be the default.

I mean what the hell is the point of a MS Desktop Certified Tech? Should you really be in IT Support - even in a Managed Services company where they draw the line between level 1 support and desktop support - if all you can do is manage a Windows Desktop OS? Yes you could argue its a starting point - but so is flipping burgers. It's arguably the 2008 equivalent to the mid-90's Tafe IT Diploma that teaches you how to drive internet explorer.

 

What the fuck do you expect when some Sysadmins get paid 40k?

 

over-commercialization, IT as a commodity, production-line IT, corporate governance, over-regulation, prescriptive processes.....

.....I think this relates to the maturity of IT in general. I'm sure similar things happened with other industries.

We have progressed from the stage where you needed an 'architectural' mindset to put systems together, to an 'ikea' frame of reference where any monkey can put it together.

I don't have an answer for you, but I do sympathize.

Out of interest, have you looked into organisations such as ACS that promote 'professional' status of IT? I have my own opinions of the ACS - I'm keen to hear others.

Cheers.

 

Base Salaries at Managed Services are 35k.

and yeah... thats a point.

 

IT is such a new area, I can't even get my head around all these new fangled certifications that you MUST have in order to get anywhere in the industry, especially on the networking side, apparently.

I think a lot of it is to do with people thinking that IT is something a monkey can do, and it pays, so the caliber of 'professionals' is severly lacking in the present day as it was back before the big bubble.

Managers want IT nownownownownow to fix all there problems, but IT is a tool, not a solution. Narrow minded managers thinking something is the be all and end all will pay IT suppliers the bucks to do fix 'everything', when the suppliers aren't actually doing that much, because they don't need to. Or they genuinely don't know what they are doing.

*sigh*

If you have the experience, you can get anywhere in the industry. It just depends on if you want to go into the lions den..

 

qft. passion is more often than not, proportionate to income.
when i started out working on phone tech support at an isp that actually needed it's phone techs to know what they were doing, on 16.50 an hour, i had vastly less drive to learn or strive for more know-how.

 

what state?

 

Hahaha.. one IT company (that will remain nameless) offered me 20k+ super for a "graduate" type role to do cisco support... this is laughable but it represents everything that is wrong with the IT industry.

 

Couple of very good points mate,

Im just starting out in the industry, and there are a lot of things i dont like. Firstly, its hard to find a decent mentor. It seems like no one has the time or the motivation to make the effort of training the younger generation.

I've also found there are a lot of IT people who know their field very well, but thats it. They lack communication skills, personality and motivation. In my very short time in IT ive tried to be as friendly and approachable as possible to non-IT colleges.

Its one thing being able to setup and maintain a network, but it you can talk to the CEO or the users then you have limited use.

I'm trying to learn a little of everything, obviously windows, linux, networking and hardware.

I guess it depends on the size of the business... A smaller business needs an all rounder, someone who can take care of multiple systems and tasks. When you look at larger firms they are able to employ a bunch of people with a specific role each.

The money is pretty poor though, these days you can earn better money as a brick layer than a systems administrator (depending where you work and how good you are).

Back in the day an IT professional would be regarded as one of the highest skilled professions, up there with medicine, engineering, law etc. These days all you need is a cheap microsoft cert ($1000 and a few hours training).


And what happened to the days of general intelligent people. I must admit my dad is one, he has the technical skills of a project manager, yet knows how to rebuild a car, build a fence, design a garden etc etc (i know this as he built our house, from plastering to tiling, the garden to the roof. Everything he was legally allowed to do, he did).

Seems to be a dying breed... Someone technically smart who can do anything. Most of the IT people ive worked with couldnt change a tire, let alone service a car.

 

feistl you are definitely doing the right thing, being able to talk to both end users and management alike is an acquired "art" in itself... a social art which does not always lend itself to those working in IT... the reason why I added the company "politics" aspect It is not to be under-estimated!

The average length of time for a helpdesk person is, I believe, 18 months... not even 2 years before burn out! Often it is this timeframe before people move on, and start skipping jobs, on their way up the corporate ladder. I believe it is hard in todays society to find someone willing to put in the hard yakka to learn about a company and stay with it. I believe finding a mentor is made that much harder because of this and it needs to change.

Personally I am looking forward to the day I get to employ my own assitant, but guaranteed he will be getting the shit kicking jobs.... to me it is the way you learn about life and the company you work for.

As they say, you do not learn by fighting on the side-lines but from the trenches!

My god, after reading that I sound like a prophet.... sorry for that....

 

My solution was to become the outside contractor giving random advice for an hourly rate, and make easilly twice what I used to being on the recieving end of this advice.

It is a about as soul crushing as it sounds, at times, however.

 

I used to be a Sysop, then a Net Admin, then a provisioner/installer (all for the same global provider), then moved into 3rd tier support and now technical sales. I like ot think I've seen the industry from a number of angles, and elvis is very right - things are going to hell in a hand basket.

Gone are the days of sysops, sys/net-admins being expected to get their hands dirty when things go wrong - now it's all about calling the provider/vendor and demanding they fix it for you. Not only does it take longer, but you're then stuck with whomever they decide to send out, and whatever fix if flavour of the month.

I have fond memories of the days in a server room where I got to roll up my sleeves and get stuck into fixing a problem. It was those days that taught me the skills that let me develop, and gave me the confidence to take on new challenges without worrying about if I had the qualification to do it.

I too had some great mentors. Unfortunately a lot of these guys are moving out of the industry, or at least out of the hands-on roles, meaning much of their knowledge is being lost. Many of the new starters only looked at IT because they thought it was easy money, and don't want to spend the time to learn. They seem to think they deserve a good mentor from day one, along with personal tutition and a corner office. It seems there is no concept of proving yourself first before expecting the rewards.

I've had the joy of interviewing "paper warriors" who have a laundry list of qualifications with absolutely no real-world experience. They have no concept of how to work with limited resources or scope whilst troubleshooting, or finding the quick fix that gives you time to work on the better solution.

Now there is an oversupply of overqualified applicants which means the more entry-level positions (and even many mid-level positions) are being under-valued. Those with the intellect and drive to succeed are finding greener pastures and leaving IT to those who are willing to take minimum wage. Still don't know why I'm here though...

 

most of SAU are i.t geeks