Consolidated Business & Enterprise Computing Rant Thread

Discussion in 'Business & Enterprise Computing' started by elvis, Jul 1, 2008.

  1. Daemon

    Daemon Member

    Joined:
    Jun 27, 2001
    Messages:
    5,399
    Location:
    qld.au
    I've been in IT now for over 20 years. Only one place (a university) used multiple domain suffixes and that's back in WinNT 3.51 days when you required WINS servers to resolve anything.

    Again, if it was such a common thing or a massive issue, it'd be patched, updated and released with support. Things like split DNS are supported (something that certainly didn't work with old systems), but not multiple search domains from DHCP (works if configured direct). It's just an indication of what's in most environments and most don't use search domains at all.
    My post came off as too much of a personal attack and for that I apologise. It sounds like you really do need a decent break and hope you get it.

    However, I will call a spade a spade. You didn't investigate the true issue correctly and were unwilling to try something new. Have a look at your original post which started this thread over 10 years ago. You were sick of people who weren't willing to learn and prided yourself on knowing things which many others didn't and pushed the industry to hold themselves to higher standards. Sometimes, we need to take a look in the mirror and decide if we're part of the problem or part of the solution.

    My last major change from working for a bit corp back to a smaller company was exactly this. I could feel myself slipping standards wise, my skill levels dropping and my ability to handle the political bullshit of a 20,000+ org slipping quick so I jumped before I climbed up the clock tower.

    So again, enjoy your break and I hope work doesn't break while you're away. Feel free to PM me if you're looking for a change or even want a temporary change in work environment, always happy to help fellow OCAU members.
     
    freaky_beeky and shredder like this.
  2. OP
    OP
    elvis

    elvis Old school old fool

    Joined:
    Jun 27, 2001
    Messages:
    34,636
    Location:
    Brisbane
    We use it extensively. As I've alluded to before, our business is complex - closer to about a dozen businesses under one roof. To go with that are complex business rules including combinations and permutations of which business units can talk to others, and their data.

    On top of that, performance is king for us. We spit things up into VLANs not only for security but also performance. From there, if I want a bunch of things to talk to very fast storage, I want them to do so on their own VLAN. I need both DNS round robin to spread load, and I need to tell dozens of different systems to mount up storage by short hostname, and from there it resolves the right long hostname to find the right interface on the right VLAN. But then there'll be shared systems on other VLANs (license servers, utility servers, etc) that need resolving too.

    Unusual? Sure. Necessary? Well it solves a lot of problems that would otherwise require an enormous amount of specific per-system, per-business unit config. So this way we write things like Puppet classes once, and cover off a bunch of business problems with the help of DNS.

    And for what it's worth, it's the second business I've worked in set up this way. The first was a large retailer with 43 sites (and growing, as I left) where every site was set up in a cookie-cutter way, again using multi-domain DNS resolving to find things in that same sort of "we need to access some of the things from all of the sites, but not always the same way" way. It worked marvellously there too.

    What these two businesses had in common was the enormous amount of work they were doing with very few IT staff. We're covering off half the globe with fewer than a dozen IT folk, doing hundreds of shows a year from shitty reality TV to blockbusters like Aquaman and something like a half dozen Netflix shows right now. The retail mob was 2 sysadmins (1 of which was me) who did *everything* - all security, networking, systems, BCP/DR, database admin, systems admin (and this was pre-cloud, so we hosted it all in-house), and all on a $0 budget thanks to open source, configuration management, and some clever DNS stuff.

    Eh, not quite. What actually happened was we relied a feature, it didn't work in testing of an upgraded OS, I solve it in about 5 minutes 6 months ago, and don't recall the specifics beyond some quick notes in a wiki.

    I know you've got a bee in your bonnet about folks who don't like systemd. I use systemd every day (and have done so for a few years now). Honestly, I don't give a shit. It's here, I've migrated, I've adapted. And I've spoken before about the fact that I think all software sucks, so systemd doesn't get an award for sucking too. (But it is fun to take the piss out of Lennart, because every film needs a good villain).

    The systemd/resolvd/netplan/ubuntu18.04 synergy I mentioned (which now appears to be blowing up into far more than it deserved) was a hiccup for a minute that broke a standard I relied on. Fixed it between coffee number 4 and 5 one morning, and I moved on.

    Problems like these happen by the minute for me. I fix them, I have a tanty in this thread to blow off steam, I move on. I think perhaps by this point it's all been blown out of proportion. It's all working fine now, and I've solved a thousand other technology and business problems in the nine months since. And sure, most of the time I don't stick around to do a deep dive on the problem. That's what happens when you're sitting at 600+ tickets in the queue. It sucks, but this is what the business wants. If they wanted me to do deep dives on problems, I'd have even one person to help. But I don't.

    Thanks. It's super stressful at the moment because we're lighter on tech staff than ever thanks in part to a few senior folk moving on (all coincidence - some following their loved ones overseas, and an explosion of work taking our in-house staff to far away lands to look after running shows). I booked this holiday a long time ago and it all seemed fine, but mere weeks before it happened we're stretched super thin by sheer surprise (honestly, creative clients can't plan to save themselves).

    The poor bastard that is doing my job while I'm away has his own mountain of shit to deal with, and now he's got mine too. So I feel guilty as fuck for dumping this on him, but at the same time I repeat what I said above - if the business wanted better, they'd give us more people. I genuinely think that our team performs miracles by the hour. And as the IT&Ops manager constantly reminds me, it's been 6 years since the last time senior management uttered a single complaint about IT. So honestly, I know that they know the sorts of bullshit we deal with, and their complete lack of criticism of our team speaks volumes. (And they're not shy about criticising people, trust me).

    A far away island with unlimited food and alcohol there and back. I've got a good book, a bunch of podcasts, and no phone or Internet access.

    And after years of stress, not much sleep, and bad eating, I'll be committing to better physical health on my return. Got a few extra kilos on me since my last health kick (and this holiday won't help), so they have to come off again. But that's February-elvis' problem. January elvis is gettin' krunk!
     
    EvilGenius, j3ll0, millsy_c and 2 others like this.
  3. Daemon

    Daemon Member

    Joined:
    Jun 27, 2001
    Messages:
    5,399
    Location:
    qld.au
    I have two issues. The first is those who attribute fault to the wrong thing. We already have customers and management for this, we don't need experienced IT staff doing it as well. You got it wrong and that's not good enough if you're going to rant about it.

    No issues with reverting back to and older solution to solve a problem and move on, that's a daily occurrence for most others as well. "Did not investigate, found a way to revert to fix" is a perfectly valid response. Blaming the wrong system for causing the fault is not. There's a clear difference.

    The second is that I exception to most of the false issues with systemd because again, I hate IT arguments that are based on feeling not fact when it comes to functionality. Different case when it's for policies, business decisions, finance and so forth but when it comes to systems we can have a pure factual discussion and rationale. Systemd exists and is the accepted solution because it solved more issues than any of the other init systems (by a long shot). This is why it gained such acceptance and the only ones who tried to buck against it were those who hate or fear change. They're wrong.

    I want to see higher standards in IT and better recognition for the work done and part of that comes from ensuring we push to be better ourselves the whole time. I also hold those who preach the same to higher standards, just the same as I expect (but get sadly disappointed) that politicians should be held to a higher standard as well. If we don't do so, then the external view is that we're no different to rambling politicians pretending they know something about their portfolio too and it's simply not good enough.
     
  4. OP
    OP
    elvis

    elvis Old school old fool

    Joined:
    Jun 27, 2001
    Messages:
    34,636
    Location:
    Brisbane
    So for the record, which of systemd, systemd-resolved or netplan is the broken component, and who was the primary author of said breakage?

    On top of multiple search domains, we also had many issues resolving anything other than A and CNAME records. It seems systemd-resolved just doesn't even bother with anything else? (Had me baffled for a while wondering where my DKIM/SPF stuff went). Whether it worked or not was entirely arbitrary it seemed (could never build a consistent test case).

    Or am I once again blaming the wrong component?

    And adding to all that, OpenVPN-fed DNS stuff to systemd-resolved seems to be a mess. The official update scripts/libraries don't appear to work (particularly when the tunnel is torn down, and it locks up network-manager in an odd way forcing you to restart the daemon to bring the tunnel up again), which is a world of fun.

    Oh yeah, and that other problem we had for a bit where switching networks and getting a new bunch of DHCP-fed settings would append the new settings rather than replace them , so laptops moving between locations needed a service restart to flush the old settings whether waking from sleep or not).

    So, my glaring incompetence aside, something as fundamental as DNS appears in a bit of a bad state at the moment. Am I the big evil here for rejecting these new tools and going back to the archaic ways? Or should I just embrace the change and not use the "unusual" features available in DNS, such as "more than one zone"?

    My biggest criticism of all of this new stuff is that it appears tested by people in an environment no more complex than a trivial home network. That's a fairly broad, blanket statement, but it appears to be proven time and time again by each new bit that appears, and fails to work outside of a very small environment. I genuinely don't notice any of these problems at home, when all that requires is basic web browsing.

    Or do I blame Ubuntu, as you did above? One would expect a latest-stable-LTS to be a bit better than this. You got pretty angry at me blaming the wrong thing. Is it right to blame Ubuntu? Or should I be pin-pointing systemd-resolved (ensuring I don't unfairly blame the parent systemd project, because that would be short sighted, right?).

    I mean, shit, this works in Windows 10 (multiple search domains, OpenVPN integration with DNS, resolving all DNS records as expected, DHCP-fed domains appending/replacing as expected - all tested and working in Win10). That's a slap in the face. Although it's worth noting that macOS is fairly inconsistent too (aggressive DNS caching is a huge problem - running system tools to flush DNS as root don't even work, and it requires a full reboot to clear). And given systemd's overall penchant for wanting to emulate macOS's launchd, I guess that is a good summary of how things are currently.

    I'm glad. Because it's not working as it should. Unless it's my incompetence at fault (whether that be the "unusual" server-side configuration I put in that's worked forever, or my inability to configure the distro defaults that don't work out of the box, or my incorrect blaming of things).

    I understand why you're upset by people unfairly hurling throw-away statements at systemd and blaming it for bad weather on the weekend and their steak being overcooked, but there are still genuine problems with this ecosystem. We all understand what systemd is trying to fix, and the good it has done so far, but we all also now have a bunch of broken shit to deal with that seems to be collateral damage. I get regressions happen, but DNS not working as expected is quite the doozy.

    My genuine apologies to you for not giving all of this the due time it deserves and ranting about it incorrectly and unfairly. Now settle down, understand I'm on the same side as you, and that all we both want is Linux working better. Even if I'm tired and overworked and can't give things enough thought in the moment, my end goal is still the same as yours. Regardless of me blaming netplan instead of systemd-resolved (assuming that was the error), shit still be broke, yo. And the answer there has to be better than "dude, *nobody* uses multiple search domains". Because they do.

    [edit]

    Checking systemd issues on github, plenty of systemd-resolved / DNS / FQDN / search domain / DHCP bugs filed there. Appears I'm not the only one who needs this, and it appears this is still a problem despite my "legacy" 2018 software.
     
    Last edited: Jan 13, 2019 at 12:33 PM
  5. wintermute000

    wintermute000 Member

    Joined:
    Jan 23, 2011
    Messages:
    1,527
    Well its not netplan, netplan is just calling whatever underlying components which in your case was systemd-rsolved and hence that is the culprit.
     
  6. OP
    OP
    elvis

    elvis Old school old fool

    Joined:
    Jun 27, 2001
    Messages:
    34,636
    Location:
    Brisbane
    So don't bash netplan, bash systemd and its ever-expanding set of code that replaces working things with non-working things. Noted.

    Adding ticket for myself to action when I get back: replace our existing business-defined, multi-domain network with a simplified single domain to keep systemd-resolved happy, and conform to "most environments".
     
    Last edited: Jan 13, 2019 at 12:49 PM
  7. NSanity

    NSanity Member

    Joined:
    Mar 11, 2002
    Messages:
    17,346
    Location:
    Canberra
    I have a lot of respect for Daemon - however i tend to agree that his point of view (and elvis is absolutely an offender too) typically ignores the fact that quite often you can't have the "perfect" solution. Whether this be running multiple Domain Name spaces vs a single unified one, VMWare vs Hyper-V vs KVM vs HCI wizards, three tier vs hci, Windows vs Linux, etc etc.

    Environments are almost always about compromise. I've been solution architecting for 2 years now as a primary focus of my role - and well over 10 at this point within my career as a whole. I struggle to remember a single environment where i was given the chance to greenfield everything "perfectly". There is always budget, or legacy, or sometimes just downright stupidity from management/stakeholders/coworkers that will introduce compromise and/or inefficiencies into an environment.

    Change is hard. Particularly for internal resources (although MSP/Consultancy suffers from it as well). Sometimes it takes a change of environment to realise that there are other ways of skinning a cat.
     
    Daemon likes this.
  8. OP
    OP
    elvis

    elvis Old school old fool

    Joined:
    Jun 27, 2001
    Messages:
    34,636
    Location:
    Brisbane
    Who me? Irrational and extreme? Nah, you must be thinking of the *other* elvis. :)
     
  9. GumbyNoTalent

    GumbyNoTalent Member

    Joined:
    Jan 8, 2003
    Messages:
    6,482
    Location:
    Briz Vegas
    I like my init systems like my women easy, fast and light, systemd is fat and bloated. ;)

    Drinkies Sunday... yay.
     
  10. Daemon

    Daemon Member

    Joined:
    Jun 27, 2001
    Messages:
    5,399
    Location:
    qld.au
    Your original post was:

    Which again, has nothing to do with the issue. Why blame it at all? It's lazy IT, if you don't understand the component, don't go on a rant about it. It's really not difficult.

    If there's a bug with systemd-resolved, then that's where the fault lies.

    I haven't had a lot to do with OpenVPN, but I forever found corner cases with it. It's a fair mess, so the fact that there are issues doesn't surprise me. I find VPN's a mess in general anyway, hopefully something like WireGuard can gather steam and replace the lot.

    Again, this does sound like a bug. Did you find any similar reports?
    No. This is a perfectly valid response. You have a complex environment with a lot of non-trivial parts and it's certainly possible / probable that there are parts which haven't been implemented nor tested.

    Aaand again with the baseless statements. Systemd is well tested and certainly NOT based on home network tests. Would you say that RHEL is home use only? What about Facebook, fairly sure they just run a server or two but only at home?

    I'm sure I've linked it before, but their talks each year on systemd have been very interesting.
    - https://www.slideshare.net/DavideCavalca1/systemd-facebook-a-year-later
    - https://www.phoronix.com/scan.php?page=news_item&px=Facebook-systemd-2018

    There's now over 30,000 code commits from over 1,000 contributors for systemd, yet it still maintains an A+ for code quality, and has a pretty comprehensive set of tests: https://github.com/systemd/systemd/tree/master/test

    They've even incorporated fuzzing into these tests (and a reasonably broad implementation), something you don't see on many other projects.

    The code is reliable and well tested, but it's software and not perfect.

    Ubuntu is and always will be quite cutting edge, regardless of the LTS tagged release. Sometimes these experiments work, sometimes they don't (ie, Unity). What is predictable is that they'll ship a product early if it offers more features and works for the majority of cases. Given the rate of change, it would mean that there's a higher risk in rollout for complex environments.

    It's why CentOS and RHEL dominate the server market, the more conservative and stable track is preferred over cutting edge. Knowing that Ubuntu push these limits is certainly something that has to be considered as part of any implementation plan. Reverting resolved back to a working config for you is a valid fix. Asserting netplan is "proper fucked" is not.

    The first iteration of Windows-10 with mDNS was also plagued with issues (which were eventually fixed). It's also got a much larger market share of the desktop world, so I'd expect more diverse environments to be working as well.

    Again, I didn't say don't use multiple search domains via DHCP, just pointing out that if they were so common then there'd be a lot more noise and therefore development effort for it. Highlighting that I haven't used it in nearly 20 years wasn't meant to dismiss the use, but that it's just not very common at all.

    Not sure if a previous post has been miscondstrued, but I've never suggested there's only one answer nor is there a perfect answer.
    Agree 100%. Everything in IT is a compromise, just as it is in life. It's why IT needs to sing with a level of competence and one I rarely get to see in other environments. After dealing with a client's "legacy" systems recently, I'm offended by the use of the word. This was a mob of IT "professionals" and still had 2k3 running along with many other old systems. "Legacy" to them was "couldn't be bothered to deal with". Already replaced and shutdown nearly a dozen of the "legacy" systems within 3 months and certainly not working on it full-time.
    Change is at least the one thing we can count on. I can't remember who originally said it, but this has always resonated for me:
    The same is especially true in IT, it's an every changing world and one where professionals adapt, rise and overcome. The clever ones will ride the change and use it to their advantage, rather than an anchor to hold them back.

    I'm not sure I'd agree that change is hard, it simply requires discipline and process. Maybe I'm just used to the fact that I work with environments which constantly change so there's a bit of Stockholm syndrome, however we don't see the adverse outcomes others do.
     
    Last edited: Jan 14, 2019 at 12:39 AM
  11. OP
    OP
    elvis

    elvis Old school old fool

    Joined:
    Jun 27, 2001
    Messages:
    34,636
    Location:
    Brisbane
    I've done my mea culpa for this. Now correcting course as advised, and ranting about the correct bit of software...

    Sounds quite advanced. Care to share your thoughts as to why, with all these developers and advanced automated testing, they let something as RFC-breaking as my highlighted problem through? Given it's a years-old implementation, and there are multiple bug reports about it, what you're saying about quality and what I'm seeing in real life aren't adding up.

    I don't want to upset you again with baseless statements, and you seem to know the project well. I'd love to hear why you think they've botched such a major part of DNS (regardless of whether it's common or not - it's part of the spec, and in use). You've hinted previously that my use case is "uncommon". If we're going to talk about lazy IT, I think cherry picking parts of an RFC to implement in enterprise software based on subjective market analysis is right up there. But heck, I don't want to assume or rant without backing it up, so if you've got a valid reason for them missing this, I'm all ears.

    Indeed. I note that none of our CentOS servers have systemd-resolved installed by default (base install), compared to Ubuntu who do (desktop, server, and minimal server installs). Sadly CentOS doesn't cut the mustard for our desktop installs, which has nothing to do with either init or DNS. Ubuntu 16.04LTS worked very well for us. Ubuntu 18.04LTS is proving far more challenging (and not just because of systemd-resolved).

    If RedHat/CentOS offered something a bit longer lived than Fedora, that would be nice. Although I genuinely don't know if Fedora are now installing systemd-resolved by default like Ubuntu do. If they are, this bug will be present as it's still unfixed upstream from what I can see (at least, according to the bug reports I sfrom the Arch users on very recent code).

    Funny, I'm feeling EXACTLY the same way about systemd-resolved (and systemd in general) right now, despite the cheerleading it's getting here.

    It genuinely feels to me like one step forwards, two steps back, every time I'm running into newly broken shit with some new systemd component or release. And while computer stuff (and Linux stuff) is complex enough as it is, having fundamental things like DNS resolution suddenly not working correctly feels to me like... well... I don't know what because outside of macOS's launchd, I don't know another system that's broken it like this. (I repeat: Windows never got this particular issue wrong). Given Linux's "king of the cloud" status, that concerns me on multiple levels.

    So sure, I'll take a break from the "why did init need to change?" and "it breaks the UNIX philosophy" stuff the greybeards normally throw at you. But shit, I'm really struggling to enjoy Linux as much as I used to, and all of the things on my irritation list (much longer than just systemd) mostly point back to the ideas of one dude who, by my baseless observation at least, appears to want to emulate a bunch of macOS's features inside Linux (and if I wanted that, I'd use macOS). What all of this has done has really shaking my core belief system around open source being superior through proven merit. Lately it really does feel just as shitty as everything else.
     
  12. NSanity

    NSanity Member

    Joined:
    Mar 11, 2002
    Messages:
    17,346
    Location:
    Canberra
    Windows has a vested interest in getting a lot of DNS right. Active Directory is essentially underpinned by it.
     
    elvis likes this.
  13. GumbyNoTalent

    GumbyNoTalent Member

    Joined:
    Jan 8, 2003
    Messages:
    6,482
    Location:
    Briz Vegas
    LOL needed a laugh to start the day.
     
  14. NSanity

    NSanity Member

    Joined:
    Mar 11, 2002
    Messages:
    17,346
    Location:
    Canberra
    isn't it time for your nap?
     
    shredder likes this.
  15. OP
    OP
    elvis

    elvis Old school old fool

    Joined:
    Jun 27, 2001
    Messages:
    34,636
    Location:
    Brisbane
    At this point in time, I genuinely feel like Windows and BSD are the only ones getting DNS right.

    macOS and Linux appear to have gone off the rails in that regard. I hope they can both put dogma aside and correct themselves.
     
    Last edited: Jan 14, 2019 at 8:25 AM
  16. GumbyNoTalent

    GumbyNoTalent Member

    Joined:
    Jan 8, 2003
    Messages:
    6,482
    Location:
    Briz Vegas
    get off my grass...

    I was LOLing because it is 2018 and not 1988, surely resolving DNS should already be a thing in ALL modern OS?

    PS -
    Today I build a debian img for Pi3 that is PXE and lives 100% in memory, fun times.
     
    Last edited: Jan 14, 2019 at 8:35 AM
  17. samus

    samus Member

    Joined:
    Jun 3, 2002
    Messages:
    1,138
    Location:
    Baulkham Hills, Sydney.
    Day 3 - Still floppy. I know we get nostalgic at times, especially when I watch Phil's Youtube channel, but damn, these things are so slow. I thought, maybe a few hours at best, no chance, this is day 3.
     
    millsy_c and GumbyNoTalent like this.
  18. OP
    OP
    elvis

    elvis Old school old fool

    Joined:
    Jun 27, 2001
    Messages:
    34,636
    Location:
    Brisbane
    I assumed so. I was wrong.
     
  19. OP
    OP
    elvis

    elvis Old school old fool

    Joined:
    Jun 27, 2001
    Messages:
    34,636
    Location:
    Brisbane
    I like old computers, but not old storage. My nostalgia ends there, and I retrofit most of my old stuff with much more modern (and much faster) storage and IO whenever I can.
     
    Last edited: Jan 14, 2019 at 9:23 AM
    samus likes this.
  20. GumbyNoTalent

    GumbyNoTalent Member

    Joined:
    Jan 8, 2003
    Messages:
    6,482
    Location:
    Briz Vegas
    Scenario.

    Device uses MicroSD and fails to boot GRUB because SD issue.
    Device will PXE boot without SD.
    SD cannot be removed and replaced.

    What to use to map bad sector?

    Device is Pi3, but it lives inside of casing that cannot ever be reopened.
     

Share This Page