Can someone explain to me why systemd is bad?

Discussion in 'Other Operating Systems' started by NSanity, Oct 12, 2015.

  1. NSanity

    NSanity Member

    Joined:
    Mar 11, 2002
    Messages:
    16,449
    Location:
    Canberra
    Like if it is *so* bad, then why are all the major distro's moving to it?

    Finding people to say nice things about it is pretty difficult - but is it just a case of next-gen grey beards complaining about times of yore?
     
  2. Doc-of-FC

    Doc-of-FC Member

    Joined:
    Aug 30, 2001
    Messages:
    2,878
    Location:
    Canberra
  3. elvis

    elvis Old school old fool

    Joined:
    Jun 27, 2001
    Messages:
    30,575
    Location:
    Brisbane
    Warning: wall of text inbound. Turn away now if you don't care.

    It makes boot times faster, and has better dependency management. These things are very popular with the laptop/mobile crowd, who are pushing new things on Linux more than sysadmins who don't need these new things.

    Not that simple.

    Firstly, systemd was buggy as hell for a long time, and did really stupid shit by default like storing logs in a binary form and throwing away stderr. That sort of shit drives people who have to fix computers for a living mental.

    Most of those things have either been fixed or set so that they can be changed to more sensible options after install now.

    The bigger problem is a philosophical one. And yeah, that matters. Let me explain why, using a totally different example.

    MacOSX sucks balls. Everything "UNIX" about it is broken. The network stack is fucked beyond belief to the point of being unusable (try to run NFS or iSCSI mounted applications, and you'll suffer death by latency), NFS is broken, the system launch daemon, "launchd" and it's management tool "launchctl" are nearly unmanageable at scale (both are a strong influence on systemd). Bonjour can bring a WiFi network of more than 50 users to its knees (Avahi under Linux is an implementation of Bonjour, which is written by the same kid who wrote systemd). MacOSX did away with the lean/fast/sensible CRON, and replaced it too with launchd/launchctl, and a ludicrously verbose XML format file per "calendar entry" you want to run, making rolling out scheduled tasks at scale to a Mac fleet a nightmare.

    OSX gets worse and worse on every release. Why? Because they've stopped following the UNIX philosophy: do one thing, and do it well. OSX now prescribes to these monolithic, incompatible, "everything and the kitchen sink" stacks that are all heavily API focussed, and forget about doing sensible things with plain text.

    Now, it's *really* easy to sit back and complain about grey-beards and times of yore, because the people complaining about this sound that way. The fact of the matter is that doing small things well, and using combinations of tools to achieve bigger and more amazing things in a "lego brick" style just works better with computers.

    There's a lot wrong with computers. There's a lot wrong with Linux. But "the UNIX philosophy" is not one of those things. In fact, where that philosophy has been ignored (and not just in UNIX, but everywhere), we've resulted with shit systems. Any major fuckup in software you can think of pretty much boils down to arrogant people thinking that by replacing lots of small moving parts with one giant thing that is "everything for everyone", it ends up being not great for most people. Every user is different, every business is different, and the bad things that happen in computing are a result of people trying to assume everyone needs "one tool to rule them all".

    The world continues to fix things in Linux every day. In fact, Linux is really fucking cool at some things. You're not going to find a faster setup for web tools on a general purpose CPU than Linux. FreeBSD, Windows and Mac don't come close for the amount of data they can push around. Ditto for OpenGL - Valve showed us that Linux smokes Windows for realtime 3D. I work at a company that makes movies, and we render on Linux because it's faster, and doing 1,000 iterations of a scene on Linux means we get a better result than the 800 iterations we can do on Windows (and that has NOTHING to do with the dollar cost of the software, but rather what we can do with it). Hell, even Microsoft run Azure's entire networking layer ON LINUX. Why? Because it's pretty objectively fucking good at doing that sort of thing.

    Systemd throws out this philosophy, and tries to do so many things that it just shouldn't. Systemd wants to replace so many small, clever, well written moving components of Linux, and replace them with an enormous hulking mega-project over the top.

    The problem is simply that systemd could have been written in a better way. Nobody's complaining that Sys V Init got replaced. That's not the drama at all. Rather, a whole bunch of stuff including networking (Network Manager was the same guy), logging, and many other critical startup components have all been captured into a very large binary monster. The complain is purely that Sys V Init got replaced by something worse, written by someone who doesn't understand what he's replacing. There's a dozen ways it could have been done so much better.

    For what it's worth, the author (one Lennart Poettering), is also responsible for afore mentioned tools like Network Manager (great for laptops, terrible for servers), Avahi (zeroconf/bonjour auto-discovery and config tool that spews all sorts of shit over your network for people who don't know what DNS and DHCP is), and Pulse Audio (a simple way to burn 100% of your CPU playing music at the same quality as ALSA can do at below a single percent). Again - notice a trend here. Monolithic, auto-configuring tools that are all very "Mac like", being written into an operating system that was made for highly technical folk (and most often servers), with the intent of making it all easier for a virtually non-existent desktop userbase.

    Systemd had a rocky start. Bugs would leave systems unable to boot, and users unable to either diagnose or rescue their systems. That's settled down a little, but Poettering's lack of willingness to take on constructive criticism either about the bad start, or the fact that systemd continues to gobble up core components of Linux makes him not very welcome in the Open Source community. Indeed, his attitude is very reminiscent of how MacOSX has been developed over the last few years (which is fine, but Linux users don't want a MacOSX clone).

    There's a running joke that soon we won't be running GNU/Linux, but rather Systemd/Linux. The difference being that GNU is made up of thousands of small, well written, fast applications. There's a somewhat joking expectation that Systemd will gobble getty and even the shell at some point soon, which given it's current design, doesn't seem totally unrealistic.

    The crying shame of all of this is that talented developers working on Linux are needed. Poettering is a very keen developer who's missing about 20 years of real world experience with computers, and a good mentor. With those two things, he could be writing some amazing software instead of changing things to suit a non-existent audience, and in the process putting a lot of very large and important systems at risk with monolithic systems that are difficult to diagnose and debug when they go wrong.
     
    Last edited: Oct 12, 2015
    terrabyte likes this.
  4. gdjacobs

    gdjacobs Member

    Joined:
    Apr 3, 2007
    Messages:
    678
    Location:
    MB, Canada
    The problem is the close coupling Lennart and his cabal are utilizing for all their service replacements as well as his contempt for standards. He can claim (rightly) that systemd, logind, and all the rest are not monolithic (they aren't), but users are still robbed of choice because any other combination outside of the baseline ends up in some variation of broken as Lennart just doesn't care about outside use cases.

    With consoled on the way, it only seems a matter of time. I'm sure something will have to be changed in consoled which will result in getty being "broken" and needing replacement.

    I think Poettering's strength is his weakness. He's a developer who writes code very rapidly, but he has no great talent for software architecture. Because he is so prolific, I think he fails to realize this. His go-to answer to criticism is always to write something better, as if that excuses ignoring others and making bad decisions.
     
  5. elvis

    elvis Old school old fool

    Joined:
    Jun 27, 2001
    Messages:
    30,575
    Location:
    Brisbane
    Precisely. Poettering is a product of the "devop" era - write lots of code, ship it fast, worry about quality later.

    From this is born an arrogance that "new is better", and that anything not understood about the previous generation is wrong. That's fine for crappy wordpress blogs. Not so fine for something as critical as the core OS.

    As the old saying goes, "Those who do not learn history are doomed to repeat it." That is as true for software as anything else.

    It's been a common thing much higher up the stack in the last 5 years - I see a new API or a new language invented every day that doesn't actually seem to fix problems, but rather just re-invent the wheel. I'm all for people writing code to learn - that's fine. Shit, I've written a tonne of bad code that re-implements core things like network stacks and even malloc (that was a uni project), and it taught me HEAPS about computers and UNIX/Linux. They key difference was I was never arrogant enough to believe that my implementation was better than everyone else's.

    As you've mentioned, the problem is Poettering ships code. Lots of it. Regardless of quality, it's there, and his enthusiasm is confused with knowledge in this case. Like I said before, the kid needs a good mentor - someone who objectively criticises his bad ideas, encourages the good, and helps him get better at critical reasoning. With those skills (things that have nothing to do with writing code), he'd actually be a great help to Linux. But right now, he's just an inexperienced kid re-writing things badly to fix problems in a way that suit only himself, and not his audience. That in itself is not a bad thing, but sadly he's managed to convince enough important but non-technical people around him that he's doing the right thing.

    I'm somewhat resigned to the fact that systemd isn't going away. My only hope is that the meritocracy of open source prevails, and that the bugs will be fixed, the design altered, and the project eventually turns from terrible to great. There have been notable other open source projects that have done the same. I remember when ALSA was terrible compared to OSS, and that seemed to change after a major re-write. OpenOffice was pretty awful for ages, and the LibreOffice team have taken it in strides and bounds since forking. systemd isn't unfixable, but it's going to take a team of good people, not Poettering (unless he gets a sudden burst of maturity), to fix it.
     
    gregzeng likes this.
  6. ^catalyst

    ^catalyst Member

    Joined:
    Jun 27, 2001
    Messages:
    11,642
    Location:
    melbourne
    Thanks for the info, I knew some of the aspects about binary logs (!) and whatnot but it had not gelled that the same developer has made a lot of other things that have shit me to tears in the last few years, PulseAudio (horrific), Network Manager (horrific) and Avahi (horrific).

    Ideas on how to avoid systemd and not become a vegan/rms?
     
  7. gdjacobs

    gdjacobs Member

    Joined:
    Apr 3, 2007
    Messages:
    678
    Location:
    MB, Canada
    I'm not sure that systemd in it's present form can ever be fixed. Rewriting init to make it even more simple would be beneficial if it resulted in a leaner, meaner process reaper, but Lennart wants process tracking from the get go, so instead starts the prime systemd daemon (including the kitchen sink) after startup. This design has drastically increased the failure and attack surface and eliminating any possibility of healing from a successful denial of service attack or catastrophic fault at the service management level.

    Try Slackware? You might inexplicably want to buy a stuffed Goldy Gopher and have lots of cravings for Lutefisk and tater-tot hot dish, but there's no necessity to become vegan.
     
  8. elvis

    elvis Old school old fool

    Joined:
    Jun 27, 2001
    Messages:
    30,575
    Location:
    Brisbane
    Debian-based Linux: https://devuan.org/

    Or any BSD (I really like the look of DragonFly BSD: https://www.dragonflybsd.org/ )

    I'm unsure as to what Gentoo and LinuxFromScratch are offering, or if they're maintaining sys-v init as well as other init systems.

    It is definitely difficult not to come across as a full-blown RMS type character here. It seems that simplicity is not value like it once was, nor is it understood why text based scripts, configuration and manipulation is so powerful at scale. There is a lot more API-driven stuff these days, which is great for some things, but not all things. Trying to get the message across that "some of the time, not all of the time" is the right approach is difficult, particularly on a generation that has really captured management's attention with their "ship code all day every day without concern" approach to development, and what that means for getting customers to shell out dollars for lines of code written.

    At this stage, I sadly don't think it is avoidable at all, given that the two "grandfather" distros of Linux have both embraced it.
     
  9. OP
    OP
    NSanity

    NSanity Member

    Joined:
    Mar 11, 2002
    Messages:
    16,449
    Location:
    Canberra
    Look I hear a ton of people saying "well fuck them, i'll go BSD".

    And this is usually a good answer till they get to pf. And the fact that pf is soooo different to iptables (although many say they'd rather pull out both their eyeballs and swallow them than build firewalls out of iptables).

    Also the views here by Elvis (and pretty much everyone else) is *amazingly* different to that of the reddit thread that Doc linked... the main point of contention is that SysV Init doesn't "do it well" in terms of the Unix mantra.

    Additionally - the person you'd think would have a serious problem with it - doesn't - http://www.itwire.com/business-it-n...lds-says-he-has-no-strong-opinions-on-systemd
     
    Last edited: Oct 13, 2015
  10. elvis

    elvis Old school old fool

    Joined:
    Jun 27, 2001
    Messages:
    30,575
    Location:
    Brisbane
    Learning pf is a good thing for any sysadmin to do. I'm not suggesting everyone needs to be an expert in it, but exposure to it is healthy.

    I've adminned old Linux, new Linux, FreeBSD, HP-UX and Solaris sites. All of them taught me new and different ways to tackle old problems. None of those systems are prefect, and they all have their unique ways of tackling problems.

    Diversity is the key to success. :)

    Reddit is the epitome of this new Agile/Devops world. I don't know any sysadmin worth their salt that even bothers with it.

    I'm sure I've offended a bunch of people by saying that. Hell, my newest junior at work swears by Reddit and gets pretty pissed when I insult it. But truth be told, I don't let him anywhere near important production things because he has the wrong attitude to just about anything computer related. I don't see his love of Reddit as any sort of cause and effect, merely interesting correlation.

    The Agile/Devops opionion on systemd is "meh, it's new, and I learned Linux pretty recently, so no skin of my nose to learn more". It's a very shallow "end user" understanding of Linux, based on "monkey push button, monkey get banana".

    Anyone who's adminned Linux in anger for years knows what can go wrong and why. And when you've brought very important and very remote systems back from the brink of death, you get why systemd is risky.

    Again, systemd is fine if your application is massively distributed, and largely unimportant. If a hosed system means you reformat, rebuild, and your massively geo-disperse clustering deals with the rest, then bully for you. And again, no surprises that the Agile/Devops crowd are exactly this target audience, and simply don't care. To them, the extra 10 seconds saved on boot time with systemd means more (mostly because they consider rebooting a valid fix to problems, whereas old school UNIX admins don't, and would rather find out the root cause and not have to STONITH a system every 24 hours as a valid fix).

    And there-in lies my problem with systemd. It's not bad per se. It's just not for everybody. It's bloody fantastic for laptop/tablet users. NetworkManager (same author) is similar - bloody magic on a laptop that hibernates all the time and switches networks every few hours. But it's a total pain in the balls for a server with important uptimes and very small maintenance windows. Similarly, systemd is pissing of people who have to manage important servers and services that can't be massively distributed (i.e.: anyone who doesn't work on web/cloud stuff).

    Poettering suffers the delusions that his problems are everyone's problems. Anyone with an ounce of maturity knows that the IT industry is so incredibly diverse that this is never the case for anyone anywhere ever.

    Diversity is the key to success. :)
     
    gregzeng likes this.
  11. OP
    OP
    NSanity

    NSanity Member

    Joined:
    Mar 11, 2002
    Messages:
    16,449
    Location:
    Canberra
    Offtopic - Reddit (/r/sysadmin, /r/msp, /r/exchangeserver) is gaining rotation for me - as in i'll check it every few days - there is indeed *far* too much shit on there (kinda reminds me of when BE&C here gets "bad" periodically), but occasionally there is good stuff there which is worth looking at - and whilst reddit is getting less "timely" in terms of "this shit just happened right now, go read about it" - its still far better than most other sources for people "in the trenches".

    Ontopic - the point of "you can be *much* leaner than systemd allows" was addressed in that reddit thread (or one linked from it). Someone pointed out that their custom distro built for a *very* specific purpose sucked ass (bloat wise) w/ systemd - to which the argument was that systemd wasn't built for that, and that you'd still be better off doing things in a minimalist way.

    People have also addressed concerns about config and logs - specifically that systemd configs are *pretty easy* compared to init scripts.

    Code:
    [Unit]
    Description=The Apache Webserver
    Wants=network.target nss-lookup.target
    After=network.target nss-lookup.target
    Before=getty@tty1.service
    
    [Service]
    Type=notify
    PrivateTmp=true
    EnvironmentFile=/etc/sysconfig/apache2
    ExecStart=/usr/sbin/start_apache2 -D SYSTEMD -DFOREGROUND -k start
    ExecReload=/usr/sbin/start_apache2 -D SYSTEMD -DFOREGROUND -t -k graceful
    ExecStop=/usr/sbin/start_apache2 -D SYSTEMD -DFOREGROUND -k graceful-stop
    
    [Install]
    WantedBy=multi-user.target
    Alias=httpd.service
    Finally apparently 2 lines can get you every log entry by systemd pumped into a syslog server.

    All in all the point of "starting, stopping and managing every background service" is a hard job and thus it requires a complex tool to do it seems to feel true for me.

    Again - Its not that i have a horse in the game, my use in Unix/Linux is limited to Storage and the occasional application server that runs "best" on linux - as such i'll continue to stay in the protected lands of FreeNAS/RHEL/Ubuntu - which (with the exception of FreeNAS) are head first, balls deep in systemd.
     
    Last edited: Oct 13, 2015
  12. elvis

    elvis Old school old fool

    Joined:
    Jun 27, 2001
    Messages:
    30,575
    Location:
    Brisbane
    Linux already has numerous tools to do this. From just a shell we can background a process, foreground it again, redirect logs, get its PID, get its exit status, and more.

    I could write you an init system in BASH that handles all of that and dependency management in less than 200 lines. And not a single line of C. No kidding. Probably less in Python (assuming I'd want to risk init written in something more complex, which I wouldn't, because that makes rescuing it more difficult when it breaks).

    But, I recognise that *my* method would only really suite people like me, and not be a whole lot better than sysvinit.

    Indeed, just yesterday I fought with systemd not handling Samba4.3's nmbd process. "smbd -D" is fine, "nmbd -D" exits mysteriously. No error log, no output, no exit status that I can capture.

    When I run it from a shell, it's fine. When I run it from sysvinit, it's fine. When I run it from systemd, it just vanishes. How do I debug this? I have no idea, because I don't know what the hell systemd is doing, and why it's seemingly marking the process for death (especially weird when the smbd process fires up in exactly the same way, but works). Pure dumb trial and error shows that if I don't daemonise it (i.e.: nmbd -F), it works fine. But systemd didn't help me there. So what do I do now? (Other than hack my systemd config file? I'd really like to run a daemon as a daemon, after all).
     
    Last edited: Oct 13, 2015
  13. Doc-of-FC

    Doc-of-FC Member

    Joined:
    Aug 30, 2001
    Messages:
    2,878
    Location:
    Canberra
    time to weigh in briefly:

    systemd, I’ve seen it coming for years, seen the debian community duck and weave and in the end embrace it.

    I've tracked testing for too many years and recovered my system on too many to recount occasions due to things like gcc/libc migrations between major debian releases.

    Systemd can be scary for people who have years of experience and who are fluent with sed/awk/bash/init tooling their way through to mission accomplished. systemd also shows some merit, it's attempting to simplify some of the hair brain concepts around kde/gnome/xfce and how things are interfaced with to achieve the same operation across the major distros.

    As elvis pointed out, network adaptor management has been made much simpler, although I ran head first into network manager pissing me off because it lacked the concept that I required a KVM network bridge configuration, quite a number of local VM's during my deb7-8 migration.

    The gui isn't for everybody, there’s no one ring to rule them all, yet poettering keeps bashing away; in some respects he's making linux the OS for the desktop, he's pushing the simplification barrier for the 99%, android brought linux to the phone for the 99%, will systemd create the 'needed?' commonality between distros?

    most of my home systems run FreeBSD and when needed I use the linux compatibility shim.
     
  14. elvis

    elvis Old school old fool

    Joined:
    Jun 27, 2001
    Messages:
    30,575
    Location:
    Brisbane
    This is an argument I hear a lot, particularly from pro-systemd (and previously anti-Linux crowd). i.e.: there needs to be "one major Linux distro", or "one true way" to do things across all distros. Even some people who you think would "get Linux" suffer from this madness (Mark Shuttleworth insisted all distros and upstream authors - everyone from Samba to the Linux kernel developers - sync to the same release cycle as Ubuntu to make his life easier).

    Linux's heterogeneity is a good thing, and the design differences and unique end goals of each distro are a good thing. The argument that systemd is good for Linux because it "simplifies daemon management" for authors is a flawed one, particularly when a very small percentage of open source are daemons (by volume, there's a hell of a lot more desktop and web stuff out there today). Beyond that, it's the package maintainers, not the upstream software developers, who have to worry about the init system, and fitting in with their larger distro ecosystem of choice.

    systemd is not going to fix any of the perceived problems people have with Linux.

    It's not "scary" per se. But you've illustrated the point. sed/awk/bash/init are small tools that work together well. systemd is a monolithic tool that is one mother of a black box. What happens when part of systemd fails?

    My trivial example above has already bitten me. How do I find out why a daemonised service managed by systemd fails, when a foregrounded version passes? Debugging verbosity on the nmbd binary doesn't help, as systemd appears to be at fault, and not telling me much at all.

    This all takes me back to my Windows admin days, and is very similar to the problems I face managing a large MacOSX rollout today. Monolithic tools, poor logging, mysterious exit states not following POSIX convention with no ability to capture their output. Indeed, all of these reasons are why I ended up choosing Linux as my area of speciality - specifically because the design goals of Linux (and UNIX before it) were precisely the opposite - give plenty of information to the user, and adhere to very simple design rules (the most important one is to "fail loudly" when something goes wrong).

    It's worth noting that Poettering's personal stance is to ignore criticism and only respond to people who praise his work. It appears his software takes the same design - only give people good/happy messages. Hide failure messages, and gobble up errors. Software with a positive attitude is apparently better than software that helps people do their work and debug errors.
     
    Last edited: Oct 14, 2015
  15. Quadbox

    Quadbox Member

    Joined:
    Jun 27, 2001
    Messages:
    5,916
    Location:
    Brisbane
    Gentoo still defaults to openrc, but makes it entirely possible to choose to use systemd if you so desire. It's always seemed a bit pointless to me to bother with systemd on gentoo as openrc already DID most of the things that systemd purported to improve... dependency resolution and parallel init's been a feature of openrc for like eight years

    I agree entirely with your comments about misguided attempts to "unify" or "harmonize" distros
     
  16. elvis

    elvis Old school old fool

    Joined:
    Jun 27, 2001
    Messages:
    30,575
    Location:
    Brisbane
    From OpenRC's description:

    "Description: OpenRC is a dependency based init system that works with the system provided init program, normally /sbin/init."

    Instantly I see that OpenRC works with /sbin/init, and doesn't attempt to replace it.

    THIS is sensible design. Work with existing tools, fix only the parts that need fixing, continue the UNIX philosophy by doing small incremental changes and using small tools that work together to do bigger and more important things.

    Compare and contrast to systemd's entire design: replace all the existing small, clever tools with one monolithic tool.

    I don't see any uproar over OpenRC. I think that illustrates the point perfectly that the aguments against systemd aren't about fear of change, but rather that systemd is objectively bad software.

    Perhaps Gentoo will be my next distro of choice? Lord knows I've got enough CPUs to do compiles on, and could trivially build my own packages for binary rollout across my entire fleet. This is sounding better and better...
     
  17. GreenBeret

    GreenBeret Member

    Joined:
    Dec 31, 2001
    Messages:
    19,377
    Location:
    Melbourne
    I hate these stuff for the same reasons elvis listed: NetworkManager, Avahi, Pulse Audio, systemd etc. both on desktops (when I used to run Linux desktop fleet) and servers (including cloud backend infra).

    But I didn't know that they're all written by the same guy until now. :lol: :mad:

    This is everything I hate about OSX. If Poettering uses the same design philosophy, no wonder...
     
  18. elvis

    elvis Old school old fool

    Joined:
    Jun 27, 2001
    Messages:
    30,575
    Location:
    Brisbane
    It's funny how often I hear that response lately. When I show people the stuff he's written, the penny drops, and they realise all this bad design is the fault of one dude.

    I *really* wish Apple would just hire the guy. He's got the right attitude for their requirements/market/goals. A bit like the Tesla CEO said - when you fail at the really hard/important stuff, you can always go work for Apple instead. :lol:
     
  19. FiShy

    FiShy Member

    Joined:
    Aug 15, 2001
    Messages:
    9,682
    I have not had to deal with systemd (since its not my area) but if its like NetworkManager then fuck it and all things to do with it.
     
  20. Wolfje

    Wolfje Member

    Joined:
    Jul 9, 2007
    Messages:
    923
    Location:
    Brisbane
    I love systemd. For my intents and goals, it saves me writing god awful init.d scripts when I package things. For a vast majority of people, it's fine.
     

Share This Page