Killer Software

This Boeing air disaster story is just a nightmare. Now it seems that Boeing was warned about the software problems four days before the most recent crash.

I feel like I dodged a bullet. Boeing (nee McDonnell-Douglas) is one of the biggest employers in Saint Louis. I don't work for Boeing, but I do write software. Happily, I don't write aviation software. When my software fails, people have to ask questions instead of looking something up in a database. No one dies.

I cannot even imagine what those Boeing software developers are going through. There must be William Mulholland levels of guilt.

 

Comments

[Big Dot]

Good point. It’s easy to forget that behind the faceless institutions there are real people at fault and having to live with it. Didn’t know about Mulholland, either. Can’t imagine how he even got out of bed in the morning.

[TheQueen]

Big Dot - I sound like I’ve forgotten that there were hundreds of people who died, in my concern for the software developers.

[KC]

Something I have reassured myself with on Very Bad Days: "Even if this all goes wrong, no one will die. You are not a brain surgeon or something."

Doesn't hold true with aviation, car, medical, etc. software, though.

I do think that the people who probably ought to feel most guilty in a situation like this are the ones who pushed the process faster, who discouraged bug reports, who pooh-poohed concerns such that people stopped raising them. (They exist at most companies; I assume this sort of management/business-minded-entity/consultant also exists at a species at Boeing, although I don't know how prolific they are.) NASA and some other places have kept a pretty good "it matters more for it to work than for us to be fast at making it" ethic, but mostly "will it probably work? then push it through ASAP and forget your qualms" seems to be more prolific?

Anyway. Where people can die because of sloppiness, companies and those within them who caused the problem (not focused on the "person who wrote a specific failing line of code", although including that - but everyone who made the "move fast, break things" sorts of choices to make a bad line of code likely) should be responsible, with the responsibility shifting depending on the pressure exerted; if there's no way the developers could do a really definitely good job within the time/budget constraints they were given, but were forced to choose between trying and being fired, then it's less their fault than those who locked them in that box? But if the developers knew this wasn't quite safe but didn't chirp at all, then it's also their fault? Unless all prior chirping was also ignored, in which case, I don't even know anymore. Programming culture requires alteration; business models of productivity and proxy variables require serious alteration. And people die.

[KC]

(oh! but my favorite line from a EULA; it was in the EULA for a label-formatting software application [like Word mail-merge label-printing stuff, except standalone] and it was something to the effect of "Do not use this software to run nuclear facilities." What I'd like to know is *how* you could possibly use label-formatting software to run a nuclear facility?)(but it did give the "we wrote this, it will probably work, but it is perhaps quick and dirty and will not *certainly* work, so Don't Use It Where Things Need To Definitely Work" sort of gist of things, anyway.)

[TheQueen]

KC - It’s a nightmare all around. It’s like a real-life All My Sons.

[KC]

I didn't know what "All My Sons" was, and just looked it up, and according to Wikipedia, *that* was based on a true story, too. Eugh.

Human beings: they are pretty horrifically awful sometimes. But a lot of the time they "get away" with imposing unacceptable risks on other people in exchange for minor rewards for themselves (think of how many drunk [or texting-while-driving] drivers do *not* kill someone or even get caught by police, for instance). I definitely do not feel badly for the people who deliberately risked other peoples' lives by cutting corners or by backing other people into a position where they were near-forced to cut corners; I do feel badly for the people who honestly didn't know what to do, or honestly didn't know what the right thing to do was, and where it turned out This Badly. Oof.

[theQueen]

KC - Raisin in the Sun is another excellent play with a fascinating true story behind it.
I know programmers, and I will bet you anything they were up against an intermittent problem: the worst, because you can’t reliably recreate it - or fix it. And my guess is the money men said, “I don’t see a problem, then.”

[KC]

Intermittent problem that they've seen but are having a hard time replicating would make sense. But in that case, the system should be grounded until they've pinned that sucker down and made *sure* it's dead!

I loathe how, in our society, we not only tolerate but privilege a short-sighted, exclusively money/profit-to-the-shareholders concept of ideal corporate behavior (articles that basically say that if by abusing your employees you can get a bit more profit this quarter, then *of course* that's the right thing to do; what kind of idiot would pass up a profit? or pay more in taxes than the minimum they can get away with paying using complex shell companies and income-hiding and whatnot? that's just common-sense... without any humanity, ethics, or actual common-sense about it). The question tends to be not just "is it technically legal? then fine" but "if it's not legal, can we still get away with it? then fine" as opposed to "what is right to do?" or "what will be long-term best for this company?" even. (I'd love "what would be best for society in total?" to be included in the decision equations, but that seems like an implausibly long way away from where we are for the most part with not-small-businesses...)

Anyway. Not a fan.

(and it looks like that play and the story behind it would be fascinating, but incredibly depressing. We've come so... not far... since those bad old days.)