I've been running Solaris systems for the last 8 or so years, starting with 2.5.1. Soon I'll be rolling some Solaris 10 into production. I think that makes me fully qualified to condemn Solaris and collaterally Sun to the darkest depths of hell and declare them, officially, The Worst. (Sadly this distinction isn't singular or unique.)

There are so many problems, it's difficult to know where to begin. Let's start with something as banal as disk management. It's nothing short of shocking that Solaris 10's boot kernel still deals with hard partitions, or as they call them "slices." Sure, you can abstract them behind "meta volumes," but not without dealing with hard partitions first. Horrible. Unacceptable. 1998 called and they want their boot-disk handling back.

Why is this so terrible? It's sorta like jazz, if you don't know, it's sorta hard to explain, but I'll try. Not having a real (and by real I mean working, stable, full-featured and informative) volume management system is akin to selling laptops with DOS on them. Any OS worth the media it's printed on has feature-rich volume management. Volume management that is easy to programmatically query. VM that handles things like business continuance volumes well, VM that deals with disks, not slices. The sad sad state of SVM isn't especially surprising given that Sun mostly assumes you'll use a third-party (Veritas) filesystem/VM with their OS.

And that brings me to the sloppy wet turd that is the Solaris Volume Mangler. Yea, I know the answer to any sort of "when are you going to have real volume management that doesn't completely suck ass" question is <hushed respectful tones>ZFS</hushed respectful tones>. Yea, I'm sorry SVM has been around for, what? 15 years? more? and they still can't get even basic mirroring correct, so how the fuck am I supposed to trust this horrific mangling of filesystem and volume management?

Oh and can you boot off it? "RSN!!!"

And that page I linked above about the SVM bug is just the best. If you install the most recent release of Solaris 10, which they call "05/08" you get that patch. No way around it.

Back-story: When we installed Solaris 9, years ago on some very high production hardware there was a huge, unbelievable and, as far as I can tell, completely unpublicized "line in the sand" patch that fixed a bug where if you'd made any sort of meta volumes prior to the patch, they were irrecoverably fucked. There was no way to get from A to B other than to re-install and create all new meta databases. Of course we were the lucky ones who discovered this bug and were in production with it. Yippe! we got to re-install. As a result of that fiasco, I'm very weary of SVM bugs and patches going into these new systems.

So yea, that new bug report caught my attention pretty quickly. At least Sun made it easy for me to see how the bug manifests:

1. Impact
Solaris 10 systems using Solaris Volume Manager (SVM) where root is mirrored may experience multiple services failures (rpcbind(1M), network/physical) after installing patch 128306-03 (SPARC) or 128307-03 (x86).

What? What does rpcbind have to do with SVM? Network/physical? WTF is that? I'll read some more, surely they clarify below:

3. Symptoms
Systems that experience this issue will see multiple services failures on the order of rpcbind(1M) or network/physical.

OH! On the order of rpcbind. Right! On the order of network/physical. And again I ask: are you fucking kidding me? But it gets worse. The suggestion for how to deal with this problem is simply to restore the backup that the patch made of this certain file. Oh yea, but this patch is on the 05/08 media and there is no backup. So rather than link to a reasonable file, rather than come out with a real patch, rather than tell you what the hell the bug is, Sun leaves you knowing something (but not what) is wrong, and that there's really no way to fix it until they run their lengthy and apparently fabulous vetting process on this patch.

But wait, there's even more! If you ask anyone at Sun what patches are required to be "up to date" with SVM, the answer is unknown. Hell, they can't even tell you what packages are required for SVM -- or anything else for that matter.

The only supported way to install Solaris is to install every single package. (Don't miss the paragraph unto itself drama!)

You might think that means what Sun calls "All+OEM" but you'd be flat out wrong. "All+OEM" is not a super-set of "All" it's a whole different meta-cluster. Nice. Nice job. Good work. So do you install All or All+OEM? Who knows!? Not Sun.

You might wonder where I'm getting my information. I'm getting my information from years of yelling at the person at the other end of the support call line. And I'm talking deep, back-end people, not some poor bastard reading from a script. We've had VP's of support explain to us how we were going to get sick of them because they'd be pestering us so much to make sure that things were going well -- only to never hear from them again. We've escalated, we've pushed it to the highest levels and brought the best Sun was willing to provide to bear. What I don't do is spend time in Solaris admin forums or on mailing lists or anything else. So if you're going to mail me and explain to me the common knowledge of these or other non-official support channels, don't waste your time. I don't care.

Hell even using Sun's own website is an exercise in frustration and rage. Let's try a simple example. I wanted to get the latest version of CEDiag for the new machines we just got, so I did what any sane person would do. First on the list? Nah, Sun doesn't give a shit about stuff like that! They're Sun for fuck's sake! So I picked the fifth result and thus began my adventure.

If you go to that fifth result, you'll see a nicely formatted, official looking list of stuff that you might want to download and install on your Solaris system, you poor solaris admin sucker. And hey, wow! CEDiag is even in the list, let's click that! And I'm going to explain each step since you're probably not masochistic enough to play along. That takes you to a page that looks like you might be getting somewhere. You might have found what you're looking for! But no, not really. There's a download link, but if you read the bold text that is the third paragraph, you'll see that this is the OLD version and you probably want to click somewhere else to get the new version. Wow! Why the fuck didn't the first thing I clicked on send me where I wanted to go and then have that page tell me how to get some old-ass version that no one wants anymore? Clearly it's my fault. So then you click on the version you probably wanted link and that gets you another horrible horrible page. Firstly it's way too short and secondly it contains no useful information. Thirdly, it encourages you to use some fucking java app to "ensure successful" download of their crap.

On this horrible page the only useful thing is a download link which brings me to another huge pet peeve of mine about the modern Intertubes. If you have a link called "download" or especially a link called "Download Now!" those links had better be straight to the fucking file I want. Of course, they never are. Ever. They're always a link to a page with more ads and bullshit and another link that says "Download Now!"

But anyway, back to bitching about Sun's retarded atrocity of a website: The download link isn't a link to a file, it's a link to a license agreement. Do us all a favor, cover your OS addons with the same agreement as your OS. You've already made me log in and have a support contract. Stop wasting my time.

And just so you know this isn't an isolated case, there's Sneep. See the eeprom command in Solaris is too stupid to, ya know, read the eeprom... so you had to download this extra utility to deal with doing something esoteric like reading the serial number of the machine from the OS. So the cycle is the same. Start with a google search and wow this time it's the first link! And what an awesome page it is, complete with this text: "Sorry, no download available now, please contact the administrator." Who? Who is the administrator? I thought I was the administrator?!

So then I did something really crazy. I did something I almost never do because 90% of the time (or more) it's useless. I used the site's own search engine. I searched for 'sneep' and, depending upon which of the two search boxes(!?!?!) you use, you actually get to a vaguely useful, but completely unacceptable page. The page actually contains this text:

Downloading:

SNEEP is available for download from the Systems Management section of the Sun Download Center :

http://www.sun.com/download -> Systems Administration -> Systems Management

(http://www.sun.com/download/products.xml id=4304155a)

What? Text where links should be? Whaaaaa??! Why?! Why do you do this to me? But there's more, it's worse... the first "url" up there sorta hints at it, but check out this excerpt from this other CEDiag page I found (not as described above: searching using one of the many Sun searches yields an entirely different CEDiag page that has a 404 disguised as a random list of bullshit when you click on its link entitled "View License Agreement and Download cediag 1.3.2"):

4. How do I get cediag

In your browser enter: http://sunsolve.sun.com

Under "Diagnostic Tools" select the link, "Memory DIMM Replacement Management Tool"

The "Memory DIMM Replacement Management Tool - cediag 1.2.1" page (this page) should be displayed. Here you will find the change summary for the latest version, FTP instructions, FAQs, installation instructions, and a link to: "View License Agreement and Download".

Select "View License Agreement and Download - cediag 1.2.1 "

Review the License Agreement and select "Agree" to proceed with downloading.

This time we get a link, the wrong link, a not-even-close link, but wait, ARE YOU FUCKING KIDDING ME?!?! Go here and do this and click that and jump on your left foot?!?

Make a fucking hyperlink to the file and if you just can't do that, make a damn link to the page you want me to go to. Here's the BEST part. If you do all that shit, click here and click there and say the magic words: YOU END UP BACK AT THIS SAME PAGE!!!. So if you're stupid like me and your eyes immediately jump to the part that says "How do I get cediag" you get to spin at least once. Awesome work, Sun! Pull your heads out of your asses.

And while you're at it, make your web site way way way faster. If I wasn't forced to use your crap OS and I was thinking of doing so, a single visit to your page would make me realize that you not only don't have your shit together, but that your website and therefore YOUR HARDWARE is achingly slow.

And the best part about all of this is that, of course, you need to be logged in to have all of this fun. And, like most vendor sites, their session time-out is unforgivably small.

Wow, I could go on and on -- really. There are so many things about Solaris that drive me completely insane (dmesg is a fucking shell script?!?!?!?!?!?!??!?) that I just don't have time to cover them all here.

Now if you'll excuse me, I have to get back to fighting this bullshit OS.

Oh crap! I've been chain-lettered months ago and just now noticed. And by a bigwig no less! Shit!! Jeez, Stu, what'd I do to you? :)

So here's 5 things you don't know about me... um...

1. I've been recently diagnosed with severe sleep apnea. My number was 57.7. That means basically once a minute while sleeping I stop breathing for more than 5 seconds and have to "wake up" to start breathing again.
2. I'm the very proud owner of Travis Bean #1716. It's the most amazing guitar I've ever laid hands on and by a margin I wouldn't have believed.
3. The only alcohol I've ever consumed was 10ml (think "shot-glass") of wine at the 800th anniversary of some big Heidelberg winery. Cuz, come on, how often is it the 800th anniversary of anything?
4. Um, I hate macs/Apple? No, you know that. I'm a chest-beating athiest? Nah you know that too... this is hard! Um... Ok, how about: despite knowing a decent amount of music theory, I can't read musical notation.
5. I recently tried my hand at making some online comics. They're probably not funny to people who don't work with me, but that isn't the point of this list, now is it?

So to make the circle keep on circlin' I'll nominate Sherry cuz she'll respond. Glenn because he won't. And Michael because he's a cool guy!

permalink

I moved everything to Dreamhost. Stuff might be broken. I might fix it, I might move to some other software, I might do nothing.

I have had the blog bug a bit lately, so there's some glimmer of hope against inaction.

Today I bought Being and Nothingness and at the conclusion of the transaction the person behind the counter said "Have a nice day!" which I found terribly hilarious.