Worst Measurement Ever

I've been resisting posting about it for some time, I think since Patrick pointed me at the link almost six months ago. I've reached my breaking point. In my quest to learn Erlang, I've come across at least three blogs and/or articles that actually site this "measurement" with attribution as if it were some sort of legitimate claim as to the scalability of Erlang.

Given that I'm reading his book, I would really expect more from Joe Armstrong and by attribution Ali Ghodsi.

The Apache vs. Yaws measurement is one of the most useless pieces of information produced by the Erlang community, to the point that I'd argue it does a disservice to the Erlang community and the language.

In any sort of quasi-scientific measurement (or primary school science experiment for that matter), I would expect to see:

1. the actual code used to test the server
2. the actual Linux kernel version
3. the actual yaws server code

Instead, we see a graph of an under-documented experiment that creates conditions for a DoS test at best, not a web server scalability test.

From the looks of it, this "measurement" is not:

1. documented to any reasonable extent - what kernel was used? what was the exact Apache configuration? what was the Yaws code used to serve the files?
2. repeatable - no source, little documentation, little detail on the environment
3. peer reviewed - without the above, nobody else can discuss in detail or attempt to repeat the same results
4. valid - it simply does not repeat a real-world environment faced by any modern web server

Allow me to support some of those assertions from real-world experience helping large web sites:

1. Not knowing what Linux kernel version was used, how Apache was configured (in detail) and how Apache was built, it's impossible to know if Apache "was configured for maximum performance". Seeing as they decided not to share any of that vital information, I consider the entire experiment invalid. These things matter, just like if I were to run BEAM on an SMP system without SMP, it's easy to misconfigure the runtime. For example, Apache started using epoll in 2.6.x kernels. Anything earlier than that would be an inappropriate kernel to use with mpm_worker. We wouldn't know from the sparse detail if it is a valid configuration or not.
2. In a real-world environment, if I saw 100s of connections inactive for 10 seconds at a time, I would simply set the connection timeout to five seconds. Not sure how you would do the same on Yaws.
3. Most production web servers don't serve one byte files (the "load" requested by the clients in the measurement). I can't actually recall the last time I saw a high-volume server dish out more than one or two one byte files. Instead, real web servers serve files that measure in the KB or even MB in size. In fact, given Erlang's empirical difficulty handling basic file I/O, I'm not surprised that they chose a one byte file to use in simulating "load". If the file were any larger, Yaws would likely have been swamped under it's own unbuffered I/O implementation and exhibited substantial latency per request.
4. A one byte file would be cached upstream of the server in a high-volume site, particularly if your clients were operating over latent connections where one character per ten seconds was realistic (i.e. dialup).
5. If this were a real DoS attempt, it would be choked off at the ISP router, well before the web server saw the intentionally slow request.
6. I can personally type an HTTP request faster than one character per ten seconds, this is simply not a realistic access pattern from a benign client.

I'm not going to say that Erlang does or doesn't scale. My point is that this diagram and the corresponding writeup are utterly useless and that to site it as a valid study is irresponsible at best (especially when it's your own thesis).

I put this in the same category as Microsoft FUD - there is an agenda here, and people will read it and say "Oh, yea - told you we rock" without questioning the details. Most however, should dismiss it as pure FUD, and FUD served from an Apache 2.2.3 server no less (maybe Yaws wasn't up to the task).

My Bow-Legged Master - Comment

I tried to comment directly on Bill's Blog but comments were broken again. What I wanted to say was:

I think the real tragedy of this legacy is that we currently have installed, in nearly every networked node in existance, the potential for a simple, iterative, resource-driven client that can't really speak the protocol it was designed to handle. I repeatedly find myself grudgingly writing a curl wrapper to test a RESTful API in a pinch while my browser looks at me and laughs.

Oh, and the irony that your comments button says "Post" rather than "Put" is not lost.

Hopefully comments will be fixed soon and I'll cross post, er cross PUT.

Scale in an Ontology Vaccum

I had the pleasure today of reading Clay Shirky's Ontology is Overrated. Being in the collaboration software business, understanding how people classify artifacts is not just a science, it is vital to the evolution of the software we build.

For our purposes, I think it's a dual-edged sword. Make things too wide open, and users won't evolve their own classifications and people will stray away from the true strengths of the tools. Make things too ontology-driven and users will naturally migrate away from the structure, save everything locally and keep emailing and searching with Google Desktop. We have to build a better mouse trap to survive.

For me, a big take-away from Clay's piece was that any attempt to define classifications will fail. The Dewey 200 category is a fantastic example and one that really drives home our inability as simple humans to step out of context.

So, to paraphrase Clay's hypothesis a bit, any attempt I make at trying to define an ontology will fail, mostly because there is no way I can out-think the masses. Ok, fair enough, so how to build software that better helps the masses? Not too difficult at face value, things like tag clouds work great. Unfortunately, the implementation problems grow exponentially under scale. As the volumes of information available electronically continue to grow, and as information compounds on top of itself through links, blogs, etc, the notion of aggregating tag clouds, and more importantly, doing things like suggesting similar clouds shifts from a usability problem into an architectural one.

For collaboration software to innovate, it is faced with several interesting dilemmas. First, how do we expand beyond the bounds of our own control and pull in relevant data outside our domain? It's scary, but if you believe that collaboration software must go down that path, then we're faced with more integration concerns and less of the simplified workflow and text processing which is how most people think of their collaboration tools today.

Assuming you can go that far, how do you actually make sense of all that data? You start to run into scale problems real fast. Customers want one node to run everything and at the same time roll-up data from five internal sources, and ten external sources with partial federated identity all while being responsive. That's a tall order, and one that starts to make the "C" in "CAP" increasingly difficult to archive.

My gut tells me that Werner Vogels' and Amazon's approach to eventual consistency described in Amazon's Dynamo will win out. When you're aggregating and analysing terabytes of data looking for relevance, you have to make sacrifices.

plan for $display_id will be renewed for another year

I received an email from Yahoo! today with this in the subject line "Your Yahoo! Custom Mailbox plan for $display_id will be renewed for another year". I've seen recently that Yahoo! has been hurting and that they have been losing customers, but I can't imagine why.

I used to host all my small business domain registrations and email with Yahoo! Starting a year ago, I moved nearly all my paid services away from Yahoo!. It became clear to me over time that they didn't really intend to actually support their small business services. I challenge anyone to send an email to Yahoo! Small Business support, you can't. There is no means to contact them. At best, you can fill in a generic form that gets lost in the ether (I've tried that a few times too and nobody has ever responded).

No thanks, I'll take my money somewhere that has staff capable of correctly generating a mail merge and that will actually respond to my emails when I have a hosting issue.

Yahoo! is and will be my portal, but come on guys. Maybe Jerry can correct the course, I hope so.

Mind the Gap

I was recently fortunate enough to actually be offered a choice for my hardware and operating system at my current job. It's unfortunate that such choices aren't more common for engineering-types these days. I'm a big believer in the best tool for the job mentality and for me that's clearly some derivative of Linux on the desktop (and likely x86 Solaris on the server, depending on what I'm doing with it).

Faced with said choice, I went with Linux/x86 over Mac (that's another post all together). Having had good success in the past with Ubuntu, I decided to go with Gutsy Gibbon beta. Unfortunately, it was more beta than I would have liked.

First attempt at installation laid down the OS as normal. Being a sucker for eye candy, I wanted to get Compiz working ASAP in hopes of making the Windows users around me think twice. This turned out to be a big mistake. I ended up installing the binary ATI drivers and patching all in the same shot, and from there Gusty went into some sort of video card config hell. Despite my best attempts to hack /etc/X11/xorg.conf, I couldn't rescue my UI and had to start all over from scratch. Thankfully, I had a second hard drive available and copied all my userland tweaks there before rebuilding the whole rig.

This time around, I did a normal install. After making sure that was stable, I went after the binary drivers and got them working. Then, I allowed my updates to run, first system packages, then X11/Xde packages and binary drivers, backing-up my xorg.conf after each step so I could revert.

At the end of the day, the whole thing still isn't all that stable. I could bail on the eye candy, but that would be to admit defeat and that's not how I roll. Still though, I expected more. This experience was not all that much better from my previous Gentoo encounters with multiple reboots to the live CD to bail myself out. I was hoping for more, and didn't get it.

Finally, it's worth mentioning that Java UIs seem to struggle somewhat under my config with Compiz. IntelliJ's menus often don't show up and other Java-based UIs are also lacking.

Speaking of IntelliJ on Linux, it could really use some work. Their reliance on Java's fonts is plain painful, especially with so many nice anti-aliased fonts out there in Linux land. It's a pity that what is generally such a mature product couldn't keep-up with Eclipse and at least match SWT's handling of fonts.

Downsizing

I'm both happy and sad to announce that I'll be downsizing my career. I'm leaving behind my work as a Technologist for Liberty Mutual and assuming a core engineering position with Jive Software here in Portland.

I'm sad to be leaving the team at LM. They are immensely talented and capable of great things, I sincerely hope they are given an opportunity to act on that talent.

That said, I'm looking forward to joining Jive, more so than most of my previous career moves. If you follow the news stories around Jive, they are doing good things and doing them well. They are building on open protocols and generally lean towards Open Source Software as well - both things I highly value.