Warren Seen

It’s funny how things work out. When I made the initial version of hashjobs live back in January, I mentioned that I wasn’t really sure how or if I could make anything from it. As luck would have it, it turns out that I managed to catch the “post jobs on Twitter” trend in its infancy.

Almost immediately I started getting offers to buy the domain name. Each inquiry I got, I would knock back as I have always had plans as to what I wanted to work on for future versions of the site. Most of these seemed like tire-kickers who I never heard from again.

But there was one guy who kept coming back asking about the site. Early last month, he forwarded a very specific offer via a 3rd party to me that made me realise he was VERY serious about acquiring HashJobs. After a bit of back and forth negotiating, and consideration of what I could make from the site in the next 12-18 months if I had a serious go at it, I happily accepted his offer.

So I’m pleased to announce that as of this morning, the final installment of our transaction has cleared, and Jason Davis of recruitingblogs.com is the new owner of HashJobs. I’ve had a sneak peek at Jason’s plans for the site and I think he is going to do it far more justice than I could have, given his position in the online recruiting community. I wish him all the best.

(As a side note, I am amazed how quickly incoming wire transfers clear, compared to transfers between local banks!)

As for my plans, well at least I’ve filled a GFC-induced hole in my previous 6 months cashflow. People who I’ve told about this always ask me, “Can yo do the same again in a different field?” The answer is, “Probably not, it was more luck than good management.”

I think I’ll just start another side project (I have a few ideas) and see where the wind takes me…

I’ve just pushed the initial version of Hash#Jobs live. The site is a quick and dirty twitter filter to pull out all the job-related tweets from the twitter stream.

from the about page:

Hash#Jobs aggregates tweets that contain the following tags from twitter:
#job, #jobpost, #NAJ, #HAJ, #employment, #recruiting, #hiring
If you want your tweet to appear here, simply add one of those tags.

Some day, in the future, Hash#Jobs might be smarter, through the wonders of bayesian classifiers and other cool things like text clustering. No promises.

I built Hash#Jobs initially for myself, since I’m looking for new gigs, but figured, what the hell, for $12 bucks worth of domain registration, I might as well share it with the world. If one person gets a job out of it then it will have been worthwhile. It was also a fun excuse to play with Sinatra, I think someone said it’s like camping, but without the magic mushrooms. I’d have to agree…

I haven’t considered whether I could or should make money off something like this… I doubt I’d turn over more than the cost of the DNS or hosting, so not sure if it’s worth the effort.

I’d like to improve the system over the coming days to include clustering, as re-tweets show up and can very quickly fill a single page with repetitive information. I’m also toying with the idea of a Bayesian classifier to try and weed out what is and isn’t a genuine job post. It remains to be seen if 140 chars will be enough to get a meaningful result out of a Bayesian tool.

So anyway, there you have it, if you find it useful, feel free to drop me a comment and let me know. Also, please pass it on to someone who’s looking for a job if you think they can use it.

I’ve forked the Rails auto_complete plugin to add support for the following usage:

<% form_for(@foo) do |f| %>
  <%= f.text_field_with_auto_complete :bar %>
<% end %>

This fork also supports nested fields_for calls:

<% form_for(@foo) do |f| %>
  <% f.fields_for(@foo.baz) do |baz_f| %>
    <%= baz_f.text_field_with_auto_complete :quux %>
  <% end %>
<% end %>

You can find my fork on Github, here.

… when job ads start appearing asking for unrealistic and bordering on impossible levels of experience.

I thought we’d gotten past this back in 2000 when .Net came out… but 3 Years of Ruby on Rails experience you say?

Mathematically impossible for the average (read: non-core) rails developer, considering Ruby on Rails was only released publicly in July of 2004.

They’re obviously very picky and so are willing to pay up to a whopping US$35-US$55/hr for all those Rails rockstars who were using it before it was released to the public! I’m sure they’ll drop everything and come running to apply!

“We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil.”
Donald Knuth

Earlier today, I suggested that we should hold language and platform selection to the same standard we do with our code when avoiding premature optimisation.

The thrust of my argument was that, in relation to scalability, 99% of sites will never become so popular as to require scalability beyond that which is possible with Rails, so worrying about scalability feels like the start of big design up front.

We should aim to not be dogmatic about our selection of language or platform, instead choosing the tools to get the job done, so we can get something out there.

After all, I think it was Fred Brooks who told us to build a first version and throw it away; the inherent wisdom in this approach is that you learn far more by building something and failing fast, than doggedly sticking to a flawed design that dies a slow, painful death.

If I can return to the example of Twitter, I don’t have any references, but I think their development time was quoted at something like 4 months? That’s not a lot of time really, and therefore, it’s not a lot of time to write off in the case that Twitter doesn’t get adopted and turned out to be an idea that was DOA.

Happily for the guys at Obvious, this wasn’t the case, and now they have a scaling issue, but I bet you they know a hell of a lot more about their users and usage patterns than what they ever could have predicted if they had set out to design it up front.

Could they have not followed the same path with J2EE or ASP.Net? Sure, if you ignore the fact that they’re Rails developers and therefore are more productive with… Rails. They chose the platform that was right for them, that got out of their way and let them implement the service to see if it would even work.

Another example, Reddit. The first version was written in Lisp before being rewritten in Python – rewriting a proof-of-concept application (the cool kids call it beta software these days) into something that is “commercially ready” is not a bad thing.

I’m sure there are plenty of counter-examples where such an approach has failed, but I’m not getting into reductio ad absurdum arguments like “if your premise is true, then I could write #{compute_intensive_app} in #{slow_high_level_language}!”

When you first start using Rails, you quickly find there’s an 800lb gorilla in the room that no-one really wants to acknowledge. Yes, the ’s’ word: Scaling. Oh, they’re quick to tell you that it’s an uninteresting problem (at least for a programmer), if you do Rails right, and you’ve shared-nothing, then you just throw more hardware at the problem. That if you have a real bottleneck in your code, just rewrite the performance critical parts of it in C. Oh, and YARV will be here soon (for certain values of soon), so Ruby will be faster!

And suddenly, that gorilla shrinks; or at least it seems to.

Until someone goes and pushes the boundary way beyond those who’ve come before. Suddenly, the gorilla has grabbed your girl, climbed to the top of the Empire State and is angrily swatting light aircraft left, right and centre.

Scaling suddenly gets interesting again.

Nay-sayers and trolls cloud the issue
Those who for whatever reason want to see Rails fail, have grabbed ahold of the issue and are casting it as a reason to abandon Rails: “Behold, the savage beast that cannot be tamed!” Often, they’ve got their own agenda, whether it’s a desire to see their pet platform succeed, or to see DHH kicked to the dirt, these folks don’t really care about the scaling issue at all. They latch onto these issues not to help improve the platform, in the worst case they want to destroy it.

So forget them.

We’re here because we want to solve this without changing platforms.

It doesn’t matter to most of us anyway
The truth is, 99% of Rails projects will *never* hit a performance wall that cannot be solved using existing wisdom. So the issue is a non-issue and certainly not a reason to ditch Rails.

If perchance, you’re part of the other 1%, you should be thrilled to have this problem, it means your software has been successful! Fact is, you don’t accidentally create a site/service which generates 11,000+ requests/sec. You explicitly set out to do so. And presumably, you chose Rails for a reason. Why shouldn’t we extend the concept of “no premature optimisation” to language and platform selection? If Twitter had chosen an “enterprise” class platform and tried to architect for 11k reqs/sec, chances are they would still be building…

Obvious had a reason to choose Rails; they aren’t naive, they’ve been around the block with Rails before.

Generally, it’s much better to throw something out there NOW, and let the real problems come out in the wash, rather than sit in a dark room, worrying about what might happen and giving yourself a case of analysis paralysis.

Escalation Sucks
This issue has been blown up by a combination of strong opinions, incomplete information and folks from other camps pouring fuel on the fire. It’s time this “conflict” was de-escalated and the combative nature of the discussion toned down.

The situation itself can be summed up pretty easily

• We’re in uncharted waters in terms of this sort of throughput, although others have suggestions and maybe even solutions.
• The problem didn’t just suddenly spring up on the Twitter guys, clearly they’ve been working on it for a while – implying they’re now crying to core because Rails is broken for them is disingenuous. (In fact, the whole “they’ve forsaken that opportunity for an arms-crossed alternative.” thing smells like something of a strawman argument – I didn’t get that from the interview, so unless that’s come out of some backchannel, it seems a peculiar accusation to make.)
• Solutions that work for Twitter may not work for other Rails apps. Once you exhaust all of the conventional scaling wisdom, you start to move into application-specific optimisations. Are we there yet? I guess we’ll wait and see.

When the storm is over
This is a learning opportunity for everyone, but those who stand to learn the most are the core team and the Twitter guys. I hope each can continue to be open to the other’s position. We need to remember at the end of the day, Twitter are under no obligation to release patches, and the core team are under no obligation to accept them – I hope that doesn’t happen though, as the community would be the worse for it.

Tasmania is a small place, which is why I’m surprised that very few people in the IT industry here have heard of Morfik, a Hobart based development company. Reading this morning’s Read/Write Web however, things may be about change there. As reported by Richard, slashdot are running a post about the Morfik patent which appears to be aimed squarely at Google’s GWT.

I’m in two minds about this, as I always am about software patents. On the one hand, Morfik was clearly first to the party in terms of public release of a JavaScript compiler, according to the patent they had a provisional granted prior to the release of the GWT. On the other, I have to wonder how this is any different to the multitude of language translation techniques already out there, or at least whether it’s significantly inventive enough to pass the “not obvious to a practitioner in the field” test.

But hey, this wouldn’t be the first patent to be granted despite the obviousness of it to someone with half a brain. I mean, thank goodness the patent doesn’t mention Ruby, I guess that means RJS is safe!

The version I posted yesterday had a dodgy tag cloud implementation that I was experimenting with. I’ve reworked it this morning between other tasks and am much happier now. I imported some delicious bookmarks to populate the db a bit more and the result is this:

Much better!

(more…)

ScratchyPad Edit

Originally uploaded by Warren_Seen.

Editing an entry in the scratchpad.

ScratchyPad Tags

Originally uploaded by Warren_Seen.

Displaying all entries tagged ’scratchypad’