Almost immediately I started getting offers to buy the domain name. Each inquiry I got, I would knock back as I have always had plans as to what I wanted to work on for future versions of the site. Most of these seemed like tire-kickers who I never heard from again.

But there was one guy who kept coming back asking about the site. Early last month, he forwarded a very specific offer via a 3rd party to me that made me realise he was VERY serious about acquiring HashJobs. After a bit of back and forth negotiating, and consideration of what I could make from the site in the next 12-18 months if I had a serious go at it, I happily accepted his offer.

So I’m pleased to announce that as of this morning, the final installment of our transaction has cleared, and Jason Davis of recruitingblogs.com is the new owner of HashJobs. I’ve had a sneak peek at Jason’s plans for the site and I think he is going to do it far more justice than I could have, given his position in the online recruiting community. I wish him all the best.

(As a side note, I am amazed how quickly incoming wire transfers clear, compared to transfers between local banks!)

As for my plans, well at least I’ve filled a GFC-induced hole in my previous 6 months cashflow. People who I’ve told about this always ask me, “Can yo do the same again in a different field?” The answer is, “Probably not, it was more luck than good management.”

I think I’ll just start another side project (I have a few ideas) and see where the wind takes me…

Today they were featured on Qik, interviewed by Scoble himself. Check it out.

You may also recognise some of the code in Scheduler#run from a short post I made to check that the syntax highlighting was working

First I want to recap the goal of the scheduler before getting into the code itself. Simply put, the scheduler exists to mangage the list of URIs (web pages, RSS feeds) that need to be spidered, and to manage the spiders themselves. In particular, we want to be able to limit the number of spiders working at any one time, out of politeness if nothing else.

I’m not going to make this a tutorial in Ruby syntax by explaining things line by line, if you haven’t used Ruby before and find something you don’t understand, the PragProg book, Programming in Ruby is the place to go look.

So let’s take a peek at some code!

Class declaration

module RWS
  class Scheduler < RWS::Service
    include Singleton

    def initialize
      super
      @spider_queue = Queue.new
      @spider_threads = ThreadGroup.new
    end

Here we can see the basic structure of the RWS::Scheduler, which inherits from RWS::Service (we'll get back to that in a moment.) The key things to note here are the spider_queue, which is the basis for our URI queue, and spider_threads, a thread group which will contain separate threads for each spider instance executing.

Queue management

    # schedule a URI for spidering
    def add(uri)
      begin
        @spider_queue << URI.parse(uri).normalize
        @logger.info("Scheduled: #{uri}")
      rescue
        @logger.info("Invailid:  #{uri}")
      end
    end

    # get a URI to spider
    def get_uri
      @spider_queue.pop
    end

Here we see two extremely simple (for now) methods to work with the URI queue. Because Ruby's Queue class is inherently thread safe (it's designed specifically to allow synchronisation between threads), we can safely operate without the need for explicit synchronisation. This is not a big deal at the moment, but will become important as we build out the spidering infrastructure.

Service control

    def run
      super {
        if @spider_queue.empty?
          sleep(@@settings["timeout"])
        else
          # if we haven't reached the concurrency limit
          # schedule a spider
          if @spider_threads.list.length < @@settings["thread_limit"]
            @spider_threads.add Thread.new { Spider.new.process(get_uri) }
          end
        end
      }
    end

    def shutdown
      super {
        @spider_threads.list.each { |spider_thread| spider_thread.join }
      }
    end
  end
end

This is probably the most interesting code we'll see today. Scheduler#run and Scheduler#shutdown both override base class methods of RWS::Service only to call the base class method with a code block (the bit between the {}'s)!

Whilst it seems slightly counter-intuitive, this allows the subclass to provide its own "work block", without having to deal with the additional overhead of service management. In the case of Scheduler#run, the code block is executed repeatedly, until the service receives a signal to stop. #run will sleep when the URI queue is exhausted, otherwise it will attempt to create a new work thread for a spider, which retrieves a URI to request via #get_uri, unless the concurrent threads limit has been reached.

The block passed into the #shutdown method in contrast is executed only once. This calls join on each thread, effectively blocking until each thread in turn finishes, thus initiating a graceful termination.

The service superclass

Here's a look at the base class beneath the scheduler. As you can see it provides simplistic state management through the #run, #stop and #shutdown methods. I think it's sufficiently straightforward to not need any further explanation.

  class Service < RWS::Base
    def initialize
      super
      @state = :Stop
    end

    def run
      @state = :Running
      while @state == :Running
        yield if block_given?
      end
    end

    def stop
      if @state == :Running
        @state = :Stop
      end
    end

    def shutdown
      stop
      @state = :Shutdown
      @logger.info("#{self} Shutting down...")
      yield if block_given?
    end
  end

Issues so far:

The best thing I can say about the current code is that it works with no unintended side effects. There are a number of areas which I need to go back and assess, after all this is supposed to be a learning experience.

1. Too much object creation going on. For each URI, a new Spider instance and a new Thread instance are created. As we will see later, the Spider class is relatively stateless, it should be simple to re-use.
2. Likewise, thread pooling would be a better approach than the current thread group to which new threads are constantly being added.
3. Queue#pop can be used in a blocking fashion, it will block until there is something in the queue to process. This may be better than the fixed timeout approach currently in use.
4. Use of blocks instead of callbacks - maybe I just got a bit too excited with the ability to use a block for the Service#run method.
5. I'm not particularly fond of using a class variable @@settings, so I'm looking to extract this and inject a settings instance into the scheduler. This is somewhat difficult given that I am using the Singleton mixin so cannot simply pass settings into a constructor.
6. I'm still tainted by Java... I find myself doing things in an overly convoluted Java-esque fashion. This habit is hard to break, but I will

So there you have it, the first proper installment of code for the web spider. I hope to shortly begin uploading actual .rb files containing what I've covered here, but a couple of housekeeping jobs still to be done on that front. I'll add links here when the code is up.

So how did I start?

As I mentioned in my first post on this topic, the web spider I’ve been developing is simply a framework to test out some ideas of mine, as well as being a fun little ruby project to fill what little spare time I have.

The ideas I’m tossing around at the moment require a broad collection of documents to analyse, so it seemed pretty clear that the way to go was to narrow the initial application to blog posts, and that I would therefore need a spider that could discover not only page links, but also RSS feeds.

My initial reflex was to sketch out the few basic components I thought I would need and allow a rough architecture to evolve from there.

The initial concept called for four components: a scheduler, a spider, a harvester, and an analyser. This is a cleaned up version of my initial sketch – the first one had a lot of crossing out, so much so that even I was getting confused about what my initial “design” was.

I’ll spare you my handwriting, I’ve copied the annotations numbered 1-4, they are:

1. The scheduler provides a public interface to trigger the seeding of the URI queue, and control the spiders. Additional URIs can be submitted through the scheduler and added to the queue.
2. The spiders retrieve a URI for processing, and pass the retrieved data to the harvester. Additional URIs detected by the spider are inserted back into the URI queue.
3. The harvester collects the data retrieved by the spider and inserts this data into a “page” cache, an opaque storage service for HTML or RSS data. Currently this is filesystem based, however there is no reason that harvester subclasses could not store this data in a DB, virtual FS or other appropriate datastore (that’s why it’s called “opaque” ). This also triggers inesrtion into the analysis queue a record of retrieved data – along with a reference to the file in the page cache, the record contains additional metadata such as URI, referring URI, time of retrieval, etc.
4. An analyser retrieves data to process, removing the record from the analysis queue and using this to retrieve the file from the page cache. And then some magic we’re not talking about happens here

As you can see, there is no real mention of where each of these components will run, or indeed exactly how they will be implemented. As this series continues, we’ll build up to those details…

Well, that’s it for this exciting installment, comments, criticism and offers of high paying contracts gladly accepted below

Next: Part 1: the scheduler

Anyway, I’m almost at a point where I’m happy to share this code (probably under GPL) as it’s not exactly rocket science (and I’ve only invested a week or so of evenings into it), but it has a couple of neat tricks that made it a good exercise in Ruby. A short laundry list of features:

• Multi-process and multi-threaded
• Can parse RSS, Atom and HTML
• Capable of distributing workload across multiple machines, via a queue based scheduler (using DRb)
• Respects robots.txt and is generally well behaved. Can set process and thread limits per machine or across the pool
• disk-based storage of spidered pages (for now)
• Asynchronous analyser(s) can process pages independent of the spiders’ execution

I won’t be releasing the analysers I’m using right now (gotta keep a bit of mystery) but I will release a generic analyser that you will be able to extend (ie subclass) to implement your own analysis, classification, etc.

Given the scope of the system (even though the code itself is simple), I’ll break it down according to function, and show how I’ve fitted it all together. I hope to have the first part up by Sunday evening.

edit:

Part 0 is now up, you can read it here.

Anyway, precis on BlogCode is fairly straight forward, drop the name and url (not the feed tho) of the blog you want to code into the UI, then score the blog on a range of sliding scales covering content, tone, etc.

Neat, but essentially useless in and of itself.

The trick however is that the sliding scale scores allow statistical analysis to determine the blogs that match according to the criteria rated. eg for this blog, a range of sites are returned with a 72-78% “match”.

The underlying idea is that over time, as more people code a blog, the ratings will become a more accurate and democratic opinion of the blog in question.

I noticed that they do collect the country that the blogger appears to be from, however it doesn’t seem that you can use this to narrow searches, etc.

The problem I can see with this system however is this: there are a LOT of categories to rank, and although it doesn’t take long with sliding scales to do this, the novelty of assessing a blog over 15+ criteria is going to get old pretty fast.

Additionally, there’s no real incentive to come back to the site again and again. This feature would be more useful as an element of existing blog search (eg Technorati) or built into an aggregator (eg Rojo) as a value add. Imagine you add a blog to your feeds, and get suggestions for similar blogs, with sample content so you can decide whether you want to read them too. Not to mention being able to sort blogs according to their BlogCode “match” level.
Alternatively, it could serve as the basis of a meme tracker in which you can influence the results you see based on blogs you’ve coded.

In short, an interesting statistical experiment, but the model (which has been carried over from storycode.com), doesn’t seem to be that tight a fit with the blogging state of the art. I mean, no RSS feeds, what were they thinking?

• Steve Rubel takes the classic high school debate approach of defining the word and building an argument from there. Conclusion – yes, it’s not authority, it’s popularity, people.
• Data mining agrees with Steve, “name things for what they are, not for what they are used for”. That is quite obviously right out of Usability 101.
• Jack Krupansky leaves an excellent comment on Scobelizer: “to Technorati, “authority” is simply popularity. That makes *no* sense.“

Consensus seems to be that tracking popularity but calling it “authority” muddies the waters… There is nothing wrong with the feature itself, it’s just the name that’s misleading.

Technorati has just added ‘authority’ filtering to their search (see Scoble, TechCrunch, Dave Sifry, et cetera).

My issue with this is simple – the number of inbound links does NOT necessarily qualify the source as being an “authority” on anything. This is nothing more than a popularity contest.

Ben Barren hits the nail on the head here.

So its very hard to determine relative popularity, on a regional basis

Before pointing to this New York Metro quote:

In the blogosphere, the biggest audiences—and the advertising revenue they bring—go to a small, elite few. Most bloggers toil in total obscurity.

Popularity, that’s all it is. And that’s sad, because some of the people who deserve to be held as authorities in their field are lost amongst the noise, while the “blogosphere” (gack, I HATE that “word”) becomes more and more like a conversation between a panel of “A-list” bloggers with everyone else on the sideline. Then you’ve flipped from being “citizen” media to just media.

Disparate thoughts to follow…

• Blog search the way Technorati does it is fundamentally broken. Ranking based purely on “link love” is primitive at best. Calling it “Authority” filtering is inaccurate. That’s a big call to make, but I’m going to be dumb and stand by that one…
• It’s broken because it causes new blogs to be put in a catch 22 – if you want to get noticed, you have to get links. If you want to get links you need to get noticed in the first place.
• Even if you can get some traffic driven to your blog from eg Technorati, how do you make it sticky? The majority of my page views reported by awstats are in the 30 second region. The next highest is the 30-60 second. It tails off quickly after that.
• Have you ever wondered about the great bloggers you haven’t read? How will you ever find them if no one links to them?
• Don’t you wish you could find blogs similar to the ones you already read? Or ones that provide a counter-point? ie “Show more like this…”
• I’m not talking about just keyword matching here, it’s a far more sophisticated problem than it appears. Memetrackers DON’T do this… bet you wish they could.
• What if your aggregator sorted items from your feeds by relevance? Think a mutant cross between reddit and amazon.com’s “for you” for your feeds. Key word filtered “smart” folders are dumb by comparison.
• What if technorati did the same? Why should I have to manually create my watch list, when technorati knows what I search for regularly? Hell, they know what I tag, that’s a BFC (big effing clue) right there…

time for the (long) tail to wag the dog.

Seems they travelled a similar path to one of my previous employers, via the ANZATech conference. I’ll leave it as an exercise to the reader to figure out which company that was. Hope it works out better after Demo than it did for the guys from Foursticks
Nice work guys, keen to see the video when it’s up on the Demo website.

Zingee – an Australian-based company, who claims they’ve traveled the furthest to get to Demo.  They allow you to share files from your computer, without any uploads.  A neat way to preserve your privacy!

Not just me though, apparently Michael Arrington at TechCrunch hadn’t heard of them last week.

I finally got a look at newcomer storage service Zingee, which would have been included on my “Online Storage Gang” post if they had been around.

Trying to find out a bit more, but all I can see is a couple of phone numbers on their site (only one in Sydney), and their profile on the Demo site, which actually lists Singapore contact details.

This is not really surprising, Singapore is a better place to build to flip than Australia according to some accounting folks I’ve worked with.

In any case, this may be another local outfit to watch.