Today they were featured on Qik, interviewed by Scoble himself. Check it out.

Now I was happy with my design, on the recommendation of a friend, I shot a PDF off to Click Business Cards via their website, and within 24 hours my cards had been printed and express posted to my door (they arrived yesterday). Their website may not be much chop, but their service sure is.

With a little hesitation, I cracked open the box, hoping that I’d gotten the resolution, colours and bleeds right. I had, they look excellent, if I do say so myself!

I’m the first to admit I’m no designer, and I’ll cop to the fact that the design was largely inspired by Dave Seah’s cards, whilst the loco image and ruby red was chosen to tie into the fact that I’m moving towards specialising in Ruby on Rails.

So, with new cards, I realised that the Wordpress template I’ve been using was pretty tired.

Time for some branding re-alignment!

A quick CSS change and a new header image later, here we are. I’ve kept the layout the same for the time being, I don’t want to sink a lot of time into a new Wordpress template as I’m thinking of packing this whole site up and moving to a Slicehost slice. If I do that, it will be goodbye Wordpress, hello to either Typo or Mephisto.

For the time being however, it’s steady as she goes!

Oh, and before he was born:

So despite the fact that I’m lying on the couch there, I’ve been far from idle.

My yard still looks like this:

so it’s not as though i don’t have enough to do! However, I *am* committing to finishing up, in the very near future, the Ruby Web Spider I began to write about some 5-6 months ago.

Consider this, a dusting off of the cobwebs kinda post…

You may also recognise some of the code in Scheduler#run from a short post I made to check that the syntax highlighting was working

First I want to recap the goal of the scheduler before getting into the code itself. Simply put, the scheduler exists to mangage the list of URIs (web pages, RSS feeds) that need to be spidered, and to manage the spiders themselves. In particular, we want to be able to limit the number of spiders working at any one time, out of politeness if nothing else.

I’m not going to make this a tutorial in Ruby syntax by explaining things line by line, if you haven’t used Ruby before and find something you don’t understand, the PragProg book, Programming in Ruby is the place to go look.

So let’s take a peek at some code!

Class declaration

module RWS
  class Scheduler < RWS::Service
    include Singleton

    def initialize
      super
      @spider_queue = Queue.new
      @spider_threads = ThreadGroup.new
    end

Here we can see the basic structure of the RWS::Scheduler, which inherits from RWS::Service (we'll get back to that in a moment.) The key things to note here are the spider_queue, which is the basis for our URI queue, and spider_threads, a thread group which will contain separate threads for each spider instance executing.

Queue management

    # schedule a URI for spidering
    def add(uri)
      begin
        @spider_queue << URI.parse(uri).normalize
        @logger.info("Scheduled: #{uri}")
      rescue
        @logger.info("Invailid:  #{uri}")
      end
    end

    # get a URI to spider
    def get_uri
      @spider_queue.pop
    end

Here we see two extremely simple (for now) methods to work with the URI queue. Because Ruby's Queue class is inherently thread safe (it's designed specifically to allow synchronisation between threads), we can safely operate without the need for explicit synchronisation. This is not a big deal at the moment, but will become important as we build out the spidering infrastructure.

Service control

    def run
      super {
        if @spider_queue.empty?
          sleep(@@settings["timeout"])
        else
          # if we haven't reached the concurrency limit
          # schedule a spider
          if @spider_threads.list.length < @@settings["thread_limit"]
            @spider_threads.add Thread.new { Spider.new.process(get_uri) }
          end
        end
      }
    end

    def shutdown
      super {
        @spider_threads.list.each { |spider_thread| spider_thread.join }
      }
    end
  end
end

This is probably the most interesting code we'll see today. Scheduler#run and Scheduler#shutdown both override base class methods of RWS::Service only to call the base class method with a code block (the bit between the {}'s)!

Whilst it seems slightly counter-intuitive, this allows the subclass to provide its own "work block", without having to deal with the additional overhead of service management. In the case of Scheduler#run, the code block is executed repeatedly, until the service receives a signal to stop. #run will sleep when the URI queue is exhausted, otherwise it will attempt to create a new work thread for a spider, which retrieves a URI to request via #get_uri, unless the concurrent threads limit has been reached.

The block passed into the #shutdown method in contrast is executed only once. This calls join on each thread, effectively blocking until each thread in turn finishes, thus initiating a graceful termination.

The service superclass

Here's a look at the base class beneath the scheduler. As you can see it provides simplistic state management through the #run, #stop and #shutdown methods. I think it's sufficiently straightforward to not need any further explanation.

  class Service < RWS::Base
    def initialize
      super
      @state = :Stop
    end

    def run
      @state = :Running
      while @state == :Running
        yield if block_given?
      end
    end

    def stop
      if @state == :Running
        @state = :Stop
      end
    end

    def shutdown
      stop
      @state = :Shutdown
      @logger.info("#{self} Shutting down...")
      yield if block_given?
    end
  end

Issues so far:

The best thing I can say about the current code is that it works with no unintended side effects. There are a number of areas which I need to go back and assess, after all this is supposed to be a learning experience.

1. Too much object creation going on. For each URI, a new Spider instance and a new Thread instance are created. As we will see later, the Spider class is relatively stateless, it should be simple to re-use.
2. Likewise, thread pooling would be a better approach than the current thread group to which new threads are constantly being added.
3. Queue#pop can be used in a blocking fashion, it will block until there is something in the queue to process. This may be better than the fixed timeout approach currently in use.
4. Use of blocks instead of callbacks - maybe I just got a bit too excited with the ability to use a block for the Service#run method.
5. I'm not particularly fond of using a class variable @@settings, so I'm looking to extract this and inject a settings instance into the scheduler. This is somewhat difficult given that I am using the Singleton mixin so cannot simply pass settings into a constructor.
6. I'm still tainted by Java... I find myself doing things in an overly convoluted Java-esque fashion. This habit is hard to break, but I will

So there you have it, the first proper installment of code for the web spider. I hope to shortly begin uploading actual .rb files containing what I've covered here, but a couple of housekeeping jobs still to be done on that front. I'll add links here when the code is up.

So how did I start?

As I mentioned in my first post on this topic, the web spider I’ve been developing is simply a framework to test out some ideas of mine, as well as being a fun little ruby project to fill what little spare time I have.

The ideas I’m tossing around at the moment require a broad collection of documents to analyse, so it seemed pretty clear that the way to go was to narrow the initial application to blog posts, and that I would therefore need a spider that could discover not only page links, but also RSS feeds.

My initial reflex was to sketch out the few basic components I thought I would need and allow a rough architecture to evolve from there.

The initial concept called for four components: a scheduler, a spider, a harvester, and an analyser. This is a cleaned up version of my initial sketch – the first one had a lot of crossing out, so much so that even I was getting confused about what my initial “design” was.

I’ll spare you my handwriting, I’ve copied the annotations numbered 1-4, they are:

1. The scheduler provides a public interface to trigger the seeding of the URI queue, and control the spiders. Additional URIs can be submitted through the scheduler and added to the queue.
2. The spiders retrieve a URI for processing, and pass the retrieved data to the harvester. Additional URIs detected by the spider are inserted back into the URI queue.
3. The harvester collects the data retrieved by the spider and inserts this data into a “page” cache, an opaque storage service for HTML or RSS data. Currently this is filesystem based, however there is no reason that harvester subclasses could not store this data in a DB, virtual FS or other appropriate datastore (that’s why it’s called “opaque” ). This also triggers inesrtion into the analysis queue a record of retrieved data – along with a reference to the file in the page cache, the record contains additional metadata such as URI, referring URI, time of retrieval, etc.
4. An analyser retrieves data to process, removing the record from the analysis queue and using this to retrieve the file from the page cache. And then some magic we’re not talking about happens here

As you can see, there is no real mention of where each of these components will run, or indeed exactly how they will be implemented. As this series continues, we’ll build up to those details…

Well, that’s it for this exciting installment, comments, criticism and offers of high paying contracts gladly accepted below

Next: Part 1: the scheduler

Speaking of things that are red, I *love* the expressiveness of Ruby. The following is the run method of my Scheduler class, Ruby makes it a piece of cake to understand what is going on here, even without the full context, I think the following is fairly intuitive…

def run
  while not stop?
    if @spider_queue.empty?
      sleep(Timeout)
    else
      # if we haven't reached the concurrency limit
      # schedule a spider
      if @spider_threads.list.length < ThreadLimit
        @spider_threads.add Thread.new {
          Spider.new.process(remove)
        }
      end
    end
  end
  # join threads
  @spider_threads.list.each {
    |spider_thread| spider_thread.join
  }
end

(For those wondering, yes, it runs in its own thread).

Anyway, this was really only a test of the SyntHihol plugin for Wordpress. It took a bit of setting up as to get Ruby highlighting, I needed the latest version of GeSHi, and I had to turn off TinyMCE, because it kept stealing my indentation

Anyway, I’m almost at a point where I’m happy to share this code (probably under GPL) as it’s not exactly rocket science (and I’ve only invested a week or so of evenings into it), but it has a couple of neat tricks that made it a good exercise in Ruby. A short laundry list of features:

• Multi-process and multi-threaded
• Can parse RSS, Atom and HTML
• Capable of distributing workload across multiple machines, via a queue based scheduler (using DRb)
• Respects robots.txt and is generally well behaved. Can set process and thread limits per machine or across the pool
• disk-based storage of spidered pages (for now)
• Asynchronous analyser(s) can process pages independent of the spiders’ execution

I won’t be releasing the analysers I’m using right now (gotta keep a bit of mystery) but I will release a generic analyser that you will be able to extend (ie subclass) to implement your own analysis, classification, etc.

Given the scope of the system (even though the code itself is simple), I’ll break it down according to function, and show how I’ve fitted it all together. I hope to have the first part up by Sunday evening.

edit:

Part 0 is now up, you can read it here.