Warren Seen

This is the second part of a series of posts covering the development of my web spider in Ruby. You can read about the initial idea here, and the architecture in Part 0: Concept.

You may also recognise some of the code in Scheduler#run from a short post I made to check that the syntax highlighting was working

First I want to recap the goal of the scheduler before getting into the code itself. Simply put, the scheduler exists to mangage the list of URIs (web pages, RSS feeds) that need to be spidered, and to manage the spiders themselves. In particular, we want to be able to limit the number of spiders working at any one time, out of politeness if nothing else.

I’m not going to make this a tutorial in Ruby syntax by explaining things line by line, if you haven’t used Ruby before and find something you don’t understand, the PragProg book, Programming in Ruby is the place to go look.

So let’s take a peek at some code!

Class declaration

module RWS
  class Scheduler < RWS::Service
    include Singleton

    def initialize
      super
      @spider_queue = Queue.new
      @spider_threads = ThreadGroup.new
    end

Here we can see the basic structure of the RWS::Scheduler, which inherits from RWS::Service (we’ll get back to that in a moment.) The key things to note here are the spider_queue, which is the basis for our URI queue, and spider_threads, a thread group which will contain separate threads for each spider instance executing.

Queue management

    # schedule a URI for spidering
    def add(uri)
      begin
        @spider_queue << URI.parse(uri).normalize
        @logger.info("Scheduled: #{uri}")
      rescue
        @logger.info("Invailid:  #{uri}")
      end
    end

    # get a URI to spider
    def get_uri
      @spider_queue.pop
    end

Here we see two extremely simple (for now) methods to work with the URI queue. Because Ruby’s Queue class is inherently thread safe (it’s designed specifically to allow synchronisation between threads), we can safely operate without the need for explicit synchronisation. This is not a big deal at the moment, but will become important as we build out the spidering infrastructure.

Service control

    def run
      super {
        if @spider_queue.empty?
          sleep(@@settings["timeout"])
        else
          # if we haven’t reached the concurrency limit
          # schedule a spider
          if @spider_threads.list.length < @@settings["thread_limit"]
            @spider_threads.add Thread.new { Spider.new.process(get_uri) }
          end
        end
      }
    end

    def shutdown
      super {
        @spider_threads.list.each { |spider_thread| spider_thread.join }
      }
    end
  end
end

This is probably the most interesting code we’ll see today. Scheduler#run and Scheduler#shutdown both override base class methods of RWS::Service only to call the base class method with a code block (the bit between the {}’s)!

Whilst it seems slightly counter-intuitive, this allows the subclass to provide its own “work block”, without having to deal with the additional overhead of service management. In the case of Scheduler#run, the code block is executed repeatedly, until the service receives a signal to stop. #run will sleep when the URI queue is exhausted, otherwise it will attempt to create a new work thread for a spider, which retrieves a URI to request via #get_uri, unless the concurrent threads limit has been reached.

The block passed into the #shutdown method in contrast is executed only once. This calls join on each thread, effectively blocking until each thread in turn finishes, thus initiating a graceful termination.

The service superclass

Here’s a look at the base class beneath the scheduler. As you can see it provides simplistic state management through the #run, #stop and #shutdown methods. I think it’s sufficiently straightforward to not need any further explanation.

  class Service < RWS::Base
    def initialize
      super
      @state = :Stop
    end

    def run
      @state = :Running
      while @state == :Running
        yield if block_given?
      end
    end

    def stop
      if @state == :Running
        @state = :Stop
      end
    end

    def shutdown
      stop
      @state = :Shutdown
      @logger.info("#{self} Shutting down...")
      yield if block_given?
    end
  end

Issues so far:

The best thing I can say about the current code is that it works with no unintended side effects. There are a number of areas which I need to go back and assess, after all this is supposed to be a learning experience.

1. Too much object creation going on. For each URI, a new Spider instance and a new Thread instance are created. As we will see later, the Spider class is relatively stateless, it should be simple to re-use.
2. Likewise, thread pooling would be a better approach than the current thread group to which new threads are constantly being added.
3. Queue#pop can be used in a blocking fashion, it will block until there is something in the queue to process. This may be better than the fixed timeout approach currently in use.
4. Use of blocks instead of callbacks - maybe I just got a bit too excited with the ability to use a block for the Service#run method.
5. I’m not particularly fond of using a class variable @@settings, so I’m looking to extract this and inject a settings instance into the scheduler. This is somewhat difficult given that I am using the Singleton mixin so cannot simply pass settings into a constructor.
6. I’m still tainted by Java… I find myself doing things in an overly convoluted Java-esque fashion. This habit is hard to break, but I will

So there you have it, the first proper installment of code for the web spider. I hope to shortly begin uploading actual .rb files containing what I’ve covered here, but a couple of housekeeping jobs still to be done on that front. I’ll add links here when the code is up.

This entry was posted on Wednesday, March 8th, 2006 at 12:05 am and is filed under programming, ruby, spiders, web, web2.0. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.