Warren Seen

The web spider writeup continues. I am no longer best friends with red wine after the weekend however.

Speaking of things that are red, I *love* the expressiveness of Ruby. The following is the run method of my Scheduler class, Ruby makes it a piece of cake to understand what is going on here, even without the full context, I think the following is fairly intuitive…

def run
  while not stop?
    if @spider_queue.empty?
      sleep(Timeout)
    else
      # if we haven't reached the concurrency limit
      # schedule a spider
      if @spider_threads.list.length < ThreadLimit
        @spider_threads.add Thread.new {
          Spider.new.process(remove)
        }
      end
    end
  end
  # join threads
  @spider_threads.list.each {
    |spider_thread| spider_thread.join
  }
end

(For those wondering, yes, it runs in its own thread).

Anyway, this was really only a test of the SyntHihol plugin for Wordpress. It took a bit of setting up as to get Ruby highlighting, I needed the latest version of GeSHi, and I had to turn off TinyMCE, because it kept stealing my indentation

This entry was posted on Tuesday, February 28th, 2006 at 12:03 am and is filed under programming, ruby, spiders, web. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.