http://warrenseen.com/blog/2006/03/03/ruby-web-spider-part-0-concept/ freelance software developer Tue, 02 Dec 2008 22:15:31 +0000 http://wordpress.org/?v=2.5.1 http://warrenseen.com/blog/2006/03/03/ruby-web-spider-part-0-concept/#comment-858 Phill Midwinter Fri, 23 Feb 2007 12:17:15 +0000 http://warrenseen.com/blog/2006/03/03/ruby-web-spider-part-0-concept/#comment-858 Not a bad start, you may find it worth putting the cache into DB if you want to search it effectively, you should also try breaking down the page into what are known as 'barrels'. Basically a fat list of all the keywords on the page and their associated attributes (density, colour, font size whatever floats your boat). This makes it a lot easier to search later on. Not a bad start, you may find it worth putting the cache into DB if you want to search it effectively, you should also try breaking down the page into what are known as ‘barrels’. Basically a fat list of all the keywords on the page and their associated attributes (density, colour, font size whatever floats your boat). This makes it a lot easier to search later on.

]]>