Learning To Crawl Web Forums

Present Forum Crawler Under Supervision (FoCUS), a supervised web-scale forum crawler. The goal of FoCUS is to crawl relevant forum content from the web with minimal overhead. Forum threads contain information content that is the target of forum crawlers. Although forums have di erent layouts or styles and are powered by di erent forum software packages, they always have similar implicit navigation paths connected by speci c URL types to lead users from entry pages to thread pages. Based on this observation, we reduce the web forum crawling problem to a URL-type recognition problem. And we show how to learn accurate and e ective regular expression patterns of implicit navigation paths from automatically created training sets using aggregated results from weak page type classi ers. Robust page type clas-si ers can be trained from as few as ve annotated forums and applied to a large set of unseen forums.