Check out Grab-A-Site software from Blue Squirrell. You can find it through google. Danielle Wiese Florida Staet University

From: "rafel Lucea" <rafel@MIT.EDU> Reply-To: air-l@listserv.aoir.org To: <air-l-aoir.org@listserv.aoir.org> Subject: [Air-l] Mapping the net with crawlers/robots Date: Wed, 20 Oct 2004 10:50:27 -0400

Hi all,

I am trying to analyze the relationships between organizations on the web. In particular, I want to map the linking behavior of a set of organizations subjectively defined.

I have explored a number of software packages (Website Watcher, Sphinx, MnoGoSearch, issuecrawler...) but they are either not thought for this specific purpose (WW, Issuecrawler) or require coding abilities that are beyond my knowledge (Sphinx -OS).

I would be most grateful if someone could indicate me whether there exists some web crawler that allows to define - a set of URLs from where to start the crawl - the depth - how many levels one wants to look in a given target domain - and number of iterations -how far from the original URL domain one wants to go. - and a few filters -limit specific types of pages (pdf for example)

and returns either a map, a table of relationships (some sort of adjacency matrix) or both.

Thanks in advance,

Rafel Lucea MIT - Sloan School of Management

_______________________________________________ The Air-l-aoir.org@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org

Join the Association of Internet Researchers: http://aoir.org/airjoin.html