Sunday, March 18, 2007

Suzanne Shell and the Spiders on the Web

A woman by the name Suzanne Shell has sued Archive.org for the spiders crawling her site. The Colorado woman says she posted a notice on the site saying it she doesn't allow the site to be crawled.She sued for civil theft, breach of contract, and violations of the Racketeering Influence and Corrupt Organizations act(hell, what are they, never heard of them, after all she is an American ...phew).However, this is a late report by me and a court ruling last month dismissed all charges on Archive.com , except for the breach of contract claim. If Shell prevails on that claim, sites like Google will have to get online publishers to 'opt-in' before it can crawl their sites.

It seems it is more of her mistake.As a web-designer or an owner ,she shud've known all the ways to prevent crawling of the site.One of the important thing is to put up a 'robots.txt' file which she didn't. And it is too foolish to argue that the notice shudve served the purpose when it is well known that spiders don't understand the text which they crawl.

The case seems to be completely stupid.If she wanted to protect the page from being crawled , why didn't hse obtain a security certificate for it and why didnt she password protect it giving IDs to people who want to visit the site.Once a page is put up on the Web as a 'public' page ,which a page is by default, there is no use saying it is not intended for public.It is akin to a book once published and sold and then banned.

And what was the Internet for? When she launched the site without password protection, SHE ,infact had ignored the rules of the Net.

If I owned Archive.com , I wud've looked up the rules and sued her instead(ummm,they already know the rules...I'm waiting for the lawsuit to come out. American style :P). And if Miss Shell (or her lawyer,btw) was any more sensible ,they wudve withdrawn the case for the simple reason that there was no robots.txt instead of dropping 'shell's .It is simply the way the system works. Ignorance is no excuse.

I know it too, the hard way though.If she wins the case, it would only result in the emergence of new 'shell's to throw more 'shell's at such companies which crawl the Net.