[Welcome to Sensei's Library!]

RecentChanges
StartingPoints
About


Referenced by
Wikinews

 

Access Blocked
    Keywords: SL description

Sensei's Library tries to protect itself from dumb mirroring scripts that issue some thousand requests within minutes bringing our server to its knees.

A first meassure is to block access to any function other than viewing a page, if there is no referrer information present[1]. What does this mean?

Every time you click on a link your browser sends a request to our server to get the desired page and pictures. This request not only contains the pagename which you would like to see, but also which page you are coming from (referrer information).

However, mirroring scripts don't send this information. So checking for the referrer information is an easy way to distinguish between scripts and regular browsers.

If there is no referrer information than everything but viewing a page is blocked (e.g. diff, edit, save, search, pageinfo, ...).

If you get the "AccessBlocked" message as a regular user than either you are not using a standard browser, or have configured your browser in a way to not send the referrer information, or a proxy you are using is removing this information. Solution: change your settings.

As the referrer information has to originate from within SL it is no longer possible to link to diff, pageinfo, etc. from other websites. Note: you can still link to pages themselves.

As a second messure, if the misbehaving script insists on requesting such pages over and over it will be dynamically added to a block list for 48 hours. There's also a trap link on the pages for scripts to follow. Users should not normally be able to see this link. (You can see it in the source, but don't try it out or your address will be blocked for 48 hours. Really. We mean it.)

The above meassures should shield SL from the most offensive scripts. What if you would still like to mirror/download SL? Use a friendly script such as wget which obeys robots.txt. If you use wget don't forget to specify a wait period between the requests (at least "-w 3"). Yes, it will take some hours, but that way our server will still be accessible to others as well. If this advice isn't followed we may think of even more restrictive meassures. You have been warned.

Contact ArnoHollosi or MortenPahle if you have further questions.

[1] This check is circumvented, if you have the SLPrefs cookie set. (e.g. Mozilla currently doesn't send referrer information if you open a page in a new window).


Gorobei Dumb Question: you do have a robots.txt files to keep well-behaved spiders from hammering the site?

Arno: yes we do: [ext] http://senseis.xmp.net/robots.txt But those dumb scripts used recently don't obey robots.txt.



I'm getting 'Access blocked because of missing referrer information' if I try to use 'open in new window' to get at a diff page. Would it perhaps be possible to allow access to agents which send an SLPrefs cookie, even in the absence of referrer information? (using oldish Mozilla (0.9.1) on Gnu/Linux.)

--Matthew Woodcraft

Added by-pass of check for users who have the cookie set. I verified this with Mozilla 0.9.4 - bug is still there. Actually it's known to the Mozilla team as bug #48902 for over a year now. It seems that the next build will contain a fix. We will see.... --Arno



This is a copy of the living page "Access Blocked" at Sensei's Library.
(C) the Authors, published under the OpenContent License V1.0.