Access Blocked
Having problems with edit, search, diff and are being referred to this page?
See solution below.
Sensei's Library tries to protect itself from dumb mirroring scripts that issue some thousand requests within minutes bringing our server to its knees. (Please use SLSnapshot instead of mirroring the whole server.) Three measures have been taken:
- checking for referrer information
- hidden trap link
- limiting requests/minute
Referrer information
A first measure is to block access to any function other than viewing a page, if there is no referrer information present. What does this mean?
Every time you click on a link your browser sends a request to our server to get the desired page and pictures. This request not only contains the pagename which you would like to see, but also which page you are coming from (referrer information).
However, mirroring scripts don't send this information. So checking for the referrer information is an easy way to distinguish between scripts and regular browsers. On the other hand, many standard configurations of browsers and firewalls refuse to send referrer info too. See below for solutions.
If there is no referrer information then everything but viewing a page is blocked (e.g. diff, edit, save, search, pageinfo, ...).
If you get the "AccessBlocked" message as a regular user than either you are not using a standard browser, or have configured your browser in a way to not send the referrer information, or a proxy you are using is removing this information.
As the referrer information has to originate from within SL it is no longer possible to link to diff, pageinfo, etc. from other websites. Note: you can still link to pages themselves.
Hidden trap link
As a second measure, there's a trap link on the pages for scripts to follow if they don't obey the robots.txt file. Users should not normally be able to see this link. (You can see it in the source, but don't try it out or your address will be blocked for 48 hours. Really. We mean it.)
Limit requests/minute
Some scripts issue many requests per minute. We now enforce a limit of 35 requests/minute. Any IP issuing more requests is first warned and subsequently blocked for 6 hours.
If you are a regular user and got blocked by this, then send us an email. Maybe you should reconsider your browsing habits :o)
Comments
The above measures should shield SL from the most offensive scripts. What if you would still like to have a local copy of SL?
-> Download a ready packed snapshot at SLSnapshot.
See also: Why mirroring is bad.
Contact Arno or Morten if you have further questions.
SirLyric: Arno, would it be possible to add something like an additional GET parameter that could override the referrer check? I've written a Mozilla/Firefox search plugin that would allow the user to search SL from the Firefox search bar (something I often find myself wanting to do), but the referrer check makes this impossible to do without the user having set the magic cookie. Checking for something like 'sourceid=Mozilla-search' would work just fine as that string is standard for search plugins. See http://mycroft.mozdev.org/contribute.html for more information on what I'm trying to accomplish. Thanks!
Arno: email me.
Has something been done about this, or could it be? I made a web shortcut to search the site in Konqueror but it doesn't work for the same reason.
SirLyric: Best to contact Arno directly on this one.
togo I would suggest to rather check the user-agent instead of the referrer. Same profit, less collateral damage.
Solutions to missing referrer problem
If you are using your normal browser and cannot perform edits, searches, or view diffs then either your browser, your personal firewall or some proxy in between removes the referrer information (explanation see above). The easiest way to circumvent this check is to go to UserPreferences and just click "Set preferences". If you have an account you can log in too. Both actions set a cookie and the referrer check is circumvented if you have a cookie set.
With Norton Internet Security on a Windows platform: Start --> Programs --> Norton Internet Security --> Security Configuration (my translation from Dutch) --> Privacy control (same remark, please correct) --> <unmark> enable privacy
Phelan: To make Norton Personal Firewall send referrer info only to Sensei's Library, open Norton Personal Firewall, and click on the "Options" tab. There, in the "Web Content" tab, you can add Sensei's Library to the list of sites, and then change "Information about visited sites" from "Use default settings" to "Permit". This works on NPF 2003.
With elinks, changing a line in $HOME/.links/links.cfg from "http_referer 0" to "http_referer 1" worked for me.