How to handle Web Crawler Activities ?
Moderators: chulett, rschirm, roy
How to handle Web Crawler Activities ?
What should be done in future for crawler activities in our posts?
- Ignore them? Let Walter look into it.
- Should anybody reply with comments?
- Should someone just mail Walter, when such activities happen?
Some rules need to be set up as to what action needs to be taken in such conditions. (Till these crawlers are handled more cleverly by the site)
- Ignore them? Let Walter look into it.
- Should anybody reply with comments?
- Should someone just mail Walter, when such activities happen?
Some rules need to be set up as to what action needs to be taken in such conditions. (Till these crawlers are handled more cleverly by the site)
Narasimha Kade
Finding answers is simple, all you need to do is come up with the correct questions.
Finding answers is simple, all you need to do is come up with the correct questions.
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
We email Walter, but he is entitled to his weekends. The first to encounter the problem should email Walter and post that he/she has done so, so that Walter does not get deluged with emails.
It might be worth seeking legal advice. I don't know what US law is; such activity is illegal here in Australia, under the Spam Act. There have been successful prosecutions.
It might be worth seeking legal advice. I don't know what US law is; such activity is illegal here in Australia, under the Spam Act. There have been successful prosecutions.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
For a signal for other, maybe start a new post with the heading "Email sent to Walter regarding Spam" or something along those lines. I sent him an email at the first sigt of spam activity and later realized that Craig had already sent one. I got to know that when i went through all the posts.
Creativity is allowing yourself to make mistakes. Art is knowing which ones to keep.
Ditto observation and conclusion.narasimha wrote:I also noticed along with others, that the forum which gets hit the most is the "Editor's Corner".
The reason could be because it the first one to appear in the Forum list.
Creativity is allowing yourself to make mistakes. Art is knowing which ones to keep.
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Perhaps a new first forum is required, in which posts are automatically deleted and the poster barred. Flamed would be nice, but this never gets back to the perpetrator.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Only the brain-dead spambots stay in the first forum and reply to existing threads. I've seen plenty of other occurrences on other sites, new posts can show up in any forum.
Others have suggested a hidden first forum as Spam Bait, like a Roach Motel if that means anything to people out there ( spam checks in but it doesn't check out ) but it's not a silver bullet.
You need to deny them their ability to auto-register and post right away. Besides the image recognition you've probably seen elsewhere (What does this blot look like? Umm... bacon!) another such way is a manual approval step for new posters, but that puts a burden on Walter and may prove a barrier to growth. Anyone sign up with ADN way back when it was a one man manual process and remember how long you had to wait before you could start posting?
Ray, I don't think anything can get back to the perpetrator - unless you click on the link and I don't think that's the 'get back' you had in mind. Keep in mind the fact that if you do click on the link, that fact gets back to the spammer and let's them know it is working - and your site name goes on 'the list'. Please resist the urge to see exactly what she may or may not have inside waiting just for you...
Others have suggested a hidden first forum as Spam Bait, like a Roach Motel if that means anything to people out there ( spam checks in but it doesn't check out ) but it's not a silver bullet.
You need to deny them their ability to auto-register and post right away. Besides the image recognition you've probably seen elsewhere (What does this blot look like? Umm... bacon!) another such way is a manual approval step for new posters, but that puts a burden on Walter and may prove a barrier to growth. Anyone sign up with ADN way back when it was a one man manual process and remember how long you had to wait before you could start posting?
Ray, I don't think anything can get back to the perpetrator - unless you click on the link and I don't think that's the 'get back' you had in mind. Keep in mind the fact that if you do click on the link, that fact gets back to the spammer and let's them know it is working - and your site name goes on 'the list'. Please resist the urge to see exactly what she may or may not have inside waiting just for you...
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Roach Motel - Thats a nice one!
We subscribe to a Belgium company called fantomaster to get a list of valid world wide search engine IPs, User Agents and domain names.
We restrict the others, from crawling our site.
HTH
We subscribe to a Belgium company called fantomaster to get a list of valid world wide search engine IPs, User Agents and domain names.
We restrict the others, from crawling our site.
HTH
Narasimha Kade
Finding answers is simple, all you need to do is come up with the correct questions.
Finding answers is simple, all you need to do is come up with the correct questions.
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
I'd be out of here as well since I can't afford my own license, I just leech off the clients that I work forkumar_s wrote:...validate Datastage License...
<a href=http://www.worldcommunitygrid.org/team/ ... TZ9H4CGVP1 target="WCGWin">
</a>
</a>
Actually we can have new registrations only through two channels :chulett wrote:... another such way is a manual approval step for new posters, but that puts a burden on Walter and may prove a barrier to growth. Anyone sign up with ADN way back when it was a one man manual process and remember how long you had to wait before you could start posting?...
1> Through reference from an existing user
2> Manual Approval from Walter
That way Walter won't get killed with requests
It took me fifteen years to discover I had no talent for ETL, but I couldn't give it up because by that time I was too famous.