How to handle Web Crawler Activities ?

narasimha · Post by **narasimha** » Sun Jan 28, 2007 12:59 pm

What should be done in future for crawler activities in our posts?

- Ignore them? Let Walter look into it.
- Should anybody reply with comments?
- Should someone just mail Walter, when such activities happen?

Some rules need to be set up as to what action needs to be taken in such conditions. (Till these crawlers are handled more cleverly by the site)

ray.wurlod · Post by **ray.wurlod** » Sun Jan 28, 2007 2:07 pm

We email Walter, but he is entitled to his weekends. The first to encounter the problem should email Walter and post that he/she has done so, so that Walter does not get deluged with emails.

It might be worth seeking legal advice. I don't know what US law is; such activity is illegal here in Australia, under the Spam Act. There have been successful prosecutions.

DSguru2B · Post by **DSguru2B** » Sun Jan 28, 2007 2:11 pm

For a signal for other, maybe start a new post with the heading "Email sent to Walter regarding Spam" or something along those lines. I sent him an email at the first sigt of spam activity and later realized that Craig had already sent one. I got to know that when i went through all the posts.

narasimha · Post by **narasimha** » Sun Jan 28, 2007 2:13 pm

I also noticed along with others, that the forum which gets hit the most is the "Editor's Corner".
The reason could be because it the first one to appear in the Forum list.

DSguru2B · Post by **DSguru2B** » Sun Jan 28, 2007 2:19 pm

narasimha wrote:I also noticed along with others, that the forum which gets hit the most is the "Editor's Corner".
The reason could be because it the first one to appear in the Forum list.

Ditto observation and conclusion.

ray.wurlod · Post by **ray.wurlod** » Sun Jan 28, 2007 3:21 pm

Perhaps a new first forum is required, in which posts are automatically deleted and the poster barred. Flamed would be nice, but this never gets back to the perpetrator.

chulett · Post by **chulett** » Sun Jan 28, 2007 4:24 pm

Only the brain-dead spambots stay in the first forum and reply to existing threads. I've seen plenty of other occurrences on other sites, new posts can show up in any forum.

Others have suggested a hidden first forum as Spam Bait, like a Roach Motel if that means anything to people out there ( spam checks in but it doesn't check out ) but it's not a silver bullet.

You need to deny them their ability to auto-register and post right away. Besides the image recognition you've probably seen elsewhere (What does this blot look like? Umm... bacon!) another such way is a manual approval step for new posters, but that puts a burden on Walter and may prove a barrier to growth. Anyone sign up with ADN way back when it was a one man manual process and remember how long you had to wait before you could start posting?

Ray, I don't think anything can get back to the perpetrator - unless you click on the link and I don't think that's the 'get back' you had in mind. Keep in mind the fact that if you do click on the link, that fact gets back to the spammer and let's them know it is working - and your site name goes on 'the list'. Please resist the urge to see exactly what she may or may not have inside waiting just for you...

DSguru2B · Post by **DSguru2B** » Sun Jan 28, 2007 4:26 pm

Thats what I was thinking as well. The forum could be named as "Spam Catchers", a general alert can be made about the forum. Something to that effect.

ray.wurlod · Post by **ray.wurlod** » Sun Jan 28, 2007 6:45 pm

Roach Motel is a great name in this context, and a weird memory.

narasimha · Post by **narasimha** » Sun Jan 28, 2007 7:43 pm

Roach Motel - Thats a nice one!

We subscribe to a Belgium company called fantomaster to get a list of valid world wide search engine IPs, User Agents and domain names.
We restrict the others, from crawling our site.

HTH

kumar_s · Post by **kumar_s** » Sun Jan 28, 2007 11:52 pm

Is there any way to validate Datastage License if that been entered during process of registration. That will allow only those who have access to Datastage.

ray.wurlod · Post by **ray.wurlod** » Mon Jan 29, 2007 12:57 am

That would count me out some of the time. I am a self-employed consultant; there are times when I am not doing DataStage things.

ArndW · Post by **ArndW** » Mon Jan 29, 2007 2:29 am

kumar_s wrote:...validate Datastage License...

I'd be out of here as well since I can't afford my own license, I just leech off the clients that I work for

sud · Post by **sud** » Mon Jan 29, 2007 2:38 am

chulett wrote:... another such way is a manual approval step for new posters, but that puts a burden on Walter and may prove a barrier to growth. Anyone sign up with ADN way back when it was a one man manual process and remember how long you had to wait before you could start posting?...

Actually we can have new registrations only through two channels :

1> Through reference from an existing user
2> Manual Approval from Walter

That way Walter won't get killed with requests

DSguru2B · Post by **DSguru2B** » Mon Jan 29, 2007 8:01 am

I think most of us who are working consultants would not be able to use DSXchange if a license number is required

. Merci merci