We need some bug spray

[Q]

There're a bajillion spiders crawling around here. I saw them snooping around when I was checking who's on line.

 

[IDLEchild]

What?...I am sorry what do you mean by that? You mean too many guests or hidden users etc?

 

[A.B.Normal]

Leslie posted something in the shoutbox yesterday about it ,fury went "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA ,so he's aware of the problem.Hopefully he can add a "No_bot" or something,I would hate to have everything we post be available via Google

 

[IDLEchild]

Oh you mean Spiders from google indexing this sites every thread?

 

[PT]

That is just a little creepy, I don't want EVERYTHING available via search engines either. Are we doing something about this?

 

[fury]

That is just a little creepy, I don't want EVERYTHING available via search engines either. Are we doing something about this?

A lot of our traffic comes from people finding us through search engines, so I don't see a problem with it. Besides, search engines don't make EVERYTHING available to their users, they make a preview of the page with the search terms they're looking for and then link them to the actual page, so they will click on over to OTC.

A.B.: I was going AAAAAAAAAAAAAAAAAAAAH for totally unrelated reasons.

 

[Dave]

yeah, but you didn't have any spiders checking your profile....

 

[A.B.Normal]

A.B.: I was going AAAAAAAAAAAAAAAAAAAAH for totally unrelated reasons.

My bad ,thought maybe the prospect of being probed was making you anxious .


Doesn't the constant Spider visits use up bandwidth that could be better spent on useless drivel?

 

[fury]

We've only used 875 megs so far this month (that's over 4 days). I think we're doing fine in the bandwidth department.

(GZIP compression helps out a lot, plus I don't think spiders even download images except maybe attachments. )

 

[Q]

I don't know too much about these spider-bot thingys...but the fact that they broke our record for users online and then broke their own record the last two days in a row...does indeed, creep me right the fuck out. That's just waaaay too many spiders.

 

[alex]

Goddamn spiders wont probe my site

 

[catocom]

Just give it alitte time Pad.
And-to you could join a few yahoo groups, and mention your
site once or twice, and you'll have all the crap you can handle.

 

[alex]

I did submit a form for an exchange of links to everythingpcb.com.....that should pick things up a little soon.

 

[PT]

A lot of our traffic comes from people finding us through search engines, so I don't see a problem with it.

Yeah, I guess so. After thinking about it I guess it isn't that strange. I do find it odd that 22 of their little spiders would be here on the same day though.

 

[fury]

They do that for efficiency, so they don't have one spider crawling through the whole site (can you imagine how long that would take, even for a computer?)

 

[Raven]

still it is freaky though if they start going through our profile pages...if thats possible

 

[PT]

Alright, all these spiders are starting to creep me out. I can understand they would need to catalog everything, but damn, they're still showing up 20 and 30 at a time. Is this going to be from now on since we got listed at Google?

 

[Raven]

looks that way....can fury make a raid script?

 

[fury]

No, just once every week or 2, depending on how often Google updates their indexes. They use so many because it would be inefficient for just one spider to crawl through the site, it would never get done.

 

[Raven]

is it a roaming IP or does each one have a temporary IP or is there a fixed IP for each spider?