This may sound like a dumb question, but, how much time does a spider usually spend on a site? I would assume that it depends on how many pages they need to crawl?

The reason I ask is that I've been watching an IP which I've traced to MSN search that's been on my site for almost 45 minutes now and it's all over the place, page wise that is. Do they use up allot of bandwidth or slow down the site?

Thanks!

Mike

Depending on how big your site is 45 mins could be ok.

You can also edit your robots text file and put CONTENT="nofollow" for the msn bot. This will reduce the impact of the bot.

Ok....now I have a dumber question....why would we want to keep that particular kind of bot off our site?

Thank you for the info. I guess it's a give to get deal. I checked my webstats and found that all spiders combined for this month have used almost 300 meg of bandwidth with Google being the biggest eater. Still can't get more then a dozen listings in Google so I guess I need to research this to see why they take so much and give so little.

Thanks again!

Mike

Ok, my bad....I had you confused for someone else. I clicked on your link and see you're an auction site.

If I may ask, why wouldn't you want the spiders & bots indexing your site? You say they "take so much and give so little." But I disagree. They are indexing. As an auction site owner, why wouldn't you want this done?

Now, if you want to limit the directories that the spiders/bots can index, you can disallow them from those directories (your administrator, includes, or any other directory they don't need to index).

Am I missing something? Other site owners? PSU? Why chase the spiders/bots away to preserve bandwith? Isn't that like cutting off your nose to spite your face? I wanna learn here....

sylvia

Hi angelsolutions. What I meant by " take so much and give so little" was that Google is a hungry little bugger but I've got almost no listing in Google. I just don't understand what they are doing there if they won't index me. Last week HotBot and another had almost 2000 results each for my site, Google, 4. I don't mind the bandwidth, I just wish I could get some results for it.

Mike

Thanks for the response. I'll be darned if I'll ever "get it." LOL

I get ALOT of searches from google itself, and I can see that many of those end up as buys. But the keyword strings they're using *aren't* the ones I would use to find my stuff. go figger.... Not only that...but I don't find me there.

oh well....if I figgered that one out, I'd be rich, huh? LOLOLOL

Best wishes & thanks for your patience,
sylvia

No problem angel. I don't get it either. Keep content fresh, keep keywords accurate, keep description under 200 characters, and blah blah blah, to get good rankings. All the research I've done just leads me in circles. I've found websites in Google that rank #1 and have no meta tags at all. If you check ebay at this site: http://www.widexl.com/remote/search-engines/metatag-analyzer.html for meta tag analysis it shows ebay's tags are not very good. But you can't help running into their links all over Google, MSN, and others. I swear I'd give my big toes to have someone honestly explain this to me!!

O well, I'll just keep beating my head on the desk!

Mike

300MB is not bad at all. Googlebot alone has used over 15GB of bandwidth so far this month at Satisfaction.com.

IMO, bring it on! The more indexing the better!

I was doing a little research and found that there are apparently some spiders out there that hammer your site for the sole purpose of stealing content, images, email address's, code, and so on. Google is a supposed to be pretty good at image harvesting. I guess that makes sense since you can search images on their site and find pretty much anything you want. I've read that the bad part is some spiders don't obey the robots.txt file and they can slow down the site to a crawl.

Here's one of the articles I read for anyone interested;

http://www.revenews.com/wayneporter/archives/001072.html

Cya later

Mike

Mike, if your worried, talk to your host. They can tell you whats going on with the bots.

And just remember that one of the reasons we would want the googlebot to index images is that the images go with our products/auctions.

Yes, I've read about bad bots...good reason to keep an eye on what bots are visiting you.

Somehow, I don't get a bad feeling about the search engine spiders, though, dude. I dunno.

sylvia

I asked my host about spiders/bots and pretty much any info I can get is in my control panel log files. Anyway, there are 9 different bots that have been to my site and when I check my "who's online" there is always at least one bot/spider on the site, sometimes as many as 4. I only wanted to know about them slowing down my site and using allot of bandwidth. I guess it's no big deal. I'm going to edit my .htaccess file to allow only Google, MSN, Jeeves, and Alltheweb access to my site.

My site runs slow enough now let alone when it's full of bots/spiders!

Take care all

Mike