The Rancor Pit Forum Index
Welcome to The Rancor Pit forums!

The Rancor Pit Forum Index
FAQ   ::   Search   ::   Memberlist   ::   Usergroups   ::   Register   ::   Profile   ::   Log in to check your private messages   ::   Log in

Site Outages / Call for Assistance
Post new topic   Reply to topic    The Rancor Pit Forum Index -> The Rancor Pit Users/Forums Info -> Site Outages / Call for Assistance
View previous topic :: View next topic  
Author Message
Whill
Dark Lord of the Jedi (Owner/Admin)


Joined: 14 Apr 2008
Posts: 10286
Location: Columbus, Ohio, USA, Earth, The Solar System, The Milky Way Galaxy

PostPosted: Sat Jan 08, 2022 10:04 pm    Post subject: Site Outages / Call for Assistance Reply with quote

Lately, you may have experienced one or more service outages lasting up to a few minutes at the Rancor Pit. I have been informed by my web hosting company that it is because the Rancor Pit has become so massive with pages that some web spider bots crawling the site are using a lot of resources, so the host temporarily throttles the service.

Some bots are good, like the ones that crawl for the major search engines that keep Rancor Pit web search results current. Those use minor resources and aren't actually the problem. Others are just website analytic services that crawl the entire web so they have the data should you elect to subscribe to their services. I don't.

The below bots are the main offenders here. From my research so far, none of these are important to the Rancor Pit. I was shown how to block IPs from the entire domain, but I need help. I am still recovering from surgery. I am in pain, very tired, and have minor dizzy spells. In anyone could find out the range of IP addresses these bots use, please report it to me (with a web page source for the info) so that I may go in and block them all. This should eliminate the throttling. Thank you.

AhrefsBot
SemrushBot
MJ12bot
YandexBot
DotBot
_________________
*
Site Map
Forum Guidelines
Registration/Log-In Help
The Rancor Pit Library
Star Wars D6 Damage
Back to top
View user's profile Send private message Visit poster's website
cheshire
Arbiter-General (Moderator)


Joined: 04 Jan 2004
Posts: 4833

PostPosted: Sun Jan 09, 2022 3:20 pm    Post subject: Re: Site Outages / Call for Assistance Reply with quote

Whill wrote:


AhrefsBot

51.222.152.133
54.36.148.1
195.154.122.

Quote:

SemrushBot

46.229.168.0/24

Quote:

MJ12bot


The best I can figure is that this one uses a dynamic IP range, and most recommend using robots.txt to disallow.

Quote:

YandexBot

FROM TO
100.43.64.0-100.43.79.255
100.43.80.0-100.43.80.255
100.43.81.0-100.43.81.255
100.43.82.0-100.43.82.255
100.43.83.0-100.43.83.255

Quote:

DotBot

This one again uses a dynamic IP range, and it looks like they'll tell you how to exclude them in the robots.txt file.

https://moz.com/help/moz-procedures/crawlers/dotbot
_________________
__________________________________
Before we take any of this too seriously, just remember that in the middle episode a little rubber puppet moves a spaceship with his mind.
Back to top
View user's profile Send private message Visit poster's website
Whill
Dark Lord of the Jedi (Owner/Admin)


Joined: 14 Apr 2008
Posts: 10286
Location: Columbus, Ohio, USA, Earth, The Solar System, The Milky Way Galaxy

PostPosted: Sun Jan 16, 2022 3:57 pm    Post subject: Reply with quote

Thank you. I blocked all the IPs above, but I'll have to figure out how to do the robot text thing.
_________________
*
Site Map
Forum Guidelines
Registration/Log-In Help
The Rancor Pit Library
Star Wars D6 Damage
Back to top
View user's profile Send private message Visit poster's website
cheshire
Arbiter-General (Moderator)


Joined: 04 Jan 2004
Posts: 4833

PostPosted: Sun Jan 16, 2022 4:19 pm    Post subject: Reply with quote

I'm imagining you in a lineup of droids saying,

"What I really need is a droid that understands the programing language of php."

Though seriously, I've always outsourced the person who does our robots.txt, but I know a gal who can probably give some pointers. I'll be having a sit down chat with her in a couple of weeks, I think. I'll see if I can bring this up.
_________________
__________________________________
Before we take any of this too seriously, just remember that in the middle episode a little rubber puppet moves a spaceship with his mind.
Back to top
View user's profile Send private message Visit poster's website
Display posts from previous:   
Post new topic   Reply to topic    The Rancor Pit Forum Index -> The Rancor Pit Users/Forums Info All times are GMT - 4 Hours
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group


v2.0