Badly lagging forum this morning

We are getting random blips in resource usage which slows it down for a minute before it goes back to normal. I rebooted incase it was a hung service causing it and then the whole server became inaccessible. The recovery console showed it had no external connection. Tried loads of things to fix it including even trying old backups and they had the same issue. Was just in a support queue and suddenly it started working again.

It's still getting the random resource usage blips though so I'm trying to work out the cause

 

Is it working better for everyone now? I hope so as it took me over 13 hours of tweaking things to get on top of it. The problem is we are in an annoying middle ground when it comes to servers.

We have way more concurrent connections than our server setup is recommended for out of the box (It has taken alot of tweaking but it generally able to handle way more connections that it can out of the box) and in an ideal world we would move to a more advanced setup for future proofing but the growth isn' fast enough to justify the extra cost and complexity (and neither does the income support it) so its just about continually tweaking things to combat malicious traffic better and keep making it run more efficiently for legit traffic so we can put needing to do that off until users reach that critical mass that we have no choice.

 

They are loading with a bit of a delay , we are getting 5-6k viewing at any one time compared to 5-600 a few years ago... I think thats the issue

 

Yeah basically for that amount of traffic the recommended server setup is

1 x Low powered server as Load Balancer
2/3 x Mirrored Server Web server nodes speced atleast double the resources we are using for 1 right now (Basically you would land on the Load Balancer machine, which would then look at the Server nodes and send you to the one with the least load)
1 x Database Server (Speced higher than the current server)
1 x Cache Server that stores all the static files and can serve these quickly

That recommended setup though would be very expensive, beyond what a site like this can really jusify (maybe adding a cache server or a dedicated DB server might be viable especially if the slight ad rate rebound that basically every site has seen over the last couple of months after basically the steady reduction the industry has seen post covid) but its meant alot of tweaking of settings to make this server as efficient as possible so it generally can handle it fairly well. There are still times it can slow down, especially if there's a wave of malcious traffic which since last year has seen a real increase internet wide (Although we can even deal with that a bit better right now with all the tweaks that allow us to automatically manage malicious more efficiently so its less likely I need to intervene)

 

Just a moment ago, we had over 6,000 'guests' (alongside 40 or so genuine members) looking at the site! It then reduced to around 2,000. A quick sample of the guests appeared to suggest we have many thousands of City fans in Bejing which was a little surprising .. at least to me! If having so many 'guests', probably bots, is geniinely part of the speed issue, maybe part of the solution is to stop site viewing from China and some other countries?

 

Bloody hell! It's worse than I thought. One of these si-called guests even has my log-in details on a yellow post-it. The nerve of it!

 

The problem is banning full on countries isn't foolproof. The server doesn't actually know where a visitor is from, it just knows the IP address. To find the country you have to run a lookup and if it was running a lookup on every single IP that would just overload the server more (and get the server banned from lookups)

So what you have to do is use a local database that translates IP's to Countries but you need to have have that database to call upon and as its country IP's aren't sequential and what country an IP is assigned to can change the database has to be regularly updated. There used to be one that was completely free but that has switched to a tiered system, there is still a free version which is what we use but it updates less frequently and has less data.

And this less data can then become a problem as alot of VPN's have exit nodes in the problem counters. Wth the full data you can identify when it's from a legit source better but as it is we can't so we have to be more cautious.

And even if we just fully blocked them the problem is it still has to be processed by the server which requires resources to do and under too heavy load this processing can actually just make the situation worse.