We went through different infrastructure options and drew conclusions. Below we briefly describe what we tried up (and with what result).
VPS - of the many tested, unfortunately none worked. Mainly due to resource management by the hypervisor and also the restrictions (often hidden) imposed by the hosting.
Dedicated server - seems to be a good solution, but here the challenge is IOPS (disk access time). On harddrives, IOWAIT could reach 30% of the time. SSD/NVMe drives, on the other hand, have another disadvantage: the minimum block size is 4KB, and with a lot of small files ~300B, yes, the space is there, but no inodes :/
We wenth with both NVMe and harddrives, in pairs in RAID1. The NVMe drives hold what needs to be accessed quickly (database, user records, images) and the harddrives hold what might run slower (php and perl scripts, json files for images etc).
Apache? - works well with perl (more on that later) and php, but is too slow.
Nginx? - can't handle perl properly.
OpenLiteSpeed? - crashed under heavy load.
What is left? NGINX + Apache - it's almost perfect. Almost, because users on the other side of the globe are queuing for images etc. CDN solves this problem and, in the process, relieves the load on the server.
Every hour the server downloads all the data from all active InnoGames servers and loads it into the database. Our proprietary bash+perl mechanism, in addition to downloading and loading data, detects if there is nothing to download from a server (because it has been closed) and also whether a new server has been opened. The data is updated in such a way that unavailability to users is less than 1s, while the actual time it takes to download and load it into the database (currently 1591 files for all active worlds) is: ~15m
We tried various database engines, but MySQL proved to be the fastest (table partitioning, text file loading speed)
We tried to get any support from InnoGames, but they were not interested. Instead, they were happy to implement our solutions in the game (for example, the city resercation module or the hero coin count display) 😉
There are no ads for two reasons. Firstly, there were advertisements for adults which we didn't want, but also had no control over. Secondly, InnoGames banned the display of in-game ads. We decided that the site didn't need to be one big billboard and bet on donations - which kind of worked out for the first 2 years.
CPU: Intel Xeon E3-1230v6 – 4c/8t – 3.5 GHz/3.9 GHz
RAM: 32 GB ECC 2133 MHz
drives: 2×450 GB SSD NVMe (Soft RAID), 2×2 TB HDD SATA (Soft RAID) – just to store ~3,000,000 files.
as it is not a subject to as much load as the main server, a fraction of its parameters will suffice. It can't be an Atom though, because it's too slow. So any Intel Xeon with sufficient disk space. The amount of RAM does not matter.
The current parameters of the backup server, are:
CPU: Intel Xeon E3-1225v2 - 4c/4t - 3.2 GHz/3.6 GHz
RAM: 16 GB 1333 MHz
drives: 3×2 TB HDD SATA Soft RAID (RAID5)
Reach us out on our Discord (link in the footer of this page)
|Cities||4 105 862|
|Conquests||7 290 266|
|Players using GRCRTools||10 694|
|Saved settings GRCRTools||114 735|
|Stored images for reports||993 086|
|Reports in the format HTML||704 816|
|Reports in the form of an image||763 473|