Jump to content
  • 1

How to optimize server-FPS and server performance


Tobias Solem

Question

It's a fact that ARMA has a dated architecture that only recently received some optimizations for multiple cores and utilizing them for different purposes (multi-tasking and multi-threading). There are several ways for you to optimize your server-side performance to get the maximum speed out of your Epoch server (because Epoch loads so many building items, it requires even more processing power).

 

Before I continue there is a fundamental thing you have to know and consider when you configure your Epoch server: 

 

* Every client will load and have to calculate everything that occurs on your server, meaning:

- Every client has to load ALL base building items

- Every client has to load ALL the vehicles

- Every client has to load and calculate ALL AI movement

- Every client has to load and calculate ALL clan logos/custom graphics

 

Another thing to consider is the fact that there is something called ragdoll and physics in ARMA 3, basically it is a set of calculations applied to each player body, each AI-body, and each car, and for a moment, also every placed building block. All vehicles run physics-calculations in real-time 24/7 (100% all the time) this means that for every car you have on your server, the server and each client will have to run the calculations for physics on that vehicle all the time. Naturally more calculations are required when something happens to that vehicle (driving, getting shot at, colliding, etc.) but essentially the computing power for all these physics & ragdoll items will take a toll.

 

Furthermore, all inventories and contents of traders, tents, vehicles, etc. are fed from the database in real-time, essentially the more items a container holds, the heavier the load on the server for each time that inventory is loaded. So if you have, say 300 items in a container, all those 300 will load in a big packet, which sucks some server performance. The less a container can hold, the less strain on the server.

 

Now, multiply physics, and inventory calculations per client, the more clients connected, the more vehicles driving, swimming, running, flying, inventories getting opened, the more the strain on each client, and on the server. A lot of this information is also sent from the server, and Arma 3 Epoch is one of the most bandwidth-intensive games out there. When the bandwidth is maximized on either the server-side or the client-side, you will see a yellow chain link in your bottom right corner, or a red chain link, depending on how overloaded the server or the client is with data. This is also when you will see lag happening. Whereas "desynch" (client and server not having the same information, and thus weird things happen) and rubberbanding (which looks like teleporting/warping) is usually a combination of over-stressed CPU (server and or client-side).

 

It is also important to know that in player interaction, the weakest link is usually the cause of the issues, for example: It doesn't matter if you are connected with a 1Gbit connection, with an awesome CPU and GPU, if the server is overloaded, that's the problem, you can't fix it clientside. But the same thing is if you and your friends are connected and your friend has a shitty Internet connection and/or computer and happens to be driving the car. He will lag/desynch based on his low-performance equipment, and that will affect you, and anyone trying to blow that car up (because the car will behave erratically). 

 

Virtual Machines

 

It should also be known that the arma3 server-file has issues with running on virtual machines (VMWare, or XenServer, etc.) essentially the performance output gets negated by as much as 30% if you run it on the same hardware as a dedibox, ARMA 3 is currently (2014-12-30) not well-optimized for running on Virtual Machines. Dedicated, native servers will see better performance than a VM.

 

 

How do I combat/fix this?

 

With these basic fundamentals in knowing how the server works, you as a player need to know that connecting to a server with hundreds of vehicles, or tens of thousands of buildables, or tons of AI-missions will not necessarily be a good thing for your game experience. While there may be plenty of content to have fun with, the experience with all that content will soon turn sour when your client FPS sinks (because the server has low performance, or is under too much strain (yes this affects clients too, if the server is the weakest link, all clients suffer)) your gameplay and experience will suffer too. 

 

Currently, the thoroughly tested, recommended max settings for cars on an Epoch server is 150, and the max amount of buildables is 1500 and the max amount of container content is 1500. This is the default settings in Epoch and it is what makes it possible for a server like Mellys to run 100 clients connected, with little to no issues. Yes, there will be a struggle to find a car, but the experience for each player will be as optimal as ARMA 3 Epoch can be with 100 players connected simultaneously.

 

The only possible compromise if you want to have more vehicles, more buildables, more missions, etc. is to allow for fewer clients (less than 100 players). It doesn't matter if the server has SSD-drives, or 1Gbit up and down, or 32GB DDR5 RAM on a super-high cache Xeon-processor. The arma3server-binary can't handle more than a certain amount of load anyways. 

 

 

No logs, memory handlers, and toggles

 

If you run a server with hyperthreading you should use the following toggle (do NOT combine it with cpuCounts=X or threads=X, etc.):

-enableHT

 

If you want to squeeze some more performance out of your server with an optimized memory allocator, use the following (download fred's dll here https://github.com/fred41/tbbmalloc_arma/raw/master/dwarden/uptodate_bins.zip ) and place its contents in your dlls-directory where you have arma3server:

-malloc=tbbmalloc 

 

IF you want to squeeze some extra FPS out of the server you can also turn off RPT-logging (not recommended if you are having issues or are actively tracking other problems/reports) by using:

-nologs

 

Setting the processor priority higher is also recommended:

-high

 
Server shutdowns
 
The recommended shutdown hours currently for optimal performance is 3 hours, the reason why we have to shut down the server with certain intervals is because of essentially two reasons: 1) Memory leaking (binaries leak RAM, and start performing poorly and thus need to be terminated and restarted) and 2) Clutter (basically a bunch of loops that get stuck due to poor code, vehicle explosions that take up unnecessary bandwidth, corpse clutter, and other things that are a detriment that increases over time on server performance).
 
Hardware
 
- CPU
ARMA 3 is processor clock dependent first (both client- and server-side). Having a CPU clock of 3.9GHz or higher is recommended. Intel Xeon-processors have been proven to perform better than regular non-server CPU:s. 
 
- RAM
More than 8GB per server has not been reported to make any difference whatsoever unless you run a lot of other things or multiple servers on the same machine
 
- Hard Drive Speed
SSD-hard drives improve loading performance severely, which means quicker reboots and startup-times for the server, but after the server is loaded everything is handled from the computers RAM, which is much faster than SSD-drives currently. 
 
- Bandwidth
 
Expect an ARMA 3 Server have been reported to use around 1Mbit per connected client (server-to-client upstream bandwidth). This means that with 50 players, you will need about 50Mbit upload. The server downloads much less data than this.
 
--------------------------------------------------
 
Bottom line
 
As a server owner you really need to consider the factors involved:
 
Amount of vehicles (more than 150 will mean you will have to lower your max allowed connected clients from 100)
Amount of buildables (more than 1500 will mean you will have to lower your max allowed connected clients from 100)
Amount of storage space (more than 1500 will mean you will have to lower your max allowed connected clients from 100)
 
 
Your server performance is measured in FPS and CPS - the standard FPS without load will be about 49-50FPS server-side, server-FPS and client-FPS are not the same thing, it is not as vital for server-FPS to be so high, everything 15FPS or higher will leave little to no issues. The same thing goes for CPS, which is the processing power used for Artificial Intelligence and non-gamebased calculations (like the AntiHack). The same principle for AI applies to this, the two values should be about the same normally. These values are heavily affected by (in order of performance-drain): 1) Connected clients, 2) Amount of vehicles 3) Amount of buildables 4) Contents of inventory
 
--------------------------------------------------
 
Link to comment
Share on other sites

Recommended Posts

  • 0

Lowrey - how many players are on your servers and what stats are u getting?

 

Psot 1.40 we see:

 

20 FPS @ 40/45 players (+/- 4 FPS).

12 FPS @ 60/65 Players (+/- 3 FPS) This is a 4 Core + 8 logical running at 80+% CPU usage with -enableHT and affinity set (using a third party app).

 

Pre 1.40 (i.e 1.36).

 

30 FPS @ 45/50 Players (+/- 5 FPS).

+12 FPS with 70+ players - This was an 8 Core + 8 Logical running at 35/46% CPU usage with -enableHT and affinity set (using a third party app).

 

 

The 8 Core even had local AI running server side.

 

I've said it be fore I'll say it again I'm happy for people to test on the 8 and 4 core servers to see if they see similar results. I think it would be good to get to a definitive place on this topic.

 

I'm also going to look at Linux this weekend. However results may take some time as I need go get the sever up to good numbers.

 

How about you?

Link to comment
Share on other sites

  • 0

Richie

 

Not sure if you saw this. 

 

E5-2690 v2 is a 10 core CPU with 20 threads. I see from your task manager you have 10 CPUs listed.. So I would say you have HT disabled in the bios... Or your OS does not have the support for that many CPU's

I would say if you don't have HT enabled (I don't think you do but I would check) then don't enable HT.

What is your current affinity for the a3's you are running.

Is this OS on tin or do you have a hypervisor between it ?

Link to comment
Share on other sites

  • 0

Just to chime in, we've been seeing around 11-14 Server FPS @ 92 players running stock epoch with just a few tweaks and no missions. The server at that point is utilizing approx 85-90% of resources. This is on a < week old hive with default or lower limits. It would seem 80 players is the sweet spot, with 17-20 Server FPS. Under 60 players we run FuMS HC and see between 18-20 FPS at peak before removing missions for the night. At 40-50 players with no missions on a fresh reboot the FPS is in the low 40s and after 2+ hours, in the mid 30s.

 

We use hyperthreading. 

 

Hardware

Xeon E3-1271v3

32 GB RAM

240GB  SSD

 

In our experience, the biggest issue with epoch is not its out of the box performance, but rather its performance two weeks down the road when database objects reach critical mass. A switch to MYSQL from Redis would be a major step in the right direction.

 

We are also looking into separating @epochhive and scripts therein to HC instances which we believe would significantly improve performance by allowing for a better distribution of server resources. 

Link to comment
Share on other sites

  • 0

Im testing a little with a big Base where many shelf´s (about 60) inside. Every where when i was inside the Base the FPS goes down to 35. When i´m teleporting away the fps goes up to 48 after 1-2 Minutes. I do it 3 Times u can see it on the Screenshot.

 

c4aanlcstl8v.png

Link to comment
Share on other sites

  • 0

Richie,

That is great to hear. So have you now set the affinity for wasteland and epoch now to be diff ??. If you did not have this set initially I can see why you may have been having issues as you would probably be fighting for resources shared.

If I was you I would do 4 for each server and then 2 for the HC.

Also have you tried -ExThreads ? 1,3 and 7 ? I no in theory 1 is what you want for server but I've seen a little diff with 5 3-4 fps

Link to comment
Share on other sites

  • 0

One more information regarding HT:

 

 

Hyper-threading support was 'disabled' in Armas until now which basically means that when the game detected HT it spawned threads only on one half of the CPU cores. With HT enabled Arma 3 will spawn threads on all CPU cores which means better parallelization and possibly slightly better performance.

It's not clear how much can hyper-threading help (depends on CPU architecture, threads scheduling and what is being simulated) but since we have done more parallelization optimizations lately (especially on dedicated servers) it might be worth trying this thing out. I would personally expect better results on the dedicated servers with high amount of players (the higher the better) since it's a little bit better parallelized but either way, don't expect any magical jumps in FPS.

It should also help non-HT CPUs with 8 or more cores. If you have non-HT CPU with less cores than there's simply no change - the game will run as usual.

And I second the encouragement to test and post back the results - I expect there will be differences based on you CPUs, SP/MP/dedicated servers, player count and missions used - but any feedback will be very welcome.

Source

 

I think you guys are on the right way to test without HT.

 

 

If you have a Core i7 or a 4 "real" core Xeon, for example, 4 real cores, 8 virtual "cores/threads" (with HT), and 1 servers on the machine, I would NOT use enableHT. Arma spawns then threads for 4 cores, you have 4 cores. Match. Maybe use cpucount 4.

 

If you have 2 servers on such a machine, I would use cpucount 2 (cpuCount / exThreads) for each server instance. Then set the affinity for the first server to the first half of the cores (0-1 / 0-3) and the affinity for the second server to the second half of the cores (2-3 / 3-7).

 

exThreads I would not touch in the beginning.

 

This is only information which configuration I would test. Only if somebody want's to test a bit and is looking for a place to start.

 

EDIT: In Richies case (10 real cores) I would try cpucount=4 on the first server, and set the affinity for core 0-3. Cpucount=4 on the second server, with the affinity 4-7, and cpucount=2 for the HC with affinity 8-9. No enableHT, no exThreads.

 

EDIT2:

 

Actually do not use -cpuCount and -enableHT together because -cpuCount will always override -enableHT. For example if you have a 4 core CPU with HT (8 cores in total) and you will use following parameters "-cpuCount=4 -enableHT" Arma 3 will use only 4 cores. Use -cpuCount if you want to explicitly specify how many cores should Arma 3 use (might be useful when running several dedicated servers on one machine or in similar setups) or use -enableHT if you simply want to use all the CPU cores.

Source

 

So enableHT just spawns threads for all the cores it can see. I could imagine, that if you have to much threads, the overhead becomes bigger and the performance goes down again.

 

BR

Rob

Link to comment
Share on other sites

  • 0

Had a play with not using ht last night and it doesn't make a huge difference but def drops 4 frames when disabling the HT 1,3,5,7.

Last night less players and server performed worse than the night before with identical settings. CPU just didn't want to work sitting at 50% all night and 15fps average.

 

Did you set the affinity only to 0,2,4,6? If you just have one arma instance on the machine, you should not need to fiddle around with the affinity.

Just try to not use the -enableHT option and use cpucount 4.

Link to comment
Share on other sites

  • 0

Ok have it all setup properly and testing it as we speak - 46 players online - 17fps on average. with it set as cpucount 4 and affinity set to only physical. 
If I then manually override the affinity to include the hyperthreads there is negligible difference except it seams that the fps in @ASM is a little more stable in regards to jumping up to 19 then down to 16, with using the hyperthreads it sits alot more settled on 17/18fps.

Update - 51 Players, Stable at 11fps :'(

Link to comment
Share on other sites

  • 0

Ok have it all setup properly and testing it as we speak - 46 players online - 17fps on average. with it set as cpucount 4 and affinity set to only physical. 

If I then manually override the affinity to include the hyperthreads there is negligible difference except it seams that the fps in @ASM is a little more stable in regards to jumping up to 19 then down to 16, with using the hyperthreads it sits alot more settled on 17/18fps.

 

 

When you run only one arma instance on the whole machine, and no headless client, then give the arma instance access to all ressources. This means, don't change the affinity, or set the affinity to all cpus.

However, I would just not use the -enableHT command line parameter when starting arma. But set the affinity to all cpus, when there is no need ro reserve ressources for other arma instances.

Do you use a second arma instance and/or a headless client on the same machine?

Link to comment
Share on other sites

  • 0

 

Brian Soanes - does that require clients to have the client file to see any benefit from it?

 

 

No, but they will see an improvement if they run client perf build

 

 

 

1. Close game

2. Download and unziphttps://www.dropbox.com/sh/582opsto4mmr8d8/AAASKQ8hTVRn69XL9VtVs9O_a/arma3client_performance_144_130768_v2.7z?dl=0

3. Rename file to arma3.exe

4. Copy file to your Arma 3 root folder

5. Start game as normal

Link to comment
Share on other sites

  • 0

Ok tested so many different combos today. Fact of the matter is the server performed better with HT than CPU Count. Server FPS was more stable and the FPS was 5-10 higher with 40 players. I tried several combos of settings across multiple restarts with only 1 instance running. Running it with just cpucount performed worst with the server quite regularly dropping 10fps for about 10 minutes for no visible reason - players not joining, network traffic usual etc. only factor was CPU usage dropped 20%.

Link to comment
Share on other sites

  • 0

Ok tested so many different combos today. Fact of the matter is the server performed better with HT than CPU Count. Server FPS was more stable and the FPS was 5-10 higher with 40 players. I tried several combos of settings across multiple restarts with only 1 instance running. Running it with just cpucount performed worst with the server quite regularly dropping 10fps for about 10 minutes for no visible reason - players not joining, network traffic usual etc. only factor was CPU usage dropped 20%.

 

There is no question that hyper threading improves performance on a dedicated server. I'm not sure why people are saying otherwise, but I guess it doesn't hurt to experiment. 

 

On our dedicated box it equates to about an additional 8-10 server FPS @ 50 players. 

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...