Jump to content
  • 1

How to optimize server-FPS and server performance


Tobias Solem

Question

It's a fact that ARMA has a dated architecture that only recently received some optimizations for multiple cores and utilizing them for different purposes (multi-tasking and multi-threading). There are several ways for you to optimize your server-side performance to get the maximum speed out of your Epoch server (because Epoch loads so many building items, it requires even more processing power).

 

Before I continue there is a fundamental thing you have to know and consider when you configure your Epoch server: 

 

* Every client will load and have to calculate everything that occurs on your server, meaning:

- Every client has to load ALL base building items

- Every client has to load ALL the vehicles

- Every client has to load and calculate ALL AI movement

- Every client has to load and calculate ALL clan logos/custom graphics

 

Another thing to consider is the fact that there is something called ragdoll and physics in ARMA 3, basically it is a set of calculations applied to each player body, each AI-body, and each car, and for a moment, also every placed building block. All vehicles run physics-calculations in real-time 24/7 (100% all the time) this means that for every car you have on your server, the server and each client will have to run the calculations for physics on that vehicle all the time. Naturally more calculations are required when something happens to that vehicle (driving, getting shot at, colliding, etc.) but essentially the computing power for all these physics & ragdoll items will take a toll.

 

Furthermore, all inventories and contents of traders, tents, vehicles, etc. are fed from the database in real-time, essentially the more items a container holds, the heavier the load on the server for each time that inventory is loaded. So if you have, say 300 items in a container, all those 300 will load in a big packet, which sucks some server performance. The less a container can hold, the less strain on the server.

 

Now, multiply physics, and inventory calculations per client, the more clients connected, the more vehicles driving, swimming, running, flying, inventories getting opened, the more the strain on each client, and on the server. A lot of this information is also sent from the server, and Arma 3 Epoch is one of the most bandwidth-intensive games out there. When the bandwidth is maximized on either the server-side or the client-side, you will see a yellow chain link in your bottom right corner, or a red chain link, depending on how overloaded the server or the client is with data. This is also when you will see lag happening. Whereas "desynch" (client and server not having the same information, and thus weird things happen) and rubberbanding (which looks like teleporting/warping) is usually a combination of over-stressed CPU (server and or client-side).

 

It is also important to know that in player interaction, the weakest link is usually the cause of the issues, for example: It doesn't matter if you are connected with a 1Gbit connection, with an awesome CPU and GPU, if the server is overloaded, that's the problem, you can't fix it clientside. But the same thing is if you and your friends are connected and your friend has a shitty Internet connection and/or computer and happens to be driving the car. He will lag/desynch based on his low-performance equipment, and that will affect you, and anyone trying to blow that car up (because the car will behave erratically). 

 

Virtual Machines

 

It should also be known that the arma3 server-file has issues with running on virtual machines (VMWare, or XenServer, etc.) essentially the performance output gets negated by as much as 30% if you run it on the same hardware as a dedibox, ARMA 3 is currently (2014-12-30) not well-optimized for running on Virtual Machines. Dedicated, native servers will see better performance than a VM.

 

 

How do I combat/fix this?

 

With these basic fundamentals in knowing how the server works, you as a player need to know that connecting to a server with hundreds of vehicles, or tens of thousands of buildables, or tons of AI-missions will not necessarily be a good thing for your game experience. While there may be plenty of content to have fun with, the experience with all that content will soon turn sour when your client FPS sinks (because the server has low performance, or is under too much strain (yes this affects clients too, if the server is the weakest link, all clients suffer)) your gameplay and experience will suffer too. 

 

Currently, the thoroughly tested, recommended max settings for cars on an Epoch server is 150, and the max amount of buildables is 1500 and the max amount of container content is 1500. This is the default settings in Epoch and it is what makes it possible for a server like Mellys to run 100 clients connected, with little to no issues. Yes, there will be a struggle to find a car, but the experience for each player will be as optimal as ARMA 3 Epoch can be with 100 players connected simultaneously.

 

The only possible compromise if you want to have more vehicles, more buildables, more missions, etc. is to allow for fewer clients (less than 100 players). It doesn't matter if the server has SSD-drives, or 1Gbit up and down, or 32GB DDR5 RAM on a super-high cache Xeon-processor. The arma3server-binary can't handle more than a certain amount of load anyways. 

 

 

No logs, memory handlers, and toggles

 

If you run a server with hyperthreading you should use the following toggle (do NOT combine it with cpuCounts=X or threads=X, etc.):

-enableHT

 

If you want to squeeze some more performance out of your server with an optimized memory allocator, use the following (download fred's dll here https://github.com/fred41/tbbmalloc_arma/raw/master/dwarden/uptodate_bins.zip ) and place its contents in your dlls-directory where you have arma3server:

-malloc=tbbmalloc 

 

IF you want to squeeze some extra FPS out of the server you can also turn off RPT-logging (not recommended if you are having issues or are actively tracking other problems/reports) by using:

-nologs

 

Setting the processor priority higher is also recommended:

-high

 
Server shutdowns
 
The recommended shutdown hours currently for optimal performance is 3 hours, the reason why we have to shut down the server with certain intervals is because of essentially two reasons: 1) Memory leaking (binaries leak RAM, and start performing poorly and thus need to be terminated and restarted) and 2) Clutter (basically a bunch of loops that get stuck due to poor code, vehicle explosions that take up unnecessary bandwidth, corpse clutter, and other things that are a detriment that increases over time on server performance).
 
Hardware
 
- CPU
ARMA 3 is processor clock dependent first (both client- and server-side). Having a CPU clock of 3.9GHz or higher is recommended. Intel Xeon-processors have been proven to perform better than regular non-server CPU:s. 
 
- RAM
More than 8GB per server has not been reported to make any difference whatsoever unless you run a lot of other things or multiple servers on the same machine
 
- Hard Drive Speed
SSD-hard drives improve loading performance severely, which means quicker reboots and startup-times for the server, but after the server is loaded everything is handled from the computers RAM, which is much faster than SSD-drives currently. 
 
- Bandwidth
 
Expect an ARMA 3 Server have been reported to use around 1Mbit per connected client (server-to-client upstream bandwidth). This means that with 50 players, you will need about 50Mbit upload. The server downloads much less data than this.
 
--------------------------------------------------
 
Bottom line
 
As a server owner you really need to consider the factors involved:
 
Amount of vehicles (more than 150 will mean you will have to lower your max allowed connected clients from 100)
Amount of buildables (more than 1500 will mean you will have to lower your max allowed connected clients from 100)
Amount of storage space (more than 1500 will mean you will have to lower your max allowed connected clients from 100)
 
 
Your server performance is measured in FPS and CPS - the standard FPS without load will be about 49-50FPS server-side, server-FPS and client-FPS are not the same thing, it is not as vital for server-FPS to be so high, everything 15FPS or higher will leave little to no issues. The same thing goes for CPS, which is the processing power used for Artificial Intelligence and non-gamebased calculations (like the AntiHack). The same principle for AI applies to this, the two values should be about the same normally. These values are heavily affected by (in order of performance-drain): 1) Connected clients, 2) Amount of vehicles 3) Amount of buildables 4) Contents of inventory
 
--------------------------------------------------
 
Link to comment
Share on other sites

Recommended Posts

  • 0

Let's try to keep this constructive, guys.  Everyone can point a finger from their respective side of the fence and say 'They should have done X or Y'.  Let's just work together and try to help make it better for everyone.  

 

Thing is, it is constructive for builds to be tested on populated servers, what happens on servers with >10 players is completely different to what happens on servers 50> players. With performance going down since 1.40/0.2.5.2 performance is a big thing as players lose interest.

Link to comment
Share on other sites

  • 0

Thing is, it is constructive for builds to be tested on populated servers, what happens on servers with >10 players is completely different to what happens on servers 50> players. With performance going down since 1.40/0.2.5.2 performance is a big thing as players lose interest.

 

 

I totally understand and agree.  However, once you make that suggestion people can't help themselves but to start pointing fingers back and forth between the private/public development debate.  There's pros and cons both ways.  Yes, more large scale testing is needed.  But what are the devs supposed to do when they can't get more than 20 people on any of the testing servers?  Giving the unfinished code to a public, more populated server isn't a good answer either, since if it's buggy then players will complain and leave, and likely start bad-mouthing the mod since they don't fully understand the process.  

 

I've said for a long time that the devs should have a dedicated group of testers who volunteer for the process and know what they're getting into and how to actually test things.  The idea of 'Official' servers clearly isn't working as planned, since they just don't get the traffic needed to really debug anything.  There are dozens of dedicated server owners on here (myself included) who would be willing to donate our time and efforts to join a closed, passworded server run by the Epoch team where they can get a bunch of we server owners on it and do proper testing.  If everyone brought one or two people we trust with us they would have a full server in no time and the devs could get some good, real-world performance data and could make adjustments accordingly.  That wouldn't be feasible for cases like we had last week where a BI update necessitates a knee-jerk Epoch update to keep things running, but for the major Epoch updates it would definitely help.  

 

See what I mean?  People can't help but turn into an armchair-developer.  lol!

Link to comment
Share on other sites

  • 0

There was a very very busy official server that did the performance testing once  :P

 

 

I'm gonna keep my dog out of that fight.  :P  Regardless, we're getting off topic.  The point of this thread is to provide info and feedback to help folks keep their servers running well.  Debating the "coulda woulda shoulda" belongs someplace else.  

Link to comment
Share on other sites

  • 0

ARGH so frustrating... have everything cut to the bone, everything optomised and right down low, and still once it hits 40 players + only get 10 server fps on average. monitoring my server bandwidth, and with average 50 players am uploading data to them at about 3.5gig an hr... looking at the spikes its all the network traffic to the players that has suddenly increased again with the 1.44 update. First 0.3 killed server fps, add the dlc's to it and eugh!

 

We appreciate your hard work and the work of others in this thread.

 

cheers,

 

Drink.

 

2dmin.org

Link to comment
Share on other sites

  • 0

Ok so I have trimmed EVERYTHING, no epoch events occuring, virtually no sappers (like 4 across the whole map with 60 players) hardly any UAV's, Missions all removed. 1gbps connection on dedi tested to be giving 200mbps upload atm, cpu only used 50%, only 20% Ram being used, - server fps after 1hr 10fps  with 60 players. Have played with the basic.cfg and it has made no difference to performance.

Link to comment
Share on other sites

  • 0

Ok so I have trimmed EVERYTHING, no epoch events occuring, virtually no sappers (like 4 across the whole map with 60 players) hardly any UAV's, Missions all removed. 1gbps connection on dedi tested to be giving 200mbps upload atm, cpu only used 50%, only 20% Ram being used, - server fps after 1hr 10fps  with 60 players. Have played with the basic.cfg and it has made no difference to performance.

 

infiSTAR or standard AH?

Link to comment
Share on other sites

  • 0

ok so this is a strange one but iwas 32min into a server restart with 35 players currently online - had 40fps consistently since server restart. at 32min, server fps dropped to 21fps (no clients joined). whats interesting is that the server was using 75% cpu with all cores fairly consistent with the 40fps, then at the drop, cpu usage dropped to about 50% and along came the 21fps, and the first core is blazing at around 95%.

heres the kickeer - Disable the first core, server fps jumps back up to 33fps and 75% cpu usage. Note I am using server startup commands - -autoinit -nologs -enableHT -malloc=tbbmalloc.    

ow get this... if i reenable the first core after a minute or so of running without it, the server cpu stabilises the first core and keeps pumping along at 75% and server fps increases back up to 40...

 

Note during this whole time no players were joining and this is over a 15min test period. going to see if its repeatable.

 

Oh also using a custom basic.cfg but that had made little to no difference the past 24hrs to server performance

Link to comment
Share on other sites

  • 0

ok so this is a strange one but iwas 32min into a server restart with 35 players currently online - had 40fps consistently since server restart. at 32min, server fps dropped to 21fps (no clients joined). whats interesting is that the server was using 75% cpu with all cores fairly consistent with the 40fps, then at the drop, cpu usage dropped to about 50% and along came the 21fps, and the first core is blazing at around 95%.

heres the kickeer - Disable the first core, server fps jumps back up to 33fps and 75% cpu usage. Note I am using server startup commands - -autoinit -nologs -enableHT -malloc=tbbmalloc.    

ow get this... if i reenable the first core after a minute or so of running without it, the server cpu stabilises the first core and keeps pumping along at 75% and server fps increases back up to 40...

 

Note during this whole time no players were joining and this is over a 15min test period. going to see if its repeatable.

 

Oh also using a custom basic.cfg but that had made little to no difference the past 24hrs to server performance

 

 

From monitoring my server sometimes it uses almost full CPU load and other times it's coasting at 50% total, there doesn't seem to be any reason for it (it's not thermal throttling either, the server is in a fully air conditioned room at 18c).

 

The CPU load seems to be completely random, the FPS will just drop for 5 minutes then shoot back up again, I'm monitoring logs but can't see anything corresponding to the times it drops.

Link to comment
Share on other sites

  • 0

David "Dwarden" Foltyn had this to say (regarding sudden drops in Server FPS):

 

 

[2015-05-12 13:33:06] David Foltyn: might be that when load on all cores is too high it decrease the first core clock

[2015-05-12 13:33:48] David Foltyn: i5 and i7 works that way, if there is primary core / single therad load then it has highest clock
in moment there is high load on all cores the max clock of primary core lowers
[2015-05-12 13:34:20] David Foltyn: not sure about Xeons, it might be similar but as those are CPUs for multicore aim it should not be that major

[2015-05-12 13:40:00] David Foltyn: most of performance degradations were usually linked to callextensions
[2015-05-12 13:40:06] David Foltyn: usually missing sleeps stalling the binary 
[2015-05-12 13:40:43] David Foltyn: another thing, are you using BattlEye filters ?
[2015-05-12 13:40:50] Tobias Solem: Yes.
[2015-05-12 13:40:57] David Foltyn: pro tip, remove them, completely
[2015-05-12 13:41:08] David Foltyn: compare the avg FPS afterward, if it's more than 5-10% difference then there are some wrong/overcomplex rules, especially too long and and too complex scripts.txt is suicide
Link to comment
Share on other sites

  • 0

well i dont want to count my chickens before they hatch and server hasnt had 60 players on it tonight yet... but the fps is not going below 30 with about 35 players online... maybe all my fixes together have finally worked... then to start turning things back on to find the culprit...

Link to comment
Share on other sites

  • 0

I have noticed doing a defrag on the windows server has positive affect on constant fps/cps once it completes the 1st one I do it again around 3-4 more times. Just something to add to the pot

Link to comment
Share on other sites

  • 0

From Dwarden on Reddit, replying to my comments about the community needing better education on editing the basic.cfg:

[–]DwardenTech Community Manager[S] 1 point an hour ago 
there are several factors which play role but the thing is,
after 1.46 the old basic.cfg might not be true (since 1.42 we optimize networking)
also by my fastest servers usually up to 40-50 players the server had flatline 50fps
over 50 players the FPS slowly drops, over 72 players negative effect gets multiplied each 2 players
playable 20+ fps threshold is 100-110 players
which I would like to push to 128 in future if optimizing goes well

So by having this knowledge and adding in the knowledge of our current performance levels that we're talking about here in this thread, we can extrapolate that by using the Epoch mod and whatever other addon scripts we have, it's dropping server performance by about 40-50% more than a stock Arma server.  One thing that we have to keep in mind is that BI can only optimize their own code so much.  They can't possibly take into account whatever mods or scripts are out in the wild.  If they're aiming for 20 server FPS with 128 connected players on a stock server, then probably the best we can hope for is going to be around 70 players with our current script packages.  Granted, the script addons package (welcome messages, death messages, status bar, respawn loadouts, etc.) will vary from server to server, and consequently the performance and player count will vary right along with it, but that's probably a solid baseline to shoot for.  As Epoch development continues and they have a chance to optimize the final code it may get better over time, leaving a little more headroom, but we will have to wait and see.  

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...