How to optimize server-FPS and server performance

[Tobias Solem]

It's a fact that ARMA has a dated architecture that only recently received some optimizations for multiple cores and utilizing them for different purposes (multi-tasking and multi-threading). There are several ways for you to optimize your server-side performance to get the maximum speed out of your Epoch server (because Epoch loads so many building items, it requires even more processing power).

 

Before I continue there is a fundamental thing you have to know and consider when you configure your Epoch server: 

 

* Every client will load and have to calculate everything that occurs on your server, meaning:

- Every client has to load ALL base building items

- Every client has to load ALL the vehicles

- Every client has to load and calculate ALL AI movement

- Every client has to load and calculate ALL clan logos/custom graphics

 

Another thing to consider is the fact that there is something called ragdoll and physics in ARMA 3, basically it is a set of calculations applied to each player body, each AI-body, and each car, and for a moment, also every placed building block. All vehicles run physics-calculations in real-time 24/7 (100% all the time) this means that for every car you have on your server, the server and each client will have to run the calculations for physics on that vehicle all the time. Naturally more calculations are required when something happens to that vehicle (driving, getting shot at, colliding, etc.) but essentially the computing power for all these physics & ragdoll items will take a toll.

 

Furthermore, all inventories and contents of traders, tents, vehicles, etc. are fed from the database in real-time, essentially the more items a container holds, the heavier the load on the server for each time that inventory is loaded. So if you have, say 300 items in a container, all those 300 will load in a big packet, which sucks some server performance. The less a container can hold, the less strain on the server.

 

Now, multiply physics, and inventory calculations per client, the more clients connected, the more vehicles driving, swimming, running, flying, inventories getting opened, the more the strain on each client, and on the server. A lot of this information is also sent from the server, and Arma 3 Epoch is one of the most bandwidth-intensive games out there. When the bandwidth is maximized on either the server-side or the client-side, you will see a yellow chain link in your bottom right corner, or a red chain link, depending on how overloaded the server or the client is with data. This is also when you will see lag happening. Whereas "desynch" (client and server not having the same information, and thus weird things happen) and rubberbanding (which looks like teleporting/warping) is usually a combination of over-stressed CPU (server and or client-side).

 

It is also important to know that in player interaction, the weakest link is usually the cause of the issues, for example: It doesn't matter if you are connected with a 1Gbit connection, with an awesome CPU and GPU, if the server is overloaded, that's the problem, you can't fix it clientside. But the same thing is if you and your friends are connected and your friend has a shitty Internet connection and/or computer and happens to be driving the car. He will lag/desynch based on his low-performance equipment, and that will affect you, and anyone trying to blow that car up (because the car will behave erratically). 

 

Virtual Machines

 

It should also be known that the arma3 server-file has issues with running on virtual machines (VMWare, or XenServer, etc.) essentially the performance output gets negated by as much as 30% if you run it on the same hardware as a dedibox, ARMA 3 is currently (2014-12-30) not well-optimized for running on Virtual Machines. Dedicated, native servers will see better performance than a VM.

 

 

How do I combat/fix this?

 

With these basic fundamentals in knowing how the server works, you as a player need to know that connecting to a server with hundreds of vehicles, or tens of thousands of buildables, or tons of AI-missions will not necessarily be a good thing for your game experience. While there may be plenty of content to have fun with, the experience with all that content will soon turn sour when your client FPS sinks (because the server has low performance, or is under too much strain (yes this affects clients too, if the server is the weakest link, all clients suffer)) your gameplay and experience will suffer too. 

 

Currently, the thoroughly tested, recommended max settings for cars on an Epoch server is 150, and the max amount of buildables is 1500 and the max amount of container content is 1500. This is the default settings in Epoch and it is what makes it possible for a server like Mellys to run 100 clients connected, with little to no issues. Yes, there will be a struggle to find a car, but the experience for each player will be as optimal as ARMA 3 Epoch can be with 100 players connected simultaneously.

 

The only possible compromise if you want to have more vehicles, more buildables, more missions, etc. is to allow for fewer clients (less than 100 players). It doesn't matter if the server has SSD-drives, or 1Gbit up and down, or 32GB DDR5 RAM on a super-high cache Xeon-processor. The arma3server-binary can't handle more than a certain amount of load anyways. 

 

 

No logs, memory handlers, and toggles

 

If you run a server with hyperthreading you should use the following toggle (do NOT combine it with cpuCounts=X or threads=X, etc.):

-enableHT

 

If you want to squeeze some more performance out of your server with an optimized memory allocator, use the following (download fred's dll here https://github.com/fred41/tbbmalloc_arma/raw/master/dwarden/uptodate_bins.zip ) and place its contents in your dlls-directory where you have arma3server:

-malloc=tbbmalloc 

 

IF you want to squeeze some extra FPS out of the server you can also turn off RPT-logging (not recommended if you are having issues or are actively tracking other problems/reports) by using:

-nologs

 

Setting the processor priority higher is also recommended:

-high

 

Server shutdowns

 

The recommended shutdown hours currently for optimal performance is 3 hours, the reason why we have to shut down the server with certain intervals is because of essentially two reasons: 1) Memory leaking (binaries leak RAM, and start performing poorly and thus need to be terminated and restarted) and 2) Clutter (basically a bunch of loops that get stuck due to poor code, vehicle explosions that take up unnecessary bandwidth, corpse clutter, and other things that are a detriment that increases over time on server performance).

 

Hardware

 

- CPU

ARMA 3 is processor clock dependent first (both client- and server-side). Having a CPU clock of 3.9GHz or higher is recommended. Intel Xeon-processors have been proven to perform better than regular non-server CPU:s. 

 

- RAM

More than 8GB per server has not been reported to make any difference whatsoever unless you run a lot of other things or multiple servers on the same machine

 

- Hard Drive Speed

SSD-hard drives improve loading performance severely, which means quicker reboots and startup-times for the server, but after the server is loaded everything is handled from the computers RAM, which is much faster than SSD-drives currently. 

 

- Bandwidth

 

Expect an ARMA 3 Server have been reported to use around 1Mbit per connected client (server-to-client upstream bandwidth). This means that with 50 players, you will need about 50Mbit upload. The server downloads much less data than this.

 

--------------------------------------------------

 

Bottom line

 

As a server owner you really need to consider the factors involved:

 

Amount of vehicles (more than 150 will mean you will have to lower your max allowed connected clients from 100)

Amount of buildables (more than 1500 will mean you will have to lower your max allowed connected clients from 100)

Amount of storage space (more than 1500 will mean you will have to lower your max allowed connected clients from 100)

 

 

Your server performance is measured in FPS and CPS - the standard FPS without load will be about 49-50FPS server-side, server-FPS and client-FPS are not the same thing, it is not as vital for server-FPS to be so high, everything 15FPS or higher will leave little to no issues. The same thing goes for CPS, which is the processing power used for Artificial Intelligence and non-gamebased calculations (like the AntiHack). The same principle for AI applies to this, the two values should be about the same normally. These values are heavily affected by (in order of performance-drain): 1) Connected clients, 2) Amount of vehicles 3) Amount of buildables 4) Contents of inventory

 

--------------------------------------------------

 

[1Man]

I use defaults and I have no lag or Desync so I see no need to touch it. I would recommend some of the tips up top for VPS and maybe only have max 150 vehicles and low AI missions

[Richie]

Well made guide, I vote for sticky :)

[RC_Robio]

Very, very nice! Maybe you could add that to monitor server performance that it is recommended to use ASM or Arma Server Monitor. https://github.com/fred41/ASM Good tool for addon makers to see what the additional addon's are doing to the server. Also a great tool for monitoring you server remotely.

[Hikage]

hello i would like to use your tbbmalloc link but when i download it, he have a password and i cant find it.

What the password please

 

https://github.com/fred41/tbbmalloc_arma/blob/master/dwarden/uptodate_bins.zip

[psychosis]

yep.. need password)

[RC_Robio]

tbbmalloc  Drop into your dll folder in your server directory as stated above and add the parameters -malloc=tbbmalloc to your server.

 

Also get it here. https://github.com/fred41/tbbmalloc_arma Click download zip on the right and the dll will be in the Binary folder.

tbbmalloc.zip

[Hikage]

@rc_robio i have this one but i want the one said by Tobias Solem in his post but it has password

[machine6fd]

Whats an avg. Server FPS when a server is freshly booted up and mission is running. 0 players online. 

[RC_Robio]

@rc_robio i have this one but i want the one said by Tobias Solem in his post but it has password

 

It's the same one.

 

Whats an avg. Server FPS when a server is freshly booted up and mission is running. 0 players online. 

My box usually runs at 47Fps

[Richie]

My box starts at 50 fps, as time goes by and the players join it drops, I wish i could get it to use all the server resources :(

 

Currently the server has been running 2hrs 30 mins, I have 50 players online, server fps is 19, CPU usuage is 67% and Ram 30%

[Ryansgotacbr]

very nice guide man

[cen]

My box starts at 50 fps, as time goes by and the players join it drops, I wish i could get it to use all the server resources :(

 

Currently the server has been running 2hrs 30 mins, I have 50 players online, server fps is 19, CPU usuage is 67% and Ram 30%

 

Just wait until they start building :D

[Richie]

Just wait until they start building :D

 

They already are, we hit the build limit last week :lol:

[BenR]

They already are, we hit the build limit last week :lol:

Whats your build limit set to?

[cen]

3k/8k for me

[second_coming]

Just installed ASM and getting a solid 50 FPS/CPS with about 10 people online 40 minutes after restart.

 

Server is a Mac Pro running Windows 7 Pro :)  (dual CPU quad Xeon 2.8GHz E5462)

[happysan]

Awesome info.  Any tips on basic.cfg specifically for Epoch?

[Tobias Solem]

Editing the basic.cfg should NOT be done without some basic understanding of TCP/IP and bandwidth limitations, etc.

MinBandwidth

This is your bits per second value of your absolute minimum bandwidth available, note, base this on your UPLOAD limit. I recommend doing a bandwidth check during a Sunday peak to determine your minimum upload speed. Never set this too high or you will have red chains, CPU-issues, and A LOT of rubberbanding, etc. - your upload is your megabit value followed by six zeroes (10Mbit = 10000000).

MaxBandwidth

This is your bits per second value of your absolute maximum bandwidth available, note, base this on your UPLOAD limit. This should be your maximum capability of upload speed.

MaxMsgSend

This is the maximum number of messages that can be sent in one simulation cycle. Default is 128, with a 50+Mbit upload you can multiply this base value by 3 or so (384).

MaxSizeGuaranteed

The maximum guaranteed packet sent in bytes (without headers). This will highly affect your bullet impact and reactions from bullets. Default value is 512, we run 696.

MaxSizeNonguaranteed

The maximum guaranteed packet sent in bytes (without headers). These packets aren't prioritized, essentially it affects how vehicles behave when you're a passenger and witness the vehicle's behavior. I recommend making it slightly smaller than default (232)

MinErrorToSend

The error needs to be how small between client and server to send updates? The smaller the value the more accurate movements become. Setting it to zero will have adverse effects however. I recommend 0.0015

MinErrorToSendNear

Same as above except now it affects closer units (close range combat). Personally I keep this the same as the above.

MaxCustomFileSize

If you want players to have custom faces this might be a cool gimmick, however - these come with some laggy side-effects. Recommend 0. to connect.

[second_coming]

Is it normal for the FPS/CPS to dip regularly like this?

[larsmos2011]

Really nice tutorial!

 

I have just one question, how is it with the CPU Cores. I have an I7 2600K Clocked at 3.4ghz and I don't really know how I should use the cores. When I use all cores for 1 Epoch Server, 50 Slots, Full, the CPU Load is about 65% when the Server is full, which is too much to set up a 2nd Server. If I set it down to 2-3 Cores the Load is at arround 25-35%. I couldn't notice any big lags or sth. but the Database is almost empty.

 

So how many CPU cores should I use so that I can still have good Performance with arround 8K Database Objects and 2 Servers on the same Box. Talking about 2 50 Slots Servers on Chernarus.

[Stranger]

Is it normal for the FPS/CPS to dip regularly like this?

Go into your Energy Settings on your Windows machine and set it to "High Performance" - CPU 100% all time.

[second_coming]

I've just installed this and it seems very good for optimising and monitoring CPU usage (https://bitsum.com/processlasso/)

[guttall]

Hey Guys,

 

I´m looking for basic.cfg parameters based on my infrastructure... I have a VPS with 5 cores of 2.4Ghz from a E5-645 processor, 6gb of RAM, network of 20mbs (up/down).  [i know it´s a VPS  :-(  thats what we can pay from here]

 

 

Parameters from basic.cfg are like follow:

 

MinBandwidth = 5242880;

MaxBandwidth = 20971520;

 

MaxMsgSend = 256;

MaxSizeGuaranteed = 512;

MaxSizeNonguaranteed = 256;

 

MinErrorToSend = 0.002;

MinErrorToSendNEar = 0.02;

 

MaxCustomFileSize = 0;

 

With that configuration, server running with 40 players is showing average of 7 FPS at @ASM, CPU around 65% of utilziation and memory with 2.7Gb used. Although there is no major lag, vehicles are moving without desync, etc... can someone point a better value for the parameters mentioned ahead ?

 

Thanks in Advance!

[second_coming]

I've been using these settings the last few days:

// These options are created by default
language="English";
adapter=-1;
3D_Performance=1.000000;
Resolution_W=800;
Resolution_H=600;
Resolution_Bpp=32;
 
// These options are important for performance tuning
 
 
// Bandwidth the server is guaranteed to have (in bps). This value helps server to estimate bandwidth available. 
// Increasing it to too optimistic values can increase lag and CPU load, as too many messages will be sent but discarded. 
// Default: 131072 old 131072
MinBandwidth = 15728640; // 15 MBps
 
 
// Bandwidth the server is guaranteed to never have. 
// This value helps the server to estimate bandwidth available.
MaxBandwidth = 50000000; // 50 MBps
 
 
// Maximum number of messages that can be sent in one simulation cycle. 
// Increasing this value can decrease lag on high upload bandwidth servers. 
// Default: 128 old 256 (384)
MaxMsgSend = 1024;
 
 
 
// Maximum size of guaranteed packet in bytes (without headers). Small messages are packed to larger frames. 
// Guaranteed messages are used for non-repetitive events like shooting. 
// Default: 512 (696)
MaxSizeGuaranteed = 384;
 
 
// Maximum size of non-guaranteed packet in bytes (without headers). Non-guaranteed messages are used for repetitive 
// updates like soldier or vehicle position. Increasing this value may improve bandwidth requirement, but it may increase lag. 
// Default: 256 (232)
MaxSizeNonguaranteed = 192;
 
 
 
// Minimal error to send updates across network. Using a smaller value can make units observed 
// by binoculars or sniper rifle to move smoother. 
// Default: 0.002 (0.002)
MinErrorToSend = 0.001;
 
 
 
// Minimal error to send updates across network for near units. Using larger value can reduce traffic sent for near units. 
// Used to control client to server traffic as well. 
// Default: 0.01
MinErrorToSendNear = 0.01; 
 
 
 
// (bytes) Users with custom face or custom sound larger than this size are kicked when trying to connect.
MaxCustomFileSize = 0; 

[second_coming]

Is it normal for the FPS/CPS to dip regularly like this?

 

If anyone was wondering and has the same thing, it's Lootspawner that causes the dips. Each dip coincides with the cleanup routine that Lootspawner runs every 60 seconds.