Jump to content
  • 1

How to optimize server-FPS and server performance


Tobias Solem

Question

It's a fact that ARMA has a dated architecture that only recently received some optimizations for multiple cores and utilizing them for different purposes (multi-tasking and multi-threading). There are several ways for you to optimize your server-side performance to get the maximum speed out of your Epoch server (because Epoch loads so many building items, it requires even more processing power).

 

Before I continue there is a fundamental thing you have to know and consider when you configure your Epoch server: 

 

* Every client will load and have to calculate everything that occurs on your server, meaning:

- Every client has to load ALL base building items

- Every client has to load ALL the vehicles

- Every client has to load and calculate ALL AI movement

- Every client has to load and calculate ALL clan logos/custom graphics

 

Another thing to consider is the fact that there is something called ragdoll and physics in ARMA 3, basically it is a set of calculations applied to each player body, each AI-body, and each car, and for a moment, also every placed building block. All vehicles run physics-calculations in real-time 24/7 (100% all the time) this means that for every car you have on your server, the server and each client will have to run the calculations for physics on that vehicle all the time. Naturally more calculations are required when something happens to that vehicle (driving, getting shot at, colliding, etc.) but essentially the computing power for all these physics & ragdoll items will take a toll.

 

Furthermore, all inventories and contents of traders, tents, vehicles, etc. are fed from the database in real-time, essentially the more items a container holds, the heavier the load on the server for each time that inventory is loaded. So if you have, say 300 items in a container, all those 300 will load in a big packet, which sucks some server performance. The less a container can hold, the less strain on the server.

 

Now, multiply physics, and inventory calculations per client, the more clients connected, the more vehicles driving, swimming, running, flying, inventories getting opened, the more the strain on each client, and on the server. A lot of this information is also sent from the server, and Arma 3 Epoch is one of the most bandwidth-intensive games out there. When the bandwidth is maximized on either the server-side or the client-side, you will see a yellow chain link in your bottom right corner, or a red chain link, depending on how overloaded the server or the client is with data. This is also when you will see lag happening. Whereas "desynch" (client and server not having the same information, and thus weird things happen) and rubberbanding (which looks like teleporting/warping) is usually a combination of over-stressed CPU (server and or client-side).

 

It is also important to know that in player interaction, the weakest link is usually the cause of the issues, for example: It doesn't matter if you are connected with a 1Gbit connection, with an awesome CPU and GPU, if the server is overloaded, that's the problem, you can't fix it clientside. But the same thing is if you and your friends are connected and your friend has a shitty Internet connection and/or computer and happens to be driving the car. He will lag/desynch based on his low-performance equipment, and that will affect you, and anyone trying to blow that car up (because the car will behave erratically). 

 

Virtual Machines

 

It should also be known that the arma3 server-file has issues with running on virtual machines (VMWare, or XenServer, etc.) essentially the performance output gets negated by as much as 30% if you run it on the same hardware as a dedibox, ARMA 3 is currently (2014-12-30) not well-optimized for running on Virtual Machines. Dedicated, native servers will see better performance than a VM.

 

 

How do I combat/fix this?

 

With these basic fundamentals in knowing how the server works, you as a player need to know that connecting to a server with hundreds of vehicles, or tens of thousands of buildables, or tons of AI-missions will not necessarily be a good thing for your game experience. While there may be plenty of content to have fun with, the experience with all that content will soon turn sour when your client FPS sinks (because the server has low performance, or is under too much strain (yes this affects clients too, if the server is the weakest link, all clients suffer)) your gameplay and experience will suffer too. 

 

Currently, the thoroughly tested, recommended max settings for cars on an Epoch server is 150, and the max amount of buildables is 1500 and the max amount of container content is 1500. This is the default settings in Epoch and it is what makes it possible for a server like Mellys to run 100 clients connected, with little to no issues. Yes, there will be a struggle to find a car, but the experience for each player will be as optimal as ARMA 3 Epoch can be with 100 players connected simultaneously.

 

The only possible compromise if you want to have more vehicles, more buildables, more missions, etc. is to allow for fewer clients (less than 100 players). It doesn't matter if the server has SSD-drives, or 1Gbit up and down, or 32GB DDR5 RAM on a super-high cache Xeon-processor. The arma3server-binary can't handle more than a certain amount of load anyways. 

 

 

No logs, memory handlers, and toggles

 

If you run a server with hyperthreading you should use the following toggle (do NOT combine it with cpuCounts=X or threads=X, etc.):

-enableHT

 

If you want to squeeze some more performance out of your server with an optimized memory allocator, use the following (download fred's dll here https://github.com/fred41/tbbmalloc_arma/raw/master/dwarden/uptodate_bins.zip ) and place its contents in your dlls-directory where you have arma3server:

-malloc=tbbmalloc 

 

IF you want to squeeze some extra FPS out of the server you can also turn off RPT-logging (not recommended if you are having issues or are actively tracking other problems/reports) by using:

-nologs

 

Setting the processor priority higher is also recommended:

-high

 
Server shutdowns
 
The recommended shutdown hours currently for optimal performance is 3 hours, the reason why we have to shut down the server with certain intervals is because of essentially two reasons: 1) Memory leaking (binaries leak RAM, and start performing poorly and thus need to be terminated and restarted) and 2) Clutter (basically a bunch of loops that get stuck due to poor code, vehicle explosions that take up unnecessary bandwidth, corpse clutter, and other things that are a detriment that increases over time on server performance).
 
Hardware
 
- CPU
ARMA 3 is processor clock dependent first (both client- and server-side). Having a CPU clock of 3.9GHz or higher is recommended. Intel Xeon-processors have been proven to perform better than regular non-server CPU:s. 
 
- RAM
More than 8GB per server has not been reported to make any difference whatsoever unless you run a lot of other things or multiple servers on the same machine
 
- Hard Drive Speed
SSD-hard drives improve loading performance severely, which means quicker reboots and startup-times for the server, but after the server is loaded everything is handled from the computers RAM, which is much faster than SSD-drives currently. 
 
- Bandwidth
 
Expect an ARMA 3 Server have been reported to use around 1Mbit per connected client (server-to-client upstream bandwidth). This means that with 50 players, you will need about 50Mbit upload. The server downloads much less data than this.
 
--------------------------------------------------
 
Bottom line
 
As a server owner you really need to consider the factors involved:
 
Amount of vehicles (more than 150 will mean you will have to lower your max allowed connected clients from 100)
Amount of buildables (more than 1500 will mean you will have to lower your max allowed connected clients from 100)
Amount of storage space (more than 1500 will mean you will have to lower your max allowed connected clients from 100)
 
 
Your server performance is measured in FPS and CPS - the standard FPS without load will be about 49-50FPS server-side, server-FPS and client-FPS are not the same thing, it is not as vital for server-FPS to be so high, everything 15FPS or higher will leave little to no issues. The same thing goes for CPS, which is the processing power used for Artificial Intelligence and non-gamebased calculations (like the AntiHack). The same principle for AI applies to this, the two values should be about the same normally. These values are heavily affected by (in order of performance-drain): 1) Connected clients, 2) Amount of vehicles 3) Amount of buildables 4) Contents of inventory
 
--------------------------------------------------
 
Link to comment
Share on other sites

Recommended Posts

  • 0

Can we clarify the benefits / usage of -enableHT. My first question would be what difference does it make when hyper threading is disable in the bios (if available), is it chip architecture or bios setting that is important ?

 

Have had a reliable report of going back to -cpuCount on a hyperthread enabled server being more beneficial.

Link to comment
Share on other sites

  • 0

HT is always a hot topic.. However I have done some test that if you have a HT box don't use -enableHT use cpu4 or something the. Use affinity to bind to CPU 0,2,4,6. This if you have ht enabled in the bios in theory should stop any CPU overloading

HT in this day and age should give more CPU on any multi threaded application like windows, MySQL, MSSQL how've I'm still unsure on the single thread bang for buck that arma uses.

Link to comment
Share on other sites

  • 0

Can we clarify the benefits / usage of -enableHT. My first question would be what difference does it make when hyper threading is disable in the bios (if available), is it chip architecture or bios setting that is important ?

 

Have had a reliable report of going back to -cpuCount on a hyperthread enabled server being more beneficial.

 

I've seen very little real world FPS using -enableHT (server only) the client is a different story. disabling HT in the BIOS means -enableHT is redundant! 

 

Dwarfer is correct "Use affinity to bind to CPU" is your best option.

 

"is it chip architecture or bios setting that is important ?" - In short both. Lets not forget that you have a OS between your hardware and software. This is why the answer is both. 

 

At work we disable HT on all hour hosts, windows is "optimised" for HT but not perfect!

 

"if you have ht enabled in the bios in theory should stop any CPU overloading" - Not true, but more importantly you should not be overclocking a server - Ever! Xeons are great processors for a reason they do not require overclocking. If you think this is untrue take a look at the manufacturing process for processors .

 

"HT in this day and age should give more CPU on any multi threaded application like windows, MySQL, MSSQL how've I'm still unsure on the single thread bang for buck that arma uses." - Agreed!

Link to comment
Share on other sites

  • 0

This is a good read.

http://superuser.com/questions/347982/how-does-windows-processor-affinity-work-with-hyperthreaded-cpus

But in essence CPU 0 and 1 are the same Core. You can send a thread to 0 or 1 and it would execute at the same speed (note or). However if you send to both at once in theory (I'm no genius here) you would see half the speed..

Using affinity on the odds OR evens you basically cheat and disable HT from the OS scheduling rather than bios..

HT in some cases does work better when you doing lots of light executions and a multi threaded app. However with Arma being the CPU hog we all love I wonder if HT is good or bad.. My test are inconclusive however client HT does seem to help but still not sure on server... Ooo to be able to spin up fake clients to generate load..

Link to comment
Share on other sites

  • 0

See Wasteland at the top with more players than Epoch, then look at the servers FPS :o the middle one is a HC on the same machine for Wasteland

Epoch had been running for 3 hrs, Wasteland for 7hrs, server had plenty spare resources but not being used

 

Server spec :

Intel® Xeon® Processor E5-2690 v2

32 GB DDR3 EEC Ram

6 x15k 600GB SAS

 

Launch parameters  -enableHT  -loadMissionToMemory

 

FmN4O3F.png

 

With all due respect your running multiple instances of Arma on the same severs.

 

Each instance fighting for CPU. Queue times, parking and priory is going to be very hard to diagnose.

 

For clear results you'll want to run one at a time like we do.

 

 CPU_A3.PNG

 

One instance using 82+% CPU with + 30 FPS. 

Link to comment
Share on other sites

  • 0

E5-2690 v2 is a 10 core CPU with 20 threads. I see from your task manager you have 10 CPUs listed.. So I would say you have HT disabled in the bios... Or your OS does not have the support for that many CPU's

I would say if you don't have HT enabled (I don't think you do but I would check) then don't enable HT.

What is your current affinity for the a3's you are running.

Is this OS on tin or do you have a hypervisor between it ?

Link to comment
Share on other sites

  • 0

Is it established that clients with really shitty fps/cps are dragging the whole network down. Maybe introduce fps/cps limits along with minimum ping. I remember the days of high or low ping servers..

 

FPS as in Frames Per Second or FPS as in simulation cycles per second?

Link to comment
Share on other sites

  • 0

Client FPS and CPS. Both don't always drop together and CPS is dependent on what FSM is doing at the time. Question is, can we agree that one or two 'bad' clients can cause issues for everyone, server included? 

 

I don't know... lol

 

I know what doesn't help the topic but... our servers are running as steady as a rock. I say this because we must have had some (lets call them bad apples) connected I I've not seen any dips that would suggest they are affecting the whole server.

 

Whats more we have 2 servers with the same mod(s) and a similar amount of players and the results are almost the same (a 5/7% delta).

 

The data I'm seeing is that if the hardware is right and conducive to Arma 3 requirements there are little issues. That said this release (1.40 has somehow upset the applecart). The Balance is all off.

Link to comment
Share on other sites

  • 0

With all due respect your running multiple instances of Arma on the same severs.

 

Each instance fighting for CPU. Queue times, parking and priory is going to be very hard to diagnose.

 

For clear results you'll want to run one at a time like we do.

 

It's a 10 core server, it should be more than capable of running 2 servers, the point i was making was Wasteland had more players and ran longer yet still ran well, Epoch with less players and had only been running for 3 hrs was dead on its feet and had to be restarted.

Link to comment
Share on other sites

  • 0

I would also suggest, to completely disable HT in the BIOS for debugging purposes.

 

HT "just" enables that idle resources of a core can be used from a 2nd thread.

 

As example a Core i3 has 2 "full" cores. With HT enabled it looks like 4 cores to the OS. However, these are not 4 "real" cores.

 

The effect of HT really depends on the calculations that need to be made. When you have a mainly single threaded calculation, it _can_ be better, to disable HT, so that this thread has the "full" core for itself and no scheduling of data needs to be done.

 

Just to rule out this kind of problems, I would completely disable HT in the BIOS and dont use the enableHT option.

 

With HT off each CPU in windows is a "full" core and if you manually set the affinity to one of the cores, the thread has a "full" core to work with.

 

 

PS: I dont want to say in general, disable HT for more performance! HT normally works good. But as I read the problems here, I would disable it for debugging purposes to rule out thread scheduling / affinity issues.

 

See also the first 3-4 post for a (imho) good writeup: Link1

and, if interested: Link2

 

BR

Rob

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...