Jump to content
  • 0
DNightmare

Performance problems with 1.0.3?!

Question

No clue what happened yesterday but anyone else having serious performance problems with 1.0.3 (with and without custom sutff)

- When our lil' server was running 1.0.2.2 we had like 3-4 FPS @ 35 players

- At the time of 1.0.2.5 this dropped to ~3-4 FPS @ 20 players but we also had custom stuff running

- Yesterday with 1.0.3 we hit 0.3 FPS @ 21 players

 

I'm running out of ideas what exactly is causing those issues apart from Epoch Code. Anyone around having some trouble with performance? Or general tipps what may improve it?

Normally our hardware should be able to handle it :/

 

http://pastebin.com/Kha2Sm9x < RPT from yesterday when everything went downhill... we ended up with locking the server, allowing 1 player at a time to join and filtering the ones with a bad connection, kicking them to at least allow the others to play without massive desyncs.

Share this post


Link to post
Share on other sites

28 answers to this question

Recommended Posts

  • 0

At the time of the RPT we had stripped everything out and only used 1.0.3 files..

Right now we're at 24 players with 5fps... let's hope it doesn't go down again. Still seems strange that the server can't handle it / I'm running out of ideas what can maybe be tweaked on the server/windows/setup side of things. If any guru has some ideas, feel free to join this thread :D

Share this post


Link to post
Share on other sites
  • 0

I do hope we did not go backwards as all tests showed otherwise.  You do have over 8000 objects you may want to consider looking at your cleanup times. The object count is a real performance metric as we found on our dev server that 12k objects takes the server down to 25 fps from 50 and that is with 1 player.

Share this post


Link to post
Share on other sites
  • 0

Well a point I thought of yesterday was the cleanup.fsm and the use of missionobjects...

normally [] spawn should use a dedicated thread when called, but does that also happen when it's spawned from a fsm? 

because maybe that's what's causing my issue here.. sure to automation/fsm/thingy is nice with it's "triggered every frame" but by the looks of it, the cleanupcalls and the iteration though all objects seem to hang the server

 

Right now i've stripped the cleanup.fsm down to only sync-time, check for hackers, update objects.. and that has helped surviving the night with 20ppl.

I could try it with a complete empty DB w/o any bases, but the problem is that my players prolly don't wanna join that server without all their cool stuff in big enough numbers to see how the performance is there.

Share this post


Link to post
Share on other sites
  • 0

Hmm, debug fps doesn't show in my RPT anymore, is there a setting to turn it on now? :)

 

 

DZE_DiagFpsFast = true; // reports every 1 Min
DZE_DiagFpsFast = true; // reports every 5 Min
DZE_DiagVerbose = true; // include objects & players, false = only FPS

Share this post


Link to post
Share on other sites
  • 0

Still init.sqf inside the mission.

 

 
btw. ToejaM, may I ask what your serverspeccs are for your 52 Slot machine?
I'm running:
- Core i7-4770 [3,4GHz but as I'm only running 1 server right now, it's running on 3,8 in Turbo]
- 32gb ram
- S-ATA 3 HDD
- 1Gbps connection
Windows 2012 Standard x64 / MySQL 5.5.30 x64 @ InnoDB / ArmA 2 OA @ BETA 103718 and attached to Core 1&2
 
Tried running with optimized / different maxSendMessage etc.. in server.cfg but that didn't make any difference either :/

Share this post


Link to post
Share on other sites
  • 0

Great will get this added on the next restart, thanks!

 

Sure my specs are this.

 

3.8Ghz Xeon Ivybridge (8 Cores)

32GB DDR3 Ram

240GB SSD

Windows 2008 R2 Standard (64bit)

1Gbps (100down/100up)

 

Two servers up, only one is full at the moment using 13-17% CPU and 1.4GB(19% total with all applications) Ram

 

The SSD is the part that seriously improves performance across the board. Wouldnt go back to a HDD even if you paid me :)

Share this post


Link to post
Share on other sites
  • 0

 

DZE_DiagFpsFast = true; // reports every 1 Min
DZE_DiagFpsFast = true; // reports every 5 Min
DZE_DiagVerbose = true; // include objects & players, false = only FPS

 

 

Could you clarify this, seems that there are two commands for different things.

 

Does setting just the verbose one bring up objects + fps + players?

 

Edit:

 

Putting this in the init.sqf works:

DZE_DiagFpsFast = true;
DZE_DiagVerbose = true;
 
and you duplicated a setting, should have been: DZE_DiagFpsSlow = true;
 
cheers tho! :D

Share this post


Link to post
Share on other sites
  • 0

We are also seeing major FPS drops, desyncs/lag spikes since the update ..

 

Mostly getting spammed with this in the RPT log;

 

"15:54:26 NetServer::SendMsg: cannot find channel #840074523, users.card=15
15:54:26 NetServer: users.get failed when sending to 840074523
15:54:26 Message not sent - error 0, message ID = ffffffff, to 840074523 (EvGen-[-__-]-27Ru)
15:54:26 NetServer::SendMsg: cannot find channel #840074523, users.card=15
15:54:26 NetServer: users.get failed when sending to 840074523
15:54:26 Message not sent - error 0, message ID = ffffffff, to 840074523 (EvGen-[-__-]-27Ru)
15:54:26 NetServer::SendMsg: cannot find channel #840074523, users.card=15"

That is for EVERY player; end of the server restart the RPT file was 3MB +

And

"16:04:37 Server: Network message 16279 is pending
16:04:37 Server: Network message 1627a is pending
16:04:37 Server: Network message 1627d is pending
16:04:37 Server: Network message 1627e is pending
16:04:37 Server: Network message 1627f is pending
16:04:37 Server: Network message 1628e is pending
16:04:37 Server: Network message 16291 is pending
16:04:37 Server: Network message 16292 is pending
16:04:37 Server: Network message 16293 is pending"

 

 

Not sure what's going on and NO clue on how to fix the horrible lagspikes/fps drops from the update ..

Share this post


Link to post
Share on other sites
  • 0

 

Could you clarify this, seems that there are two commands for different things.

 

Does setting just the verbose one bring up objects + fps + players?

 

Edit:

 

Putting this in the init.sqf works:

DZE_DiagFpsFast = true;
DZE_DiagVerbose = true;
 
and you duplicated a setting, should have been: DZE_DiagFpsSlow = true;
 
cheers tho! :D

 

 

slow/fast defines on which timers you want the FPS to get logged

verbose just changes how much info you want, but first you have to decide if you want it at all ;)

Share this post


Link to post
Share on other sites
  • 0

Yeah I was confused because this is your original post:

 

 

DZE_DiagFpsFast = true; // reports every 1 Min
DZE_DiagFpsFast = true; // reports every 5 Min
DZE_DiagVerbose = true; // include objects & players, false = only FPS

 

 

Two lots of fast ;)

Share this post


Link to post
Share on other sites
  • 0

Try to disable in init.sqf:

1) [] execVM "\z\addons\dayz_code\system\antihack.sqf";

2) [] execVM "\z\addons\dayz_code\external\DynamicWeatherEffects.sqf";

3) execVM "\z\addons\dayz_code\compile\local_lights_init.sqf";
4) #include "\z\addons\dayz_code\system\REsec.sqf"

 

And just try https://github.com/denisio/Dayz-Epoch-Linux-Server

No lags and fps drops!  :)

Share this post


Link to post
Share on other sites
  • 0

Try to disable in init.sqf:

1) [] execVM "\z\addons\dayz_code\system\antihack.sqf";

2) [] execVM "\z\addons\dayz_code\external\DynamicWeatherEffects.sqf";

3) execVM "\z\addons\dayz_code\compile\local_lights_init.sqf";
4) #include "\z\addons\dayz_code\system\REsec.sqf"

 

 

Should only need to disable #3 as the lights got enabled by default on Chernarus and maybe causing some issues. You should only disable the anithack.sqf or resec.sqf if you have another security system.

Share this post


Link to post
Share on other sites
  • 0

Still trying to find the issue... we reverted our mainserver down to 1.0.2.5 and problems seem to be gone even with 9k objects.

Now I've setup our testserver with 12k objects and am trying to locate what is causing the drop from 47fps @ 4players to 30fps @ 4 players... disabling the cleanup.fsm did nothing.

Any ideas what else may be causing it / any code you cann recall from out of your head that monitors missionobjects / etc.?

 

On top of that, I get some red chains (even on a complete empty, fresh server) with 1.0.3 when I get bandwidth spike of 40-50k UP-stream on my client... 1.0.2.5 seems to not have those spikes :/

Share this post


Link to post
Share on other sites
  • 0

We did change the periodic save time to have a min and max of 10 - 60 seconds. Before this change a full 50 slot server would have typically a max save time of 110 seconds.

try changing this line in the server_cleanup.fsm:

PVDZE_plr_SetSaveTime = round(60 - diag_fps);

to:

PVDZE_plr_SetSaveTime = (playersNumber west * 2) + 10;

or

PVDZE_plr_SetSaveTime = round(110 - diag_fps);

Share this post


Link to post
Share on other sites
  • 0

Running with a steady ~8-9 fps with 35 players on panthera. Ocassionaly drops to ~6fps with 40-42 players.

 

Xeon 1240v2 @ 3.8ghz (HT disabled!)

16Gb ram (matters not)

Samsumg Pro 840 SSD

Database on ramdisk

100/100MB

 

Only ''mod'' shreding the performance is AH, mostly vanilla except for self bb, repair stations and tow&lift

Share this post


Link to post
Share on other sites
  • 0

Ok, tried some more things today, still can't really find the problem :/

http://pastebin.com/aBhAyn6t

That's ArmA 2 running from a 22GB RAMDisc with 1 player, fresh 1.0.3 files and our old DB (1.0.2.5) migrated to 1.0.3 (delete all tables, execute epoch.sql, import only object_data).

Why do we have a drop from 48 to 38fps out of nowhere :/ when nothing's going on.

 

@shinobu: Only thing that you & Toejam have different is the SSD (well and you're running xeon's but would that really be the issue that it can't even handle 20players correctly?)

 

Ok, looks like the very first time-sync causes the 1 drop... just tried it with more objects to see if we have a breaking-point when FPS go down and the drop from 48 to 38 happens at the same time:

14:06:25 "DEBUG FPS : 47.619 OBJECTS: 8820 : PLAYERS: 1"
14:07:25 "TIME SYNC: Local Time set to [2013,8,3,11,7]"
14:07:25 "DEBUG FPS : 38.1862 OBJECTS: 8821 : PLAYERS: 1"
14:08:25 "DEBUG FPS : 47.4777 OBJECTS: 8823 : PLAYERS: 1"
Edited by DNightmare

Share this post


Link to post
Share on other sites
  • 0

Please keep in mind that the verbose fps command actually uses a more resources and should only be used for temp debug of the relationship of # of objects and players vs fps.

 

Fast mode reports fps once every minute, before 1.0.3 you would be lucky to get an fps report every 5 -10 minutes.

 

To still show fps with less overhead use these settings:

DZE_DiagFpsSlow = true;
DZE_DiagFpsFast = false;
DZE_DiagVerbose = false;

Share this post


Link to post
Share on other sites
  • 0

Panthera 30-50 players:

 

Line 1449: 17:01:49 "DEBUG FPS : 6.92641"
Line 1529: 17:06:50 "DEBUG FPS : 5.42925"
Line 1619: 17:11:50 "DEBUG FPS : 4.23168"
Line 1948: 17:16:51 "DEBUG FPS : 4.42356"
Line 2378: 17:21:52 "DEBUG FPS : 4.14186"
Line 2627: 17:26:53 "DEBUG FPS : 3.83877"
Line 2897: 17:31:53 "DEBUG FPS : 3.79957"
Line 3109: 17:36:54 "DEBUG FPS : 3.38338"
Line 3395: 17:41:55 "DEBUG FPS : 2.86893"
Line 3651: 17:46:56 "DEBUG FPS : 3.14218"
Line 4162: 17:51:57 "DEBUG FPS : 2.85663"
Line 4770: 17:56:58 "DEBUG FPS : 2.90539"
Line 5345: 18:01:59 "DEBUG FPS : 3.22711"
Line 5782: 18:07:01 "DEBUG FPS : 3.20064"
Line 6045: 18:12:02 "DEBUG FPS : 3.17523"
Line 6293: 18:17:03 "DEBUG FPS : 3.24346"
Line 6756: 18:22:04 "DEBUG FPS : 3.29354"
Line 7282: 18:27:05 "DEBUG FPS : 3.06866"
Line 7915: 18:32:05 "DEBUG FPS : 2.9901"
Line 8473: 18:37:07 "DEBUG FPS : 3.00582"
Line 9063: 18:42:08 "DEBUG FPS : 2.98285"
Line 9590: 18:47:08 "DEBUG FPS : 2.54372"
Line 10387: 18:52:08 "DEBUG FPS : 3.615"
 
Taviana 20-40 players:
 
Line 1928: 17:05:22 "DEBUG FPS : 6.37959"
Line 1976: 17:10:23 "DEBUG FPS : 5.49073"
Line 2063: 17:15:24 "DEBUG FPS : 5.17632"
Line 2182: 17:20:25 "DEBUG FPS : 4.2861"
Line 2464: 17:25:26 "DEBUG FPS : 4.293"
Line 2756: 17:30:27 "DEBUG FPS : 3.85264"
Line 3112: 17:35:27 "DEBUG FPS : 3.68579"
Line 3448: 17:40:28 "DEBUG FPS : 3.56665"
Line 4086: 17:45:30 "DEBUG FPS : 3.05752"
Line 4467: 17:50:31 "DEBUG FPS : 3.08404"
Line 5029: 17:55:33 "DEBUG FPS : 3.07692"
Line 5315: 18:00:34 "DEBUG FPS : 3.05227"
Line 5685: 18:05:35 "DEBUG FPS : 2.48524"
Line 6246: 18:10:35 "DEBUG FPS : 2.55102"
Line 6693: 18:15:36 "DEBUG FPS : 2.32423"
Line 7325: 18:20:36 "DEBUG FPS : 2.41255"
Line 7697: 18:25:39 "DEBUG FPS : 2.36581"
Line 8040: 18:30:41 "DEBUG FPS : 2.34158"
Line 8732: 18:35:41 "DEBUG FPS : 2.11977"
Line 9717: 18:40:43 "DEBUG FPS : 2.4623"
Line 10198: 18:45:44 "DEBUG FPS : 2.25511"
Line 10919: 18:50:45 "DEBUG FPS : 2.94931"
 
Overwatch 30-50 players:
 
Line 1396: 16:54:16 "DEBUG FPS  : 11.611"
Line 2462: 16:58:18 "DEBUG FPS  : 26.7112"
Line 2678: 17:01:22 "DEBUG FPS  : 17.5824"
Line 2801: 17:04:25 "DEBUG FPS  : 11.544"
Line 3038: 17:07:33 "DEBUG FPS  : 9.48429"
Line 3509: 17:11:01 "DEBUG FPS  : 7.84698"
Line 3898: 17:14:54 "DEBUG FPS  : 7.90123"
Line 4458: 17:18:47 "DEBUG FPS  : 7.22022"
Line 4795: 17:23:11 "DEBUG FPS  : 6.55469"
Line 5266: 17:27:58 "DEBUG FPS  : 6.58165"
Line 5635: 17:32:59 "DEBUG FPS  : 6.01956"
Line 6069: 17:38:14 "DEBUG FPS  : 5.24075"
Line 6631: 17:44:32 "DEBUG FPS  : 4.2072"
Line 7708: 17:52:08 "DEBUG FPS  : 3.89959"
Line 8579: 18:00:06 "DEBUG FPS  : 3.76205"
Line 9872: 18:09:26 "DEBUG FPS  : 2.86021"
Line 11259: 18:19:15 "DEBUG FPS  : 3.51339"
Line 12771: 18:28:30 "DEBUG FPS  : 3.0888"
Line 14049: 18:37:41 "DEBUG FPS  : 3.91007"
Line 15006: 18:45:29 "DEBUG FPS  : 3.79417"
 
Sahrani 10-25 players:
 
Line 2887: 13:52:19 "DEBUG FPS  : 12.3457"
Line 3991: 13:56:51 "DEBUG FPS  : 47.9042"
Line 4155: 13:59:53 "DEBUG FPS  : 46.2428"
Line 4236: 14:02:54 "DEBUG FPS  : 50"
Line 4343: 14:05:55 "DEBUG FPS  : 50.1567"
Line 4578: 14:08:56 "DEBUG FPS  : 49.6894"
Line 4663: 14:11:57 "DEBUG FPS  : 50.1567"
Line 4735: 14:14:58 "DEBUG FPS  : 49.5356"
Line 4948: 14:17:59 "DEBUG FPS  : 48.3384"
Line 5145: 14:21:00 "DEBUG FPS  : 48.9297"
Line 5283: 14:24:01 "DEBUG FPS  : 48.4848"
Line 5408: 14:27:02 "DEBUG FPS  : 43.956"
Line 5544: 14:30:03 "DEBUG FPS  : 39.0244"
Line 5629: 14:33:04 "DEBUG FPS  : 36.6972"
Line 5691: 14:36:05 "DEBUG FPS  : 34.8584"
Line 5807: 14:39:06 "DEBUG FPS  : 39.1198"
Line 5914: 14:42:07 "DEBUG FPS  : 35.1648"
Line 6066: 14:45:08 "DEBUG FPS  : 35.3982"
Line 6197: 14:48:09 "DEBUG FPS  : 40.5063"
Line 6430: 14:51:10 "DEBUG FPS  : 39.801"
Line 6510: 14:54:11 "DEBUG FPS  : 43.2432"
Line 6661: 14:57:13 "DEBUG FPS  : 44.5682"
Line 6750: 15:00:14 "DEBUG FPS  : 46.2428"
Line 6790: 15:03:15 "DEBUG FPS  : 42.1053"
Line 6876: 15:06:16 "DEBUG FPS  : 39.2157"
Line 6962: 15:09:17 "DEBUG FPS  : 41.3437"
Line 7026: 15:12:18 "DEBUG FPS  : 43.3604"
Line 7166: 15:15:19 "DEBUG FPS  : 30.7692"
Line 7347: 15:18:20 "DEBUG FPS  : 25.1177"
Line 7617: 15:21:21 "DEBUG FPS  : 26.7112"
Line 7798: 15:24:22 "DEBUG FPS  : 32.7869"
Line 8025: 15:27:23 "DEBUG FPS  : 42.5532"
Line 8106: 15:30:24 "DEBUG FPS  : 43.956"
Line 8206: 15:33:25 "DEBUG FPS  : 43.8356"
Line 8317: 15:36:26 "DEBUG FPS  : 48.9297"
Line 8346: 15:39:27 "DEBUG FPS  : 49.0798"
Line 8453: 15:42:28 "DEBUG FPS  : 50"
Line 8483: 15:45:29 "DEBUG FPS  : 49.2308"
Line 8569: 15:48:30 "DEBUG FPS  : 41.7755"
Line 8724: 15:51:31 "DEBUG FPS  : 46.9208"
Line 8810: 15:54:32 "DEBUG FPS  : 43.1267"
Line 8887: 15:57:33 "DEBUG FPS  : 41.9948"
Line 8935: 16:00:34 "DEBUG FPS  : 32.9218"
Line 9104: 16:03:36 "DEBUG FPS  : 30.4183"
Line 9187: 16:06:37 "DEBUG FPS  : 27.027"
Line 9350: 16:09:38 "DEBUG FPS  : 22.1914"
Line 9496: 16:12:39 "DEBUG FPS  : 26.9815"
Line 9680: 16:15:40 "DEBUG FPS  : 25.974"
Line 9885: 16:18:41 "DEBUG FPS  : 24.024"
Line 9983: 16:21:42 "DEBUG FPS  : 21.2766"
Line 10067: 16:24:43 "DEBUG FPS  : 23.7037"
Line 10123: 16:27:44 "DEBUG FPS  : 19.9005"
Line 10374: 16:30:45 "DEBUG FPS  : 16.2933"
Line 10571: 16:33:46 "DEBUG FPS  : 13.1579"
Line 10786: 16:36:48 "DEBUG FPS  : 11.7302"
Line 10980: 16:39:57 "DEBUG FPS  : 9.20598"
Line 11314: 16:43:17 "DEBUG FPS  : 9.06516"
Line 11546: 16:46:34 "DEBUG FPS  : 8.47907"

Share this post


Link to post
Share on other sites
  • 0

On our EGL server we had 9500+ objects and with 10 people we were getting around 17 FPS.  We removed 3000 objects (two cities that were hardly used) and the FPS went up to 25 FPS with 10 players.

 

We'll be pruning the objects even further until we reach a nice sweet spot where content and the players ability to have building retention meets our need to support 30+ players i.e. stay above 10 FPS.  By the way it's on a dedicated box.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...

  • Advertisement
  • Supporters
  • Discord

×
×
  • Create New...