Attacker crashing Arma2OA game server at will

[mgm]

Hey guys,

 

Does anyone know a 'hack' in which the attacker can cause the Arma 2 OA game server to crash close? 

 

Since I already *know* it is possible I'm actually looking  for the potential fix actually so will be much appreciated if anyone can share a working fix.

 

Thanks!

 

 

 

--

Background: A few dupers got caught & insta/perm banned today. In the following hours our game server crashed 4 times.

This is the exact same files that we've been running over a week now with zero unscheduled crash closes of arma2oa server.

The Windows server/game server files/host have been stable for over 2 months now. The only factor changed is dupers getting banned earlier tonight.

 

After each restart I've received comments indicating some of the people in the game server are "in the know". I know some of the friends of the dupers are on the server still (they did nothing wrong and I will not ban people because they hang out with dupers) I will not name names or provide further info on this - while I'm not closing the door to a miracleous 1% chance of 4 server crashes, I can say I am 99% sure it is "attackers work" causing the server crashing out of sudden 4 times in an hour.

 

They stopped after a while since they either got bored or got the message accross (that they can shut us down any time they wish).

 

I'm not sure if it's an XML attack or what (have been reading BI forums in the background) but it seems related to network as we see huge traffic jumps in server's outgoing traffic.

Each peak point (like the one you see below) is a server crash point in time - I'm sure as I was online and had the chance to check it

 

I hope someone can shed a light on this as this is not a good sitation to be in now...

 

 

 

[TNT]

you could try to block the ports , related to XML files  Port 80,8080 and 443 just to be safe.

 

you can also monitor server traffic and see if you can find the IP thats causing the issue and block it with a firewall 

[mgm]

you could try to block the ports , related to XML files  Port 80,8080 and 443 just to be safe.

 

you can also monitor server traffic and see if you can find the IP thats causing the issue and block it with a firewall 

Thanks for the response.

 

As per posts #13 & #15 from the same thread, port block is not working if you allow the game EXE (and yes it is already blocked on my Windows server firewall - and the rule is at the top).

#13

#15

 

Any other known workarounds? I heard the attacking 'kids' were talking (bragging) about "ping bombs" on our TeamSpeak just before the attack.

Not sure if it is even real information or misinformation to throw admins off track (because some suspicious players got banned tonight, following the attacks) but if it is, I can't find anything on google that suggest arma 2 server ping flood.

[MGT]

Set the temp folder in your instance folder to read only, also get some anti ddos.

Only way I've heard of in game players crashing or lagging a server is with satchel charging certain buildings or loading draggees into suvs

[mgm]

Set the temp folder in your instance folder to read only, also get some anti ddos.

Only way I've heard of in game players crashing or lagging a server is with satchel charging certain buildings or loading draggees into suvs

I checked both directories and both are already set to read-only.

 

They cannot be using satchel charges because satchel charges are not allowed on our server.

 

I do not know if the "server-crasher" guys was in-game or out of the game. There were 20+ players, some of them definitely attacker's friends (my deduction from their comments) but no clue whether the crash happened from within the game or outside. The only hint is excessive network traffic and it says nothing about attacker being in/out-side the game server.

[Defent]

The spike you're showing is too inconsistent to resemble a ddos attack and or any other form of packet throttle and or overflow. If you were being ddos'd you would probably not be able to connect to the machine during the spike. I do not however know of any functions that allow players to send massive packets to the arma 2 server, other than with squad .XML files.

 

You can always ask your host and see if they notice any increase in sent packets during that specific time and or dos/ddos related activites. They should be able to tell rather quickly.

Try setting the -malloc to system, that solved most of my random crashes. You can also download the program WindDBG, it can open the crash dumps created and you can tell if it is a memory allocation error or other. (I do not know if bad squad xml's cause memory errors aswell)

 

Any other known workarounds? I heard the attacking 'kids' were talking (bragging) about "ping bombs" on our TeamSpeak just before the attack.

Not sure if it is even real information or misinformation to throw admins off track (because some suspicious players got banned tonight, following the attacks) but if it is, I can't find anything on google that suggest arma 2 server ping flood.

 

Ping bombs, or ping of death has been disabled for a long time on windows machines. It is a dos attack which goes through telnet(?). This could be done if your machine is not protected correctly, however, I believe that most server machines have a default protection against masssive ping packets.

 

Another way to tell if it is a dos or ddos is to check teamspeak (if its on the same server). If the teamspeak latency goes haywire, crashes and is unreachable, then it is probably either a ddos or dos attack.

 

Check your logs for script errors too, especially the .rpt. How many network message errors do you have?

 

P.S Don't buy software DDOS protection, it is not useful. 

http://www.hetzner.co.za/helpcentre/index.php/articles/content/category/ddos-attacks/can-a-ddos-attack-be-prevented

 

P.P.S if you really want to go in depth, get a packet sniffer and see where packets come from and what they are during these spikes (wireshark for example).

[mgm]

The spike you're showing is too inconsistent to resemble a ddos attack and or any other form of packet throttle and or overflow. If you were being ddos'd you would probably not be able to connect to the machine during the spike. I do not however know of any functions that allow players to send massive packets to the arma 2 server, other than with squad .XML files.

 

You can always ask your host and see if they notice any increase in sent packets during that specific time and or dos/ddos related activites. They should be able to tell rather quickly.

Try setting the -malloc to system, that solved most of my random crashes. You can also download the program WindDBG, it can open the crash dumps created and you can tell if it is a memory allocation error or other. (I do not know if bad squad xml's cause memory errors aswell)

 

 

Ping bombs, or ping of death has been disabled for a long time on windows machines. It is a dos attack which goes through telnet(?). This could be done if your machine is not protected correctly, however, I believe that most server machines have a default protection against masssive ping packets.

 

Another way to tell if it is a dos or ddos is to check teamspeak (if its on the same server). If the teamspeak latency goes haywire, crashes and is unreachable, then it is probably either a ddos or dos attack.

 

Check your logs for script errors too, especially the .rpt. How many network message errors do you have?

 

P.S Don't buy software DDOS protection, it is not useful. 

http://www.hetzner.co.za/helpcentre/index.php/articles/content/category/ddos-attacks/can-a-ddos-attack-be-prevented

It is an attack that quickly increases the stress on the server till eventual crash. I was logged in as #login <pwd> followed by #monitor 2. I saw the increased outgoing bandwith :

450kbps

700 kbps

1100 kbps

1700 kbps

...

...

9900 kbps

 

A few days ago (on another attack instance) I received an email from game server provider confirming that it was a DDoS - no emails yet today and today's attack signature was different anyway.

At the time of the game server provided DDoS attack, outgoing data xfer speed went as high as 24000 kbps which is crap and excessively out of proportion to what was inbound (2000 kbps or less).

Today 9K was the peak I think. Two different signatures, two different attacks.

 

Today's attack could be ping bomb as the people on TS -supposedly- were bragging about "ping bomb" attacks. What did you do to disable "ping bomb" on your servers?

There's nothing on RPT files.

[Defent]

Are you with a game server host? If so, then there is not much you can do. It's up to them and sometimes these attacks are not directed for your server specifically but for the cluster that your server is hosted on. (I.e they hit one server but inadvertedly hit all of the servers within that cluster).

 

I did not disable it or anything as it should be stopped automatically by windows (I believe).

I doubt it is a ICMP ping or other type of ping of death since those were pretty much active during '96 and '97. Mainstream OSes should not have those vulnerabilities. 

 

The outgoing data you are showing is indeed suspicious and should not climb that high. Are you sure you have no network leaks or something that could cause this on the server side?

The bandwith increase is a sign of a ddos/dos attack but if it is ongoing then you should not be able to access the server at all (if it is powerful enough). If it is just a small scale attack then there is not much you can do except ask the host to add some sort of physical block or that you check where the attacks are coming from and IP block that area or those ports. 

 

If you have a webserver up, close it. They can easily be targeted and cause massive increase of load on the server.

[mgm]

Are you with a game server host? If so, then there is not much you can do. It's up to them and sometimes these attacks are not directed for your server specifically but for the cluster that your server is hosted on. (I.e they hit one server but inadvertedly hit all of the servers within that cluster).

 

I did not disable it or anything as it should be stopped automatically by windows (I believe).

I doubt it is a ICMP ping or other type of ping of death since those were pretty much active during '96 and '97. Mainstream OSes should not have those vulnerabilities. 

 

The outgoing data you are showing is indeed suspicious and should not climb that high. Are you sure you have no network leaks or something that could cause this on the server side?

The bandwith increase is a sign of a ddos/dos attack but if it is ongoing then you should not be able to access the server at all (if it is powerful enough). If it is just a small scale attack then there is not much you can do except ask the host to add some sort of physical block or that you check where the attacks are coming from and IP block that area or those ports. 

 

If you have a webserver up, close it. They can easily be targeted and cause massive increase of load on the server.

Yes with a game server host and no I don't think it is targetted to the entire "game server hosting" company's server cluster as we have a group of upset people attacking us rather than the whole server cluster.

(I have no evidence to show you that other hosts have not been attacked but I am assuming from chat messages).

 

We do have a webserver on the host but there are not enough requests to cause anything like this - so the web server is not being utilized in the attack.

 

I know about the old ping-o-death but also read about the resurrected NEW one:

http://gcn.com/blogs/cybereye/2013/08/microsoft-patch-ping-of-death-ipv6.aspx

https://technet.microsoft.com/library/security/ms13-065#section22

 

Reading MS bulletin however it does not seem like our issue as the Windows host did not require a restart. Our issue is a specific attack crashing arma2oaserver.exe but not crashing the O/S.

So my deduction is, it is an undocumented Arma 2 OA bug these kids somehow got their hands on.

[Defent]

I could have figured IPV6 would come with new exploits.. But yea, I still doubt that a ddos is the case here. Try checking the memory usage of the server, what is it at? For reference, my server, 52 players, 1 hour and 30 mins uptime uses 1390 mb of RAM.

[mgm]

I could have figured IPV6 would come with new exploits.. But yea, I still doubt that a ddos is the case here. Try checking the memory usage of the server, what is it at? For reference, my server, 52 players, 1 hour and 30 mins uptime uses 1390 mb of RAM.

It's not a DDoS, it's a targetted attack by someone who knows how to crash Arma2OA game servers.

Attack is crash closing the game server exe:

- On the server desktop there is no error message.

- On clients desktops, naturally, they get "lost connection" message

 

Our server's memory consumption is 815 MB at 100 minutes runtime with 15 players.

[Snackbar_Jones]

Hey mgm, I created this account just to reply to your post.

 

It is completely possible to crash any Arma2 server remotely. There is no patch and it's present on both Windows and Linux server versions.

 

It is related to leveraging a part of the game.

 

Essentially what happens is an attacker causes the server to immediately read something it shouldn't causes the server to crash. It is not necessary for the attacker to even connect to the game server.

 

The spike you are seeing is server resources consuming themselves due to the bad request. There is no fix.

After reading your post and description I knew exactly what to look for.

 

I could go into more detail, but I think for all our sakes and to keep this from malicious turd-gurglers, finer details shouldn't be posted. Those that know could find it, but I'm not going to give anyone a hint or tip where to look.

 

If this forum supports PM's you can PM me and I will give you more details as well as offering a work around. It will cripple a feature of the game, but will keep you online during these skiddie attacks. I'd also be willing to meet up with you on your TS or any TS and help you harden your server.

[mgm]

Hey mgm, I created this account just to reply to your post.

 

It is completely possible to crash any Arma2 server remotely. There is no patch and it's present on both Windows and Linux server versions.

 

It is related to leveraging a part of the game.

 

Essentially what happens is an attacker causes the server to immediately read something it shouldn't causes the server to crash. It is not necessary for the attacker to even connect to the game server.

 

The spike you are seeing is server resources consuming themselves due to the bad request. There is no fix.

After reading your post and description I knew exactly what to look for.

 

I could go into more detail, but I think for all our sakes and to keep this from malicious turd-gurglers, finer details shouldn't be posted. Those that know could find it, but I'm not going to give anyone a hint or tip where to look.

 

If this forum supports PM's you can PM me and I will give you more details as well as offering a work around. It will cripple a feature of the game, but will keep you online during these skiddie attacks. I'd also be willing to meet up with you on your TS or any TS and help you harden your server.

Sure, I'll contact you now. Thanks.

[Snackbar_Jones]

I'm currently at work, but I get off in an hour and a half or so, I should be home within 3 hours.

[icomrade]

Hey mgm, I created this account just to reply to your post.

 

It is completely possible to crash any Arma2 server remotely. There is no patch and it's present on both Windows and Linux server versions.

 

It is related to leveraging a part of the game.

 

Essentially what happens is an attacker causes the server to immediately read something it shouldn't causes the server to crash. It is not necessary for the attacker to even connect to the game server.

 

The spike you are seeing is server resources consuming themselves due to the bad request. There is no fix.

After reading your post and description I knew exactly what to look for.

 

I could go into more detail, but I think for all our sakes and to keep this from malicious turd-gurglers, finer details shouldn't be posted. Those that know could find it, but I'm not going to give anyone a hint or tip where to look.

 

If this forum supports PM's you can PM me and I will give you more details as well as offering a work around. It will cripple a feature of the game, but will keep you online during these skiddie attacks. I'd also be willing to meet up with you on your TS or any TS and help you harden your server.

I suggest you and other people that are affected contact BIS with this issue. Email dwarden or PM him on the BIS forums with instructions to repro the issue, perhaps a crash report from the server, .rpt, .mdmp, .bidmp and if you want or asked the netlog as well (netlogging must be enabled via -netlog startup param). BIS are usually pretty good at fixing security vulnerabilities.

[Snackbar_Jones]

This bug has been reported and has yet to be fixed.

 

It's actually existed since Arma.

[mgm]

This bug has been reported and has yet to be fixed.

 

It's actually existed since Arma.

Snackbar,

 

Thanks  for sharing the exploit via Private Messaging. I never thought I would say that but I attacked my server with the exploit you shared. It did not go down.

The exploit our current favorite attacker is using must be something different. Please do share if you have any theories. Thanks

 

 

 

For other server admins,

 

Please be aware this bug is still out there.

We did have an automated process of restarting the server every 3 hours but if it crash-closed it did not automatically launch (because it never crashed and we never needed it till now!). As a result our server was down for the last 4 hours while I was AFK. I will be adding the check_if_running_and_launch_if_not batch script in the next 15 minutes.

I suggest if any of you don't have the same yet, perhaps you should add it as my assumption is the attackers can take down any Arma2OA server if they chose to target you.

 

[mgm]

This is getting interesting: The second attack method he is using now is making all clients go back to lobby.

 

Anyone know a fix for this one perhaps?

[mgm]

So I wrote a small batch file so that the server will be restarted in under 45 seconds if the attacker comes back ( ).

 

The script above takes corrective action on the first issue: "attacker crashes the game server exe"

It does not take corrective action on the second issue: "attacker makes everyone go back to the lobby". <== anyone have a script/solution for this?

[mgm]

I found a mention of the second issue ("attacker makes everyone go back to the lobby") here: http://opendayz.net/threads/players-forced-to-lobby-bug.18884/ by bFe (who is also a forums member here).

I will contact him now via PM to see if there's a fix he came up with.

 

Anyone else have a fix, please post - thanks.