Vehicles disappearing then reappearing after restarts

[Kroenen]

1. The server you were playing on. - 178.32.49.230:2302

2. What arma3 build and what Epoch build you are using. - 0.3.0.2b/1.42

3. What it was you were doing - Sometimes following a restart, players (and myself) have had vehicles go missing. Like a lot of times, we assume it has glitched and exploded since that seems to be quite prevalent at the moment. However, recently, we've noticed sometimes the vehicles reappear, with their inventory intact after one or possibly even two restarts. I myself have seen this with a HEMMT (bench type) which was parked on normal ground, but within a built 'garage', and also with a Taru which was parked on wooden floors.

4. Is it repeatable. - It has happened more than once, but there seems to be no pattern or means to reproduce it.

 

 

EDIT: It would appear that these vehicles are randomly moving to locations they have been previously, sometimes up to 3 restarts ago. Have since found several vehicles which the players recognise as being in locations they had been before.

[Scaris]

I have had this happen as well. I can find the vehicle in the database, however it is not rendering in the world.. and can go several restarts before it shows back up. And then it could very well dissapear again after another restart. 

[Kroenen]

Are you running FuMS by any chance?

[Scaris]

Are you running FuMS by any chance?

 

I am not

[Dwarfer]

Do the players also randomly go back to other locations ?.

If so I would say this is something todo with epoch not updating redis.

What about now with 0.3.0.3 ?

[Kroenen]

No just vehicles. I actually cured the problem by taking FuMS off (had been trying to get it running server side), and since then, no more missing vehicles. Put FuMS back on last night....lo and behold, vehicles disappearing again.

 

Not sure really what's going on....it has something to do with redis I'm sure, but what I can't get my head around is how it manages to 'lose' a vehicle (or move it to a position it was in the day before), and then remember all these positions and put it back following a restart or two.

 

Not getting any communication errors with redis at all, or RPT errors (other than the usual unrelated ones).

[Darth_Rogue]

This has been reported to me as well but I haven't had any time to dig into it.  Interesting observation about FuMS, but I don't see how the two could be connected.  FuMS does nothing with the database at all, nor does it call any Epoch vehicle spawning functions that I've seen, but anything is possible.  The FuMS code is quite complex.  Only Horbin knows what all is in there.  :P  

[Kroenen]

I know Darth, it's had me totally bamboozled!

[DirtySanchez]

Guys that problem was occurring on Donkey Punch prior to my rebuild(1 week ago), but it was happening to my custom mission I created, once I removed my epoch event mission all went back to normal.

I do currently run FuMS, so I really don't think that can be to blame, since this problem is not apparent as of this morning.

Long suggestion short, test test fail test succeed test some more.

I would start with the smaller scripts that provide vehicles and work my way up.

Making sure each one has the proper calls and saves for being able to spawn in vehicles to the server and keep em!

[Rob2]

IF I read the code correctly, this happens:

 

You have a vehicle table with, for example, 10 vehilce types, each 10 allowed per type.

Epoch calculates the vehicle limit from the vehicle table (10x10 = 100).

 

At restart Epoch loads "vehicle limit" items out of the database.

 

The problem is, when you have missions that spawn vehicles, or admins, that spawn vehicles, you maybe have 120 vehicles in the database already.

But at the next restart only 100 vehicles will be loaded.

 

I assume we need to wait until epoch goes further.

 

Again, thats only, IF(!) I read and understand the code correctly.

 

BR

Rob

[Darth_Rogue]

In my mind the way it should work is that when the server restarts it should consult the database and load all stored vehicles then look at the Epoch config and go down the vehicles list and keep spawning vehicles until it reaches the max specified amount of each class. Vehicles stored in trader inventories should be ignored. That way you have a consistent amount of vehicles on the server at all times. That's not how it works currently, but it should.

[horbin]

Yea, and I will say FuMS could be a possible contributor...so guys if you can verify.

 

The code I am using to get the vehicles to 'become sellable' is NOT getting them to stay on the map after restarts, but, based on how I am manipulating the Epoch vehicle counter it is possible some sort of 'overflow' is occuring where the vehicle IS being saved, but after a restart, Epoch will only spawn in say 100 vehicles, while there where 110 saved to the map.

 

Maybe after you see one missing..go around remove 15-20 vehicles...restart and see what appears?

 

More testing needed, or wait until we can do writes ourselves to the DB.

[Kroenen]

That sounds like exactly what is happening, which to be honest if that is the case, is a bit of a design flaw in Epoch. As Darth said, it should load the DB vehicles first, then decide if it needs to spawn any more on reset.

 

After taking FuMS off, the problem went away...HOWEVER, I still have SEM installed which is set to save the vehicles, and I've had no reports of vehicles disappearing while that's been on.  Also, on my last install of FuMS, i only had Madscience (no vehicles), static (no vehicles), and the coastal/aquatic missions (no vehicles), and I still noticed helis/cars disappear, then reappear when I took FuMS off again.

[Rob2]

.

[Kroenen]

OK, so a little more info on this one. Today, even with FuMS removed, we had a Zamak Transport Covered disappear that was there with a player, right before the restart. The limit for this vehicle is set in EpochConfig.hpp is '2'.

 

We have found 2 on the map, none of which are his, and also found at least 1 more on a trader. This makes at least 3.

 

So why is it the database seems to pick and choose how many of the vehicle are loaded in or not? Ultimately, we now need to disable vehicle saving from SEM, or any other mods because of the way the DB seems to load vehicles.

 

 

EDIT: Little more info that may help;

 

Just checked our DB, and we have a vehicle limit in EpochConfig of 173

We currently have 190 vehicles in the vehicles section of the DB, but a few of those don't even have coords, just [ ]

 

I thought maybe it only loads the 1st 173 vehicles, but a Zamak that went missing was ID 186, and a Taru that went missing was ID 46.

 

Trying to find even this information in redis has taken about 90 minutes......bring back SQL, all is forgiven! :)

[Darth_Rogue]

Would be nice if we could hear from one of the devs on this to get some official clarity on how the system is intended to work and what variables could cause it to not work properly.

[Rob2]

We currently have 190 vehicles in the vehicles section of the DB, but a few of those don't even have coords, just [ ]

 

 

As far as I know, [] are deleted/destroyed vehicles. Just empty slots. I still try to fully understand the vehicle spawn code. But at the moment I assume, when you have a class limit of 2, the server only spawns 2.

 

My Mozzie limit is 7. My vehicle limit is 212. I have 212 Redis Keys. According to ZUPAS Redis Tool / Live Map, I only had 202 vehicles spawned.

So I spawned 10 Mozzies via EAH. Then I had 212 vehicles on map according the ZUPA tool. I was not able to spawn a 11th mozzie. So at least the EAH seems to respect the vehicle limit.

Then I did a restart. And all 10 mozzies are gone and I was back at 7 Mozzies. So after this first test, I assume, the spawn management respects the vehicle class limit.

 

However, I think, this is not a "error". The vehicle management code is only not yet that mature. So I think we just need to wait. And deal with it for now.

The CRC check from the server side PBOs is removed, afaik. Which means, we could improve the code already. However, that is no trivial task.

 

EDIT: 2 Restarts later I now have at least 222 vehicles (17 mozzies) in the database (and in the Zupa livemap, that reads the database) but not spawned on the map.

I see I still not fully understand this.

 

 

BR

Rob

[axeman]

Give us a clear report and will be happy to help :) is it FUMS ? Certainly seems to be helping from what I have read.

 

The traders inventory, as far as I am aware and will check, shouldn't add to the amount of spawned vehicles as they are, well, trader inventory and not spawned vehicles on the map.

 

At some point during the loading limits will be hit, currently each part is spawned in its own thread, I expect that may be optimised at some point but for now it certainly makes for quick server loads..

 

it may make for some unpredictable orders upon load but general limits should be adhered to.

[Rob2]

Axeman: From much servers I hear from "disappearing vehicles". From the server code  I know, that epoch loads as much vehicles that fit into the vehicle limit.

But what is, when you have a vehicle limit off 100 but 120 vehicles in the database? In my actual understanding of the situation, 20 vehicles will not spawned then.

And I assume that are the "disappearing vehicles at server restart". Player says, he lost a vehicle, admin spawns a new one, player parks at his base, restart, vehicle gone again.

 

I think it would be better not to load #vehiclelimit vehicles, but query the database for the number of vehicles in the database and load them.

But I assume, that is not trivial, cause would it be trivial, the devs would have done it that way already.

However, we have still the problem, that it can be, that there are more vehicles in the database than the vehicle limit is.

 

BR

Rob

[Darth_Rogue]

The server should always load whatever vehicles are in the database. Say there are already 4 Off road Trucks active on the map and a player buys another one that's stored in a trader. So what happens if the Epoch config file has a max of 4 off road vehicles set? One of those 5 trucks isn't going to spawn in on the following restart. That's where the disappearing vehicles are being reported.

The only time the vehicle list in Epoch config should come into play is if all the vehicles have been loaded from the database but the max amount of a given class has NOT been reached. Then the server will spawn more of that vehicle class until the max is reached as specified in the config.

I'm guessing it was coded the way it currently is to try to keep the vehicle count down so as to not contribute to lower server performance. That makes good sense, but me personally, I'd rather have too many vehicles on the map and have to thin them out now and then versus not having enough vehicles for my players to find and use.

[Rob2]

That source for reference: https://github.com/EpochModTeam/Epoch/issues/255

 

I see it from the same point. Auto-Spawn vehicles when we have LESS than the table. But I would prefer not to have a hard limit. At least, mission and admin spawned vehicles should join the game+database correctly.

 

BR

Rob

[Scaris]

Give us a clear report and will be happy to help :) is it FUMS ? Certainly seems to be helping from what I have read.

 

The traders inventory, as far as I am aware and will check, shouldn't add to the amount of spawned vehicles as they are, well, trader inventory and not spawned vehicles on the map.

 

At some point during the loading limits will be hit, currently each part is spawned in its own thread, I expect that may be optimised at some point but for now it certainly makes for quick server loads..

 

it may make for some unpredictable orders upon load but general limits should be adhered to.

 

I actually have the same issue and I am running A3EAI and SEM but not FUMS. I have gone through the redis database and verified that there are vehicles in my DB with full data there (coordinates, inventory, locked) that are not rendering on the map. They usually stay gone, however that is not completely consistent. I have had them come. In one case it came back and was occupying the same parking spot in my base as the new vehicle I put there which resulted in big badda boom.

[Nemaconz]

Yea I'm with Kroenen on this one, for the love of Jane Russell's big fat horse's ass on toast, bring back SQL for the database!!! I too have this vehicle problem and it's driving me up a friggin wall. I'm afraid that after even a short period of time I'm gonna have 2000 vehicles on my server of which only 50 or so are actually spawning on the map. One thing of importance I have noticed is that I get vehicles spawning vertically inside of buildings, they will not be shown on the map but they are in fact there in the real world, so perhaps that is where a lot of these "missing" vehicles are going. They're not showing up on the map but they are there because the DB for some insane reason puts them there. But, another odd thing about these vehicles is that you can walk right through them or shoot right through them like they're not even there. There is no ability to inspect them or get in them. This should do wonders for performance. Editing and modifying the database is the equivalent of someone using a dentist drill on your molars without the aid of novocaine.

[DirtySanchez]

Well if you are that determined to use a different database structure, just use profile name space. It was used by some for a short period of time last year.

Or you could just use Google and I have even found through random searches some very interesting things in relation to alternative database structures for arma.

As much as I liked mysql for arma 2, would it truly make the difference we need.

is there truly any overhead from Redis that hurts us performance wise? No

is Redis configurable in terms of saving times based on # of changes? Yes

is Redis good enough for this game? Yes

how can we stop the loss of progress at the end of restart? Stop shutting down the dB after restarts with high pop. Or adjust question 3 to work for your scheduler.

how can we control Redis? Use Redis desktop manager.

how can we remove the [] vehicles? Use Redis desktop manager

how can we straighten out a players bank account? Use Redis desktop manager

how can, STOP!

I am getting at take control of your database, MAINTAIN your database, treat it like your baby and it will respect you back as best it can. Yes we are all getting some issues here and there,

but we can't expect 1.0 when we are at 0.3!

Everything we are going through here can probably be compared to something that went on during DayZ epoch.

What everyone fails to see here is this is an alpha test with the developers, this is not buy my early alpha and suffer while I give you more hats and single bullet rifles, instead they gave us a very playable game during a true Alpha mod and said, hey we are right here working with you and for you. Hell show me a day that one of the devs haven't replied to a post, replied to git, showed face at a popular server or just general PR.

So thanks devs for the hard work, keep it up and see ya on the flip side of 0.4!

[Scaris]

Well if you are that determined to use a different database structure, just use profile name space. It was used by some for a short period of time last year.

Or you could just use Google and I have even found through random searches some very interesting things in relation to alternative database structures for arma.

As much as I liked mysql for arma 2, would it truly make the difference we need.

is there truly any overhead from Redis that hurts us performance wise? No

is Redis configurable in terms of saving times based on # of changes? Yes

is Redis good enough for this game? Yes

how can we stop the loss of progress at the end of restart? Stop shutting down the dB after restarts with high pop. Or adjust question 3 to work for your scheduler.

how can we control Redis? Use Redis desktop manager.

how can we remove the [] vehicles? Use Redis desktop manager

how can we straighten out a players bank account? Use Redis desktop manager

how can, STOP!

I am getting at take control of your database, MAINTAIN your database, treat it like your baby and it will respect you back as best it can. Yes we are all getting some issues here and there,

but we can't expect 1.0 when we are at 0.3!

Everything we are going through here can probably be compared to something that went on during DayZ epoch.

What everyone fails to see here is this is an alpha test with the developers, this is not buy my early alpha and suffer while I give you more hats and single bullet rifles, instead they gave us a very playable game during a true Alpha mod and said, hey we are right here working with you and for you. Hell show me a day that one of the devs haven't replied to a post, replied to git, showed face at a popular server or just general PR.

So thanks devs for the hard work, keep it up and see ya on the flip side of 0.4!

 

Actually Redis Desktop Manager is the worst DB tool I have ever used and does not offer the ability to create records. You just edit some of the data. There are no good tools out for Redis, you have to do everything but hand in SQL at the query command prompt.