[suggestion] Database

[SpiRe]

I have made and setup 100s of websites. I have sites that get millions of hits a month. These sites run PHP and use MySQL for storage of information. Some also use redis for storing cached data that normally has a TTL of a few minutes at most.

 

I can tell you from experience, MySQL AND Redis can both deal with ALOT of data IO, even concurrently. 

And yes Redis is faster than mysql at pure data IO but were it falls down is trying to manipulate the data once its in there.

 

The fact that Redis has no scheme/structure so having no data validation, makes it fast. In my opinion this is terrible for a game server like ARMA, were structure is everything, one var datatype miss match and the whole server can go to shit.

 

Redis IS NOT FASTER than MySQL when you consider the day to day management of an ARMA server, were hackers can and do spawn crap in which is near enough impossible to find in Redis, without an application to load to the data set into memory and search it. (This is were MySQL is faster).

 

The reason the dev's are using Redis I suspect is because it is easy to change the "database* schema" on the fly and you don't need much additional knowledge to interact with Redis, MySQL is more involved.

 

* Redis IS NOT a database!

 

Please don't flame me, I really do have a lot of experience with this crap :P

 

The thing with data manipulation is real... Do everyone remember "WIPE WILL BE NEEDED"... With MySQL you can just apply DB structure update and you are fine. Or what you will do when your Redis crash your DB data? all that you really can do is big rollBack. Like that happen to me: ( ). I can't imagine how wow server would survive with Redis.... And if WoW server with really really big DB and 1000 players can run smoothly then A3 Epoch with 100 can too...

 

So using excuses isn't really necessary.. Just DEVs wants Redis. Thats it... When Epoch will be open then there will be sure someone who will port it to MySQL.

[SpiRe]

Somehow this debate seems more and more like a debate about maybe building the Eiffel tower after construction is about 1/3rd done in construction steel rather than puddling iron.

 

It's already built there, everything beyond that point is mere trivia at best.

 

I almost saw there: Hey lock the topic, people wants something different.

[Darth_Rogue]

What if they're only using Redis until such time as they remove the server security features and obfuscation? Maybe the whole point of using Redis was to intentionally make it difficult for people to edit stuff until the devs are "ready" to allow it? Just a thought.

[Stranger]

Somehow this debate seems more and more like a debate about maybe building the Eiffel tower after construction is about 1/3rd done in construction steel rather than puddling iron.

 

It's already built there, everything beyond that point is mere trivia at best.

 

Drama Queen. Havent seen any Post or Quote you made for users...stop being that arrogant pls. It is a mod that could only exist by the users and administrators doing a good Job.

So your statistic poll you set up says nothing! The most users hosting on Vilayer and so on - thats not real administering a server and these users dont have a clou about how fucked up it is to work with redis.

 

Its fast - we are all sure about that. But its a Pain in the ass to get soemthing done there (not user Friendly) and Redis is not made to be used in such environments. Redis is made for much bigger stuff.

[maca134]

What if they're only using Redis until such time as they remove the server security features and obfuscation? Maybe the whole point of using Redis was to intentionally make it difficult for people to edit stuff until the devs are "ready" to allow it? Just a thought.

This is not the case, if you look through and understand how the backend of Epoch works, its clear that there is no intention of using anything else. Unless Awol intends to re-write alot of the backend.

 

The way vehicles get loaded, it is based on the allow vehicle list, it totals them up and there is your vehicle world limit and max index key in redis.

 

The code literally queries EACH KEY separately to pull data from Redis in a "for _i from 1 to 152".

(So 152 vehicles = 152 callExtension calls)

 

This is how is all works, the implementation of Redis dll also seems to lack the "KEYS" command too. Which would allow you to get a list of keys based on a pattern. (Vehicles:Instance:* would return all keys matching that pattern)

[SpiRe]

What if they're only using Redis until such time as they remove the server security features and obfuscation? Maybe the whole point of using Redis was to intentionally make it difficult for people to edit stuff until the devs are "ready" to allow it? Just a thought.

 

The obfuscation makes no good when the code is public. You mean like driving a prius at driving school and then change to camaro ( MySQL isn't camaro.. it was just example ) :D :D :D.. Nah i don't think so.

[Tobias Solem]

Very good point made there ;)

So once some software is written and does the job, it shouldn't be improved upon then by that logic?

 

Here's the thing about most people who code stuff I know: They all go with what they are comfortable with and know how to work with best. I have a friend who refuses to go away from Access Database when he works with databases. Who refuses to use anything that isn't IIS, and Microsoft-made, not because he's convinced that's "the best", but because he doesn't want to waste the time adapting what he knows to a platform based on something he doesn't know. Every time I show him WAMP he's like: "I could spend time learning that, and maybe I could do what I spent years on doing right now but on a different platform, or I'll just stick to what I know best".

 

Every argument against Redis keep ignoring the simple fact that AWOL made Epoch for redis built on the simple basis that it's what he knows best. The choice of database most likely isn't a "what's best for what most people are familiar with, or what is more comfortable to them", but simply what he works best with and is more efficient at using.

 

The whole "improvement"-discussion is naturally being perpetuated by a bunch of people who are better at/more familiar with MySQL, and none of the arguments are untrue. Yes it is easier to use, yes there are way more tools for it, yes it's a real database system, yes it can probably be as fast as Redis, etc. etc. - but it's simply not what AWOL decided to use.

 

And as you've noted yourself, there are few signs that this is expected to change. So if someone wants to port Epoch to MySQL, they most likely can, but don't expect AWOL to do it for you... there is no silent majority expecting him to either, so the "What's best for the community"-card that usually gets pulled in this situation doesn't really work either. 

[XerxesIv]

I would rather have mysql but that's my preference as a-lot of server owners would like it as well

[BenR]

So if someone wants to port Epoch to MySQL, they most likely can, but don't expect AWOL to do it for you... 

 

How exactly are you meant to do that without breaking the licence (redistributing modified code)

[maca134]

Here's the thing about most people who code stuff I know: They all go with what they are comfortable with and know how to work with best. I have a friend who refuses to go away from Access Database when he works with databases. Who refuses to use anything that isn't IIS, and Microsoft-made, not because he's convinced that's "the best", but because he doesn't want to waste the time adapting what he knows to a platform based on something he doesn't know. Every time I show him WAMP he's like: "I could spend time learning that, and maybe I could do what I spent years on doing right now but on a different platform, or I'll just stick to what I know best".

 

Every argument against Redis keep ignoring the simple fact that AWOL made Epoch for redis built on the simple basis that it's what he knows best. The choice of database most likely isn't a "what's best for what most people are familiar with, or what is more comfortable to them", but simply what he works best with and is more efficient at using.

 

The whole "improvement"-discussion is naturally being perpetuated by a bunch of people who are better at/more familiar with MySQL, and none of the arguments are untrue. Yes it is easier to use, yes there are way more tools for it, yes it's a real database system, yes it can probably be as fast as Redis, etc. etc. - but it's simply not what AWOL decided to use.

 

And as you've noted yourself, there are few signs that this is expected to change. So if someone wants to port Epoch to MySQL, they most likely can, but don't expect AWOL to do it for you... there is no silent majority expecting him to either, so the "What's best for the community"-card that usually gets pulled in this situation doesn't really work either. 

 

Hang on a minute, are you one of the devs? Can you speak on behalf of them?

Awol has always been open to suggestions from the community, many of the changes in this last update is clear evidence of this.

This post is more about explaining to the devs the pitfalls of using Redis from a SERVER OWNERS point of view.

 

Many of the changes in both A2 and A3 epoch have come about from feedback from server owners.

 

Stop trying to derail/flame/etc and please only post with valid points for or against.

[Tobias Solem]

Drama Queen. Havent seen any Post or Quote you made for users...stop being that arrogant pls. It is a mod that could only exist by the users and administrators doing a good Job.

So your statistic poll you set up says nothing! The most users hosting on Vilayer and so on - thats not real administering a server and these users dont have a clou about how fucked up it is to work with redis.

 

Its fast - we are all sure about that. But its a Pain in the ass to get soemthing done there (not user Friendly) and Redis is not made to be used in such environments. Redis is made for much bigger stuff.

 

Haha, good post. I've not enjoyed myself in reading something this ironic since when infiSTAR tried to grab the moral high-ground on ARMA 3-cheating. Didn't you ragequit from hosting ARMA3 over not being treated "properly" by the Epoch devs btw? ^^

[Darth_Rogue]

I get what you're saying Tobias.  But in a case where you're releasing a product to the public where it is meant to be worked with and manipulated, then to me the developers comfort zone should carry less weight than what's in the best interest of the community.  By now the dev team knows full well that there is a substantial need to clearly see and easily edit the data.  I've got nothing but love and respect for the devs, but to intentionally choose a product that makes managing stored server data this difficult for the majority of server admins simply because it's easier for them would be very selfish and short-sighted.  I would like to seriously hope that's not their primary motivation.

[Tobias Solem]

How exactly are you meant to do that without breaking the licence (redistributing modified code)

 

Epoch devs have stated several times that the majority of these restrictions are while Epoch is still in alpha. 

[maca134]

I want Epoch mod on ACCESS 95 !!! 

At least with Access you can search the data :P

[Tobias Solem]

Hang on a minute, are you one of the devs? Can you speak on behalf of them?

Awol has always been open to suggestions from the community, many of the changes in this last update is clear evidence of this.

This post is more about explaining to the devs the pitfalls of using Redis from a SERVER OWNERS point of view.

 

Many of the changes in both A2 and A3 epoch have come about from feedback from server owners.

 

Stop trying to derail/flame/etc and please only post with valid points for or against.

 

You know I'm not, I say what's on my mind.

 

I've not ever heard AWOL's voice on this, other than through axeman, I do know that he reads and listens to what everyone says. But there must be a reason why he hasn't said anything (that I know of) directly on this subject. And to that, my speculation is as I presented: That he simply does what he does best architecturewise.

 

Personally I have found few reasons not to stick with redis.

[maca134]

You know I'm not, I say what's on my mind.

 

I've not ever heard AWOL's voice on this, other than through axeman, I do know that he reads and listens to what everyone says. But there must be a reason why he hasn't said anything (that I know of) directly on this subject. And to that, my speculation is as I presented: That he simply does what he does best architecturewise.

 

Personally I have found few reasons not to stick with redis.

Well making points like "We are not having MySQL because AWOL cant/wont do you" when you dont know yourself is silly.

 

And looking at the wider picture, how many things have been added because server owners have made good points to get it changed. There are a few...

 

There is a wealth of knowledge and experience in the community that Awol and co could utilize and that would allow them to work on cooler things, while someone else writes a persistence layer that is cross-platform (Works with MySQL, Redis and more)

 

Hell, you have me and Grim to thank for the last security patch ;) Which if it was exploited, you would have to wipe redis to fix it. With mysql, it would be a 10 minute job to fix hehe

[Brian Soanes]

Personally I have found few reasons not to stick with redis.

 

AI inventory cleanup doesn't work, the command which I assume is issued by Epoch, when it's running of course doesn't clean the AI inventory in Redis, I assume this is because it needs the game to be stopped to achieve this.

If this were MYSQL it would take 1 minute to write an SQL command to do it.

 

With AI inventory not clearing, vehicles that are sold to traders stay in trader inventories, even when bought, a replacement is in trader inventory after restart, therefore after time the map has less and less vehicles.

 

There's 1 major reason to not stick with Redis.

[Tobias Solem]

Well making points like "We are not having MySQL because AWOL cant/wont do you" when you dont know yourself is silly.

 

And looking at the wider picture, how many things have been added because server owners have made good points to get it changed. There are a few...

 

There is a wealth of knowledge and experience in the community that Awol and co could utilize and that would allow them to work on cooler things, while someone else writes a persistence layer that is cross-platform (Works with MySQL, Redis and more)

 

Hell, you have me and Grim to thank for the last security patch ;)

 

My point was that the building is 1/3 parts built already. Considering the building material at this stage is (too) late.

 

It's one thing to add security to the current code, quite another to change Epoch into using MySQL instead of Redis. You know this

 

I don't know AWOL well, but I know him well enough to be sure that he hasn't considered the strength and weaknesses of Redis vs. MySQL and come to some conclusion. He doesn't simply go with Redis based on some whim. 

[Darth_Rogue]

There's many reasons to switch to SQL.  But my biggest gripe is the lack of ability to reliably back up and restore individual server instances.  Currently in order to back up the database, you have to shut down Redis, copy the dump file to another location and then restart the service.  This is a dodgy operation at best when talking about a production server.  What if Redis fails to restart for some reason?  We all know how Windows servers can be sometimes.  Also, what if I'm running multiple servers and want to restore just one of them, but I'm running them both off the same Redis server?  I can't restore just that one server.  They are both restored at the same time since the data for both resides in the same dump file.  There's no way to separate them.  

 

Also, here's a thought for you....with the 0.3 update a few weeks ago many server owners did a db wipe, which if we were using SQL, would not have been necessary.  We could have simply wiped the traders table and left the character and stored items data alone.  That would have kept players way happier and kept their interest higher in the days and weeks leading up to the update.  Many servers were nearly empty for weeks because players didn't want to invest the time and effort on a server that was just going to be starting fresh soon.  Yes, the wipe was optional, but it was recommended, and many servers that chose not to wipe initially were later faced with problems and had to end up doing one anyway.  A SQL back end would have greatly mitigated the impact and scope of the maintenance needed for that update.  

 

And finally, what if I want to merge two databases?  Say two communities decide to join forces and we both have a player base on our respective servers and we want to combine them into one.  How is that even going to work?  Each key and entry in Redis is linked to an instance ID.  BHM_1 for instance is mine.  Say the other community's server instance is named XYZ_2.  If I try to combine both into the same instance I have three options - first, either endure an incredible volume of work to go through and have to manually update every key and every entry to the correct instance ID, or second, wipe them both and force everyone to start fresh, or third, make a choice and use one or the other, which isn't fair to either community.  If we were using SQL I could do a simple query and simply add the second community's data to mine and we're done.  Both sides retain all their data and everybody wins.  In a scenario like this, with Redis, everyone loses, no matter what you do.  

[maca134]

My point was that the building is 1/3 parts built already. Considering the building material at this stage is (too) late.

 

It's one thing to add security to the current code, quite another to change Epoch into using MySQL instead of Redis. You know this

 

I don't know AWOL well, but I know him well enough to be sure that he hasn't considered the strength and weaknesses of Redis vs. MySQL and come to some conclusion. He doesn't simply go with Redis based on some whim. 

This may be true but Awol/Co may not have had experience when it comes to running a community server, so therefore may not have considered all the potential pitfalls of running populated server(s). I can tell you now, running the many servers I do, I do not look forward to dealing with Redis.

 

You cant do shit with Redis without writing a program first. (Except search by key ;))

 

The community is clearly split down the middle, so this issue needs addressed by the devs and not just swept under the carpet.

[Darth_Rogue]

 

 

The community is clearly split down the middle, so this issue needs addresses by the devs and not just swept under the carpet.

 

 

No offense to anyone, but the folks who are "OK" with Redis likely don't actually do much of anything to their servers.  They set it and forget it and don't really care as long as it "just works". And that's fine up to a point, but any serious and knowledgeable server owner knows full well that there are times when accessing and manipulating stored data is necessary, and that's where Redis fails miserably.  

[maca134]

No offense to anyone, but the folks who are "OK" with Redis likely don't actually do much of anything to their servers.  They set it and forget it and don't really care as long as it "just works".  Any serious and knowledgeable server owner knows full well that there are times when accessing and manipulating stored data is necessary, and that's where Redis fails miserably.  

I agree but im trying to be diplomatic :P

[Darth_Rogue]

Haha!  Well I tend to call a spade a spade.  It is what it is.  

[maca134]

Haha!  Well I tend to call a spade a spade.  It is what it is.  

In my honest opinion of Redis+Epoch. Its utterly stupid, using inidb would have been better...

[Tobias Solem]

No offense to anyone, but the folks who are "OK" with Redis likely don't actually do much of anything to their servers.  They set it and forget it and don't really care as long as it "just works".  Any serious and knowledgeable server owner knows full well that there are times when accessing and manipulating stored data is necessary, and that's where Redis fails miserably.  

 

None taken. 

But I'm not someone who doesn't do anything to our Epoch-server either. 

Nor am I someone who has run into enough situations where I miss ie. phpMyAdmin. Sure, there's been an instance or two where I've missed it (eg. finding a specific jammer/jammer owner in the DB) but nothing I can't either work around, or know that eventually there will be tools for it to be accomplished.

 

AWOL still has a few aces in his sleeve to help admins/server owners with these things. But they are WiP.