I will back MySQL for Epoch !!

[BenR]

Yep, we did. AWOL has been using it for years. What features do you need from a database engine beyond the fastest possible representation of the data you requested ?

Possibly the ability to take a database backup easily without the need to then shutdown and restart the server to restore a database, which I have found is not always a safe way of doing it, quite a few things not saving. If theres a better way to take a backup than copying the dump.rdp then I would like to know

[Havoc302]

Yep, we did. AWOL has been using it for years. What features do you need from a database engine beyond the fastest possible representation of the data you requested ?

So he's been using it for years and knows it's only just a few months ago finally reached a stable build production release and he decided to use that over a well known, trusted and easily MANAGED database platform. What good is a fast database to me if I can't manage the data without having to learn a whole new scripting language. Sure it's great for those 2% of database engineers in the world, not so great for the guys who just want to run a freaking GAME server.

[Tobias Solem]

Can someone maybe make something to get Epoch working with SQL, I'm over this Redis shit. In my database right now I have 352 AI_ITEMS entries yet I only have 50 traders, also the traders are currently broken, the items they have are frozen and it's causing no more vehicles to spawn because they appear to have duplicated trader lists between traders. Killing them doesn't seem to clear it, buying all the items off them doesn't clear it, it's generally broken. I'm having to manually go through the database clearing them. If this was SQL...

What? A feature in version 0.2.5 that doesn't function properly? This must be the first time of that ever happening in the history of software development.

Very frustrating, aye. I agree. As is the duping, the glitching, the unfinished building system, the non-stacking items in the trader inventories, the lack of upgrades, etc. etc. - and they're all related to the stage of development of the mod. Which is perfectly unfinished.

I'm not even sure why you are raging (because that's the way you're coming off). Is it because you think there's a better way to do it? Then do it! Once Epoch reaches a final stage of development you can probably tear it apart and rewrite everything to fit MySQL. Or did you expect AWOL to just take all the current development progress and rewrite it because some people (you) would prefer MySQL, because it is more convenient to you?

Exactly why would he do that? Really? The man is experienced enough to have written the original Epoch for ARMA2 using MySQL, and written Epoch for ARMA3 to this point, using Redis. And you believe he isn't qualified to make a judgment what database backend is best suited for the mod? Are you paying him to do this, or is there some other reason for your complete and utter entitlement?

[Glenn]

For everyone complaining about Redis - why not try to understand what it is first, then figure out if you still feel the same way?

 

The best way to learn what Redis is all about is by taking the interactive tutorial for Redis: http://try.redis.io/

[axeman]

Easy management, I don't think, had more than a few minutes of discussion time.

 

Rule Number One: Performance ! Is it Faster ? End of discussion :)

 

Management suites and tools to access the data come later from developers. MySQL not a bottleneck ? I will never be convinced, don't bother trying, and I am one of those web developers that uses it. Don't get me wrong it has served me well, for websites.

 

Embrace the progression.. Pointy clicky tools wlll come later.

[itsatrap]

They are just mad because they don't spaek redis, come on people learn something new

[Havoc302]

What? A feature in version 0.2.5 that doesn't function properly? This must be the first time of that ever happening in the history of software development.

Very frustrating, aye. I agree. As is the duping, the glitching, the unfinished building system, the non-stacking items in the trader inventories, the lack of upgrades, etc. etc. - and they're all related to the stage of development of the mod. Which is perfectly unfinished.

I'm not even sure why you are raging (because that's the way you're coming off). Is it because you think there's a better way to do it? Then do it! Once Epoch reaches a final stage of development you can probably tear it apart and rewrite everything to fit MySQL. Or did you expect AWOL to just take all the current development progress and rewrite it because some people (you) would prefer MySQL, because it is more convenient to you?

Exactly why would he do that? Really? The man is experienced enough to have written the original Epoch for ARMA2 using MySQL, and written Epoch for ARMA3 to this point, using Redis. And you believe he isn't qualified to make a judgment what database backend is best suited for the mod? Are you paying him to do this, or is there some other reason for your complete and utter entitlement?

1. It's not just me.

 

2. I shouldn't have to rewrite it.

 

3. Before any development of any kind begins on any software a smart developer identifies his target market. The target market here is Epoch server admins. Owning a freaking game server does not automatically make you a database engineer, nor does it mean you should have to go and learn how to be one just to run your game server.

 

4. He is being paid to do this, all GSP's pay $10 a month for each Epoch server that runs (which only exists because I'm paying for it). If the servers are allowed to be monetised now then they'll probably charge for that too.

 

5. Yes, at this stage of development it'd be a perfect time to rewrite it for SQL so EVERYONE ON THE FREAKING PLANET can use it, there's a load of resources available on how to throw together easy SQL scripts without having to learn how to write the whole damned language.

 

6. Do I think developers are the most qualified people to judge what the best back end is for software that's going to primarily be used by non-developers? No I don't think they are and neither does the whole software development industry, it's why they have teams of managers, marketing people and engineers that work that out. It's easy to decide on a feature that's above the level of your user base when you're a specialist in one field. It's also why software engineers often don't decide the final look of the high end games that are released.

[nedfox]

MySQL is not the bottleneck with ARMA ffs, any web developer will tell you, mysql deals with 1000s of concurrent connects with no problems at all...

The reason epoch used redis is because its easier to develop backend dlls+code.

 

With redis, all structure goes out the window and you can throw whatever you like in the DATASTORES (not database)

 

This is very good when caching data but for storing data which needs managing its shit.

 

Mysql: storing data in a structured manner

Redis: storing any data in whatever format

 

This. Redis is an easy way out for future epoch modifications since you can drop just any string into it.

 

Problem is that Epoch devs do not care about redundant, or even obsolete data in there.. I too have 150 trader entries in there, with less then 50 traders.

I need to buy Visual Studio to compile all kind of crap command line crap, to use some scripts that might or might not work.

 

Epoch devs forgot that the gaming community is run by a lot of self made admins, who are used to SQL code.. Not some crap Perl / Unix / whatever commandline stuff.

 

Also : MySQL on my SSD is fast enough to do anything Epoch can throw at it.

[Torndeco]

I had a long post but i try and rewrite it smaller

Redis + MySQL both have advantages / disadvantages

Redis Storage
Key-Value:
Simplier to use, but you need to update the entire string (normally an array containing all the starts
Easier for people to write scripts to save in the Database

Harder to parse info from outside, requires parser to read Arma Arrays

MySQL:
Storage == SQL:
Harder to update the Database Schemes + perserve Data in the changes.
Ability to update individual Info like player inventory, without needed to update the entire string

--------------------------------------
Speed wise

Redis it is faster
MySQL: fast enough if its done right imo

ArmaServer will spend most of its time sending updates to Backend to save...
This can easily be offloaded to a worker pool todo without blocking arma engine.
Hell it might be slightly faster than asking Redis Server ;)

Plus you cant compare Trader Speeds with A2, since A3Epoch loads trader lists and saves it via setvariable on Traders

Redis also has
Advantage when it comes to speed is the ability to callExtension to fetch results in unscheduled enviroment without worry about it blocking engine to long.
Other methods using worker threads i.e like extDB, require sleep to avoid blocking arma engine. Means you need to call it in scheduled enviroment to be able to sleep.
Plus side to extDB method is you can workaround the size limitations of callExtension (last i checked 10k)

---------------

One of the main issues i believe is people need to write a parser to read the info from Redis. <----------
Since its normally an arma array of info stored in string form.

When most people are used to MySQL with the easier ability to change individual values.
Or run a simply query to check player money etc for people that might be cheating etc...


--------------

Regardless both storage schemes have different layouts and it would be a pain to support both offically.
Basicly you are stuck waiting for epoch version 1 and a public release of code.


Sidenote:
You guys couldn't pick a backend with an offical library that supports both Linux + Windows.
Instead of an offical library that supports Linux only and the port that MS maintains separately.

[itsatrap]

Well let's all just do the same old thing and never try new things because some people can't be bothered to learn something new!!!

There is a lot of language the speaks redis, I use php works 100% and they do think about unused data since almost every thing have TTL players, bank, vehicles and buildings moste arma2 server have a lot of unused data and we had to write our own cleanup script, delete old players, unused bases and crap

I welcome redis

[Havoc302]

Well let's all just do the same old thing and never try new things because some people can't be bothered to learn something new!!!

There is a lot of language the speaks redis, I use php works 100% and they do think about unused data since almost every thing have TTL players, bank, vehicles and buildings moste arma2 server have a lot of unused data and we had to write our own cleanup script, delete old players, unused bases and crap

I welcome redis

"The target market here is Epoch server admins. Owning a freaking game server does not automatically make you a database engineer, nor does it mean you should have to go and learn how to be one just to run your game server." - Did I need to explain it some other way for you to get it?

[Glenn]

3. Before any development of any kind begins on any software a smart developer identifies his target market. The target market here is Epoch server admins. Owning a freaking game server does not automatically make you a database engineer, nor does it mean you should have to go and learn how to be one just to run your game server.

 

4. He is being paid to do this, all GSP's pay $10 a month for each Epoch server that runs (which only exists because I'm paying for it). If the servers are allowed to be monetised now then they'll probably charge for that too.

 

5. Yes, at this stage of development it'd be a perfect time to rewrite it for SQL so EVERYONE ON THE FREAKING PLANET can use it, there's a load of resources available on how to throw together easy SQL scripts without having to learn how to write the whole damned language.

 

6. Do I think developers are the most qualified people to judge what the best back end is for software that's going to primarily be used by non-developers? No I don't think they are and neither does the whole software development industry, it's why they have teams of managers, marketing people and engineers that work that out. It's easy to decide on a feature that's above the level of your user base when you're a specialist in one field. It's also why software engineers often don't decide the final look of the high end games that are released.

 

NOTE: The following comments are all my own knee-jerk reaction to reading the above post. Emphasis placed in quote are my own.

 

3. Umm... I'm sorry, but I dont agree with your assessment of hte target market for A3:Epoch. Target market is a player that want to play Arma 3 Epoch, with all the features that the mod have to offer. Easy Plug'n'PLay for Server Admins should not be the first, second, fifth or the 27th concern at this stage - the Gameplay and the Mod itself should be.

 

4. I would certainly hope the Devs (any dev) gets paid for the hours and hours of work put in. It is not my place to judge business decisions that I am not a part of, cause I am not affected by it. And even if I were renting a GSP hosted server, he cost from a GSP to a Dev is not of my concern.... it is between the GSP and me that the business transactions takes place. I have not seen any indication that the Server side code for Epoch would not be free in the future... but now that you brought it up, maybe they should charge a token fee as a way to grant a license.

 

5. Just because people will have an easier time because they already know something is not a valid reason to force a change on something that works _right now_. Are there resources out there to learn Redis? Yes. 

Main Site: http://redis.io/

Interactive Tutorial: http://try.redis.io/

Documentation: http://redis.io/documentation

Is it easier to learn? Probably not - so I suggest you stop raging and start reading up on things.

 

6. Your opinion on company organization and optimal structure on how established development team is commendable - but in this case out of scope for the current conversation. When Awol and the team forms a company, moves into a high-tech nindustrial park and plan to release "Epoch - The Game", then it will probably be a good time to talk about this.

 

To summarize - how about we assume that the mod is still in Alpha, not nearly finished and that we are here right now arguing about things that really should not be argued about?

Feedback has its own section in the forum, and it is up to Devs to act on it or not.

[Havoc302]

BTW Sorry if I seem like I'm raging, I'm just pissed off at what seems to be a very poor choice in development. I love what the Epoch Devs do (95% of the time) but there's also some decisions I disagree with and I'm willing to voice them. I'm not going to be one of those people who just rages without reason though, I'm providing strong, proven reasons why I think the decision is wrong and why it should be rectified before any further development continues.

 

It all ties back to that original problem people had with the Epoch devs not releasing the files in a way which can be edited. They're making it harder and harder to customise. If Epoch was designed to be released with every server being perfectly vanilla then this approach would be ok, but Epoch is more than that, it's almost like a framework, it's an amazing base that we all love to build upon, if this approach was taken in it's development then we'd have something amazing.

 

If you look at other "complete" mods (I say that as in they're completely locked down) like Breaking Point and Altis Life they're interesting and fun for a bit but they quickly grow old, play them for a few months and you start to feel like all the servers are identical, what makes Epoch great is that each server is a reflection of what the owner wants it to be, small things that make them all have their own flavour.

 

Putting in Redis makes it harder to manage, harder to mod for and rules out 90% of server owners from being able to easily manipulate the data store for their server. It pushes it into the realm of only manageable by those who either know or want to invest the time to learn Redis.

 

Personally I'd love to learn how to work databases better but guess what, I've got enough stuff to do, I have a family, expecting twins in 7 weeks, I work full time and I spend a lot of my spare time working on my server and maybe occasionally trying to play.

 

To say that the mod should be written with the players in mind first and the admins 27th is a very stupid approach, this isn't Breaking Point, Epoch isn't going to be paying for the running of all the servers, they're not going to be a central hive, they're not going to all be vanilla and I wouldn't want them to be, it'd be boring and pointless. Do you think A2 Epoch would have ever reached 4000 servers globally if all the servers were identical and admins couldn't and didn't customise them?

 

The Devs might write the mod but the admins make the servers.

[nedfox]

To say that the mod should be written with the players in mind first and the admins 27th is a very stupid approach, this isn't Breaking Point, Epoch isn't going to be paying for the running of all the servers, they're not going to be a central hive, they're not going to all be vanilla and I wouldn't want them to be, it'd be boring and pointless. Do you think A2 Epoch would have ever reached 4000 servers globally if all the servers were identical and admins couldn't and didn't customise them?

 

This ^^

 

The community is powered by admins and driven by players. You need both and putting admins that low is just plain BS.

 

Us admins need to be able to read the DB in a reasonable way, in one glance (for instance by running queries) and intervere accordingly.

ATM a lot of admins are locked out from this since they don't know Redis.. The answer is not "learn redis" since it's way more then redis alone, PLUS most hosted servers will not allow all kind of ported Linux crap to run on their server.

 

The whole idea of redis is easy mode for the devs, without taking ANY account for normal game admins.

 

The reasons posted (fast, that's it) should be balanced against what will *work*  for the community (easy access to data, that's it).

 

IMO the balance is towards access; I have not seen a properly mysql server config that wouldn't cope with 70 player servers (And I've run a dozen).

[RIE_JACKTE(new)]

personally i like this less tools for casuals thing , too many arma 2 epoch servers were shit running on virtual servers with people running them only being able to do what tools they had enabled them to do  , hopeful  the less "pointy clickly" tools thing will keep the server count down and the population per server higher .

[Havoc302]

personally i like this less tools for casuals thing , too many arma 2 epoch servers were shit running on virtual servers with people running them only being able to do what tools they had enabled them to do  , hopeful  the less "pointy clickly" tools thing will keep the server count down and the population per server higher .

That's not how you bring people to your server, only the well ran servers were popular anyway, moot point.

[itsatrap]

Well the mod is only at version 0.2.5 so when it reach 1.0 I'm sure there will be a lot of tools, when I hosteds my first epoch server version 0.9.x there was no tools no script other then default epoch look where we are now

I'm sure there will come toll to the admin who don't have time the learn new things

[RIE_JACKTE(new)]

That's not how you bring people to your server, only the well ran servers were popular anyway, moot point.

not a moot point i've played on quite a few popular ones that were ran by idiots such as val's , panda's server they managed to get 35 to 50 on week ends and both the owners were incompetent . etc val banning me for camping stary when i was in zelengorse .

and zero remorse  is quite popular atm and that server runs like arse .

 

[Havoc302]

Well clearly they are doing something right or players wouldnt play there...

What they're doing "right" is giving players free stuff, weapons on spawn in, teleports to whatever starting point they want, etc. Free stuff usually beats out a well run server with the kiddies.

[lowrey]

OK guys! 

 

Back to the topic at hand!

 

Redis is the best solution for Arma 3. Redis in fact is an all round great solution so good that it's being endorsed by one of the largest software companies in the world... Watch this space!

 

Perhaps we can close this thread as it's getting a little off topic?

 

M

[elbabar]

"The Devs might write the mod but the admins make the servers." 

 

"Admin" it's a big word for just extract an zip, and follow a readme ...   

[RimBlock]

FIrstly, these threads, from a attempt to force a change, are doomed to failure just like the threads requesting the A2 Epoch hiveext.dll be amended with lots of good reasons for doing so were.  History shows that the Epoch team are very unlikely to change anything to do with the undelying core tech choices once that choice is made.

 

Bottom line is that it is their mod and their choice.

 

These threads can, however, bring forward some interesting discussion points amoung the chaff.

 

Yep, we did. AWOL has been using it for years. What features do you need from a database engine beyond the fastest possible representation of the data you requested ?

 

Any chance of these benchmarks being published so we can see them.  Would be very interested.

 

Easy management, I don't think, had more than a few minutes of discussion time.

 

Rule Number One: Performance ! Is it Faster ? End of discussion :)

 

Management suites and tools to access the data come later from developers. MySQL not a bottleneck ? I will never be convinced, don't bother trying, and I am one of those web developers that uses it. Don't get me wrong it has served me well, for websites.

 

Performance should not always measured purely on speed.

 

Rule number one: Best fit for the purpose.

 

Most sensible people would not buy, even if they could, a rocket car to get their monthly supermarket shopping :) .

 

Personally, off the top of my head, I would measure performance for an Arma mods persistant layer on (in no particular order);

Speed of communication (TPS)

Supportability (within A3 Epoch server management scope)

Stability

Support (for the technology by the releasing entity)

Ease of connectivity / maintenance of connectivity (between persistant store and Arma).

 

We chose extDB for our A3 mod based on these above criteria.

 

====================

 

@Everyone.

 

I have yet to see an investigation that points to MySQL being a bottleneck to A2 Epoch that is able to rule out;

1. The A2 engine callext function being a limiting factor.

2. The Hiveext.dll being a limiting factor.

3. The underlying server hardware being a limiting factor.

 

If anyone knows of any investigations that have then I would love to know as they would be very interesting.

 

Unfortunately when there is a slowdown with data that has to be pulled from an external source, in A2 at least, the common knee jerk reaction is that it is the DB server that is the problem.  The DB being the choke point may not be the case and TBH I would be very surprised if it was with MySQL.

 

Would be interesting to see;

1.  if there is a difference between the hiveext.dll and extDB.dll (I believe someone was porting over A2 EPoch to extDB) for trader slowdown using the same MySQL DB. 

2. If a MySQL DB hosted on a ram disk still has the same issue.

[Havoc302]

"The Devs might write the mod but the admins make the servers." 

 

"Admin" it's a big word for just extract an zip, and follow a readme ...   

If you actually run an Epoch server and all you've done is that then no, you're not really an admin.

 

My A2 Epoch server had 33 scripts loaded into it, many of them I had to go through line by line to get them to work together, RimBlocks P4L with Snap Building Pro, Added Vectors for Snap Building Pro and a couple of other scripts that utilised some of the same files had to be merged and tested over and over, line by line, checking log files for errors and making adjustments. If that's not being an admin of a server then I don't know what is. Almost no Epoch server was vanilla and the installation and troubleshooting of those mods was what an admin does. Not just that there was maintenance scripts to create in the SQL database to keep the DB clean, not all of them were released with Epoch up until about 1.0.4. Then there's the player support for all the glitches that ARMA provides, any good server had admins that were johnny on the spot when a glitch happened, trying to rectify the issue and restore the normal state of play (at least on my servers anyway).

 

RimBlock we did a fair bit of testing with Vert Hosting on the A2 Epoch server and when we converted it to an Overpoch server. We noticed that there was some trader and database lag when we had about 4000 items in the traders, certain traders had a lot of weapons when we added the Overwatch weapons in but that was easily rectified by dividing them up a little more, adding a couple more traders and dropping the items down to 3600 (removed non-essentials and some caused lag just being on the server).

 

I hope you're wrong RimBlock, I'd like to think the Devs can take criticism so long as it's constructive and backed up by logic and agree to at least think about it, do a needs analysis and actually assess how their mod is going to be used. If they want it to be another breaking point, then why release it. If they want us to use it as the amazing base it was in A2 and build upon it then facilitate that. We shouldn't have to study database engineering to be able to manage the daily operations of a bloody game server, this isn't an enterprise server in a large business where I can spend all day learning how to operate it properly, it's for gamers who want to run it and play it in their spare time.

[RimBlock]

RimBlock we did a fair bit of testing with Vert Hosting on the A2 Epoch server and when we converted it to an Overpoch server. We noticed that there was some trader and database lag when we had about 4000 items in the traders, certain traders had a lot of weapons when we added the Overwatch weapons in but that was easily rectified by dividing them up a little more, adding a couple more traders and dropping the items down to 3600 (removed non-essentials and some caused lag just being on the server).

 

Not wanting to take this off-topic but how do you define DB lag ?.  Was the DB server showing IO or processing heavy load ?.  It would be interesting to know if the 4000 items caused the response to go over the 10k extension call limit (requiring multiple calls to retreive all the data ) and if the 3600 was under it.

 

The other possibility is that the tables and indexes could possibly be better optimised.  I have seen some trader lag on my own test server with just two of us playing on a local lan.  I have no reason to believe that issue was with the DB.

[Havoc302]

Not wanting to take this off-topic but how do you define DB lag ?.  Was the DB server showing IO or processing heavy load ?.  It would be interesting to know if the 4000 items caused the response to go over the 10k extension call limit (requiring multiple calls to retreive all the data ) and if the 3600 was under it.

 

The other possibility is that the tables and indexes could possibly be better optimised.  I have seen some trader lag on my own test server with just two of us playing on a local lan.  I have no reason to believe that issue was with the DB.

We never really got to the bottom of it, I had a hunch that it was a network bottleneck (or possibly buffering lag) as they run a central SQL server and all the ARMA servers talk to it, but we could never confirm that and they couldn't see anything wrong.