Jump to content

REDIS Database Instability


ClontarfX

Recommended Posts

After running the mod for a week now I have noticed that the redis server process is prone to crashing when reaching memory constraints.

 

This is an issue on servers which do not have a high amount of memory available to be dedicated to the redis database file. We run multiple servers, however our game servers are NOT configured to be dedicating over 1GB of memory to a flat-file database system. I would prefer to not have to host the DB via an external DB host...

  • Why was REDIS (a bleeding-edge experimental DMBS) chosen over the proven reliability and performance of MySQL?
  • Is it possible to move back to MySQL (or provide an alternative Hive DLL) to utilise our existing DB infrastructure?

Is there a way for server administrators to configure REDIS not to crash and stop accepting connections, or to automatically restart the redis server when a failed fork is detected? There seem to be many design decisions made with this mod that haven't taken into account the requirements for server owners. I can't stress enough that I think REDIS is a very poor choice for game servers...

Link to comment
Share on other sites

For the record, it seems we can set a parameter in order to make REDIS remove data from the memory table once limits are being reached.

  • maxmemory-policy allkeys-lru

This should probably be in the default server configuration... I'm going to give this a trial run now. Cross fingers this makes it a bit more stable...

Link to comment
Share on other sites

I had this problem too. Crashed after 2h of uptime. I resolved it by increasing the Page File size in Windows. It was set to 1GB,and I upped it to 8GB. This instantly resolved the issue. Hasn't crashed in days, and is rarely restarted.

The one DB instance is also serving 2 independent servers at this point as well.

Link to comment
Share on other sites

I'm also running this on Win2008R2, and I am finding this new DB system very unstable. Redis is constantly crashing.

 

I personally am not pleased about using this new database system. How are we suppose to troubleshoot such things if we know little to nothing about it? Not everyone has the time or ability to learn a new database systems. We are loosing players due to playing for a few hours just to loose their progress. Since I see this as something out of my control, I just tell them to go have a chat with the Developers, as we are not the ones that chose this database system. All my settings in regards to the DB are default settings, except for the access password.

 

As a Server Manager, I can't manage parts of my server, that's an issue.

 

There is an old Technical Acronym that I must reflect on and suggest everyone do the same "KISS" (Keep, It, Simple, Stupid).

This works as it reduces the ID10T errors.

Link to comment
Share on other sites

After running the mod for a week now I have noticed that the redis server process is prone to crashing when reaching memory constraints.

 

This is an issue on servers which do not have a high amount of memory available to be dedicated to the redis database file. We run multiple servers, however our game servers are NOT configured to be dedicating over 1GB of memory to a flat-file database system. I would prefer to not have to host the DB via an external DB host...

  • Why was REDIS (a bleeding-edge experimental DMBS) chosen over the proven reliability and performance of MySQL?
  • Is it possible to move back to MySQL (or provide an alternative Hive DLL) to utilise our existing DB infrastructure?

Is there a way for server administrators to configure REDIS not to crash and stop accepting connections, or to automatically restart the redis server when a failed fork is detected? There seem to be many design decisions made with this mod that haven't taken into account the requirements for server owners. I can't stress enough that I think REDIS is a very poor choice for game servers...

 

Redis is not experimental and is used in many large scale web sites as a cache between mysql and the frontend.

 

Redis is much faster than mysql and has lower latency. It is mature and well supported and has many tools and resources available. 

 

If your server is running low on diskspace/memory reduce the max memory value in the redis.conf. The redis database only uses a small amount of actual memory but does allocate a page file about 1.5x size of the max memory setting.

 

After we release the sources I am sure you will see other options pop up.

Link to comment
Share on other sites

  • 1 month later...

Is there any way to move this back to SQL?

 

It's not mature at all. In IT terms it's a baby and it's a long way from being anything else. I dare say you wouldn't find this in use in any production environment without it requiring a dedicated support engineer to babysit it all the time, one with experience with Redis, which I imagine is very hard to get. In the datacentre I used to work in if people wanted more speed between a web application and a database they upgraded or added hardware didn't band-aid with a mid way database.

 

Performance difference is minimal on SSD hardware or accessing a SAN over a network, network limitations will impact it before disk access time.

 

If my server is running out of memory or disk space then my provider had better bloody fix it and if you're running dedicated then how small of a box can someone seriously be trying to run it on?

 

Is it even possible to look at SQL as an option? I hate this DB and can't do shit with it, the tools out are very basic crap.

 

Curious why you moved away from what's essentially been an industry standard for databases for about 30 years?

Link to comment
Share on other sites

 

Curious why you moved away from what's essentially been an industry standard for databases for about 30 years?

 

Because...

 

"Redis is much faster than mysql and has lower latency."

 

 

http://byterot.blogspot.se/2012/11/nosql-benchmark-redis-mongodb-ravendb-cassandra-sqlserver.html

 

 

UNLESS you memcache ... 

 

http://blog.ulf-wendel.de/2013/using-phps-memcache-interface-to-query-mysql-5-7/

 

however...

 

"Redis is single threaded. One Redis instance cannot utilize more than one CPU thread. Redis likes fast CPUs but does not know what to do with a multi-core monster. "

 

and

 

"Here’s the result: MySQL 5.7 InnoDB Memcache beats a single Redis 2.8.2 on read performance. If and only if, you use Redis inapproriately."

Link to comment
Share on other sites

Since when has the database been the bottleneck in Arma2/Arma3?

 

I hate redis up to this point. No easy way to edit players gear/storage and no scripts to run cleanups.

 

Well, we always had database lag with traders in DayZ:Epoch for example. I'm not sure about the specifics, but I do know that redis is used in conjunction with MySQL, which means you could always pipe data between redis and MySQL if you prefer it. That will probably be like all other mods that aren't supported by the devs (at least yet) though, so you'd have to make something like that yourself.

 

Going back right now, even if AWOL changed his mind for some reason would take precious time out of other things. 

Link to comment
Share on other sites

The answer for the slow traders was to make them config based via SQF instead of using SQL for them.  It could be coded so that a SQL table is only needed to store inventory quantities.  Querying or writing a single number value is way quicker than querying item name, item class, tid, price, etc.

Link to comment
Share on other sites

What I missed about MySql was the ability to "run it and forget it". Redis feels much more temperamental and will roll itself back several days in case of a single improper shutdown. This is just my view as someone who runs a personal test server on their own PC, but there's probably a few server owners out there that feels that way.

Link to comment
Share on other sites

Make sure you stop the redis DB in console when you restart your server or anything like that.

Command: shutdown save

 

Why are you suggesting this rather than just letting it run as a service like MySQL?

Serious question. I know nothing about redis :D

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Discord

×
×
  • Create New...