My Redis backups don't work

[Richie]

Late last night for some reason Redis crashed, nothing saved and everyone was a fresh spawn.

So i attempted to restore the DB using the backups created in the log-rotator.cmd file, it runs every 15 minutes.

 

To restore it i stopped the Epoch server, stoped Redis and then copied over the dump.rdb file, then restarted Redis and launched my server, nothing :( everything is gone, the backups don't work.

 

Does the log-rotator.cmd not work or have i missed something ?

 

[lowrey]

I'm not sure dude I'm really sorry to here that!

 

I don't use log-rotator.cmd to backup my DB. However I have restored the DB in the same way you have just today (hacker) and it worked OK.

 

I know this may be little help going forward but try using xcopy to backup you DB.

 

I have a test server if you want me to try and restore you DB in a different environment. Up to you.

 

Good luck.

 

Mike.

[Nic]

There is that old IT rule that goes something like this...your backups don't exist unless you test them. Hopefully in light of what happened to Richie, other server admins will test to make sure their backups actually work.

[Richie]

I agree with you Nic, however i'm using the tools Epoch provide so i (wrongly) presumed they worked, I again (wrongly) presumed they had been tested.

So now i just have hundreds of useless backup files that can't actully be used :( I suggest other admins test their backups, if you're using the log-rotator.cmd file then chances are you have useless backups.

 

Anyone have other options for making database backups ? oh how i miss MySQL :(

[Face]

I used to have those Redis errors on my test server and had massive 7+-day roll backs, changing the maxmemory value from 4gb to 2gb in redis.conf seemed to solve everything.

[FastPoint Gaming]

Download redis-cli.exe - http://redis.io/download

 

Here's a batch script that will copy your dump.rdb to a new location:

@echo off
set root=C:\A3
set arma3bkpath=D:\backups


set Redispass=123
set RedisIP=127.0.0.1
set RedisPort=6397


 :: Set Time and Date
    for /f "tokens=2 delims==" %%a in ('wmic OS Get localdatetime /value') do set "dt=%%a"
    set "YY=%dt:~2,2%" & set "YYYY=%dt:~0,4%" & set "MM=%dt:~4,2%" & set "DD=%dt:~6,2%"
    set "HH=%dt:~8,2%" & set "Min=%dt:~10,2%" & set "Sec=%dt:~12,2%"
    
    set "datestamp=%YYYY%%MM%%DD%" & set "timestamp=%HH%%Min%%Sec%"
    set "fullstamp=%YYYY%-%MM%-%DD%_%HH%-%Min%-%Sec%"
    SET BACKUP_FILE=Epoch-%fullstamp%.rdb


D:\redis-cli -h %RedisIP% -p %RedisPort% -a %Redispass% save


copy "%root%dump.rdb" "%arma3bkpath%\%BACKUP_FILE%"

 

Just run that every 15 minutes for example using the windows scheduler.

 

some of this was taken from the rotate-logs script however modified.

[umfufu]

Maybe some just went wrong on the 2.8.12 version ( had some hickups lucky non fatal )

And found this https://github.com/MSOpenTech/redis iam running 2.8.18

[prone]

Ever get it to work, Richie?

[Richie]

Ever get it to work, Richie?

 

Nope, at least when things went wrong with MySQL it was an easy fix, Redis not so.

I did ask Axle to inform Awol of this thread, guess that fell on deaf ears.

 

Right now anyone using the backup script that comes with Epoch should test the backup actully works, otherwise like me you end up with hundreds of useless backup files and a very unhappy community :(

[nedfox]

You actually just might have run out of mem/harddrive space ?

 

Errors like these normally solve themselves when using 2Gb memory in redis.conf and extending your OS swapfile.

[Richie]

16 GB of Ram and 731 GB disk space free :unsure: it's not a resource problem

The backups are being made, they're just useless as none work.

[Liam]

longshot but had this issue once ages ago try to set this maybe thats not what fixed it but i've never had an issue since.

 

set deloriglogs=0

[ilganna]

Hey Richie,

 

have you tried to use the the Redis Desktop Manager to see if your DB contains useful data? Also, did you tried to restore the DB with an older file, like for example the day before the crash?

 

I'm using the standard tools (two instance of Redis opened the same time) and I never had issues...

 

Thanks

[nedfox]

This :

 

redis-cli -a **PASS** save

 

Should force to update dump.rdb, which then actually can be renamed to contain timestamp and moved for safe keeping.

 

Our Chernarus dump.rdb is about 4.5 Mb in size (popular server), our (far less populated) Tavi file is 30 kb :)

 

What I normally do in testing is force the dump.rdb, copy it to our test server and test there; hasn't failed ever.

[lowrey]

Dude! give me a copy of your DB and I'll cast my eyes over it!

 

I have Azure based Redis and I Microsoft Premium support account.

 

I can at least get MS support to look at it if I have no luck.

 

Just want to help, I feel your pain!

 

Mike 

[RC_Robio]

 

Read this should explain why you are getting the error: http://blog.marcgravell.com/2014_03_01_archive.html

 

Basically you need to adjust your pagefile. Especially on Windows Server 2008 r2 and below.

 

"From reading the "Redis Release Notes.docx" and "redis.windows.conf" file in the packages/Redis-64.2.8.4 folder, the appoach they used to solve the fork problem was to introduce a shared, memory-mapped file, and it defaults to the size of your physical memory (and that isbefore it has forked, when it can grow due to copy-on-write). So whether you are using it as a production server or a dev tool, you might want to pay particular attention the the “maxheap” setting in your .conf file. For my local dev work I usually spin up 5 servers, and I currently have 24GB physical memory in my machine – so by default, when running 5 completely empty databases, it instantly chewed up 120GB of hard disk space (you get it back when the server exits). If you don’t have enough disk space: the server won’t start. You'll probably see:

QForkMasterInit: system error caught. error code=0x000005af, message=VirtualAllocEx failed.

Fortunately it is simple to configure a memory bound:

maxheap 4gb # or whatever"

[nedfox]

quoting myself from 2 days ago.

 

Errors like these normally solve themselves when using 2Gb memory in redis.conf and extending your OS swapfile.

[Floyd]

The following is an interesting read for windows users concerning memory and disk usage. (I found specific information about database corruption in the Windows environment that I was going to post, but I lost it):

 

fork()

The POSIX version of Redis uses the fork() API. There is no equivalent in Windows, and it is an exceedingly difficult API to completely simulate. For most of the uses of fork() we have used Windows specific programming idioms to bypass the need to use a fork()-like API. The one case where we could not do so was with the point-in-time heap snapshot behavior that the Redis persistence model is based on. We tried several different approaches to work around the need for a fork()-like API, but always ran into significant performance penalties and stability issues.

Our current approach is to simulate the point-in-time snapshot behavior aspect of fork() without doing a complete simulation of fork(). We do this with a memory mapped file that contains the Redis heap. When a fork() operation is required we do the following:

• Mark every page in the memory mapped file with the Copy on Write page protection

• Start a child process and pass it the handle to the memory mapped file

• Signal the child to start the AOF or RDB persistence process on the memory shared via the memory mapped file

• Wait (asynchronously) for the child process to finish

• Map the changes in the Redis heap that occurred during the fork() operation back into the memory mapped file.

The upside with this implementation is that our performance and stability is now on par with the POSIX version of Redis. The down side is that we have a runtime disk space requirement for Redis equal to the size of the Redis memory mapped heap. The disk space requirement defaults to:

• The size specified by the –maxheap flag if present, otherwise

• 50% more than the --maxmemory setting if present, otherwise

• The size of physical RAM

We also have a runtime page file commit requirement that varies depending on the amount data in the Redis heap during the quasi-fork operation. The maximum for this is about 3 times the size of the memory mapped file. This is usually not a problem because the default configuration of Windows allows the page file to grow to 3.5 times the size of physical memory. There are scenarios where 3rd party programs also compete for system swap space at runtime.

And running redis as a service (and multiple instances on the same server):

Running Redis as a Service

In order to better integrate with the Windows Services model, new command line arguments have been introduced to Redis. These service arguments require an elevated user context in order to connect to the service control manager. If these commands are invoked from a non-elevated context, Redis will attempt to create an elevated context in which to execute these commands. This will cause a User Account Control dialog to be displayed by Windows and may require Administrative user credentials in order to proceed.

Installing the Service

--service-install

This must be the first argument on the redis-server command line. Arguments after this are passed in the order they occur to Redis when the service is launched. The service will be configured as Autostart and will be launched as "NT AUTHORITY\NetworkService". Upon successful installation a success message will be displayed and Redis will exit.

This command does not start the service.

For instance:

redis-server --service-install redis.windows.conf --loglevel verbose

Uninstalling the Service

--service-uninstall

This will remove the Redis service configuration information from the registry. Upon successful uninstallation a success message will be displayed and Redis will exit.

This does command not stop the service.

For instance:

redis-server --service-uninstall

Starting the Service

--service-start

This will start the Redis service. Upon successful start, a success message will be displayed and Redis will begin running.

For instance:

redis-server --service-start

Stopping the Service

--service-stop

This will stop the Redis service. Upon successful termination a success message will be displayed and Redis will exit.

For instance:

redis-server --service-stop

Naming the Service

--service-name name

This optional argument may be used with any of the preceding commands to set the name of the installed service. This argument should follow the service-install, service-start, service-stop or service-uninstall commands, and precede any arguments to be passed to Redis via the service-install command.

The following would install and start three separate instances of Redis as a service:

redis-server --service-install –service-name redisService1 –port 10001

redis-server --service-start –service-name redisService1

redis-server --service-install –service-name redisService2 –port 10002

redis-server --service-start –service-name redisService2

redis-server --service-install –service-name redisService3 –port 10003

redis-server --service-start –service-name redisService3

Both from the Redis on Windows.docx file in the redis-2.8.19.zip

I was looking into running multiple instances, independent database backups of the .rdb files and using a batch file with the taskkill command. I didn't want to kill all instances of the redis server in a batch file that was setup to restart redis database on a different server (Altis vs Cherno for example.)

Anyway, thought others might find the information interesting.

[Darth_Rogue]

In other words, REDIS is meant for Lunix, and running it in a Windows environment is not without potential issues.  I'd still like to see Epoch migrate to SQL, for these and other reasons which have already been well documented by myself and others.