DayZ Logger - detailed character logs

[Axe Cop]

Hey guys, I started developing my own DayZ admin tools a few days ago, because I think there are not many good tools for DayZ around and I could not find anything close to what I did so far.

 

So here it goes, I came up with a "simple" logger tool, which connects to your DayZ database and essentially writes all character changes of all active players to log files. It is easily customizable what character fields to log and how often the tool should check for changes (default every minute) among many other config options.

 

But the big question is: why is logging character data useful? Well you can do many things with the logged data, e.g. detecting hackers, make a nice map with player position history because you now have the data to see who was where at what time. Also player heat maps are possible for each day, week, whatever. I might develop a companion app to read the log files and display player position on a map and humanity changes as a graph over time or something like that, maybe later. I think you guys might have some other (even better?) uses for log files like this. Anyway it is of course possible to log other things like vehicles/objects as well, but for now only character data is implemented.

 

Every player will get his own log file with the name being the PlayerUID, and each day the logs will automatically rotate which means you have small log files which you can open with a text editor and possibly archive the logs from last week or whatever. The default file structure is something like this:

- log\ (base log directory)
     + 2013-10-27\ (folder)
                 + 12345678.log (file)
                 + 87654321.log (file)
                 + ... for every player this day
     + 2013-10-28\ (folder)
                 + 12345678.log (file)
                 + ...
     + ... for every day

The logger supports data pruning (disabled by default, see config file) what means it will omit duplicate values in subsequent log entries to safe storage space. So if you would for example log the character inventory and position (Worldspace) and run around the map alot without picking up stuff or changing your inventory (which might happen in DayZ :P), the logger will only log the position and keep the inventory value empty until it changes again.

 

As an example I have uploaded a log file from my server: http://pastebin.com/NHXTQ0D5

The logged columns in that file are: CharacterID, Inventory, Backpack, Worldspace, KillsZ, KillsH, KillsB, Humanity

It might look weird at first, but this is how log files look always look like haha. E.g. around time 02:29:53 you see only position changes for some time.

 

Keep in mind this is a beta release, but it should run pretty stable unless you lose the connection to the database server (no recovery build in yet), the programm is running on my own server for some days now without any problems and we have on average 30-50+ players around the day.

 

Download link: http://dl.bintray.com/vos/dayz/DayZLogger1.0b.zip

The DayZ Logger is a console application only, all configuration is done in the config.cfg file before start-up. 

It should be enough to fill in your MySQL database connection settings at first, but take a look at the other config options.

Attention: you might need to add an additional column to your character data table in the database, for more details see the linked README below.

You can then start the tool with the provided BAT files on Windows (assuming you have a working Java Runtime installed).

 

For more information about the DayZ Logger please take a look at the README file I put up on GitHub: https://github.com/vos/dayzlogger/blob/master/README.md (I don't like to repeat everything here, please understand).

My tool is completely open source and written in plain Java, you can find the source here: https://github.com/vos/dayzlogger

 

For any questions, ideas or suggestions please feel free to use this thread, I'm sure I forgot to mention the one or other thing... :)

[-shadow-]

Hi Axe Cop,

 

First of all I'd like to say nice work on the logger, simple yet very useful.

It's very useful for tracking player data.

 

Is it possible to track the object_data table by adding it into the config?

Because what I would like to see it do is ...

 

* Tracking for vehicles (object_data) table. Which would include static (tank traps, sandbags etc) and moving vehicles (cars, planes ,heli etc)

 

This would be extremely useful for tracking destroyed or sold vehicles.

[Granok]

I can't make this work,
I use dayz.ST and a bliss DB 

I got this error when I launch the debug 

12:14:27.079 DEBUG - localTimestamp = 2013-11-18 12:14:27.0

12:14:27.082 DEBUG - serverTimestamp = 2013-11-18 05:14:26.0

12:14:27.082 DEBUG - timeDiff = -25201000 ms

12:14:27.094 DEBUG - ================================

12:14:27.095 DEBUG - localTimestamp = 2013-11-18 12:09:27.092

12:14:27.095 DEBUG - timestamp = 2013-11-18 05:09:26.092

12:14:27.096 INFO  - querying character updates

Exception in thread "main" com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorExcep

tion: Table 'bliss_XXXX.character_data' doesn't exist

 

I have set 

#character_data_table = Character_DATA

 

But it seems to automaticly lower and unfortunally my SQLDB is case sensitive

Which is really weird because the collation name is latin1_swedish_ci

but I couldn't alter the table using 

ALTER TABLE `character_data` ADD `LastUpdated` TIMESTAMP on update CURRENT_TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP;

I had to use

ALTER TABLE `Character_DATA` ADD `LastUpdated` TIMESTAMP on update CURRENT_TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP;

to make it worked.

[-shadow-]

I have set 

#character_data_table = Character_DATA

 

 

I'm not expert but remove the # as that is used to comment a line which means it WON'T be used so change the line to this ...

 

character_data_table = Character_DATA

 

and then save the config file.

Edited November 18, 2013 by -shadow-

[Granok]

I'm not expert but remove the # as that is used to comment a line which means it WON'T be used so change the line to this ...

 

character_data_table = Character_DATA

 

and then save the config file.

 

I am soooooooooooooo stupid xD

 

It works Thank you man :)

[-shadow-]

I am soooooooooooooo stupid xD

 

It works Thank you man :)

 

:D hey we all make mistakes and no problem at all, glad I could help you out  :)

Oh and I also forgot to add I am pretty sure all the commented lines are the DEFAULT settings.

So there is no need to UNCOMMENT the lines if the tables/setting match yours.

[Axe Cop]

First: I thought nobody cared about my logger, this is the first anwer in over 2 weeks haha :P

 

:D hey we all make mistakes and no problem at all, glad I could help you out  :)

Oh and I also forgot to add I am pretty sure all the commented lines are the DEFAULT settings.

So there is no need to UNCOMMENT the lines if the tables/setting match yours.

Yeah it's the default settings, you can remove dthe leading "#" to uncomment all lines but it's not doing anything different unless your change the values! D

 

About the logging of vehicles and objects, it is possible as I've already said that in the first post, but not implemented in this version.

Anyway if you just want to log destroyed and sold cars, there are maybe better ways of doing that. I don't even know if sold cars get saved to the database or just deleted, what I'm saying is those things are hard to detects with the database alone, but easy for the server so you could just log it there in the server report file with a simple server script I guess. If you want I can look into that.

 

Btw: The logger in running on my server for weeks without the need to ever restart the program.. just saying it runs no matter of the DayZ server restarts or whatever, as long as the database is running! :)

75 MB log files in 24 days, we had to enable a whitelist on the server but there are still 25+ players every day.

[Granok]

First: I thought nobody cared about my logger, this is the first anwer in over 2 weeks haha :P

 

Dude your logger is Awesome :) !

[-shadow-]

First: I thought nobody cared about my logger, this is the first anwer in over 2 weeks haha :P

 

Yeah it's the default settings, you can remove dthe leading "#" to uncomment all lines but it's not doing anything different unless your change the values! D

 

About the logging of vehicles and objects, it is possible as I've already said that in the first post, but not implemented in this version.

Anyway if you just want to log destroyed and sold cars, there are maybe better ways of doing that. I don't even know if sold cars get saved to the database or just deleted, what I'm saying is those things are hard to detects with the database alone, but easy for the server so you could just log it there in the server report file with a simple server script I guess. If you want I can look into that.

 

Btw: The logger in running on my server for weeks without the need to ever restart the program.. just saying it runs no matter of the DayZ server restarts or whatever, as long as the database is running! :)

75 MB log files in 24 days, we had to enable a whitelist on the server but there are still 25+ players every day.

 

aww we care  :D

Sorry I didn't see that part about 'can' but not 'implemented' my bad.

But yes I would very much like to see it added.

So pretty please?  :P

 

In regards to destroyed cars they are kept in the database and the damage is set to 1

Usually what happens if you have to manually clear this up with a SQL query or admin tool.

I think in the last version or two for epoch there was a SQL fix/update to automatically clean up destroyed vehicles but we opted out of it.

But don't quote me on that  :D

 

A few examples of destroyed vehicles on my server (that have not been cleaned up)

'6973', '129157127607018575', '11', 'UH60M_EP1_DZE', '2013-11-08 17:16:10', '5992', '[237,[9702.05,9378.32,1.12]]', '[[[],[]],[[],[]],[[],[]]]', '[["glass1",1],["glass2",1],["glass3",1],["glass4",1],["motor",1],["mala vrtule",0.091],["velka vrtule",1],["munice",1]]', '0.76781', '1.00000'
'7410', '48420102946085', '11', 'Lada2', '2013-11-12 12:07:24', '0', '[297,[4839.97,10296,-0.032]]', '[[[],[]],[[],[]],[[],[]]]', '[["motor",0.92],["sklo predni P",0.968],["sklo predni L",0.985],["karoserie",1],["palivo",1],["wheel_1_1_steering",1],["wheel_2_1_steering",1],["wheel_1_4_steering",0.979],["wheel_2_4_steering",1],["wheel_1_3_steering",0.902],["wheel_2_3_steering",0.989],["wheel_1_2_steering",1],["wheel_2_2_steering",1],["glass1",1],["glass2",1],["glass3",1],["glass4",1]]', '0.15987', '1.00000'
'7582', '55309247932134', '11', 'GNT_C185', '2013-11-14 12:18:09', '0', '[1,[2566.75,14139,-1.01]]', '[[[],[]],[[],[]],[[],[]]]', '[["telo",0.904]]', '0.20425', '1.00000'

With sold vehicles I am 99.9% sure they are deleted from the database.

 

You set it so its only logged if the damage is set to 1 for logging in a vehicle log file.

Not sure how you would go about deleted vehicles though.

Arh yeah a script in the server PBO file would do the trick. 

If you could that would be great, I'd say someone out there has done it maybe?!

But it would be nice if it could somehow be implemented into your logger so it saves time trolling through RPT files  :wacko:

 

And yeah the log file size is impressive indeed!  :o

 

Dude your logger is Awesome :) !

 

What he said

[Axe Cop]

As I said it's possible to log it with my logger but much better for those cases with a server side script, since the database logger can only check the damage value, it is hard to see when the vehicled was destroyed or by whom (impossible), in the server you can do that.. the logging of vehicles would be better suited for position and inventory logging not for something like this I guess.. :D

Other then that in some cases the object_data table might change a lot, e.g maintenance of bases will add new objects and remove the old ones and stuff like that, so I would use the logger for players and not vehicles for now.

 

server side logging in the report file is not that bad, would ne nice if your could log to a different file instead of just one big log file but I understand why it is not possible with the engine.

Since your can write unique log entries to the file it would be possible to just read the Arma2OAserver.RPT file and extract the log entries you need with a special tool, i mean only copy you log entries with sold vehicle into another log file every 15 minutes or whatever, becaus ethe Arma2OAserver.RPT will be overridde after some time, I guess every 12 hours or somehting your cant look back further!?

 

Well that is just my idea, I will look into the server scripts to see where you can log those things, when I have some time :)

[Granok]

the Arma2OAserver.RPT will be overridde after some time, I guess every 12 hours or somehting your cant look back further!?

 

 

Yes ! that's a shame !!

What kind of log files get reset @ every restart ? xD 

[Axe Cop]

Yes ! that's a shame !!

What kind of log files get reset @ every restart ? xD 

the Arma2OAserver.RPT gets reset after some time or size (its a rolling log i guess, so old entries will be deleted, not the whole file), I am not sure it is not after every restart but you should notice that if your never delete if yourself its pretty small and you can only look back a few hours.. at least in my server check your file :)

[Axe Cop]

Logging of destroyed vehicles: file "dayz_server\compile\server_updateObject.sqf", line 101 funcrtion "_object_killed", you could just log destroyed vehicles at the end of that function, with diag_log (http://community.bistudio.com/wiki/diag_log) in case your don't know that, "_object" should be the vehicle so you can log the vehicle class and position or whatever  :)

[Granok]

When you shut down your server everything get reset, I tried this. I'm not sure about restart but I'm sure about shutdown 

 

By the way, I thought about a small bot in perl to open a ftp connection, and download the file.

 

Gotcha Antihack use a RPT logger but it doesn't work great and when the FTP connection is open and there is a shutdown, the server get stuck because it can't create a new RPT file...

Problem is, the RPT file increase constantly, and the FTP connection never end... So if one of your admin let an instance open and is afk you can't shutdown

[Axe Cop]

When you shut down your server everything get reset, I tried this. I'm not sure about restart but I'm sure about shutdown 

That's not true I think, I restart my test server a lot and the file won't be deleted if you shut down the server!?

[-shadow-]

As I said it's possible to log it with my logger but much better for those cases with a server side script, since the database logger can only check the damage value, it is hard to see when the vehicled was destroyed or by whom (impossible), in the server you can do that.. the logging of vehicles would be better suited for position and inventory logging not for something like this I guess.. :D

Other then that in some cases the object_data table might change a lot, e.g maintenance of bases will add new objects and remove the old ones and stuff like that, so I would use the logger for players and not vehicles for now.

 

server side logging in the report file is not that bad, would ne nice if your could log to a different file instead of just one big log file but I understand why it is not possible with the engine.

Since your can write unique log entries to the file it would be possible to just read the Arma2OAserver.RPT file and extract the log entries you need with a special tool, i mean only copy you log entries with sold vehicle into another log file every 15 minutes or whatever, becaus ethe Arma2OAserver.RPT will be overridde after some time, I guess every 12 hours or somehting your cant look back further!?

 

Well that is just my idea, I will look into the server scripts to see where you can log those things, when I have some time :)

 

It depends on how your sever is setup.

I think there is a rotate_logs.bat file that comes with epoch.

Of course again it depends on your server setup and if you have full access etc etc

 

I use my own batch file that rotates my log files with data and time stamp in the name every restart (which is every 6 hours).

 

But if you could get the server code to post a message in the RPT file then get your logger to check the file that would work great.

[-shadow-]

Logging of destroyed vehicles: file "dayz_server\compile\server_updateObject.sqf", line 101 funcrtion "_object_killed", you could just log destroyed vehicles at the end of that function, with diag_log (http://community.bistudio.com/wiki/diag_log) in case your don't know that, "_object" should be the vehicle so you can log the vehicle class and position or whatever  :)

 

Sweet that would work great, thanks :)

But it would be nice if your logger would check the RPT file and log it.

Yeah I know I am lazy :D

 

edit: 

I did know about the diag_log command as i've used it in the past to debug missions I've made for arma 2.

But when it comes to DayZ code plus database stuff it starts to make my head hurt. :D

[Axe Cop]

Sweet that would work great, thanks :)

But it would be nice if your logger would check the RPT file and log it.

Yeah I know I am lazy :D

My logger is a simple tool i dont want to include ftp access or something like that, since many users dont have access to the server file directly.. And it might keep the server from restarting if you even open the file at the localhost, but i can try that later maybe

[-shadow-]

My logger is a simple tool i dont want to include ftp access or something like that, since many users dont have access to the server file directly.. And it might keep the server from restarting if you even open the file at the localhost, but i can try that later maybe

 

Arh yes sorry I keep forgetting that most people don't have full access to files etc. so completely understandable, 

But keep us updated on logger tool, Looking forward to see where it goes and what you add to it next etc  :)

[Axe Cop]

Yeah I am busy with other DayZ scripts and stuff atm :D

I was looking into logging for your sold vehicles, problem is most of the stuff is client side in the file "dayz_code\actions\trade_any_vehicle.sqf", the onyl thing send to the server script "dayz_server\compile\server_deleteObj.sqf" is the ID of the object, not the objects itself so you can't check if it was a vehicle or whatever, I don't know if there is a way to get the object from the ID, but the best way would to change the client script to send a public variable with log details to the server and just log it that way maybe? I am no arma/DayZ scriping expert, just doing that for some weeks now.. :p

In the file trade_any_vehicle.sqf at line 172 "if(_okToSell) then {" the vehicle will be sold...

[-shadow-]

Yeah i went for a poke around in the server PBO scripts after reading your post and came up with the same conclusion.

It would be possible to replace the client script by running an edited version, putting it in the mission file and pointing it towards that.

I think you can track deleted objects from the server side script but involves some SQL queries.

Still looking into it also :)

[Alucard88]

Thank you for this tool i like !

 

You have add an option for log vehicle ?

[Axe Cop]

Thank you for this tool i like !

 

You have add an option for log vehicle ?

No and I don't think I will do that, please read the topic. If you want do it yourself, the source code is public :)

[Alucard88]

Ok thank you

[Aristoi]

But the big question is: why is logging character data useful? Well you can do many things with the logged data, e.g. detecting hackers, make a nice map with player position history because you now have the data to see who was where at what time. Also player heat maps are possible for each day, week, whatever.

Thanks for taking the time, Axecop!

 

Has anyone actually built a heatmap?  That would be deeply awesome.  If the data in MySQL would be better for that purpose, it could also be done easily that way.