Epoch Inventory Auditing Tool

[hambeast]

STATUS:  Released
 
Description:

This is a tool I wrote in my spare time to help with analysis of the inventory in the DayZ Epoch Database.  As many of you know, the inventory colunn of the dayz_objects table is quite hard to read.  When we have objects such as safes, or vehicles with large cargo capacity, it is almost impossible to get a clear view of what and how much is contained within said object.  This tool will allow you to do analyze the contents of the inventory to help prevent duping or to get an overview of how much of item X is on your server, and who has what.
 
Features:

• Log Generation - CSV and XML format.
• Red Flagging - Mark item classnames and maximum allowed qty to be logged for future analysis.  Want to know who has 200 briefcases? or who has 20 of an item you've only handed two out? Easy with red flagging.
• Total Inventory Counts - Get the total count of each inventory item in an easy to read format.  Want to know how much of each item is in a safe?  Easy!
• Graphing - Generate beautiful(*) graphs of the top inventory items on your server!
• Open Source - My code is released under GPLv3 which means in short, you are free to edit, share, decompile my code without fear of DCMA takedown notices.  If you choose to edit or use my code in another project, you MUST ensure that the project is open source!

Planned Features:

• Historical Analysis - Compare XML or CSV log files run over a period of time and generate graphs/logs to show historical item growth
• Auto-Lock-Flagged-Safes - Automatically set a safe to a random or predetermined combination (5 digit or more) if it is flagged.  This can be used by admins to help prevent duping in the event they suspect something is wrong.

 

Screenshot of CSV red flag output:

 

Screenshot of a beautiful(*) graph:

 

 

 

Download here: https://github.com/deadfred666/dze_inventory_parser_public

 

* beauty is in the eye of the beholder...

[insertcoins]

how would one use this against duping?

[cen]

Anything to use to catch dupers is a big YES in my mind.

 

I would use it hands-down.

[fr1nk]

I'd use it.

[hambeast]

how would one use this against duping?

 

this is an analytic tool, meaning it gathers data for analysis by a human (or a script if you so desire).  It is up to the user to determine if the information gathered is useful.  As I said, this program basically checks all of the inventories on the server and counts the # of a specific item in that object's inventory to see if it is above the warning threshold.  if it is, flag it for future analysis.

 

For instance, on my server, there is a limit to the high end gems such as rubys.  Currently we do not hand them out at all but I have been thinking of incorporating them as a higher end currency.  If within the first few weeks we see a huge spike in their count we know something is wrong.  Heck you could even use it for rarer items like combo locks etc.

 

 

 

Anything to use to catch dupers is a big YES in my mind.

 

I would use it hands-down.

 

I'd use it.

 

I'm going to work more on this since I got a bit of down time today.  Today's task is integrating XML config support for DB connections as well as item watch limits.

 

How do you guys prefer the output to be generated?  XML, SQL, CSV/Flatfile? or perhaps an option of the above.

[fr1nk]

CSV/SQL would be fine for me, but I'm open to whatever's popular/easiest.

[cen]

Same here, whatever is easiest for you.

[hambeast]

Just a quick update, here's an example of the config xml.  You will be able to define an XML for each of the databases you wish to poll.  You will set it like so:

 

DZE_Item_Analyzer.exe --configFile=EpochCherno.xml

 

<?xml version="1.0" encoding="utf-8" ?>
<config>
  <DB>
    <ip>localhost</ip>
    <port>3306</port>
    <database>dayzepochnapf</database>
    <username>dayz</username>
    <password>SECRET</password>
  </DB>
  <RedFlagList>
    <Items>      
      <Item ClassName="ItemTopaz">1</Item>
      <Item ClassName="ItemObsidian">6</Item>
      <Item ClassName="ItemSapphire">6</Item>
      <Item ClassName="ItemAmethyst">1</Item>
      <Item ClassName="ItemEmerald">1</Item>
      <Item ClassName="ItemCitrine">1</Item>
      <Item ClassName="ItemRuby">1</Item>
      <Item ClassName="ItemComboLock">8</Item>
      <Item ClassName="ItemBriefcase100oz">75</Item>
      <Item ClassName="metal_floor_kit">25</Item>
      <Item ClassName="cinder_wall_kit">25</Item>
      <Item ClassName="cinder_door_kit">25</Item>
      <Item ClassName="cinder_garage_kit">25</Item>
      <Item ClassName="ChainSaw">2</Item>
      <Item ClassName="ChainSawB">2</Item>
      <Item ClassName="ChainSawG">2</Item>
      <Item ClassName="ChainSawP">2</Item>
      <Item ClassName="ChainSawR">2</Item>
    </Items>
  </RedFlagList>
</config>

 

edit:  also optimized the query to only look for objects that contain items in the red flags list.  Should speed up the processing quite dramatically.

[hambeast]

OK, new update...  I optimized the scanning process by switching from nested foreach loops to LINQ queries.  Holy crap does it make a difference in performance.

 

old parsing code:

// loop thru magazines to see if they meet criteria
foreach (Item magItem in obj.inventory.magazineItems)
{
// check against the badItems list
foreach (Item badItem in badItems)
{
try
{
// match our current item against the rule we created
if (magItem.className.Equals(badItem.className) && magItem.count >= badItem.count)
{
// add to our list of suspicious objects
//suspiciousObjects.Add(obj.characterID, magItem);
Console.WriteLine(string.Format("ObjClass: {0} - CharID: {1} - ItemType: {2} - ItemQty: {3}", obj.className.ToString(), obj.characterID.ToString(), magItem.className.ToString(), magItem.count.ToString()));
countBadMagItemsSTD++;
}
}
catch (Exception ex)
{
throw ex;
}
}
}

 

new parsing code:

                    // Check to see if our mags show up in the bad list and if they are above the limit
                    var badMagItems = from m in obj.inventory.magazineItems
                                      join b in badItems on m.className equals b.className
                                      where m.count >= b.count
                                      select m;
                                        
                    // add our badMagItems to our temp obj
                    foreach (Item record in badMagItems)
                    {
                        tmpObj.inventory.magazineItems.Add(record);
                    }

 

 

Here's the output of the program in XML (sorry csv to come) it was way too easy to just serialize it straight to XML: https://gist.github.com/deadfred666/8120def20aefb3ad66a5 (charid or safecodes X'd out since this is a live DB)

sample:

 

<DZObject>
  <className>VaultStorageLocked</className>
  <characterID>XXXX</characterID>
  <objectUID>1290811599324306</objectUID>
  <objectID>1060</objectID>
  <worldSpace>[305.723,[12908.1,15993.2,0.417]]</worldSpace>
  <rawInventory>[[["BAF_LRR_scoped","Sa61_EP1","ItemToolbox","ItemEtool","ItemKeyKit","vil_uzi_SD","FHQ_ACR_WDL_RCO_SD_F","FHQ_XM2010_WDL_CAMO","RH_anacg","FHQ_ACR_BLK_HAMR_SD","FHQ_MSR_NV_SD_DESERT","FHQ_XM2010_NV_DESERT"],[1,1,3,1,2,1,1,1,1,1,1,1]],[["5Rnd_86x70_L115A1","5x_22_LR_17_HMR","100Rnd_556x45_M249","MAAWS_HEAT","6Rnd_45ACP","30Rnd_545x39_AK","30Rnd_762x39_AK47","ItemBandage","17Rnd_9x19_glock17","7Rnd_45ACP_1911","SmokeShell","10Rnd_762x54_SVD","WoodenArrow","6Rnd_HE_M203","5Rnd_762x51_M24","100Rnd_762x51_M240","20Rnd_B_765x17_Ball","ItemAntibiotic","PartGeneric","ItemTentDomed2","FlareWhite_M203","CinderBlocks","ItemWire","PartGlass","ItemGenerator","ItemAluminumBar","ItemLRK","ItemSandbag","vil_32Rnd_uzi_sd","100Rnd_556x45_BetaCMag","ItemBloodbag","ItemMorphine","ItemSodaCoke","ItemBriefcase80oz","100Rnd_762x54_PK","RH_12Rnd_45cal_usp","cinder_wall_kit","FHQ_rem_30Rnd_680x43_ACR_SD","ItemBriefcaseS50oz","RH_6Rnd_44_Mag","HandGrenade_East","FHQ_rem_5Rnd_300Win_XM2010_NT_SD","FHQ_rem_5Rnd_300Win_XM2010_NT","FHQ_rem_7Rnd_338Lapua_MSR_NT","FHQ_rem_7Rnd_338Lapua_MSR_NT_SD","ItemTopaz","ItemRuby","30Rnd_556x45_StanagSD"],[2,1,1,1,3,2,4,12,2,3,2,2,2,1,1,4,2,3,5,1,1,1,1,1,1,1,1,1,1,1,4,3,1,1,5,2,1,4,1,1,2,1,1,3,1,4,2,1]],[["DZ_LargeGunBag_EP1"],[1]]]</rawInventory>
  <inventory>
    <weaponItems />
    <magazineItems>
      <Item>
        <className>ItemTopaz</className>
        <count>4</count>
      </Item>
      <Item>
        <className>ItemRuby</className>
        <count>2</count>
      </Item>
    </magazineItems>
    <backpackItems />
  </inventory>
</DZObject>
<DZObject>
  <className>VaultStorageLocked</className>
  <characterID>XXXX</characterID>
  <objectUID>1291091599732305</objectUID>
  <objectID>1061</objectID>
  <worldSpace>[304.794,[12910.9,15997.3,0.184]]</worldSpace>
  <rawInventory>[[["RH_mk2","RH_hk417sdsp","USSR_cheytacM200","FHQ_XM2010_NV_SD_DESERT","FHQ_MSR_NV_DESERT","FHQ_RSASS_TAN","FHQ_MSR_DESERT"],[2,8,1,6,1,1,1]],[["ItemHotwireKit","10Rnd_127x99_m107","ItemTNK","ItemORP","ItemRuby","ItemObsidian","ItemComboLock","ItemAVE","USSR_5Rnd_408","fuel_pump_kit","FHQ_rem_5Rnd_300Win_XM2010_NT_SD","FHQ_rem_7Rnd_338Lapua_MSR_NT","ItemBriefcaseEmpty","ItemLRK","FHQ_rem_20Rnd_762x51_PMAG_NT","FHQ_rem_7Rnd_338Lapua_MSR_NT_SD","FHQ_rem_5Rnd_300Win_XM2010_NT","ItemSapphire","FHQ_rem_20Rnd_762x51_PMAG_NT_SD","ItemSilverBar9oz","ItemSilverBar10oz","ItemGoldBar2oz","ItemBriefcaseS80oz","30Rnd_9x19_UZI_SD","RH_10Rnd_22LR_mk2","ItemGoldBar"],[4,7,1,5,1,4,2,4,10,1,32,4,4,1,2,2,3,2,1,1,1,1,1,2,2,1]],[["DZ_LargeGunBag_EP1"],[10]]]</rawInventory>
  <inventory>
    <weaponItems />
    <magazineItems>
      <Item>
        <className>ItemRuby</className>
        <count>1</count>
      </Item>
    </magazineItems>
    <backpackItems />
  </inventory>
</DZObject>

[cen]

Great progress!

[hambeast]

OK!  small update today but a bit more progress...

 

Additions

• Option to choose between XML and CSV output.  CSV is actually tab delimited... the stupid worldspace commas were messing things up!
• Option to turn on or off Raw inventory output (useful for decreasing memory footprint and output file size)
• More Exception Handling
• Console now reports how many objects were scanned and how many objects had flagged inventory items
• Added cool new colors to cli

 

new config XML:

<?xml version="1.0" encoding="utf-8" ?>
<config>
  <DB>
    <ip>localhost</ip>
    <port>3306</port>
    <database>dayz_epoch</database>
    <username>dayz_user</username>
    <password>xxxxx</password>
  </DB>
  <Output>
    <PreserveRawInventory>False</PreserveRawInventory>
    <!-- OutputType Must be XML or CSV -->
    <OutputType>CSV</OutputType>    
  </Output>
  <RedFlagList>
    <Items>
      <!-- Classname is object you want to search for, value in brackets is qty to flag on -->
      <Item ClassName="ItemComboLock">8</Item>
      <Item ClassName="ItemBriefcase100oz">75</Item>
      <Item ClassName="ItemTopaz">1</Item>
      <Item ClassName="ItemObsidian">6</Item>
      <Item ClassName="ItemSapphire">6</Item>
      <Item ClassName="ItemAmethyst">1</Item>
      <Item ClassName="ItemEmerald">1</Item>
      <Item ClassName="ItemCitrine">1</Item>
      <Item ClassName="ItemRuby">1</Item>
      <Item ClassName="metal_floor_kit">25</Item>
      <Item ClassName="cinder_wall_kit">25</Item>
      <Item ClassName="cinder_door_kit">25</Item>
      <Item ClassName="cinder_garage_kit">25</Item>
      <Item ClassName="ChainSaw">2</Item>
      <Item ClassName="ChainSawB">2</Item>
      <Item ClassName="ChainSawG">2</Item>
      <Item ClassName="ChainSawP">2</Item>
      <Item ClassName="ChainSawR">2</Item>
    </Items>
  </RedFlagList>
</config>

 

Here's a pic of the CSV when imported into Excel. Note RawInventoryOutput is turned off here.

[hambeast]

OK guys,  I think the program is ready for a beta release.

 

I've posted the source and binaries to github.  Remember this is free and OPEN software so feel free to branch, share, modify, decompile and post on dayzepoch.com without fear of a DCMA takedown notice, whatever just make sure you keep it free and open, that's all I ask.

 

Bug reports and pull requests for fixes/performance enhancement is appreciated.  

 

I think you guys will be happy with just how fast this program is.  I tested it out, and it processed 140 objects in less than a second from an object table of 9849 records to create the CSV you see in the above screenshot.

 

Here's the link to the github: https://github.com/deadfred666/dze_inventory_parser_public

[StiflersM0M]

Mhhh for me it doesnt work, get these error:

Unhandled Exception: System.IO.FileNotFoundExpection: Could not load file or assembly "Mysql.Data, Version=6.6.4.0, Culture=neutral, PublicKeyToken=c5687fc88969c44d" or one of its depedcies.

[hambeast]

Mhhh for me it doesnt work, get these error:

Unhandled Exception: System.IO.FileNotFoundExpection: Could not load file or assembly "Mysql.Data, Version=6.6.4.0, Culture=neutral, PublicKeyToken=c5687fc88969c44d" or one of its depedcies.

 

shoot... Forgot to include that assembly in the build....

[hambeast]

FYI, I've pushed the mysql assembly up to the bin directory in git.  Can you try placing this dll in the folder you moved the exe to?  If this doesn't work I'll add the mysql.net connector as a requirement to use the program and include a link on how to install it

[cen]

This is really awesome Ham!

[hambeast]

This is really awesome Ham!

Thanks dude!  did you get it going without having to install mysql.net connector?  or did you already have it installed?

 

Also with a bit of modification you could get it to spit out every single item and its count for graphing/stats page.  how cool would that be?  Maybe in the next release :)

[cen]

I just downloaded the zip, unzipped it and pulled the \bin folder contents into another folder, set up config.xml, opened command prompt and executed.

 

Would be nice to just double click the exe or export multiple output files based on server (think like a mysqldump does with multiple databases) but it's great so far!

[hambeast]

I just downloaded the zip, unzipped it and pulled the \bin folder contents into another folder, set up config.xml, opened command prompt and executed.

 

Would be nice to just double click the exe or export multiple output files based on server (think like a mysqldump does with multiple databases) but it's great so far!

 

hmm... You mean like double click and have a GUI interface? 

 

The multi-db aspect is also possible as well.  maybe something like --configfile=blah.xml --configfile2=blah.xml

 

edit... derp can't believe I didn't think of this...

  <DB Name="Chernarus">
    <ip>localhost</ip>
    <port>3306</port>
    <database>dayzepoch</database>
    <username>dayz</username>
    <password>XXXXX</password>
  </DB>
  <DB Name="Napf">
    <ip>localhost</ip>
    <port>3306</port>
    <database>dayzepochnapf</database>
    <username>dayz</username>
    <password>XXXXX</password>
  </DB>

edit 2: yep... now I remember why I didn't do that.  We might want separate rules for our servers, think the better way would be to do this:

 

<config ServerName="Napf">
...
</config>

 

this way we can keep separate configs and it cuts down on the coding I have to do ;)

[hambeast]

looking at this charting library: http://www.codeproject.com/Articles/5431/A-flexible-charting-library-for-NET

 

How cool would it be to have a pie graph showing the item distribution by qty on your server?

 

if this tool was run once a day (with the "catch all" flag turned on (wip)) we could get a line graph of QTY per item, all sorts of cool stuff.  Would probably use RRDtool http://www.codeproject.com/Articles/28763/C-Hooks-For-RRDtool for line graphs tho to maybe create a cool admin interface.

[hambeast]

I'm lazy soo... here's a chart of the items by qty on my napf server.  waaaay too much to show but its kind of interesting...

[hambeast]

Aaaaand... got even more bored... Decided why the hell not add a graphing feature...

 

Note: this only works if you have "useRedFlagList" set to false.  this is kind of slow because I gotta loop thru the whole master list to get the counts and then graph them out.  You'll be able to pick out the top(x) items you want to graph.  Anyone with more graphing experience who wants to help me make these prettier is welcome to help out.  also going to add logging of item quantities like

 

ItemclassName Qty

DMR_Magazine 1000

Briefcase 50

etc...

etc...

 

This is generated by the program.  The top 10 items by quantity from my live napf server:

[Sandbird]

I am trying this and get:

Something is wrong with your DB connection string...
System.OverflowException: Value was either too large or too small for an Int32.
   at System.Convert.ToInt32(UInt64 value)
   at System.UInt64.System.IConvertible.ToInt32(IFormatProvider provider)
   at System.Convert.ToInt32(Object value)
   at dze_inventory_parser.Program.Main(String[] args)

all my msql details are fine though.

Tried this on my pc and the real server...same error

 

[hambeast]

I am trying this and get:

Something is wrong with your DB connection string...
System.OverflowException: Value was either too large or too small for an Int32.
   at System.Convert.ToInt32(UInt64 value)
   at System.UInt64.System.IConvertible.ToInt32(IFormatProvider provider)
   at System.Convert.ToInt32(Object value)
   at dze_inventory_parser.Program.Main(String[] args)

all my msql details are fine though.

Tried this on my pc and the real server...same error

 

hmm... well did you check your db connection string ;)

 

 

do you get any other console messages? 

 

here is the code that is throwing the error most likely:

                    DZObject record = new DZObject();
 
                    record.className = rdr["classname"].ToString();
                    record.characterID = Convert.ToInt32(rdr["characterID"]);
                    record.rawInventory = rdr["inventory"].ToString();
                    record.objectID = Convert.ToInt32(rdr["objectID"]);
                    record.objectUID = Convert.ToInt64(rdr["objectUID"]);
                    record.worldSpace = rdr["worldspace"].ToString();
 
                    masterObjectList.Add(record);

did you change your db schema at all?  I'm guessing that you are using a plot pole for life perhaps?

 

Can you run the following command in mysql and display the output to me? (run this on your epoch DB, replace dayzepoch with the name of your db. probably dayz_epoch)

DESCRIBE dayz_epoch.object_data;

[Sandbird]

 

Yeah i am using my own custom CharacterID, not plotforlife.

I rewrote all functions about object_data and use Arma2Net cause the hiveext.dll doesnt use a bigint for CharacterID.

Also i made ObjectUID cell unique, for some other reasons :P