[Video] Guide to server structure, variables, functions and Diffmerge

[raymix]

 

What is this guide for?

Actually for everyone who ever wanted to learn about file structure of Arma 2 MP environment (Epoch in this case), from beginners to a bit more advanced users alike, including script makers and editors.

 

What is this guide not for?

If you ever created your own @mod, this guide is probably not for you.

 

Damn 1 hour long video, is it worth watching it?

No, create new topic and ask someone to help you with scripts if you came here to ask that question

 

What are benefits from watching this video?

If you actually follow along, you will be able to better troubleshoot problems with server and custom scripts. You will also be introduced to file structure of Epoch server, custom variables and compiles files as well as diffmerging files for script updates or compatibility issues.

 

Are there any requirements to follow this guide?

No, grab a coffee and chill. Don't try to replicate anything, instead just try to follow along, this is a theoretical guide rather than a tutorial.

 

What's in the video?

Introduction: I am explaining reason behind this video in full detail and talking about further contents, so skip ahead to what suits you best.

 

Structure (03:00): We will be exploring file structure from first press of a button that lets client join the server. I will be explaining about different types of files used to run a server and how they chain load each other to make up a fully working server. Topics covered: Config files, mission config (ext) file, bin files, sqm and include files. I will be covering certain less essential parts like how RAM and HDD is used while loading scripts or PBO contents on demand.

 

Variables (16:51): This section is for admins and script makers who brings over custom variables.sqf over to mission file. I will be explaining how variables are being defined to avoid RPT errors as well as introducing to common mistakes of redefining already defined variables that are present with many scripts and their guides and how to reduce mission file size by only reusing what is required. For all modders out there, please keep your own vars within your scripts, lets stop this madness.

 

Functions (33:15): Precompiling functions is what actually makes our servers work. In this part of guide I will explain how they make up the server structure and how you are literary loading client and server files starting from init.sqf in your mission file. Yes you read that correct. I will be explaining common mistakes server admins commit by using custom compiles file server side and how majority of admins would blindly follow bad advices/guides and force loading their client and server files twice. We will be diving into more detail about compiles.sqf client side and server_functions.sqf files explaining how common mistakes are made when installing several addons that reuse same function and how to avoid future problems.

 

Diffmerging (47:55): Last major part of this guide. I will be explaining how to work some magic with Diffmerge (or similar tools) to keep your addons up-to-date, as well as ironing out any incompatibilities between your custom scripts. We will be looking at how to diffmerge single file, whole folders and even merge two functions together to make them compatible. Experience gained here should allow you to merge any mod (or multiple mods) out there that uses same function making them compatible again.

 

Final words (1:08:35): Conclusion and quick review of what we have learned within an hour of this guide.

 

 

 

 

 

[raymix]

Reserved

[RimBlock]

Nice info on the order of load.  Good to see you are using quality mods for the guide also  ;) .

 

Man you are passionate on the variables and custom compiles.sqf files  :D .

 

I do have a different view with bringing over the custom compiles or variables though if done correctly.  

 

The guide mentions mod makers requiring a copy of the compiles / variables etc which are then amended and an extra line is added to load these amended files as well as the original files in the PBOs.   100% agree this is a bit silly for all the reasons you mention.  I was unaware that people were suggesting this but then I use very few mods  :) .

 

ie.

changing

call compile preprocessFileLineNumbers "\z\addons\dayz_code\init\variables.sqf";

to

call compile preprocessFileLineNumbers "\z\addons\dayz_code\init\variables.sqf";
call compile preprocessFileLineNumbers "customfiles\variables.sqf";

If any files are brought over from the base Epoch PBOs then the lines originally referencing those files in the PBOs should be changed to repoint them to the custom versions.  This avoids duplication of code being run at the sacrifice of a larger mission file size.

 

ie.

call compile preprocessFileLineNumbers "\z\addons\dayz_code\init\variables.sqf";

to

call compile preprocessFileLineNumbers "customfiles\variables.sqf";

I like pulling the files over as I know those have been amended, exactly what is being run and that each function / code block is only being defined once.  I also find it easier to explain to new server owners how to do this especially as there are numerous guides already out there.

 

The idea of creating a custom compiles / variables or anything else with only the changes and additions has the advantage of a smaller mission size but anything already defined in the original PBO file that is changed is being defined twice (with the newer version taking precedence).  Not a big deal but not overly efficient.

 

It is just down to the choice between defining the code once but having a larger mission file against a smaller mission file but possibly defining the same functions more than once on server / client startup.

 

All told though, either of the last two ways work and are reasonably efficient. 

 

TBH for common mod files like variables.txt and compiles.sqf I tend to put them in a custom/init folder to make them seem more mod neutral.  A lot of confusion with people wanting to put our mods on the same server was that they would create a custom compiles and link it in to the init.sqf and then chain the SBP compiles off the end (overriding the APfL overrides).  If the files from the init pbo folder were placed in a mod neutral init custom folder it may have been clearer although there is no standardisation between mods  ;) .

 

Just one other thing on the variables and init.sqf and variables.  I usually explain the variable.txt as defining the base variable values and variables in init.sqf (or elsewhere) as overriding that initial definition, usually to turn features on or off.

 

Anyway, great video.  Thanks for making it and it is always good to learn some new stuff.

[calamity]

Mr.Raymix, what you've just said is one of the most insanely idiotic things I have ever heard. At no point in your rambling, incoherent response were you even close to anything that could be considered a rational thought. Everyone in this forum is now dumber for having listened to it. I award you no points, and may God have mercy on your soul.

 

Just kidding. Great video and as always great helpful information....

 

 

howz about a video about github ?

what it's for, who its for, how its used and who should use it..

wtf is a Repository, a pull,  gist. 

a nice tutorial would be very expert..

 

 

simple things are sometimes not soo simple. (sometimes too simple)

[raymix]

Looking nice at first sight, but I hope HD will come soon, it's pretty important for me since my eyes aren't that good in the morning :D

 

Thanks bud, yeah it's 1440p 6GB file, usually takes a while for youtube to process it, I had to push it asap because it was time to head to work, lol. Review would be nice as well as point out any mistakes I might have made. Altho rewatching it again I wish I could just go in and edit some of stuff

 

 

call compile preprocessFileLineNumbers "\z\addons\dayz_code\init\variables.sqf";

to

call compile preprocessFileLineNumbers "customfiles\variables.sqf";

I like pulling the files over as I know those have been amended, exactly what is being run and that each function / code block is only being defined once. 

Good point, actually never occurred me to also mention this method, but thanks for mentioning it here. Obviously there's no one single best way of doing things, but hey.

Quite honestly I just remembered that I am using this method on my custom mod for interiors, it is using seperate PBO to make it a bit easier to oversee the project, in fact it contains customized variables file being called from init.sqf

\rmx_interiors\variables.sqf

However I just realized that i've been doing it wrong. Since I've to update variables.sqf file every time I need it for epoch 1052, I edit both files... while I could just do what I told everybody else to do in the video, haha... oh the irony.

It's just 2 arrays I am manipulating really - allowed objects and ruins. Genius :D

 

 

Mr.Raymix, what you've just said is one of the most insanely idiotic things I have ever heard. At no point in your rambling, incoherent response were you even close to anything that could be considered a rational thought. Everyone in this forum is now dumber for having listened to it. I award you no points, and may God have mercy on your soul.

 

howz about a video about github ?

what it's for, who its for, how its used and who should use it..

wtf is a Repository, a pull,  gist. 

a nice tutorial would be very expert..

 

 

Dude... mini heart attack right there, lol. Thanks

 

I am pretty new to github and generally only use Git Shell to accomplish what I need. The GUI application seem to be quite limited and needs a lot of work to be useful (probably I just don't understand it, funny enough). Thanks for the idea tho, when I get more experience, might as well throw a video up, since there are literary none out there on the subject. Could do a simple one, but afraid I'll get my arse handed over to me with angry comments, lol so I'll take my time with that.

[RimBlock]

Thanks bud, yeah it's 1440p 6GB file, usually takes a while for youtube to process it, I had to push it asap because it was time to head to work, lol

 

Haha, that's why my mobile overruns its data usage  ;) .

 

Git is like an octopus... seems pretty simple but the more you start looking at it the more you notice all the different arms and suckers.  I am using source tree at the moment which seams reasonible.  I stopped using GitHub desktop client a while ago.

 

I am personally quite interested in the player login chain and FSM files, how they are processed and where they fit in.  I have a hankering to amend the zombies fsm as that zig zag running is just annoying :) .

[raymix]

I am personally quite interested in the player login chain and FSM files, how they are processed and where they fit in.  I have a hankering to amend the zombies fsm as that zig zag running is just annoying :) .

That's the subject I'd love to learn, too. But anything to do with AI is... complicated to explain, since it is very variable. To be honest FSM files were the initial idea to use for SBP instead of SQF at the very beginning. After wasting 2 days studying them I gave up.

On the subject of zombies, just so you know, Zeds are not normal (west, east, civilian) AI, they are wildlife, same as dogs and cows. So they do not belong to groups and creating an AI for them could be different from what you are probably used to, if at all. Not sure tho, but subject is very interesting indeed. Afaik Dean reused civilian code, so zig zagging actually comes from there... yeah zeds are avoiding shots as if they had brains.

[Powell]

Great tutorial I watched the whole video and learned alot of new things. Thanks again for the help.

[paddy1223]

DiffMerge! The info I needed after your reply to me last week. I installed it and got lost :) Thx for video m8 diffmerge makes sense now :)

[aVa Savagerx]

First off, great video! Your videos may be 1+ hours long but they are always worth it.

 

I have a question regarding the first part of your video, in particular the way in which the server initialises and uses the MPmission. The format etc you have used is default where init.sqf initialises the custom compiles.sqf and variables.sqf but I have noticed that some groups such as Vilayer and other hosts (also CCG clan) have exceptionally different formatting of their missions. Why is this? Is there any benefit?

 

I am assuming you can host some of your mission file mods in your server files to prevent clients from being able to access them afterwards (ie theft/copying) but it comes at the cost of server performance because the files are not stored locally on the client pc?

 

Any insight will be appreciated.

[raymix]

First off, great video! Your videos may be 1+ hours long but they are always worth it.

 

I have a question regarding the first part of your video, in particular the way in which the server initialises and uses the MPmission. The format etc you have used is default where init.sqf initialises the custom compiles.sqf and variables.sqf but I have noticed that some groups such as Vilayer and other hosts (also CCG clan) have exceptionally different formatting of their missions. Why is this? Is there any benefit?

 

I am assuming you can host some of your mission file mods in your server files to prevent clients from being able to access them afterwards (ie theft/copying) but it comes at the cost of server performance because the files are not stored locally on the client pc?

 

Any insight will be appreciated.

Thank you, great question, btw.

There are several reasons they'd go this way, some are kinky, some legit.

 

Vilayer's init can be overriden by default epoch init.sqf and it will work just fine, it's not about the host, it is about the engine, and Arma is one of most versatile engines to mod out there. Vilayer still provides full access to both MPmissions and dayz_server PBOs to my knowledge, unless something got changed. The reason they'd go custom route could be automated installation tools for scripts, which helps people new to hosting server and not particularly willing to learn it themselves. While the file is not changed, they could guarantee full support for mod installations and taking care of errors.

 

 

Next part is theoretical, I haven't had successful tests on this:

Server hosts can create separate PBOs server side, which could explain changes inside init.sqf with different paths. One could be hidden or locked away from player and the second one accessible for editing. This way server hosts could re-sell stuff like Antihack, where config file is accessible and actual AH is not.

I've also noticed -noFilePatching in launch parameters, it's a param that forces only PBOs to be loaded, which means extracted folder structure could be loaded directly, too. I've tried it tho without success (would be nice while developing). This could be used for some FTP tricks, I guess.

 

Generally it does not matter how your PBOs are named or how many of them you have, as stated in video, only basic information is being read at first (like prefixes and configs), actual data is not touched unless initialized either from init.sqf or trough chain of scripts that starts from init.sqf. You should be able to have \z\dayz_server\ and \z\dayz_server_superSecret\ it should work just fine as long as prefixes are correct.

 

---

 

A lot of mods can be pushed server side indeed, but not all. This is not only problem with server performance, it includes unnecessary traffic generated. Albeit pushing scripts server side is not a simple task and to my knowledge player has to do something before any action takes place, like a keypress, trigger or public variable. A data has to travel to server so it knows who to apply the action to and when. 

Let's say you want to create a deploy script that you think is quite awesome, but don't want people peeking into your mission file to steal it. You could set up a public variable event handler to do it for you, however there would still be a trace in MPmissions that would initiate this public var (or public variable server). I've noticed Infistar forces clients to load chunk of data passed from server side somehow, then it detects stuff like key presses. I haven't cracked how to do it yet, but if anyone could share the info, that'd be great. 

So this neat deploy script - all it does is spawns a vehicle. Server is very well capable of doing that for you, too. In fact that's exactly what it does when server is first started.

Anyway, unless this script of yours is executed often, there is not much of performance or network tax to be honest. However pushing script like SBP server side would be a killer, in fact it would actually completely kill the performance if just 2-3 people would start building at a same time, since points would be generated and visible for everybody, no exceptions, what SBP does is spawns them locally, so no network traffic is done and only said player gets the performance hit, even then some admins reports dramatic performance drops and adjustments like detection/point generation range are required.

 

Please feel free to correct me where you feel I am wrong as most of said above is only from own observation while diving trough countless PBOs, forum topics and wiki. 

Thanks

[Guest]

I have a question regarding the first part of your video, in particular the way in which the server initialises and uses the MPmission. The format etc you have used is default where init.sqf initialises the custom compiles.sqf and variables.sqf but I have noticed that some groups such as Vilayer and other hosts (also CCG clan) have exceptionally different formatting of their missions. Why is this? Is there any benefit?

 

I can't speak to the way Vilayer does it as I've never looked at managed hosting, but from what I can remember of the CCG mission files, they're structured VERY cleverly to maximise modularity and keep code organised. I don't think there's any specific performance benefit to the way maca has things organised in them (I could be wrong, mind you), but the last time I took a peek it looked like he'd coded things in such a way that the same mission.pbo could be used across different servers running different maps, all while maintaining a common codebase.

 

It's a lot easier to update the scripts in one single PBO file for all your servers than have to diffmerge a dozen different ones.

[aVa Savagerx]

Very interesting and thought provoking. A bit beyond my level of technical understanding but it makes sense. I was working along the lines of increasing server performance but what you say about offloading things like the object_helpers to clients makes sense (I believe thats how dayz handles zombies too). I know of headless client solutions and others too. 

 

I suppose in the battle to keep mission file sizes down it would be nice to be able to leave a lot of the scripts serverside (like your bike example) where they can live in the server files and have little to no impact on performance and keep others such as SBP locally. Ultimately though I find that your discussion regarding taking only the CUSTOM parts of variables.sqf and compiles.sqf already makes a HUGE difference. Also diffmerging mods that use common files such as playerbuild.sqf for instance. 

 

Some other mods also seem poorly constructed and use several files to do what could be done in one file with an array (I reckon it is because not everyone is a coding wizard :P ) such as the older deployables scripts where you needed a file for every deployable you wished to create etc. Through optimisations of these mods and different version releases for compatibility (Such as Vector building releasing different compatible versions for different mod combinations). I have found that the community is starting to reap some benefit.

 

The other killer on mission file sizes is custom loading screens, custom audio (intro music ) and also custom textures for clothing or vehicles. A really simple thing but it often goes unnoticed! 

[Turtle II]

This is great...I have to say I really appreciate you making this video.

[Bob_the_K]

Excellent information... but I still have a question about calling sequences and what overrides what.

 

If I have call compile preprocessFileLineNumbers "\z\addons\dayz_code\init\compiles.sqf"; in my init.sqf file and then later have call compile preprocessFileLineNumbers "custom\compiles.sqf"; in the same file, will the one in the custom folder append or replace what was called in the PBO file?  Should I extract the original compiles.sqf from the PBO, do all my changes there, remove the line that calls the one in \z\addons..., only calling the one in my custom\ folder?

 

And the same question for fn_SelfActions.sqf and other files that need modification from what is in the PBOs?

 

Or am I better off in all cases, making the changes and then repacking them back into the original PBO, replacing the original files altogether?

 

Thanks.

Bob

[raymix]

Excellent information... but I still have a question about calling sequences and what overrides what.

 

If I have call compile preprocessFileLineNumbers "\z\addons\dayz_code\init\compiles.sqf"; in my init.sqf file and then later have call compile preprocessFileLineNumbers "custom\compiles.sqf"; in the same file, will the one in the custom folder append or replace what was called in the PBO file?  Should I extract the original compiles.sqf from the PBO, do all my changes there, remove the line that calls the one in \z\addons..., only calling the one in my custom\ folder?

 

And the same question for fn_SelfActions.sqf and other files that need modification from what is in the PBOs?

 

Or am I better off in all cases, making the changes and then repacking them back into the original PBO, replacing the original files altogether?

 

Thanks.

Bob

 

Init.sqf is the very first script file called when loading MP game. If you call compile files from client PBO, they will be loaded first. Then if you call compile same file on second line with same content but uploaded to mission file, it will load that file and overwrite any existing functions and variables.

This means if you just bring over full file, you are essentially loading the whole server structure twice.

To avoid that there are 2 methods you can go with (for epoch only):

1) comment out first line that points to client files and upload full compiles to mission and edit from there. This increases download size a tad bit and is most commonly used method

2) use both lines (this method is used in video), but do not bring over full compiles file, instead you create blank compiles file mission side and only overwrite functions you are interested in. This is more advanced method, but not necesarry the best, it saves on mission size, but still loads old functions from client just to overwrite them later. It is not as bad as it sounds, because for most of the part we are talking about reading data (overwriting funcitons) from HDD into RAM well below 50KB in most cases, lol

 

Selfactions is a bit different monster, it is best to just bring it over to mission side, because at current state it is kind of horribly created. I tried to split it up into multiple functions and add extra checks, because by default it excecutes 1000 lines of code every 0.5 seconds, which is God awful idea... I might eventually get some time to finish the modular script for this, should give us all extra speed when we want to access actions in-game... problem is, this will confuse the living **** out of people new to scripts and following tuts step by step

[Bob_the_K]

So I've taken over server development from several people before me.  All the files are a jumble.  I think I'd like to get back to the original compiles.sqf and fn_SelfActions.sqf files and then just make any updates in the ones in my "custom" folder.  I think it would be easier to manage when Epoch updates come out.  I'm also trying to address all the warnings and errors in the client and server .rpt files (of which there are many).  So would that be the best compromise if I'm trying to maximize performance and not have the system run things twice as well as keeping my missions files as compact as possible?  Right now it's approaching 2500kb in size and I'd like to trim it if possible.

 

So I can do the compares and such to do that.  But for my "custom\compiles.sqf", does it follow the same general structure as the original?  Do I start my file with "if (!isDedicated) then {" just like the original (of course closing the bracket, too)?   Given that my file is part of the missions folder/files, don't I still need to do the if(!isDedicated...  check in my configs.sqf so it'll run client side and not server side?

 

I like your thoughts about a modular approach to fm_SelfActions.sqf, as I do with your other video on a modular player build.  Given the problems so many people have with the fn_SelfActions.sqf script, I think it would be a worthwhile undertaking.

 

Bob

[raymix]

I'd suggest that you create a local server and google all the scripts your current server runs and add/update them from scratch. If a script asks you to modify epoch files, then make sure you unpack a latest one from client files and do not reuse scripts from old server. If your server runs scripts as old as 1.0.4.x then that explains tons of errors you get, since 1.63 beta loves to report undefined errors... 1.0.5 epoch and scripts that were updated with it fixes all these errors, your RPT should be clean. Some of the custom undefined global variable errors you get can be fixed by simply defining them yourself, for example: I am running an outdated script that uses action menu, it spams my local RPT with undefined errors, what I do to keep it clean is simply define this variable inside my init.sqf right before any call compiles for loading screen are called, instead of hiding these guys inside custom variables.sqf

To trim down on mission file size, redoing all scripts will probably make you drop some old or no longer used scripts.

 

Another suggestion is using photoshop or irfanview to reduce quality of your loadscreen, there is absolutely no need to use maximum quality. In scale of 1-10 I am using quality around 5 or 6.... 5 would create minor pixelation, but not that noticable, so 6 would remove pixelation, but at a cost of extra 50KB. Overall bringing it down to 5/6 lets me chop image size in half or even more. Game does not use full quality of your image anyway and different resolutions sees this image differently

 

When people brings their custom compiles/variables/etc files over, they are modifying mission file to reflect client files as much as possible.

 

In regards to !isDedicated statement - I've noticed some people are not using it in their compiles, allowing server to load functions it will never use.... bad practice in general, that's all. Long story short - it is bad idea to load extra unused functions server side as this can be used for exploits or could potentially spawn incompatibility issues.

[AWulle]

Thank you for the time and effort of making and posting this. :)