Chem O’DunReferences and Guides

Multi-version extensions

This page is also published as Multi-version extensions on the Egosoft wiki.

A game update can move exactly the piece an extension depends on: a vanilla Lua file it substitutes, a signature it calls, a menu it patches. The usual answers are both unpleasant - freeze the extension on the old version, or publish a second package and ask everyone to pick the right one.

There is a third. One package can carry different content for different game versions, in the same folder, selected by the engine at load time. The mechanism is a catalog whose name carries a version number, and it has been in the engine since X Rebirth. It is described for X Rebirth in the Steam guide Steam Workshop for X Rebirth and X4, under the -buildvcat switch of the Workshop tool, and it works in X4 - though not on quite the terms that guide implies.

The naming rule and the four rules below were established by running a probe extension on 8.00 and 9.00 and reading which catalog actually answered for each file. That package is available to download so the results can be reproduced. Diff catalogs, added in 9.00, are documented from Egosoft's own description of them rather than measured.

Contents

The naming rule

An extension folder normally holds numbered catalogs - ext_01.cat / ext_01.dat for its own files, subst_01.cat / subst_01.dat for files that replace vanilla ones. A version catalog is the same thing with a version in place of the number:

ext_v800.cat    ext_v800.dat        content for game version 8.00
ext_v900.cat    ext_v900.dat        content for game version 9.00
subst_v800.cat  subst_v800.dat      vanilla replacements for 8.00
subst_v900.cat  subst_v900.dat      vanilla replacements for 9.00

The number is the game version written as three digits, without the dot: major, then minor as two digits. 7.60 is 760, 8.00 is 800, 9.00 is 900. The build number the game reports in brackets after the version plays no part.

X4 9.00 adds a second name shape, ext_NN_diff_v###.cat, which carries a version the same way but is selected on different terms. It has a section of its own: Diff catalogs. Everything from here to the end of the four rules is about ext_v### and subst_v###.

Paths inside a version catalog are the same virtual paths as anywhere else - relative to the extension folder for ext_, relative to the game root for subst_. A version catalog is not a sub-folder at runtime; it is an alternative source for the very same paths.

↑ Contents

The four rules

1. A version catalog loads only on an exact version match

ext_v800 loads on 8.00 and on nothing else. On 9.00 it is not merely outranked, it is absent - files that exist only there cannot be found at all.

This is the rule that most needs stating, because it is not what the X Rebirth guide leads one to expect. A version catalog is not "this version and older", and not "this version and newer". Running on 8.00 with all three of ext_v760, ext_v800 and ext_v900 present, only ext_v800 answered; both of the others were as good as not shipped. On 9.00, only ext_v900.

The practical consequence is the whole shape of the technique: a version catalog is needed for every game version that requires different content, and any version without one falls back to the numbered catalogs alone. From 9.00 there is a way out of that, and it is the only exception to this rule: see Diff catalogs.

2. A version catalog is applied after every numbered catalog of its kind

Where the same path exists in both, the version catalog wins. ext_v800 overrides ext_01, and it overrides ext_02 as well, so it does not matter how many numbered catalogs a package already has - the version catalog sits above all of them.

subst_vNNN behaves identically over subst_01, subst_02 and the rest.

3. The two kinds rank only within themselves

ext_ and subst_ are separate stacks and neither reaches into the other. A subst_vNNN catalog outranks the numbered subst_ catalogs and nothing else; the ordering between ext_ and subst_ as a whole is what it always was.

4. A version catalog does not need a numbered catalog to exist

With every ext_01 / ext_02 removed and only ext_v800 shipped, the extension loaded and worked on 8.00. The version catalog is a complete, self-sufficient source, not a patch that requires a base to apply to.

That makes both models available:

↑ Contents

Diff catalogs, from 9.00

X4 9.00 added a second version-named catalog, and it is the one that does not obey rule 1:

ext_01_diff_v900.cat   ext_01_diff_v900.dat

The name is the numbered catalog it belongs to, then _diff_v, then the version in the same three-digit form. It is a layer over that numbered catalog: ext_01_diff_v900 is applied after ext_01, so wherever the two hold the same path, the diff catalog is the one that answers.

Two things separate it from ext_v###:

No version before 9.00 has this catalog type. A package that also supports 8.00 keeps ext_v800 or plain numbered catalogs for that version and uses diff catalogs only from 9.00 up.

↑ Contents

Building one

The source tree needs one folder per catalog, each rooted where that catalog's paths are relative to:

my_extension/
    content.xml                     never packed - see below
    src/
        base/                       -> ext_01
            ui.xml
            ui/my_extension.lua
        v800/                       -> ext_v800
            ui/my_extension.lua         the 8.00 variant of that one file
        v900/                       -> ext_v900
            ui/my_extension.lua         the 9.00 variant
        subst_v800/                 -> subst_v800
            ui/addons/ego_detailmonitor/menu_map.xpl
        subst_v900/                 -> subst_v900
            ui/addons/ego_detailmonitor/menu_map.xpl

Each folder is packed with the X Catalog Tool into the catalog it corresponds to:

XRCatTool.exe -dump -in "src\base"       -out "dist\my_extension\ext_01.cat"
XRCatTool.exe -dump -in "src\v800"       -out "dist\my_extension\ext_v800.cat"
XRCatTool.exe -dump -in "src\v900"       -out "dist\my_extension\ext_v900.cat"
XRCatTool.exe -dump -in "src\subst_v800" -out "dist\my_extension\subst_v800.cat"
XRCatTool.exe -dump -in "src\subst_v900" -out "dist\my_extension\subst_v900.cat"
copy content.xml "dist\my_extension\"

The Workshop tool's -buildvcat does the equivalent from v800-style sub-folders of a single tree, for anyone publishing through it. Packing each folder by hand is the same result and needs no convention about where the version folders sit.

A diff catalog is built the same way, from a folder holding only what changes from its version onward:

XRCatTool.exe -dump -in "src\diff_v900" -out "dist\my_extension\ext_01_diff_v900.cat"

That folder does not have to be maintained by hand. -diff takes a base tree or catalog and writes only what differs from it, deletions included, which is exactly the content a diff catalog wants: see Diffs, and version catalogs.

content.xml is never packed. The engine reads it before it mounts any catalog, so it has to stay loose in the extension folder. Its <dependency version="..."> is the lowest game version the package supports - the one below which the extension should not load at all, not the version any particular catalog serves:

<dependency version="800"></dependency>

↑ Contents

What belongs in a version catalog

Only the files that actually differ. Every file that is the same across versions belongs in the numbered catalogs, where it is packed once and maintained once.

The cost of forgetting this is not disk space, it is drift: a file duplicated into three version catalogs is a file that will be fixed in one of them and left stale in the other two.

For substitutions specifically, the question to answer before writing anything is whether the vanilla file changed at all between the versions in question. If ui/addons/ego_movie/movie.xpl is byte-identical in 8.00 and 9.00, then one subst_01 carrying the modified copy covers both, and no version catalog is needed for it. Comparing the extracted vanilla trees of the two versions is what settles it, and it is worth doing per file rather than per version - a game update rarely touches everything.

↑ Contents

Gotchas

A registered file that lives only in a non-matching version catalog is a hard error. Because a non-matching catalog is absent rather than outranked, a ui.xml that registers ui/thing_900.lua will, on 8.00, produce:

File I/O: Could not find file '.\extensions\my_extension\ui\thing_900.lua'
Addon::LoadLuaFile() - Failed to open specified filename: ... referenced in addon: 'my_extension'

The addon still loads and its other files still run, so this is survivable, but it fills the log with errors that look like the extension is broken. Keep ui.xml registering only paths that exist on every supported version, and let the version catalogs override those paths rather than add new ones.

ui.xml itself is a normal file and can be overridden too. If two versions genuinely need different file lists, ship a different ui.xml in each version catalog rather than one union list that is wrong everywhere.

Nothing warns about a version that has no catalog. Ship ext_v800 and ext_v900, and a player on 8.10 silently gets the numbered catalogs only. If that combination is not viable, the <dependency version> floor and a runtime version check are the only things standing between the player and a confusing failure. A diff catalog has no such gap above its own version, which is the main reason to prefer one from 9.00 onward.

Vanilla UI Lua is stored as .xpl. A subst_ catalog replacing one must use that path and extension - ui/addons/ego_movie/movie.xpl, not .lua. Plain Lua source is accepted as the content; it does not have to be compiled the way the shipped file is.

↑ Contents

The test package

Download: version-content-test.zip (11 KB) - the probe the rules above were read off, cut down to 8.00 and 9.00.

Unpack it into the game's extensions/ folder and start the game with -debug all -logfile debuglog.txt, then search the log for version_content_test. It ships eight catalogs and writes one line per question:

Deleting catalogs from the folder and restarting is what proves rule 4: with ext_01 and ext_02 removed the version catalog still answers for everything, and with subst_v800 and subst_v900 removed the numbered substitution catalogs take back over.

↑ Contents