Game data

[nemequ]

Games are a major user of compression software, so I'd like to make sure they are represented. Unfortunately I know basically nothing about game development, so I need some help with this one.

What would be very useful right now is a summary of all the data which a game needs to compress. Lots of times this seems to all be placed in an archive file (like a PK3 file, or a Unity asset bundle). Since files need to be compressed individually (to facilitate random access) we need to know about the contents of those archives, not the archives themselves. Summaries are fine; a table with the extension/content-type, count, and average size (see the second comment below for an example).

If you're in the game development community, please speak up. Even if you can't release any data, if you can tell us about the data you are interested in compressing maybe we can find some from another source (like an open-source game). If I can get a good idea of what types and sizes are common I can try to find representative redistributable data.

I'm starting threads on some engine-specific web sites for this issues:

• Unity
• CryEngine
• Unreal

[nemequ]

I had a very helpful conversation with the Xonotic people on IRC. Basically, they use PK3 files for their data, which is effectively a renamed zip. The script they use to convert from the raw input files used for development to the versions which are actually included in the PK3 file is at https://gitlab.com/xonotic/xonotic/blob/master/misc/tools/cached-converter.sh

They use

• ogg vorbis for audio
• jpeg or webp for images
• S2TC and DXT* for textures.
• BSP for map data

Random access is important to them, so we would want to include individual pieces of data not a single tar(unzip(*.pk3)).

[nemequ]

Contents of the pk3 files from unvanquished, in CSV format:

Extension,Total Size,Count,Average Size
crn,261724100,2491,105067
tga,224801388,167,1346116
bsp,164299077,11,14936279
map,95501345,10,9550134
jpg,58742932,800,73428
md5anim,53287824,225,236834
wav,50479889,365,138301
webp,38912252,131,297040
navMesh,37079543,120,308996
md3,36608771,151,242442
png,31960951,102,313342
nexe,24552252,7,3507464
iqm,22587200,21,1075580
ttf,18465833,16,1154114
ogg,15488080,92,168348
md5mesh,14748687,26,567257
ase,4573191,24,190549
qvm,2235564,3,745188
po,1931794,16,120737
obj,1656219,5,331243
opus,1148534,20,57426
shader,683191,99,6900
glsl,295189,75,3935
menu,223275,51,4377
rml,183974,85,2164
pot,139430,2,69715
cfg,111205,242,459
particle,87436,30,2914
dat,82192,4,20548
txt,80050,39,2052
def,61633,1,61633
h,46567,10,4656
rcss,10617,8,1327
skin,9454,45,210
voice,9019,1,9019
CC0,7048,1,7048
svg,5812,2,2906
trail,4806,12,400
sloth,2045,16,127
lua,1781,1,1781
minimap,1687,10,168
bt,1680,3,560
README,1534,1,1534
arena,789,11,71
CREDITS,369,1,369
directory,232,1,232
DEPS,20,1,20

Major extensions:

• crn: essentially dxt textures
• tga
• bsp: compiled map that is used by the engine, it contains geometry and collision data for the map
• map: source file for a map, a .bsp is compiled from this, it is a text file
• jpg
• md5anim: text file describing an animation for the md5 model format
• wav
• webp
• navMesh: navigation data for the AI

And some info on some of the less common data: ".rml and .rcss are html/css files for librocket (an html engine to display GUI), .shader, .trail, .particle are text files describing visual effects, .minimap is a picture of the map (for navigation purpose), I guess .menu is the old q3 menu stuff which is now unused (but still there since we still ship old pk3), .qvm is the old q3 qvm binary, now unused since we use nexe (but still there for the same reason), .arena is a metadata text file for maps (containing the displayable name etc.)"