Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proposal for a more userfriendly homepath/basepath #78

Open
dGr8LookinSparky opened this issue Apr 27, 2018 · 88 comments
Open

Proposal for a more userfriendly homepath/basepath #78

dGr8LookinSparky opened this issue Apr 27, 2018 · 88 comments

Comments

@dGr8LookinSparky
Copy link

This is a rough proposal for @Chomenor 's new file system, in regards to reorganizing the directory structure inside the homepath/basepath in a way that would hopefully be a lot more user friendly than the old file system, it involves moving the fs_game directories into a sub directory.

First, as a note, the original intended roles of the basepath and the homepath, the basepath was meant to be shared by by all user accounts on a common system, while a unique homepath was meant to be owned by each user account.

The proposed “root contents” of the homepath/basepath would be as follows:

  • mods: this would serve as the home of all fs_game directories, can contain fs_game specific downloads from game servers.

  • maps: only non-default maps would be downloaded to here, and would work fs_game independent.

  • dependencies: not sure if this directory is necessary to serve its intended purpose, but basically it would be for things like texturepacks, and other kinds of pk3 files that would only be loaded if they contain something that is actually used currently by the client. would work fs_game independent.

  • overrides: would contain things like custom HUDs, custom skins, custom texture packs, etc. Would be fs_game independent at the root of this folder, but maybe could allow for fs_game specific overrides with sub directories named for the corresponding fs_games.

  • defaults: would only be used in the basepath, cannot be overridden in the homepath, and would contain all default assets of 1.1, 1.2, and 1.3. Nothing can be downloaded to this folder, and this folder doesn’t represent any fs_game. We could maybe specify by protocol which default assets would be utilized. Onlye the pk3 files in this directory are loaded when loading the main menu or when loading the download menu. Switching mods from the main menu would only load non-default pk3 files at the time a server is created from the main menu, or when loading a map from the client with /map or with /devmap, otherwise while in the main menu, only the files in the default directory are loaded.

  • demos: would contain all demos

  • screenshots: would contain all screenshots.

  • condumps: would contain all console confdumps

  • configs: would contain the universally used config files in its root, we could consider allowing for fs_game specific config overrides by placing sub directories with names corresponding to fs_game names.

  • qkey: would still be in the root of the homepath.

@jkent
Copy link
Collaborator

jkent commented Apr 27, 2018

Keep in mind that demos are (often) mod specific, otherwise this is good!

@dGr8LookinSparky
Copy link
Author

dGr8LookinSparky commented Apr 27, 2018

@jkent , @Chomenor appears to have found a way for any demo to be played outside of any specific fs_game. Basically since the files required by a game server are communicated via the initial biginfo configstring, the demo thus records which pk3 files are needed.

@jkent
Copy link
Collaborator

jkent commented Apr 27, 2018 via email

@Chomenor
Copy link

This design should be technically possible to implement with my filesystem, but it doesn't seem like it will work with existing installations or coexist with other versions of Tremulous very well.

I would propose an alternative design that keeps the current structure, where mods are in the root of homepath, and use hash-based precedence to keep the core "default" paks correctly in line. You would have some base directories such as the following:

  • base: for any pk3s compatible with 1.1 and up
  • gpp: for any pk3s compatible with gpp and up
  • trem13: for any pk3s compatible with 1.3 and up (just making up the name, could be something else)

The engine would contain a list of pk3 hashes that would be treated as the "defaults" for the main menu or any given protocol, and those default pk3s would override other pk3s in the three base directories. This is essentially the same as what my filesystem does in ioquake3, but with automatic version selection support added to handle the difference Tremulous protocols.

The advantage of this approach is that it can coexist with other Tremulous versions and read content like maps and map dependencies from any of the base directories, but it is also stable because the default paks have precedence and are effectively the only thing loaded in the absence of a pure server or active mod.

@dGr8LookinSparky
Copy link
Author

dGr8LookinSparky commented Apr 28, 2018

I was thinking that a way that something like my proposed directory layout might be made compatible for interactions with the older file system is when paths are communicated, there could be a mapping between how the directories physically are locally for the new file system, to how they would be on the older file system. So the actual FS related messages would generally be indistinguishable in terms of whether it is coming from the new file system or the older one. While, the new client and server would sort the saving of files based on my proposed layout, and would interpret the desired behavior based on the physical locations in my proposed layout.

So communicationwise files meant to go in a given fs_game, would have their path communicated as if the fs_game is in the homepath root, but clients with the new filesystem would instead search/save/download to the fs_game directory inside the mods directory. As far as communications are concerned, files that are meant to be fs_game independent in maps, defaults, dependencies, and overrides(this folder would mainly be for semipure and unpure servers) might be treated like they are all in the base fs_game (well in the case of defaults, they might be treated like they are in multiple fs_games depending on the protocol).

Additionally, if some of the more "advanced" features might require some "non-compatible" communications, that could be done via protocol 71, so those would be features that only the new clients connected to the new servers could take advantage of (or more preferably, instead of having such communications based on the protocol, it would be desirable for there to be a way the client version to be communicated to the server and the server version to be communicated to the clients, and if such a communication is absent in either direction, it can be assumed that the old file system is used. Th reason that is preferable is that currently a new multiprotocol client can connect to any of the 3 current protocols of a multiprotocol server).

@Chomenor
Copy link

I assume it is possible to make your design work when it comes to connecting to servers of different versions, although it may be complicated. My concern is regarding an executable using the new file layout coexisting in an install directory that is currently or formerly occupied by another Tremulous version using the old file layout. If the new filesystem requires a separate install or discarding/relocating all previous paks to work, that could be a significant disadvantage from the users perspective.

What are your thoughts on the hash-based approach I described in my last comment? Do you see any issues or limitations with that type of approach? It appeals to me because it is relatively easy to implement, compatible with existing installs, and stable because pk3 hashes tend to be more robust against configuration problems than anything based on pk3 name and location alone.

@dGr8LookinSparky
Copy link
Author

I don't think my proposed design would have an issue in regards to co-existing with initial installations, with the exception of the qkey, they should for the most part ignore eachother. The new file system could ignore all other directories in the homepath/basepath root, I am unaware of any pre-existing fs_games that used any of the proposed subdirectory names. It could sorta become an issue if a/some future server(s) set their fs_game to one or more of the sub directories, and an old client connects to those servers, but really I believe that when the new file system is ironed, there would no longert be a need to use an older client when you can use the latest client (other than for testing). Additionally we could have the new file system ignore a possible instance of a "defaults" subdirectory in the homepath, and only recognizing that subdirectory in the basepath, so there wouldn't be a risk of having interference from an old client connecting to a server with a fs_game set to "defaults".

I would also like to add that at least for the current Tremulous player-base, it is not essential to reuse existing pk3 files in the current layout. That is, it wouldn't be a major concern at this point in time for them to redownload stuff that they may have already had in their old layout, or for the players who know how to get to and navigate the old layout, and had need, to move stuff from their old layout to the new. If anything, the old file system has a major inconvenience where players are forced to have to move/delete pk3 files when they get too many and all are loaded, or suffer from running out of memory on many servers. But whatever we set now will become the standard, and moving things around may become more of an inconvenience later after the game is hopefully more revived.

The hash based approach could work, and maybe even regardless that should be the way it should work (more or less) behind the scenes. Only thing is, it could be a bit inconvenient to maintain/update by either having to change a hardcoded hash list, or manually change cvars, or having to manually change some other means for a hash list, as compared to just dropping a file in a certain location that automatically determines the desired behavior (perhaps the top sub-directories inside "defaults", "dependencies", and "overrides" could be the versions, currently maps would work for all versions, which is one of the reasons I think maps should have its own directory).

What I would ideally like to see is an end-user interaction with the files that is intuitive, well organized and clean, and is as easy and fast as possible to navigate and use the file layout. Even putting aside its inefficiencies, bugs, and inconveniences for a moment, The learning curve is crazy for the old file system (although it may not help that is not very well documented). Also it doesn't take too long for the homepath to get pretty messy. It took me years to figure out what was going on with that file system, and even now I'm still learning more of its subtleties. It might not be too big of a deal to those of us who have played this game and/or other idtech3 based games for years, But one of our main aims is to attract and retain a lot of new players who may not have ever previously heard of any of these games (as well as make things generally more convenient for for existing and returning players as well).

@dGr8LookinSparky
Copy link
Author

Perhaps "dependencies" and "overrides" don't have to be separate subdirectories and could be merged into a single sub-directory (maybe named "common"), serving mostly the purpose of what "base" currently serves (with the exception of the custom maps and the default pk3 files being in the "maps" and "defaults" subdirectories respectively).

@Chomenor
Copy link

Even if it technically works, this seems like a bad experience for somebody trying to upgrade to the new filesystem on top of a previous install. Your design indicates up to 9 directories with various purposes at the root of the basepath/homepath, which will be merged on top of the base, gpp, and other mod directories from the previous install, which seems like a big mess. Practically speaking, it seems like you need a clean install to use the new file structure in a reasonable way.

A hardcoded hash list should actually be really easy to maintain, and you can use the same list to prompt and autodownload missing paks at game startup. A system that doesn't use a hash list would probably need to rely on an autodownloader system to normalize a certain directory with the correct base paks, which is probably more complicated than the hash-based system, and still requires some kind of file list from somewhere.

I don't understand how the layout I proposed is harder for new users to learn than any other layout. From my perspective it seems like one of the simplest layouts possible, but given my limited knowledge of Tremulous there might be gaps in my understanding of how things get used in practice. Could you give me some use case examples of where your design would make it easier for new users to accomplish certain tasks?

@dGr8LookinSparky
Copy link
Author

It is true that the full advantage of the new layout would be best in a homepath cleaned of the old layout (with the exception of the qkey), but I'm very confident that the very large majority of the current trem players who regularly access their homepath would consider cleaning up from their previous installations a small price to pay considering the benefits (as elaborated shortly). Regarding the players that don't bother cleaning their homepath, the multiple installations would still work together.

Still, maybe we should have a command that optionally reorganizes the homepath, moving the directories and files in the homepath (not in the basepath) from the old layout to the new layout. Of course that would mean that their old installation would then be missing all those files, but that would be an optional thing, their old installation would still be functional, and I'm pretty sure that we are at a point with the client where if we address the file system issues well, players would not want to go back to their old clients if they can use the new client.

9 sub-directories (or 8 if "dependencies" and "overrides" are merged into a "common" directory, 7 if we use the hash approach for the defaults you recommended) might seem like a lot of sub directories, but the number of fs_game sub directories can greatly surpass that, making the homepath unruly. One of the advantages of my layout is that it would keep the root of the homepath static for the most part.

In the case of demos, screenshots, and condumps, the way they traditionally work in the old file system they would be stored in whatever fs_game you happen to be on at the time. the fs_game can vary between a lot of different servers, and it can change on the same server. First, for a new player, once they find their homepath (which currently can be a difficult task in itself for new players, more so on some platforms than others), it is not immediately obvious that the screenshots/demos/condumps would be in the fs_game directories (not to mention it is not obvious to new players the purpose of all those directories is). Secondly, it can be a daunting task (actually not just for new players) to hunt down the screenshot/condump/demo you are looking for in the correct fs_game directory, especially if some time went by since those particular files were recorded.

Retrieving condumps/screenshots/demos is among the most common reasons players access their homepath, which is another reason it makes sense for them to have direct access to each when they open the homepath.

Regarding configs. There are some settings that should be set universal regardless of which fs_game you use (like renderer settings, player name, mouse sensitivity, etc). However, a single common config isn't sufficient. Tremulous has asymmetric teams (humans vs aliens for the large majority of game plays), and it is relatively common to have at least two team specific configs with team specific binds that are executed when you join the corresponding team. In addition to that, it is not uncommon to have class specific binds in some cases. What this means is that all those config files would have to be copied over to new fs_game directories. But then there are some situations where some binds may very between game versions. For example in 1.1 sprinting using the command "+boost", while in gpp and above "+button8" is used. Then there are situations that arise where some desirable binds/cvar settings are mod specific.

Setting configs is one of the other top reasons players access their homepath, combined with the need to have multiple config files, and some hierarchy of universal, "semi-universal" (version specific), and mod specific, a dedicated config subdirectory in the homepath root would be very convient. With that said, if there was such a config hierarchy, and a way via the console to set which kind of config the binds and cvars would be in, there would be much less need to access the homepath to manually adjust configs.

A problem with having the default pk3 hashes hardcoded is that anytime we might want to update that list, we would have to release new binaries for each platform. With that said, perhaps your approach could work very well if updates to the default hashes were broad casted to the clients in some way, for example checked against a text file from a url, in a similar way http://grangerhub.com/wp-content/uploads/clientnews.txt is used, but there should be a local cache for that list, in case that file can't be accessed at any time for whatever reason. The clients could auto-download new missing default files from a default download url (like from dl.grangerhub.com/base/).

@Chomenor
Copy link

It should be possible to have your demos, screenshots, condumps, and configs directories as you want them without changing the mod layout. Those directories would just need to go under one of the base directories, like say the "trem13" directory. It may be slightly less elegant, but I tend to think that is a small price to pay to maintain compatibility with previous installs, and it's a style existing players are used to anyway.

I like hardcoded hashes because I think in principle the base pak versions should match the binary. If the hash list is hardcoded you can know that the base paks loaded are always the one designed for that binary, and error reports can be addressed based on the user's binary version alone without needing to separately check what version of the hash list file they are using. As long as you have a solid build script and auto update system, it shouldn't be too big of a deal to update the binary when you need to make changes to the base paks, unless you need to do it extremely frequently. Given most servers are pure, it doesn't seem like the base paks would really need to be updated outside of version releases where you would update the binary anyway.

Of course, if you don't agree with this take, it is possible to use other methods such as a separate file to store the hash list instead. I just tend to think it would be a little more complex and have more room for errors that way.

@dGr8LookinSparky
Copy link
Author

We can give the hardcoded default hashes a try.

I was thinking further regarding the homepath, maybe we should change the default hompath locations. There are two reasons we should strongly consider that:

  1. It is important to have easy access to the homepath for accessing screenshots/demos/condumps, for editing configs, for mapping, for adding custom skins/etc, and for setting up mods that might not be available for download from an existing game server. Yet the default homepaths have always been hidden, and on macos getting to hidden folders are especially obscure. With this reason alone, we should consider moving the default homepath locations to non-hidden locations.

  2. New default homepath locations would allow us to change the homepath root directory layout without having to worry about the old layout of old installations.

Another possible layout:

  • base

  • mods

  • demos

  • screenshots

  • condumps

  • qkey

@Chomenor
Copy link

One option is to allow using the basepath without using a homepath at all. What I do in my Elite Force client is a write check to the basepath on game startup, and if it's writable the basepath is used as the main directory, and if not then the homepath is used as usual. This way as long as the game is installed anywhere with write permissions there is only one directory people have to work with.

@Chomenor
Copy link

I like this layout a bit better, because it seems a bit simpler, and with the hash-based precedence some things might not need to be organized into their own directory as much as they would otherwise. I'm still concerned that making a layout incompatible with the existing standard might not be well received by existing users, and the advantages aren't really significant enough to justify it. I'm curious if any other Grangerhub members have opinions on this?

@Chomenor
Copy link

To put things side-by-side, @dGr8LookinSparky's layout would look something like this out of the homepath or equivalent location:

  • base/base
  • base/gpp
  • base/trem13
  • mods/mod1
  • mods/mod2
  • demos
  • screenshots
  • condumps
  • configs
  • qkey

My proposal is more like this:

  • base
  • gpp
  • trem13
  • mod1
  • mod2
  • trem13/demos
  • trem13/screenshots
  • trem13/condumps
  • trem13/configs
  • trem13/qkey

@dGr8LookinSparky
Copy link
Author

I was discussing this with @Xembie , and he gave an interesting out of the box suggestion. What if we got rid of the basepath and the homepath, and just have a single working directory where the binaries are located (so basically have everything in the unzipped installation folder). Especially with the hardcoded default hash list approach, why would a seperate basepath be needed anymore?

I understand the original design intent for default multi user installations but that approach may be a source of a lot more hassle than what that corner case is worth. Everything you needed for your trem installation was in a single folder, besides simplifying the setup, and providing easy access to everything you might need related to trem, that opens new possibilities, like being able to put trem on a flashdrive, and play it on other computers with all your settings and pk3 files by just plugging in the flashdrive.

@Chomenor
Copy link

Yes, it's similar to what I described a few comments back. I think it's a good idea to have a fallback capability for when users put the executable in a privileged location from a previous install that the game can't write.

@Chomenor
Copy link

My filesystem already supports either configuration by command line cvar, by the way.

basepath (install dir) only: +set fs_dirs *basepath
basepath with homepath fallback: +set fs_dirs *basepath *homepath

To hardcode the configuration is just a matter of changing the default cvar value.

@dGr8LookinSparky
Copy link
Author

That sounds good, having a single directory with a separate homepath only as a fallback if installation doesn't allow the game to write, looks like a good way to go. But I think some intuitive and easy way for the user to get to the "working directory" would still be important to have especially in the case of the fallback homepath being used (like a button that opens the working directory with the user's default file manager).

One of the reasons I listed the qkey as being in the root is not only for convenience, but that is its location with the old file system. I should mention that the rsa key also goes in the root although we haven't really made use of it yet (but we will).

Looking over Chomenor's proposed layout alongside the most recent form of my proposed layout, I still prefer my proposed layout, for thefollowing reasons:

  • Whether you are new to the game, or are familiar with the old file system layout, as soon as the "working directory" is open, it is very clear and intuitive on where anything you might want to find is located, for the most part not requiring further explanation.

  • You have immediate access to the most common type of files the user would want.

  • The root will not be growing from game usage with additional directories (unlike the case of having the fs_game directories in the root)

  • If we set the "default fallback homepath" to be different from the current default homepath, we would no longer have to worry about even the most remote possibility of collision with old installs.

@wtfbbqhax , @jkent , @redrumrobot , @cengique , @TheDushan , what are your thoughts about the proposed "working directory layouts" and on what we have been discussing so far in this issue?

@cengique
Copy link
Member

@dGr8LookinSparky @Chomenor how about same mod on different servers? In the new scheme, they will fall into the same folder in mods/.

@dGr8LookinSparky
Copy link
Author

@cengique , that is the case with the old file system as well, where if any two servers are set to the same fs_game, they would share the same fs_game even if they may use completely different pk3 files in the same fs_game. One of the main advantages of different servers being able to share the same fs_game directory is that it can save on redundant downloads for the same mod (or at least the same base of a mod) for use on different servers.

A disadvantage is that any server can put any pk3 file in the same fs_game directory, even ones that might not have anything to do with the intended mod. For playing on pure and semi-pure servers that probably wouldn't be an issue, however that can be an issue for playing on unpure servers, also if you wanted to host a server sharing the same fs_game directory that a client on the same system used for playing on other servers with the same fs_game, you may have to clean up that folder manually (potentially multiple times if you continue to play on other servers with the same fs_game).

@cengique do you have any suggestions/proposals for managing this aspect differently? @Chomenor , any ideas for a/some practical solutions to addressing the disadvantages of multiple servers sharing the same fs_game directory?

@Chomenor
Copy link

If the mod is something very standard, you can have a hardcoded hash config that is activated by the specific fs_game value when the server is unpure. In Tremulous perhaps the "slacker" mod would be a candidate for this, since it seems to be used on many servers, but I don't know for sure.

@dGr8LookinSparky
Copy link
Author

The funny thing is, the "slacker" fs_game is a perfect example of different servers using the same fs_game name but with a lot of unrelated pk3 files. We initially used that fs_game for GrangerPub and GrangerClub which were the first servers to publicly host a multiprotocolised version of the 1.1 slacker mod (even still, there are some variations between the game logic of GrangerPub and GrangerClub, primarily related to the scrim mode on GrangerClub that GrangerPub doesn't have). After we had our initial public alpha release of the 1.3 repo (with the multiprtoocol code), other multiprotocol servers owned by other independent server owners appeared, and they decided to use the same "slacker" fs_game name, even though they were using for the most part modified versions of a multiprotocolization of the 1.1 cqvm mod.

WIth that said, there have been a lot of "standard" mods in the past that were popular among different independent servers. Some examples being the original slacker, the original cqvm, lakitu7, snake, Aussie Assault, arcade, KoRx, Tremx, Z, W, X, Ambush, ubp, etc...

@cengique
Copy link
Member

@dGr8LookinSparky maybe then concatenate the (modname+server extension) together in making subfolders in mod/ such that different servers can have slightly different versions of the same mod and users can log in with custom settings. Also, one can have the pk3/qvm checksum saved such that another server can't overwrite it with a "similar" named mod (like in the case of slacker).

@Helper-Bot
Copy link

Helper-Bot commented Apr 29, 2018

@cengique It's a good idea and it could be an easy way to organize mods with the same pk3's but have custom settings. But instead of just throwing it into a sub-folder say 'mod_name+serverExtension', should it be put in a folder that is dedicated to say sub-folder mods, it'd seem like a good way of organization from the normal VS custom.

@Chomenor
Copy link

Chomenor commented Apr 29, 2018

Generally remote servers handle conflicts via the pure list, and I think this is a good approach to continue. This imposes some limits on allowing clients to use custom content, but there are ways this can be addressed for specific cases if needed.

The only people who should really need to deal with mods are either running servers or creating mods, and in either case they should be able to set up the mod directory manually. Therefore a more advanced mod system is probably not a high priority for most purposes.

@dGr8LookinSparky
Copy link
Author

Idea:

A "servers" subdirectory in the root, which contains a subdirectory for each unique server that you have connected to. Each server's specific folder would contain a hash list file for each fs_game that server used while you were connected to that server. Each hash list file would list all of the files specific to that fs_game used by that server. Perhaps there could even be some kind of version control tracking in each hash list file, so that you can go back to previous mod versions for a specific server. For extra convenience each server's specific folder could contain symbolic links to each fs_game folder it used.

@Chomenor
Copy link

What is the proposed advantage of this system compared to the normal pure system?

@dGr8LookinSparky
Copy link
Author

the VMs can be swapped live without having to restart the tremded, it just requires a map change, the fs_game currently requires a full tremded restart (to my knowledge).

Changing pk3 files live is not ideal as it should be however. If you swap or remove any pk3 files live, clients that would attempt to download the previous pk3 files directly from the game server won't be able to, and in some cases the server can crash if it tries to access a pk3 files that is no longer there.

I think perhaps a preferable approach would be someway for the swapping/removal/adding to be prepared while a map is played, and then take place automatically upon the next map change. This would especially be useful for servers that update often.

@Chomenor
Copy link

Chomenor commented May 4, 2018

There are probably ways to deal with these server issues, but I think we should deal with the client file layout first, especially the question of whether to keep the current mod layout that is compatible with existing versions. What are your thoughts on that at this point?

@dGr8LookinSparky
Copy link
Author

I wrote an initial implementation for "folder browsing buttons". It needs some more work and testing on all the supported platforms, but so far it does function on Linux at least. There is more information here: #79 (comment) .

Given that functionality, we could try the most recent approach with the layout you suggested @Chomenor , with the dedicated subdirectories I proposed (i.e. configs/demos/screenshots/condumps ) in the same folder. As long as all demos 1.3 saves and uses are in that one dedicated demos subdirectory, all client config files written and used by 1.3 are in that one dedicated configs subdirectory, etc... Then we can add buttons in the "Browse Folders" menu for those particular dedicated subdirectories.

@Chomenor
Copy link

Chomenor commented May 4, 2018

Sounds good, I'll start working on the filesystem side of the implementation shortly.

Just to confirm, do you like the idea of the name "trem13" for the new directory, or would you prefer some other name? It can be changed later of course. Also do you think a name like this should be hardcoded, or is there a use case for making it cvar controlled?

@dGr8LookinSparky
Copy link
Author

If we don't have to include pk3 files in that directory, I think it would make sense to give it a non-version specific descriptive name (something like user-data), in which case I think hardcoding would be the way to go. I think it would be good if starting from this point, that data is always in the same place (unless would it be more beneficial in the long run for that directory to be versioned?)

Since the intended use of the data in both trem13 and gpp folders would behave differently from the data in all other non-base fs_game folders, Would there be issues with old systems if we put a trem13 subdirectory inside of the base folder, and kept the 1.3 default assets in there? likewise if we put a gpp subdirectory in base as well (although we could call it something like trem12 to keep with a better naming convention, and for all practical purposes, gpp assets can be considered 1.2 assets (1.2 never had a full release, and likely never will at this point)).

@wtfbbqhax
Copy link
Member

@dGr8LookinSparky @Chomenor, please consolidate idea's into a concise 1-2 page design document for the rest of us

@dGr8LookinSparky
Copy link
Author

dGr8LookinSparky commented May 4, 2018

@Chomenor , does the edit button work for you on this wiki page (idk if editing on that wiki requires special permission)? https://github.com/GrangerHub/tremulous/wiki/New-homepath-layout

@Chomenor
Copy link

Chomenor commented May 4, 2018

Yes, the edit button appears to work.

@dGr8LookinSparky What did you think of my points above in favor of using the trem13 directory for both data and pk3s? It would follow existing conventions better and having the version in the name helps distinguish it especially to users from directories that may be used by other applications.

I don't know why you would want to put any subdirectory under base. In general you want to follow the mod dir conventions as much as possible, both because users will be able to understand it more easily and because it allows other Tremulous versions to access the same files.

@Chomenor
Copy link

Chomenor commented May 4, 2018

To sum up my design, you just add the "trem13" directory, with the new configs/demos/screenshots/etc directories under it, and which can also be used for pk3s that other clients shouldn't load because they are only compatible with 1.3.

Everything else stays as it is, and you don't need much documentation because the extra directory is very apparent in meaning to anybody looking at the install directory, which is good because most users won't read documentation anyway.

@dGr8LookinSparky
Copy link
Author

Lets consider how future major client versions might fit into this layout. What our general overall plan is currently for post 1.3, starting with 1.4 is to have the new servers (1.4 and above) break compatibility with older clients (bellow 1.4), but still have the newest clients keep backwards compatibility with the older servers. We may take the same approach with 1.5 and so on.

So lts say someone has a 1.3 installation, and they have used a trem13 sub-directory for awhile, where all the demos, screenshots, confgis, and config statements they ever made/used with the 1.3 client (whether they play on a 1.1, gpp, or 1.3 server, or from loading a match from their client) would only be found in that trem13 sub-directory. Then later, when 1.4 clients would be available, such a new client is installed on top of the existing 1.3 install. a trem14 subdirectory is created which would be the new home of the demos/configs/screenshots/condumps made/used by the 1.4 client, even from the 1.4 client connecting to 1.1, gpp, and 1.3 servers. So if you wanted to use any of the demos/configs from 1.3 in the 1.4 client for example, you would have to copy them from the trem13 folder to the trem14 folder., as the 1.4 client wouldn't be touching the demos/etc in the trem13 folder, however, the 1.4 client still may use the default pk3 files in trem13 at least for connecting to 1.3 servers, same with the pk3 files in the gpp folder.

I do believe that it is essential that at least for what a given client would create/use, all demos/configs/screenshots/condumps should be available each in their own dedicated folders, that can be opened by dedicated folder buttons (regardless of which fs_game and server version that client might be connected to at the time). But for new version installs on top of old installs, there is potential for confusion with the layout in regards to the user data, without using the in-game folder buttons, although maybe that might not be as much of an issue having the in-game folder buttons.

There is another potential problem of clutter to consider, what if older version servers created fs_game folders with the naming convention of future versions? Like a 1.3 server (or even a 1.1 server) using an fs_game name trem13. Or for that matter, what if a 1.3 (or older) server used a trem13 fs_game with modded pk3 files that should rightfully be in a seperate fs_game? Sure with this new file system, that shouldn't interfere with the correct functioning of the new clients, but there would be clutter of pk3 files that should be in "modded" fs_game folders, but are in the versioned "baselike" folders. Is there anyway we could guard against such potential clutter that would be intuitive, or is that a necessary evil?

@Chomenor
Copy link

Chomenor commented May 5, 2018

My interpretation of the "trem13" directory is that is for data and pk3s for version 1.3 and up. So when you release 1.4 you probably can keep storing most data in trem13, and only create a trem14 if you have new data that is only compatible with 1.4. The thing to remember and convey to users is that trem13 stands for 1.3 and up, not 1.3 exactly.

base - compatible with 1.1 and up
gpp - compatible with gpp and up
trem13 - compatible with 1.3 and up
trem14 - compatible with 1.4 and up

Older versions putting pk3s in a newer version directory isn't a problem, because the hash precedence system in newer version will take care of any conflicts. On the other hand a newer version server putting incompatible files in an older versioned directory could be a problem, for people who still run an old version client, but that's not really our responsibility since the potential for incompatible pak downloads is an inherent issue for older clients.

@Chomenor
Copy link

Chomenor commented May 5, 2018

I can also understand that continuing to use trem13 for data storage on versions past 1.3 might seem counterintuitive, but for me it is actual more clear than something generic like 'user-data' because it conveys the history of the directory, i.e. the file storage location introduced in version 1.3, and disambiguates it from any storage locations used by unrelated projects.

@dGr8LookinSparky
Copy link
Author

I updated the wiki page with a design based on our discussions so far:
https://github.com/GrangerHub/tremulous/wiki/New-homepath-layout

Feel free to comment regarding it, and adjust where appropriate.

@dGr8LookinSparky
Copy link
Author

I edited the wiki page further switching to the MediaWiki format so that it now has a nice autogenerated table of contents (that the markdown format didn't have).

@Chomenor
Copy link

Chomenor commented May 5, 2018

This looks really good. Now that we have at least a provisional file layout is established, I should be able to start working on the hash precedence system. Do you think we should have a separate issue or other chat area to discuss questions about that effort?

@dGr8LookinSparky
Copy link
Author

Yes, I think we should have a separate issue for that discussion. We probably should also create a few more separate issues for some of the other things discussed in this issue.

@dGr8LookinSparky
Copy link
Author

dGr8LookinSparky commented May 5, 2018

Hmm, I was thinking, should the "versioned base like" folders instead have a prefix of "base_" for extra clarity of its function (also to ensure they stay near the top of the folder when things are ordered alphabetically)? That is: base_X-Y.

@jkent
Copy link
Collaborator

jkent commented May 5, 2018 via email

@Chomenor
Copy link

Chomenor commented May 5, 2018

@dGr8LookinSparky I don't see any reason why not. I went with "trem" in my examples I guess because it reflects the name of the project associated with it, "Tremulous 1.3", but I don't know any technical reason to favor either name.

@dGr8LookinSparky
Copy link
Author

Since the entire homepath is dedicated to the Tremulous game, it is kinda redundant to use the prefix trem_ instead of base_ .

@jkent It wasn't discussed before, but I wasn't sure if using "." in the folder name would present any issues. Would there be any issues with using a period in folder names for either new or old clients/servers @Chomenor ?

@Chomenor
Copy link

Chomenor commented May 5, 2018

"trem" and "base" kind of have the same meaning to me. It probably comes down to whatever you think looks nicer.

I don't know offhand if there are any issues with periods in mod names.

@dGr8LookinSparky
Copy link
Author

dGr8LookinSparky commented May 5, 2018

I like the idea of these other "versioned base" folders being next to base alphabetically, as well as close to the top of the homepath alphabetically, and having their prefix further indicating that they have similar properties to base. Regarding the period, lets try with it for now, and if it is found to be an issue with either the new system or with compatibility with old systems we can change it back to using hyphens instead easy enough.

I updated the wiki page accordningly: https://github.com/GrangerHub/tremulous/wiki/New-homepath-layout

@jkent
Copy link
Collaborator

jkent commented May 5, 2018 via email

@wtfbbqhax
Copy link
Member

@dGr8LookinSparky @Chomenor, A comment from Jkent above reminded me of something we need which is pk3 sorting based on semantic versioning (particularly against the code we already use external/semver) as this is how we tag our releases and subsequently our paks.

@Chomenor Does your system support pk3dir's?

@Chomenor
Copy link

Chomenor commented May 6, 2018

@wtfbbqhax How does this semantic versioning work for pk3s? Is there any documentation for this feature?

My filesystem does currently support pk3dirs.

@dGr8LookinSparky
Copy link
Author

@Chomenor , what if we were to have a user-data directory that would be top level in the homepath and contain the configs, demos, screenshots, condumps subdirectories, and like the basemod directory, not allow pk3 files to be downloaded to it, and not allow the fs_game to be set to it?

@Chomenor
Copy link

It's technically possible. IIRC I favored using trem_1.3 for this purpose because it would be more clear to users which particular version was creating this particular file layout, when multiple versions coexist(ed) with the same homepath location, and is consistent with the role of other directories in the homepath.

@dGr8LookinSparky
Copy link
Author

It would be consistent with the file layout, it would be like user-data was an fs_game directory, but we wouldn't let the client nor server use it as an fs_game, and if the unlikely situation that an old client uses that as an fs_game, there still wouldn't be any conflict, the old client would actually save the demos and screenshots in the same folders in that case. Additionally, it is the plan to make the config files functional for multiple versions, having the option to set cvars and binds for specific version(s), so it should be future proof if/when there might be a need for a new base folder version after 1.3.

@Chomenor
Copy link

trem_1.3 is presumed to be used for future versions anyway, so being future proof shouldn't be an issue. Whether it is confusing is another matter, but personally I think storing data in trem_1.3 for a hypothetical version 1.4 is still better than a generic name like user_data as long as the homepath is shared with other Tremulous clients.

@Chomenor
Copy link

Chomenor commented May 26, 2018

I understand this is less elegant, but to me different types of Tremulous clients should avoid using commonized directory names in the shared homepath folder whenever possible, especially when the layout of the directory is specific to a particular client. Otherwise things can get really messy.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

6 participants