Replies: 12 comments 10 replies
-
|
Thank you for all this info, I didn't know anything about it. I will try to take a look and implement it before RomM gets much bigger, as it seems to be a very useful standard |
Beta Was this translation helpful? Give feedback.
-
|
Maybe we can work on this together when the time comes as I will definitely need it even on RetroMan. |
Beta Was this translation helpful? Give feedback.
-
|
I am hijacking this issue and converting it to a discussion because I think it is a good start point where general ideas of how to integrate Datafiles in RomM (and presumably on RetroMan) can have place. |
Beta Was this translation helpful? Give feedback.
-
|
https://github.com/linuxserver/emulatorjs added a ideia about collect and create a unified version, full non-merged files every day become more and more the mostly used and with projects start using *.zip to encapsule everything. They unzip and check the file. I have some notes too: All emulator for arcade and the current status: https://buildbot.libretro.com/compatibility_lists/ All guys from 2020 start a big move to run all games on browsers, the webassembly make this possible and works really great. Latest compilation to webassembly to test: https://web.libretro.com/ There's no single project to aggregate the best emulator for web, mostly use retroarch and hope works, but in fact the best is doing the hard work, use the best emulator standalone which requires a unified interface. I think this project can point to these emulator and not trying to run inside, because is almost impossible to keep track of all these works. Maybe using the icon of the project when available, a unified metadata with exposed API is a must, my notes makes me think to create something like this, just a API to expose in the common interface all this data about roms. |
Beta Was this translation helpful? Give feedback.
-
|
One extra addition: all consoles from PS1 to old works really great on web, PS2/PS3/Wii(U)/Switch have emulator but don't works on web, retroarch have all but old forks, the best experience and compat is using every one in portable mode, this information is important because this project maybe can focus on emulatorjs from LSIO to PS1 and old e for the modern ones support their format, RPCS3 is the only one without file format, all others have one file format and can be used their compatibility list with the hash: Olds - https://github.com/linuxserver/emulatorjs/tree/master/metadata The most modern like yusu and citra don't work great. |
Beta Was this translation helpful? Give feedback.
-
|
For RPCS3 problem the workaround used is keep the disc on *.iso and mount to use, so the hash of *.iso file can fix the problem. The emulatorjs extract and take the hash from the one file from PSX, but better look the code or read the explanation on repo. |
Beta Was this translation helpful? Give feedback.
-
|
This project looks awesome! Can't wait to manage my library with it! (Linux doesn't have any ROM manager/renamer) FIY and for future reference, Nintendo Switch titledb : https://github.com/blawar/titledb |
Beta Was this translation helpful? Give feedback.
-
|
@Extarys FYI titleDB support just shipped with the 2.1.0 release |
Beta Was this translation helpful? Give feedback.
-
|
Not sure if this is out of the scope of this discussion, but I've yet to see a rom manager with truly good 1G1R (One game one ROM, effectively no duplicates across regions but keeping regional exclusives and notable variants) support. I'd love to have a tool that will let me know if a game lies outside of the 1G1R rules for the dat so I can choose to remove it from the library. |
Beta Was this translation helpful? Give feedback.
-
|
This is def. true - and while I think this mostly matters with Arcade roms - this could also be used to handle corralling rom hacks and variations as related to the original game that they're associated with... |
Beta Was this translation helpful? Give feedback.
-
|
Man this would be super useful. I'm thinking something of how Plex supports "versions" of a movie. |
Beta Was this translation helpful? Give feedback.
-
|
@Casuallynoted grouping same-games (multiple roms but same IGDB ID) together just shipped in 2.2.1, can you test it and let me know if you're seeing them collapse? check the video here: https://streamable.com/4qwfbj and if that's not what you're looking for let me know! |
Beta Was this translation helpful? Give feedback.
-
Wow, that's freaking awesome and works great for me! I think that largely solves the issue in my eyes, since someone can use that to filter down their library if they'd like to! |
Beta Was this translation helpful? Give feedback.
-
|
Glad to hear it! |
Beta Was this translation helpful? Give feedback.
-
|
The only alternatives to having dat management are Windows desktop applications which is archaic to anyone serving up their ROMs from a central storage like a NAS, especially if they mainly use their central services and media from non-Windows machines. RomM could be the saving grace here, but for sure the existing tools need to be studied for how they do things I guess, so probably not a weekend project by far, but all the more respectable if implemented. |
Beta Was this translation helpful? Give feedback.
-
|
We are already working on a dat management system :) |
Beta Was this translation helpful? Give feedback.
-
|
Is there a PR or branch I can track, so I can keep tabs on its development? |
Beta Was this translation helpful? Give feedback.
-
|
Not yet, we are exploring, but I'll let you know when we have something official |
Beta Was this translation helpful? Give feedback.
-
|
Sweet, thank you so much! :) |
Beta Was this translation helpful? Give feedback.
-
|
Just throwing this in here... Igir is an awesome tool for handling dat files. Would love to see RomM handle dat files and use it as a manager for my frontend :) |
Beta Was this translation helpful? Give feedback.
-
|
Any updates on this? |
Beta Was this translation helpful? Give feedback.
-
|
Lots of work that has to happen behind the scenes first like #1369 |
Beta Was this translation helpful? Give feedback.
-
Wow for only three weeks since first commit the progress here is nothing short of impressive, well done!
I hate to be the one to add yet another feature request, but this is more of a requirement. You need to be informed about this (if you don't already know), sooner rather than later.
tl;dr dat files you must start from here
The ROM Management Datafile, is a fairly basic specification, which provides you with everything you need to identify a ROM, like file names, sizes and hashes. The definition also include a name, description and usually also language, region, etc. all the information you need which is difficult or impossible to find from the file name alone. It is also what is traditionally expected from a ROM Manager, by your target audience and competition.
The tool you use to organize your ROMs, the ROM Manager. To rebuild a working ROM set from other collections or unknown files, while ensuring completeness, no corruption, modification, or manipulation. Which is also why it is championed by the preservation community, wanting to ensure every bit and original byte is accounted for. But it also enables you to compile a collection of known working ROMs, in the structure and layout as expected by different emulators.
MAME is particularly tricky, and as time progress more and more systems and their ROMs are being encapsulated by the project. It has already become the de-facto set for PSX ROMs only in a matter months. The problem is not only that their naming convention is obscure, using only a few lower case characters and numbers as file names, with no spaces, but they have three different types of ROMs for the same game(s). The ROMs can be zipped as split, full merged, or non-merged files, which may include one game, or multiple games of separate releases of the same game, or different games of the same type (hardware speaking). They also may or may not include the required BIOS dumps necessary to play the game which means these also need to be managed.
Not only are the dat files necessary to identify the actual game, but it is also required from a ROM Manager to be able to unzip and compile one type from another, as may be needed, to mitigate having to keep duplicates. I have been praying for a web manager, all the others are desktop clients which makes it difficult to combine into any sensible workflow automation, and they are singular in purpose. There is a disconnect between organizing files, and the identifying data contained therein, which is required to source the additional meta data needed to actually manage a library of media content.
These two worlds of organization vs manageability has remained in disconnect, which impairs the archivability of retro games into a singular source library by any definition. Hopefully RomM is here to finally make sense of all the chaos.
Sources
Logiqx actually came up with the dat file specification which most projects are using, there is of course always that one, no-intro has a slightly different schema but the content is similar and also well documented. Since it's all XML the data is self explaining anyway. This is not an exhaustive list...
http://www.logiqx.com/Dats/ - legacy sets
https://pleasuredome.github.io/pleasuredome/ - mame sets and others
https://github.com/danmons/datfiles - Mister set
The preservationists each with their own focus and peculiarities but each in their own right indispensable for their work cataloging our history.
http://redump.org - optical discs
https://no-intro.org - cartridges
https://www.tosecdev.org/downloads/category/22-datfiles
There is another one, trurip being rebranded as emuarc, but I am still unable to find their actual dat files, moving on...
Should probably be considered out of scope, more applicable to a lister project for sourcing roms, but for interest sake. The first, catalogs the scene releases of split archive files, the other has defined a specification called srr which catalogs the metadata required to "rescene" the extracted content back to identical split archives again.
https://dats.site
https://www.srrdb.com/
Documentation
...and other useful technical information to assist with the implementation of dat files.
https://pleasuredome.miraheze.org/wiki/DAT_File
http://www.logiqx.com/DatFAQs/DatCreation.php
https://wiki.no-intro.org/index.php?title=DAT-o-MATIC_Guide
https://www.oreilly.com/library/view/gaming-hacks/0596007140/ch01s13.html
The missing link which ties ROM files to their meta data, countless hours by a vast community capturing and cataloging this information, a saving grace unavailable to any other type of media.
Beta Was this translation helpful? Give feedback.
All reactions