Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updated Huffman code #16

Draft
wants to merge 14 commits into
base: tools
Choose a base branch
from
Draft

Updated Huffman code #16

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
1ba9e20
Updated Huffman code
dpldgr Apr 20, 2024
b3fbf71
Fixed off by one error.
dpldgr Apr 20, 2024
a821dbd
Updated name of header file that needs to be updated.
dpldgr Apr 20, 2024
8003405
Update dataSize estimate.
dpldgr Apr 20, 2024
e8ff7d9
Updated estimate for maximum pieces on the board.
dpldgr Apr 20, 2024
a73604d
Merge branch 'fairy-stockfish:tools' into tools
dpldgr Jul 25, 2024
6bd4bc1
Testing GitHub actions.
dpldgr Jul 25, 2024
33f0a04
Changed huffman code to 6 bits
dpldgr Jul 25, 2024
68f68d4
Automatically determine how many piece bits are needed in the trainin…
dpldgr Jul 25, 2024
f846997
Fixed read_board_piece_from_stream
dpldgr Jul 25, 2024
0e1c156
Fixed.
dpldgr Jul 25, 2024
21d717c
Try again.
dpldgr Jul 25, 2024
eeeb272
3rd attempt.
dpldgr Jul 25, 2024
88967a3
Updated piece type bits.
dpldgr Jul 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 18 additions & 3 deletions src/tools/sfen_packer.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -159,7 +159,22 @@ namespace Stockfish::Tools {
{0b11001,5}, //
{0b11011,5}, //
{0b11101,5}, //
{0b11111,5}, //
{0b000011111,9}, //
{0b000111111,9}, //
{0b001011111,9}, //
{0b001111111,9}, //
{0b010011111,9}, //
{0b010111111,9}, //
{0b011011111,9}, //
{0b011111111,9}, //
{0b100011111,9}, //
{0b100111111,9}, //
{0b101011111,9}, //
{0b101111111,9}, // RESERVED
{0b110011111,9}, // RESERVED
{0b110111111,9}, // RESERVED
{0b111011111,9}, // RESERVED
{0b111111111,9}, // RESERVED
};

inline Square to_variant_square(Square s, const Position& pos) {
Expand Down Expand Up @@ -259,9 +274,9 @@ namespace Stockfish::Tools {
code |= stream.read_one_bit() << bits;
++bits;

assert(bits <= 6);
assert(bits <= 10);

for (pr = NO_PIECE_TYPE; pr <= 16; ++pr)
for (pr = NO_PIECE_TYPE; pr <= 26; ++pr)
if (huffman_table[pr].code == code
&& huffman_table[pr].bits == bits)
goto Found;
Expand Down
8 changes: 6 additions & 2 deletions src/ucioption.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,16 +97,20 @@ void on_variant_change(const Option &o) {

const Variant* v = variants.find(o)->second;

const int typeCount = popcount(v->pieceTypes);
const int typeCount5bits = (typeCount > 15) ? 15 : typeCount;
const int typeCount9bits = typeCount - typeCount5bits;
const int dataSize = (v->maxFile + 1) * (v->maxRank + 1) + v->nnueMaxPieces * 5
+ popcount(v->pieceTypes) * 2 * 5 + 50 > 512 ? 1024 : 512;
+ typeCount5bits * 2 * 5 + typeCount9bits * 2 * 9
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these are the bits for the pocket counts, see https://github.com/fairy-stockfish/variant-nnue-pytorch/wiki/Technical-details#training-data-format, the v->nnueMaxPieces * 5 is the one for the pieces on the board. It could e.g. be v->nnueMaxPieces * (typeCount > 15 ? 9 : 5). I think using worst case estimates here (all pieces are of the last type) is reasonable in order to be safe.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I should have picked that one up, I broke a long-standing rule to not code just before bed! 😴

Copy link
Member

@ianfab ianfab Apr 20, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks. The other change should be reverted though, since the 5 bit for the pockets are counts per type (5 bit = [0, 31]), i.e., they are fixed size and not related to the huffman encoding. However, it is using 7 bit when using the bigger data size, see

variant-nnue-tools/src/tools/sfen_packer.cpp

Lines 200 to 202 in cc4094b

for(auto c: Colors)
for (PieceSet ps = pos.piece_types(); ps;)
stream.write_n_bit(pos.count_in_hand(c, pop_lsb(ps)), DATA_SIZE > 512 ? 7 : 5);

but that does not need to be considered here. It could just be added as an additional criterion, but this isn't really related to this PR, so no need to add it here unless you would like to.

Edit: I just realized this formula requires tons of knowledge about the code base to decipher, so I should have added a comment explaining each term. I might add that later.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry if there might have been a misunderstanding. The changes regarding the bits for the pieces in hand still needs to be reverted. The comments on this formula I will add once this PR is merged to avoid merge conflicts.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, there was a misunderstanding it seems, but I have also been busy preparing to move house so haven't gotten around to fixing this yet. I'll make it a draft again for the moment until I have some time to revert the pieces in hand.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, ok. There is no rush of course, so take your time. Or if you want I could also push the changes to the PR from my side.

+ 50 > 512 ? 1024 : 512;

if (dataSize > DATA_SIZE)
std::cerr << std::endl << "Warning: Recommended training data size " << dataSize
<< " not compatible with current version. "
<< "Please recompile with largedata=yes" << std::endl << std::endl;

std::cerr<< std::endl
<< "------ lib/nnue_training_data_formats.h --------" << std::endl
<< "---------------- variant.h ---------------------" << std::endl
<< "#define FILES " << v->maxFile + 1 << std::endl
<< "#define RANKS " << v->maxRank + 1 << std::endl
<< "#define PIECE_TYPES " << popcount(v->pieceTypes) << std::endl
Expand Down