Add codepath for computing buckets without int conversion

[ayushdg]

Description

PR has 2 enhancements:

1. Improves performance for cases where users want to skip the fp check by skipping conversion of bucket_id's to integers, only needed by map_buckets and following steps in the fpcheck path.
2. Improves error messages/cases where the data contained no duplicates. Fixes [BUG] Fuzzy deduplication fails on datasets with no duplicates #67.

Usage

        lsh = LSH(
            ..., # same params as earlier
            buckets_as_int=False, # or true if planning to go via FP check.
        )

Checklist

• I am familiar with the Contributing Guide.
• New or Existing tests cover these changes.
• The documentation is up to date with these changes.

[praateekmahajan]

Not fir this PR, but just a highlight from our google docs convo, good place to leverage fsspec

[ayushdg]

Agreed. Decided to go via this route for now (since other places also use shutil). Aligned that the refactor to be more remote friendly should leverage fsspec utilities where possible.

[ayushdg]

Variable for tracking if all the buckets were empty

[ayushdg]

Reviewers ptal at this logic. I've tried to cover most edge cases

[ayushdg]

The reason we need to do this in the first place is because there's no way to know if we're writing out an empty dataframe or not, unless we persist, or write it out, check the metadata and then overwrite on the next iteration.

[ayushdg]

This logic can probably be simplified by using a global metadata file when writing out the parquet dataset write_metadata_file=True. However this had some issues in 24.10: rapidsai/cudf#17177 and is only fixed in 24.12. Will open an issue to simplify this method once that's merged in.

[praateekmahajan]

Should this return None or an empty DocumentDataset with no id's

[ayushdg]

I prefer returning None. Empty DocumentDatasets might lead to unexplained errors downstream that might be tougher to debug/understand. Happy to hear counter points.
One thing that comes up from this is that I might update the examples/FuzzyDedup.py to handle the case where the result returned was None

[praateekmahajan]

Makes sense, but then for Sequential I think we might want to handle that behavior too?

[ayushdg]

I haven't seen Sequential being used directly with FuzzyDuplicates since the results cannot be processed downstream by any of the other modules without using to filter out the duplicates first. I'm not sure how to handle this use case. But longer term, we would probably want to add a FuzzyDeduplicate class that calls Fuzzy Duplicates and also handles removal.