Retire text_bytes_aware_shuffle and directly use shuffle

[VibhuJawa]

We should retire text_bytes_aware_shuffle as we have #77 merged in now .

That will mean we refactor below code .

NeMo-Curator/nemo_curator/modules/fuzzy_dedup.py

Lines 1119 to 1144 in c2f296c

	try:
	# NOTE: If we have more text-df partitions than bucket-map
	# partitions, we are more likely to see an OverflowError
	output_df = text_bytes_aware_shuffle(
	df=merge_left_to_shuffled_right(
	subset_text_df,
	subset_bucket_df,
	merge_on,
	),
	partition_on=partition_on,
	text_column=self.text_field,
	num_workers=num_workers,
	)
	except OverflowError as err:
	# We encountered an overflow error!
	# Let's try again with less text data
	parts_per_text_batch_retry = int(parts_per_text_batch_use / 2)
	if parts_per_text_batch_retry < 1:
	raise err
	print(
	f"\nWe encountered an OverflowError and will retry "
	f"the current batch with {parts_per_text_batch_retry} "
	f"text partitions instead of {parts_per_text_batch_use}.",
	flush=True,
	)
	continue

Update 10/28

We have WIP PR #316 however we noticed a performance regression (10-20% slower at scale). Fixing it the right way will require more investigation which means I'll put this at risk for this release (slack thread).

Update 11/04

Working on it this week in #316

[VibhuJawa]

CC: @ayushdg