Improve downloading files logic to check if a file already exists

[ek-nyc]

When I rerun my tests, it takes a long time to download the 3 files that already have been downloaded. The downloading steps should be skipped if the files already exist.

[alwayslove2013]

@ek-nyc could you please provide more detailed information?

Currently, there is a logic in VectorDBBench to skip file downloads. If we detect that a file with the same name has the same size, we will skip the download.

VectorDBBench/vectordb_bench/backend/data_source.py

Lines 57 to 66 in 1ab46dd

	def validate_file(self, remote: pathlib.Path, local: pathlib.Path) -> bool:
	info = self.bucket.get_object_meta(remote.as_posix())
	# check size equal
	remote_size, local_size = info.content_length, os.path.getsize(local)
	if remote_size != local_size:
	log.info(f"local file: {local} size[{local_size}] not match with remote size[{remote_size}]")
	return False
	return True

Additionally, please note that the default download location is the /tmp folder, which is typically cleared upon system reboot.