Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DAR-3513][External] Addition of the item_merge_mode push argument #920

Closed
wants to merge 5 commits into from

Conversation

JBWilkie
Copy link
Collaborator

@JBWilkie JBWilkie commented Aug 29, 2024

Adds the item_merge_mode argument to push() :
item_merge_mode: Enum["slot", "series", "channel"]

1: It is available through both CLI & SDK-initiated push() with this syntax:
darwin dataset push /path/to/files --item_merge_mode [slots | series | channels]

from darwin.client import Client

team_slug = "team_slug"
dataset_slug = "dataset_slug"
files_to_upload = ["path/to/dir_1", "path/to/dir_2", "...", "path/to/dir_n"]
item_merge_mode = # oneof["slots" | "series" | "channels"]

client = Client.local()
dataset = client.get_remote_dataset(dataset_identifier=f"{team_slug}/{dataset_slug}")
dataset.push(files_to_upload=files_to_upload, item_merge_mode=item_merge_mode)

2: Raises an error if preserve_folders is set:
dataset.push(files_to_upload=files_to_upload, item_merge_mode=item_merge_mode, preserve_folders=True)

3: Throws 1 non-blocking warning for each instance of:

  • A file path passed to push() that references a specific file rather than a directory, since these are ignored
  • Each folder that after being parsed according to the specified item_merge_mode rules, results in an empty directory

Technical Implementation
1: Forks the existing parsing logic so that it's only run if item_merge_mode isn't set
2: If item_merge_mode is set, use parsing logic that creates 1 MultiFileItem object (new class) per dataset item. Run validation & construct + attach a Layout object to each MultiFileItem

Changes beyond this point would relate to uploading itself, so are beyond the scope of this ticket, and will be coming in the next couple of weeks

Copy link

linear bot commented Aug 29, 2024

@wiz-inc-4ad3b29aa7
Copy link

wiz-inc-4ad3b29aa7 bot commented Aug 30, 2024

Wiz Scan Summary

IaC Misconfigurations 0C 0H 0M 0L 0I
Vulnerabilities 0C 2H 1M 0L 0I
Sensitive Data 0C 0H 0M 0L 0I
Total 0C 2H 1M 0L 0I
Secrets 0🔑

@JBWilkie
Copy link
Collaborator Author

JBWilkie commented Sep 2, 2024

Closing in favour of the multi-file-push branch PR: #923

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant