Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement from_json_to_structs #2510

Open
wants to merge 80 commits into
base: branch-24.12
Choose a base branch
from
Open

Implement from_json_to_structs #2510

wants to merge 80 commits into from
+1,324 −131

Conversation

ttnghia
Copy link
Collaborator

@ttnghia ttnghia commented Oct 17, 2024 •
edited
Loading

This implements the function from_json_to_structs, which performs parsing a strings column into a structs column with a given data schema. This is the case of running from_json SQL function in Spark with struct schema.

Closes #2468.

Contribute to NVIDIA/spark-rapids#11560.

@ttnghia ttnghia requested a review from revans2 October 17, 2024 03:07
@ttnghia ttnghia self-assigned this Oct 17, 2024
@ttnghia ttnghia mentioned this pull request Oct 17, 2024
Open
@ttnghia
Fix empty input
8d7ad2e 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Misc
9e759c4 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Add nullifyIfNotQuoted option for removeQuotes
2fff949 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Implement castStringsToDecimals
d09de41 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Copy link
Collaborator Author

ttnghia commented Oct 21, 2024

CC @karthikeyann.

@karthikeyann karthikeyann mentioned this pull request Oct 21, 2024
Open
@ttnghia
Revert test
1243599 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Remove header
6f89fcd 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Rewrite Javadoc
deb3ebf 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Rename variable
9dc641f 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Rewrite docs
53b121d 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Revert test
69265b4 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Cleanup headers
da4d1f6 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Cleanup
1d91e64 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Rewrite the conversion functions
d0fa2ae 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Move code
f375a4d 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Remove call to make_structs_column
034a5ec 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Cleanup
74d858c 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia ttnghia marked this pull request as ready for review November 14, 2024 23:08
revans2
revans2 previously approved these changes Nov 15, 2024
View reviewed changes
src/main/cpp/src/from_json_to_structs.cu Outdated Show resolved Hide resolved
src/main/java/com/nvidia/spark/rapids/jni/JSONUtils.java Show resolved Hide resolved
@ttnghia
Copy link
Collaborator Author

ttnghia commented Nov 15, 2024

build

@ttnghia
Reorganize code
5a819d0 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Handle schema mismatching
fa1946a 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Add test
553d7d0 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Add another test
8a17651 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Revert "Add another test"
3773a27 
This reverts commit 8a17651.
@ttnghia
Fix schema mismatch
34cc98d 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Cleanup
bfd461b 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Add another test
cf9d6bf 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Revert "Add another test"
11ff5a7 
This reverts commit cf9d6bf.
@ttnghia
Revert "Add test"
b3f4882 
This reverts commit 553d7d0.

Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Add prefix spark_rapids_jni::
45670a2 
Signed-off-by: Nghia Truong <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hyperbolic2346 hyperbolic2346 hyperbolic2346 left review comments

@revans2 revans2 revans2 left review comments

At least 1 approving review is required to merge this pull request.

Assignees

@ttnghia ttnghia

Labels
enhancement New feature or request feature request performance
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[FEA] Implement a function to convert a list of strings columns into a structs column with desired schema
3 participants
@ttnghia @revans2 @hyperbolic2346