Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This starts work towards supporting teh C data interface for the arrow
format, as documented
here.
Currently in this PR, it includes struct definitions and basic
methods to allow getting a pointer to an
ArrowSchema
/ArrowArray
C-compatible struct that can then be populated by another
implementation. For example, with this PR, you can do:
Currently, these
ArrowSchema
/ArrowArray
structs are pretty barebones, but it at least lays some ground work for integration. Things we
still need/want to make all this nicer to use/work with:
format strings as outlined
here
to figure out what type of data we'll get in the arrays. It'd
probably be best to add a
type
field to the ArrowSchema struct thatwe'd populate when converting from
CArrowSchema
->ArrowSchema
Arrow.ArrowVector(::ArrowSchema, ::ArrowArray)
that produced a concrete
ArrowVector
subtype, likeArrow.Primitive
,Arrow.List
, etc. This will be a bit tricky,because have to follow all the same columnar layout trickery that we
currently handle for IPC in the table.jl
build
methods. Perhaps wecan refactor all that so we can re-use some code? Otherwise, we might
just need to reimplement a bunch of that logic specific to converting
ArrrowArray
s.probably need a definition like
Arrow.ArrowSchema(a::Arrow.ArrowVector)
that produced a validArrowSchema
, and then overloads perArrowVector
subtype likeArrow.ArrowArray(x::Arrow.Primitive)
that produced the rightArrowArray
for a concrete arrow arrayfor providing a pointer to the
CArrowSchema
,CArrowArray
structsonce they're populated
If anyone would like to help out, I'm happy to provide as much guidance
as possible so others can get their feet wet in some arrow spec
nitty-gritty.