Update SchemaGen Executor to natively handle SequenceExample #5689

AlexanderLavelle · 2023-01-31T14:45:52Z

While this PR does not address the need to specify payload_format in ExampleGen (or ImportExampleGen, etc), it does address the disconnect between ExampleGen, SchemaGen, and Transform by specifying tensor representations in SchemaGen. This should be a positive benefit to most users as the tensor representations will become a natural piece of the pipeline.

This addresses 5520, 4714, and, to a lesser extent, 5361.

lego0901 · 2023-02-10T01:02:48Z

tfx/components/schema_gen/executor.py

+    # Add tensor representations to handle SequenceExamples downstream. Still need correct Payload Format.
+    tensor_representations = tensor_representation_util.InferTensorRepresentationsFromSchema(schema)
+    tensor_representation_util.SetTensorRepresentationsInSchema(schema, tensor_representations)


Can you make it formatted in Pyink Python formatter for consistency? (https://github.com/google/pyink)
ex. <=80 columns rule

I have attempted to do this, although perhaps the imports need changing for this same reason? I am unsure how one recommends to do this if the updated state is not desired, as I did not come up with the module names nor function names?

lego0901 · 2023-02-10T01:09:20Z

Thanks for your contribution! It looks beneficial for many users as well.
We are checking if your change doesn't affect other users' existing pipelines or tests. Please be patient to be completely reviewed.

github-actions · 2023-03-12T02:03:05Z

This PR is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days

AlexanderLavelle · 2023-03-13T15:07:04Z

I have been out of the country but am still interested in pursuing this commit.

lego0901 · 2023-03-14T04:18:21Z

Thanks for your contribution!
The reason we took more cautious approach about this is it changes the file payload, and any related bugs are not trivial to be caught by unit tests.

To check if it doesn't break any tests and any existing pipelines, I would
(1) First approve this PR
(2) Maybe, merge this PR into the master
(3) Run some internal continuous tests including e2e pipeline runs
(4, if problem appears) Rollback before the next releases, otherwise keep this change!

I appreciate again for your consideration.

AlexanderLavelle · 2023-04-19T15:43:24Z

@lego0901 - I have a potential recommendation for this. What if during the encoding everything was turned into a sequence example (where appropriate) anyway? This should not affect the majority of users as they wouldn't have features to encode into the feature_lists of tf.train.SequenceExample

gbaned · 2023-07-04T14:59:12Z

Hi @lego0901 Any update on this PR? Please. Thank you!

AlexanderLavelle · 2023-07-05T00:55:37Z

@gbaned I have thought about this PR more and it can be improved even beyond here. Part of it is connecting other big query column types upstream, but fundamentally I don't see downstream negatives by putting everything as sequence examples. The changes will also need to be applied in example gen such that the output will be only sequence examples and not the option for one or another. The upstream modification is only regarding something like if isinstance(pyval, FeatureLists) ...

lego0901 · 2023-07-05T00:59:20Z

Yes, this looks good to me, either. But the problem is, we are updating the testing environment (Ubuntu 16.04 -> 20.04) and it is causing some build failures.. I wanted to make sure that this newly introduced feature will not break any test breakage but cannot confirm it before making every test green. I will accept this PR as soon as possible if the tests are not failing.

Sorry and thank you.

gbaned · 2023-10-27T04:20:04Z

Hi @lego0901 / @AlexanderLavelle Any update on this PR? Please. Thank you!

AlexanderLavelle · 2023-10-29T15:04:31Z

@gbaned no update on my side -- I have used this and confirm it works when intending to use SequenceExamples in TFX downstream components (Transform, Trainer)

gbaned · 2023-11-07T05:59:28Z

Hi @lego0901 Any update on this PR? Please. Thank you!

jsalkey · 2023-11-08T13:47:20Z

Hi, is there any update on this PR?

gbaned · 2023-11-15T05:43:17Z

Hi @lego0901 Any update on this PR? Please. Thank you!

gbaned · 2023-11-27T04:57:39Z

Hi @lego0901 Any update on this PR? Please. Thank you!

gbaned · 2023-12-15T05:05:21Z

Hi @lego0901 Any update on this PR? Please. Thank you!

gbaned · 2023-12-27T04:53:13Z

Hi @lego0901 Any update on this PR? Please. Thank you!

gbaned · 2024-01-10T05:30:23Z

Hi @lego0901 Any update on this PR? Please. Thank you!

gbaned · 2024-02-23T05:16:30Z

Hi @lego0901 Any update on this PR? Please. Thank you!

gbaned · 2024-04-02T04:59:59Z

Hi @lego0901 Any update on this PR? Please. Thank you!

master into schemagenmod

gbaned · 2024-04-19T05:20:22Z

Hi @lego0901 Any update on this PR? Please. Thank you!

lego0901 · 2024-04-19T05:23:14Z

I would have to explicitly mention that this is blocked because of the TF docker issue, so the internal integration test is broken for a while. I cannot confirm this PR unless it is resolved, because this should be merged after verifying that it does not break any internal tests.

gbaned · 2024-04-19T05:35:19Z

I would have to explicitly mention that this is blocked because of the TF docker issue, so the internal integration test is broken for a while. I cannot confirm this PR unless it is resolved, because this should be merged after verifying that it does not break any internal tests.

Hi @lego0901 Thank you so much for the update.

keerthanakadiri · 2024-11-28T10:09:54Z

Hi @AlexanderLavelle, Could you possibly fetch the most recent code and run the tests? Please give an update on this PR.

…for_sequences

AlexanderLavelle · 2024-12-04T06:42:51Z

@keerthanakadiri Hello! I have brought in the new changes without conflicts. I am currently on PTO but can run tests in a week or early the following week

Update executor.py

a512a4c

gbaned self-assigned this Feb 1, 2023

gbaned requested a review from roseayeon February 1, 2023 16:53

roseayeon requested a review from lego0901 February 7, 2023 01:52

gbaned added the awaiting review label Feb 9, 2023

lego0901 reviewed Feb 10, 2023

View reviewed changes

github-actions bot added the stale label Mar 12, 2023

google-ml-butler bot removed the stale label Mar 13, 2023

update for column width

4c9c04b

AlexanderLavelle mentioned this pull request May 2, 2023

Allow providing a custom_config with beam ReadFromBigQuery arguments #5860

Merged

AlexanderLavelle mentioned this pull request May 11, 2023

Refactor Utils to support added type casting capabilities #5906

Merged

AlexanderLavelle mentioned this pull request Jul 7, 2023

Default to Sequence Examples #6003

Closed

gbaned added stat:awaiting response and removed awaiting review labels Sep 27, 2023

AlexanderLavelle mentioned this pull request Oct 14, 2023

Allow List of Lists and SequenceExample in dict_to_example #6374

Closed

google-ml-butler bot removed the stat:awaiting response label Oct 27, 2023

gbaned added the awaiting review label Nov 7, 2023

Merge pull request #3 from AlexanderLavelle/master

6415971

master into schemagenmod

Merge branch 'tensorflow:master' into AlexanderLavelle/schemagen_mod_…

70edb1e

…for_sequences

keerthanakadiri requested a review from lego0901 December 6, 2024 05:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update SchemaGen Executor to natively handle SequenceExample #5689

Update SchemaGen Executor to natively handle SequenceExample #5689

AlexanderLavelle commented Jan 31, 2023

lego0901 Feb 10, 2023 •

edited

Loading

AlexanderLavelle Mar 13, 2023

lego0901 commented Feb 10, 2023

github-actions bot commented Mar 12, 2023

AlexanderLavelle commented Mar 13, 2023

lego0901 commented Mar 14, 2023 •

edited

Loading

AlexanderLavelle commented Apr 19, 2023

gbaned commented Jul 4, 2023

AlexanderLavelle commented Jul 5, 2023

lego0901 commented Jul 5, 2023

gbaned commented Oct 27, 2023 •

edited

Loading

AlexanderLavelle commented Oct 29, 2023

gbaned commented Nov 7, 2023

jsalkey commented Nov 8, 2023

gbaned commented Nov 15, 2023

gbaned commented Nov 27, 2023

gbaned commented Dec 15, 2023

gbaned commented Dec 27, 2023

gbaned commented Jan 10, 2024

gbaned commented Feb 23, 2024

gbaned commented Apr 2, 2024

gbaned commented Apr 19, 2024

lego0901 commented Apr 19, 2024

gbaned commented Apr 19, 2024

keerthanakadiri commented Nov 28, 2024

AlexanderLavelle commented Dec 4, 2024

Update SchemaGen Executor to natively handle SequenceExample #5689

Are you sure you want to change the base?

Update SchemaGen Executor to natively handle SequenceExample #5689

Conversation

AlexanderLavelle commented Jan 31, 2023

lego0901 Feb 10, 2023 • edited Loading

Choose a reason for hiding this comment

AlexanderLavelle Mar 13, 2023

Choose a reason for hiding this comment

lego0901 commented Feb 10, 2023

github-actions bot commented Mar 12, 2023

AlexanderLavelle commented Mar 13, 2023

lego0901 commented Mar 14, 2023 • edited Loading

AlexanderLavelle commented Apr 19, 2023

gbaned commented Jul 4, 2023

AlexanderLavelle commented Jul 5, 2023

lego0901 commented Jul 5, 2023

gbaned commented Oct 27, 2023 • edited Loading

AlexanderLavelle commented Oct 29, 2023

gbaned commented Nov 7, 2023

jsalkey commented Nov 8, 2023

gbaned commented Nov 15, 2023

gbaned commented Nov 27, 2023

gbaned commented Dec 15, 2023

gbaned commented Dec 27, 2023

gbaned commented Jan 10, 2024

gbaned commented Feb 23, 2024

gbaned commented Apr 2, 2024

gbaned commented Apr 19, 2024

lego0901 commented Apr 19, 2024

gbaned commented Apr 19, 2024

keerthanakadiri commented Nov 28, 2024

AlexanderLavelle commented Dec 4, 2024

lego0901 Feb 10, 2023 •

edited

Loading

lego0901 commented Mar 14, 2023 •

edited

Loading

gbaned commented Oct 27, 2023 •

edited

Loading