[OPIK-218] Data migration

[thiagohora]

Details

Migration following previous PR: #369

Issues

OPIK-218

Testing

• Manually tested

[andrescrz]

As discussed, this has some risk when run as a migration as it'd be desirable to be run by an operator.

On the other hand, this it's useful for open source users.

The risks where discussed and accepted.

As you're planning to implement filling the data with the input, expected_output and metadata field, my recommendation would be to merge that first in order to see how critical is to perform this migration.

[thiagohora]

Hi @andrescrz,

I noticed something interesting: in dev, I tried to run the query to validate it, and it generated the new rows. However, I can still see the old ones. It only does it after running OPTIMIZE TABLE dataset_items FINAL;

Do you think this is expected?

[thiagohora]

Hi @andrescrz,

I noticed something interesting: in dev, I tried to run the query to validate it, and it generated the new rows. However, I can still see the old ones. It only does it after running OPTIMIZE TABLE dataset_items FINAL;

Do you think this is expected?

Confirmed, the behavior is expected as it's up to the engine to decide the best time to run the merge.