Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[AMORO-3387]fix: create table like using mixed_hive #3387

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

Aireed
Copy link
Contributor

@Aireed Aireed commented Jan 2, 2025

Why are the changes needed?

Enviroment:
format: mixed_hive
engine: spark3.3
catalog: sparkUnifiedSessionCatalog

bug:
table created by using create table like using mixed_hive is not a mixed_hive table.

spark-sql> create table ctl_demo(id int ,name string, dt string) using mixed_hive partitioned by (dt);
Time taken: 1.137 seconds
spark-sql> show create table ctl_demo;
CREATE TABLE spark_catalog.amoro_v7.ctl_demo (
  id INT,
  name STRING,
  dt STRING)
USING mixed_hive
PARTITIONED BY (dt)
TBLPROPERTIES (
  'base.hive.location-root' = 'hdfs://easyops-cluster/user/warehouse/amoro_v7.db/ctl_demo/hive',
  'base.write.format' = 'parquet',
  'delta.write.format' = 'parquet',
  'flink.max-continuous-empty-commits' = '2147483647',
  'schema.name-mapping.default' = '[ {
  "field-id" : 1,
  "names" : [ "id" ]
}, {
  "field-id" : 2,
  "names" : [ "name" ]
}, {
  "field-id" : 3,
  "names" : [ "dt" ]
} ]',
  'self-optimizing.group' = 'amovo_v7',
  'table.create-timestamp' = '1735817876108',
  'write.metadata.delete-after-commit.enabled' = 'true',
  'write.parquet.compression-codec' = 'zstd')

Time taken: 0.893 seconds, Fetched 1 row(s)
spark-sql> create table ctl_demo_like like ctl_demo using mixed_hive;
Time taken: 0.501 seconds
spark-sql> show create table ctl_demo_like;
CREATE TABLE amoro_v7.ctl_demo_like (
  id INT,
  name STRING,
  dt STRING)
USING mixed_hive
PARTITIONED BY (dt)

Time taken: 0.103 seconds, Fetched 1 row(s)
spark-sql>

Brief change log

  • modify provider to mixed_hive
  • support like sourceTable without databases

How was this patch tested?

  • Add some test cases that check the changes thoroughly including negative and positive cases if possible

  • Add screenshots for manual tests if appropriate

  • Run test locally before making a pull request

Documentation

  • Does this pull request introduce a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

@github-actions github-actions bot added the module:mixed-spark Spark module for Mixed Format label Jan 2, 2025
@Aireed Aireed changed the title [AMORO-x]fix: create table like using mixed_hive [AMORO-3387]fix: create table like using mixed_hive Jan 2, 2025
@codecov-commenter
Copy link

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 27.80%. Comparing base (decd55d) to head (12b2d86).
Report is 3 commits behind head on master.

Additional details and impacted files
@@             Coverage Diff              @@
##             master    #3387      +/-   ##
============================================
+ Coverage     21.64%   27.80%   +6.15%     
- Complexity     2316     3589    +1273     
============================================
  Files           427      599     +172     
  Lines         39751    48597    +8846     
  Branches       5630     6266     +636     
============================================
+ Hits           8605    13510    +4905     
- Misses        30419    34149    +3730     
- Partials        727      938     +211     
Flag Coverage Δ
core 27.80% <ø> (?)
trino ?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Comment on lines +124 to +128
provider.isDefined match {
case true =>
throw new UnsupportedOperationException(
s"format ${provider.get} does not support create table like command!!!")
case false => plan
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This implementation may impact the logic of other extensions when multiple extensions are used together.

It is recommended to handle only the scenarios you are concerned with in the case conditions =>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
module:mixed-spark Spark module for Mixed Format
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants