-
Notifications
You must be signed in to change notification settings - Fork 305
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[AMORO-3329] Add MemorySize for parsing memory value and unit #3409
[AMORO-3329] Add MemorySize for parsing memory value and unit #3409
Conversation
df49415
to
9334783
Compare
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #3409 +/- ##
============================================
+ Coverage 21.59% 27.72% +6.13%
- Complexity 2309 3566 +1257
============================================
Files 426 596 +170
Lines 39719 48528 +8809
Branches 5624 6263 +639
============================================
+ Hits 8577 13456 +4879
- Misses 30414 34133 +3719
- Partials 728 939 +211
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks a lot for the contribution!
I think this is a good start to make our configuration value more readable.
We may improve our configuration format in other place, like the AMS configuration after this PR been merged.
I left a minor suggestion, you may want have a look!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the contribution, LGTM
This patch is back-compatible, but it contains some user behavior change, maybe we need to add this in the release notes, @Jzjsnow could you please create an issue to track the release notes update, thanks.
Apart the above, do we need to add some documentation about how to configure this in the documentation? if this is yes could you please add an issue to track it. thanks.
…#3409) * [AMORO-3329] Add MemorySize for parsing memory value and unit * fixup! [AMORO-3329] Add MemorySize for parsing memory value and unit --------- Co-authored-by: jzjsnow <[email protected]> Co-authored-by: ZhouJinsong <[email protected]>
Why are the changes needed?
Currently, when we use flink optimizer groups, the memory parameters
flink-conf.jobmanager.memory.process.size
andflink-conf.taskmanager.memory.process.size
only support plain numeric strings or strings withm/g
suffix. If we configure them with mb/gb suffix, we'll get wrong parsed results. And the default unit for plain numeric strings ismb
, which is inconsistent with the default unit of byte in the flink native parameter and may cause inconvenience for users familiar with flink when using this parameter.Close #3329.
Brief change log
MemorySize
to theamoro-common
module for parsing flink's memory parameters, consistent with flink's native parsing logic.properties
of ams metadata tablesresource_group
andresource
to automatically add theMB
suffix to the flink jm and tm memory parameters that have been set and saved as plain numeric stringsHow was this patch tested?
Add some test cases that check the changes thoroughly including negative and positive cases if possible
Add screenshots for manual tests if appropriate
Run test locally before making a pull request
Documentation