Add decrease interface for aggregation #9737

xzhangxian1008 · 2024-12-20T03:13:23Z

What problem does this PR solve?

Issue Number: ref #7376

Problem Summary:

What is changed and how it works?

In order to support aggregation in window function, we need to add some interfaces for aggregation. In this pr, we add `decrease` interface for aggregation.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

None

ti-chi-bot · 2024-12-20T03:13:28Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign searise for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

xzhangxian1008 · 2024-12-24T03:31:15Z

/cc @windtalker @guo-shaoge

windtalker · 2024-12-24T09:10:35Z

dbms/src/AggregateFunctions/AggregateFunctionAvg.h

+        {
+            this->data(place).sum -= static_cast<const ColumnVector<T> &>(*columns[0]).getData()[row_num];
+        }
+        --this->data(place).count;


add assert(this->data(place).count >= 0)?

windtalker · 2024-12-24T09:14:30Z

dbms/src/AggregateFunctions/AggregateFunctionForEach.h

+            if constexpr (is_add)
+                nested_func->add(nested_state, nested, i, arena);
+            else
+                nested_func->decrease(nested_state, nested, i, arena);


why not just call nested_func->addOrDecrease<is_add>()?

why not just call nested_func->addOrDecrease<is_add>()?

Because addOrDecrease<>() is not an interface in IAggregateFunction.

windtalker · 2024-12-24T09:15:33Z

dbms/src/AggregateFunctions/AggregateFunctionGroupConcat.h

+                    if constexpr (is_add)
+                        this->nested_function->add(this->nestedPlace(place), &nested_column, row_num, arena);
+                    else
+                        this->nested_function->decrease(this->nestedPlace(place), &nested_column, row_num, arena);


Ditto

Because addOrDecrease<>() is not an interface in IAggregateFunction.

windtalker · 2024-12-24T09:17:24Z

dbms/src/AggregateFunctions/AggregateFunctionMerge.h

@@ -64,6 +64,16 @@ class AggregateFunctionMerge final : public IAggregateFunctionHelper<AggregateFu
        nested_func->merge(place, static_cast<const ColumnAggregateFunction &>(*columns[0]).getData()[row_num], arena);
    }

+    void decrease(AggregateDataPtr __restrict place, const IColumn ** columns, size_t row_num, Arena * arena)


decrease is the same as add?

decrease is the same as add?

AggregateFunctionMerge is impossible to be used in window agg. It's just a placeholder to pass the compilation. I will delete it.

windtalker · 2024-12-24T09:32:19Z

dbms/src/AggregateFunctions/AggregateFunctionNull.h

+                    if constexpr (is_add)
+                        this->nested_function->add(this->nestedPlace(place), &nested_column, row_num, arena);
+                    else
+                        this->nested_function->decrease(this->nestedPlace(place), &nested_column, row_num, arena);
                }
            }
        }
        else
        {
            this->setFlag(place, 1);


The flag actually has its meanings, you should maintain the flag for decrease

windtalker · 2024-12-24T09:35:40Z

dbms/src/Common/AlignedBuffer.h

+  * It can only be allocated and destroyed.
+  * MemoryTracker is not used. AlignedBuffer is intended for small pieces of memory.
+  */
+class AlignedBuffer : private boost::noncopyable


Looks like this class is not used?

Looks like this class is not used?

It's used in test in this pr. And will also be used in further pr.

windtalker · 2024-12-24T09:45:56Z

dbms/src/AggregateFunctions/AggregateFunctionMinMaxAny.h

@@ -43,9 +45,18 @@ struct SingleValueDataFixed
        = false; /// We need to remember if at least one value has been passed. This is necessary for AggregateFunctionIf.
    T value;

+    // It's only used in window aggregation
+    mutable std::deque<T> * saved_values;


I suggest to implement window_min/window_max function which support decrease.

guo-shaoge · 2024-12-24T02:54:14Z

dbms/src/AggregateFunctions/AggregateFunctionAvg.h

+        if constexpr (IsDecimal<T>)
+            this->data(place).sum -= static_cast<const ColumnDecimal<T> &>(*columns[0]).getData()[row_num];
+        else
+        {


remove { }?

remove { }?

okk

guo-shaoge · 2024-12-24T03:03:37Z

dbms/src/Common/AlignedBuffer.cpp

+
+void AlignedBuffer::alloc(size_t size, size_t alignment)
+{
+    void * new_buf;


If alloc() is called two times without calling daealloc() or reset(), there will be memory leak.
Maybe we can add some check to avoid it?

If alloc() is called two times without calling daealloc() or reset(), there will be memory leak. Maybe we can add some check to avoid it?

It's impossible to call alloc() two times. Because it's a private function and only be used in constructor and reset().

xzhangxian1008 · 2024-12-26T09:02:06Z

/hold

ti-chi-bot · 2024-12-27T09:19:22Z

@xzhangxian1008: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-unit-test	`ad8f002`	link	true	`/test pull-unit-test`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

guo-shaoge · 2024-12-27T10:01:49Z

dbms/src/AggregateFunctions/AggregateFunctionNull.h

+            auto tmp = prefix_size;
+            prefix_size += sizeof(Int64);
+            if ((prefix_size % tmp) != 0)
+                prefix_size += tmp - (prefix_size % tmp);


I think the final prefix_size should be divided by alignOfData. But this code doesn't look it's guaranted.

xzhangxian1008 added 19 commits November 26, 2024 16:58

init

3f80eed

add gtest

171fd46

add ft

943c578

refine test framework

e0b05a2

fix compilation phase

75d2f12

init

7022607

codes done, need tests

b011bfa

save

8628a2e

add tests

db4e399

add sum tests

a9360da

refine tests

e3c31bd

format

e3b9756

fix bugs

a159857

refine test

0d4401d

fix tests

9ff2bd3

add test for string type

a9aa879

add test for SingleValueDataGeneric type

bf652bf

tweaking

61eaef0

Merge branch 'master' into wagg

6017780

ti-chi-bot bot added the release-note-none Denotes a PR that doesn't merit a release note. label Dec 20, 2024

ti-chi-bot bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Dec 20, 2024

xzhangxian1008 added 8 commits December 20, 2024 11:41

remove something

2358efb

revoke

3227b50

add AlignedBuffer

b2772ea

remove something

37f4cc5

tweaking

dd52c9f

remove useless change

6379b0f

fix ci

9e41b3a

fix ci

e6adcbf

fix ut

b84c911

ti-chi-bot bot requested review from guo-shaoge and windtalker December 24, 2024 03:31

windtalker reviewed Dec 24, 2024

View reviewed changes

guo-shaoge reviewed Dec 24, 2024

View reviewed changes

address some comments

aa12221

ti-chi-bot bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 26, 2024

xzhangxian1008 added 3 commits December 26, 2024 18:20

create new class for window

2b9e991

add ut tests

c54f45f

add static

ad8f002

guo-shaoge reviewed Dec 27, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add decrease interface for aggregation #9737

Add decrease interface for aggregation #9737

xzhangxian1008 commented Dec 20, 2024 •

edited

Loading

ti-chi-bot bot commented Dec 20, 2024

xzhangxian1008 commented Dec 24, 2024

windtalker Dec 24, 2024

windtalker Dec 24, 2024

xzhangxian1008 Dec 25, 2024

windtalker Dec 24, 2024

xzhangxian1008 Dec 25, 2024

windtalker Dec 24, 2024

xzhangxian1008 Dec 25, 2024

windtalker Dec 24, 2024

windtalker Dec 24, 2024

xzhangxian1008 Dec 25, 2024

windtalker Dec 24, 2024

guo-shaoge Dec 24, 2024

xzhangxian1008 Dec 25, 2024

guo-shaoge Dec 24, 2024

xzhangxian1008 Dec 25, 2024

xzhangxian1008 commented Dec 26, 2024

ti-chi-bot bot commented Dec 27, 2024

guo-shaoge Dec 27, 2024

Add decrease interface for aggregation #9737

Are you sure you want to change the base?

Add decrease interface for aggregation #9737

Conversation

xzhangxian1008 commented Dec 20, 2024 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot bot commented Dec 20, 2024

xzhangxian1008 commented Dec 24, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xzhangxian1008 commented Dec 26, 2024

ti-chi-bot bot commented Dec 27, 2024

Choose a reason for hiding this comment

xzhangxian1008 commented Dec 20, 2024 •

edited

Loading