[controller] log compaction #1282

WhitneyDeng · 2024-11-05T00:00:12Z

Summary

purpose: to get stores ready for compaction based on programmatically defined criteria
added API endpoints to get compaction-ready stores

How was this PR tested?

unit test: filter all stores on a cluster for compaction-ready stores

Does this PR introduce any user-facing changes?

No. You can skip the rest of this section.
Yes. Make sure to explain your proposed changes and call out the behavior change.

…er methods VeniceParentHelixAdmin::isCompactionReady VeniceParentHelixAdmin::isLastCompactionTimeOlderThanThresholdHours

…or readability & maintainability

…ForCompaction & VeniceParentHelixAdmin::isLastCompactionTimeOlderThanThresholdHours JavaDocs description - complete description of VeniceParentHelixAdmin::getStoresForCompaction - future tense to present tense for VeniceParentHelixAdmin::isLastCompactionTimeOlderThanThresholdHours

…ionReady & helper methods

…ersion is not found

…:filterStoresForCompaction - extract filterStoresForCompaction() logic for testing - write VeniceParentHelixAdminTest::testFilterStoresForCompaction

… TestVeniceParentHelixAdmin - migrate from integration test files to unit test files

… argument order

mynameborat

Good job on putting up the first PR quickly. Looks very good. I have put in some comments.

Let me know if you have questions.

internal/venice-common/src/main/java/com/linkedin/venice/controllerapi/ControllerClient.java

...mmon/src/integrationTest/java/com/linkedin/venice/controller/VeniceParentHelixAdminTest.java

services/venice-controller/src/main/java/com/linkedin/venice/controller/Admin.java

...s/venice-controller/src/main/java/com/linkedin/venice/controller/VeniceParentHelixAdmin.java

.../venice-controller/src/main/java/com/linkedin/venice/controller/server/AdminSparkServer.java

...s/venice-controller/src/main/java/com/linkedin/venice/controller/VeniceParentHelixAdmin.java

...nice-controller/src/test/java/com/linkedin/venice/controller/TestVeniceParentHelixAdmin.java

…tionReady package private

…data type ArrayList -> List

- remodularise VeniceParentHelixAdmin::isHybridStore to function var in VeniceParentHelixAdmin::isCompactionReady - make VeniceParentHelixAdmin::isLastCompactionTimeOlderThanThresholdHours hardcoded int to class constant

...s/venice-controller/src/main/java/com/linkedin/venice/controller/VeniceParentHelixAdmin.java

- change exception message in child controller

interface for repush: Azkaban/Airflow implementation for Linkedin internal. (potentially) Spark implementation for OSS.

…iceController

…logic to VeniceHelixAdmin

- empty implementation for now

reason: since repush will be used for diff use cases (other than compaction), the term repush should be reserved for the abstracted repush action.

- instantiate in VeniceController

- this config determines the number of threads for LogCompactionService::executor

…UNT annotations & comments

…into CompactionManager - create CompactionManager & TestCompactionManager - move TestCompactionManager::testFilterStoresForCompaction from TestVeniceParentHelixAdmin

…utor setup in LogCompactionService::startInner

…g time unit from hour to millisecond

…actionManager

- create com.linkedin.venice.controller.logcompaction in tests - change CompactionManager::filterStoresForCompaction from public to package private

…g-compaction-repush`

WhitneyDeng · 2024-12-06T22:18:32Z

closed because branch renamed to whdeng/log-compaction-repush to follow branch naming convention

WhitneyDeng · 2024-12-06T23:07:13Z

reverted branch name to log-compaction for PR continuity.

…epush` to `log-compaction`

…abled feature flat purpose: control rollout of log compaction service

…ionEnabled

- log trigger source (for now, can expand later)

…QueryParams initialisation to take explicit clusterName directly

- since triggerRepush() is beyond the original scope of log compaction, this method is parked for later addition. adding TODO to remember thought process & design decision to use byte[] + data model class rather than QueryParams for repush job details

- CompactionManager::compactStore & VeniceHelixAdmin::compactStore logs & rethrows exception from repush - LogCompactionService logs exception & can be extended to perform other actions to handle exception - RepushJobResponse will be passe from RepushOrchestrator::repush through CompactionManager::compactStore & VeniceHelixAdmin::compactStore to LogCompactionService - added execution URL to RepushJobResponse, to follow repush.py's pattern

after studying information in Airflow response objects

to keep LogCompactionTask running & not bring down VeniceController

…ompactionManager if log compaction is enabled

- created TestRepushOrchestratorImpl, a dummy RepushOrchestrator implementation for testing

...controller/src/main/java/com/linkedin/venice/controller/logcompaction/CompactionManager.java

…cont'd: update CompactionManager init

- add integration test in TestHybrid::testHybridStoreLogCompaction - add Admin::getCompactionManager for testing - change scope of CompactionManager::isCompactionReady

…gCompaction - setup incomplete. decided to do test in TestHybrid::testHybridStoreLogCompaction instead

… to TestLogCompactionService

Whitney Deng added 10 commits October 29, 2024 11:52

[controller] log compaction

ff3c1ae

[controller][server] complete log compaction candidate selection help…

6a7903d

…er methods VeniceParentHelixAdmin::isCompactionReady VeniceParentHelixAdmin::isLastCompactionTimeOlderThanThresholdHours

[controller][server][style] put log compaction criterion on newline f…

62fcf64

…or readability & maintainability

[controller][server][documentation] VeniceParentHelixAdmin::isCompact…

8101408

…ionReady & helper methods

[controller][server][documentation] filter out stores where current v…

8160e5b

…ersion is not found

[controller][server][unit test] Unit test for VeniceParentHelixAdmin:…

c28728c

…:filterStoresForCompaction - extract filterStoresForCompaction() logic for testing - write VeniceParentHelixAdminTest::testFilterStoresForCompaction

[controller][server][unit test] move testFilterStoresForCompaction to…

5e481cb

… TestVeniceParentHelixAdmin - migrate from integration test files to unit test files

[controller][server][unit test][style] switch actual & expected value…

2a0f3f2

… argument order

[controller][server][unit test][style] switch actual & expected value…

f6ca080

… argument order

mynameborat reviewed Nov 6, 2024

View reviewed changes

Whitney Deng added 5 commits November 6, 2024 11:38

[controller][server][style][PR] make VeniceParentHelixAdmin::isCompac…

bf73b48

…tionReady package private

[controller][server][style][PR] revert wildcard import

77d83c0

[controller][server][style][PR] delete wildcard import

af12106

[controller][server][PR] change Admin::getStoresForCompaction return …

e82e659

…data type ArrayList -> List

[controller][server][PR]

373c208

- remodularise VeniceParentHelixAdmin::isHybridStore to function var in VeniceParentHelixAdmin::isCompactionReady - make VeniceParentHelixAdmin::isLastCompactionTimeOlderThanThresholdHours hardcoded int to class constant

majisourav99 reviewed Nov 7, 2024

View reviewed changes

...s/venice-controller/src/main/java/com/linkedin/venice/controller/VeniceParentHelixAdmin.java Outdated Show resolved Hide resolved

Whitney Deng added 13 commits November 21, 2024 09:10

[controller]

2b11f9d

- change exception message in child controller

[controller] create RepushOrchestratorProvider

1a7f015

interface for repush: Azkaban/Airflow implementation for Linkedin internal. (potentially) Spark implementation for OSS.

[controller] create LogCompactionService & initialise instance in Ven…

1df10d9

…iceController

[controller][PR] change ArrayList to List datatype in contracts

4652dc9

[controller][PR] move VeniceParentHelixAdmin::getStoresForCompaction …

da5455c

…logic to VeniceHelixAdmin

[controller] add triggerRepush endpoint

f30f705

- empty implementation for now

[controller] change repush terminology to compactStore

24c97a1

reason: since repush will be used for diff use cases (other than compaction), the term repush should be reserved for the abstracted repush action.

[controller] create LogCompactionService

3991b22

- instantiate in VeniceController

[controller][config] add SCHEDULED_LOG_COMPACTION_THREAD_COUNT config

43f5c40

- this config determines the number of threads for LogCompactionService::executor

[controller][config][style] delete SCHEDULED_LOG_COMPACTION_THREAD_CO…

f638e40

…UNT annotations & comments

[controller] Refactor VeniceHelixAdmin::getStoresForCompaction logic …

2f3966d

…into CompactionManager - create CompactionManager & TestCompactionManager - move TestCompactionManager::testFilterStoresForCompaction from TestVeniceParentHelixAdmin

[controller] add scheduled.log.compaction.interval.HR config for exec…

8655b96

…utor setup in LogCompactionService::startInner

[controller][config] change schedule.log.compaction.interval.ms confi…

1f44e45

…g time unit from hour to millisecond

Whitney Deng added 3 commits December 5, 2024 16:52

[controller][PR] revert wildcard import

2c65d3c

[controller][PR] prepend DEFAULT to public static final field in Comp…

9a9cc1d

…actionManager

[controller][PR] move logcompaction tests to same package

5457c79

- create com.linkedin.venice.controller.logcompaction in tests - change CompactionManager::filterStoresForCompaction from public to package private

WhitneyDeng closed this Dec 6, 2024

WhitneyDeng deleted the log-compaction branch December 6, 2024 19:25

test commit after renaming branch from log-compaction to `whdeng/lo…

083af27

…g-compaction-repush`

WhitneyDeng restored the log-compaction branch December 6, 2024 22:11

WhitneyDeng reopened this Dec 6, 2024

Whitney Deng added 13 commits December 6, 2024 15:08

test commit after reverting branch name from `whdeng/log-compaction-r…

dc0835e

…epush` to `log-compaction`

[controller][PR] add and update docs and logs

a9a5aeb

[controller][PR] revert wildcard imports

5ac6468

[controller][PR] add VeniceControllerClusterConfig::isLogCompactionEn…

c4dc3de

…abled feature flat purpose: control rollout of log compaction service

[controller][PR] add VeniceControllerMultiClusterConfig::isLogCompact…

7a851d9

…ionEnabled

[controller] add LogCompactionTask::triggerSource

b93c6fc

- log trigger source (for now, can expand later)

[controller][PR] in ControllerClient::getStoresForCompaction, modify …

5a5dddb

…QueryParams initialisation to take explicit clusterName directly

[controller] add/modify RepushJobResponse fields

8253c82

after studying information in Airflow response objects

[controller] catch unhandled Throwable

7eec4be

to keep LogCompactionTask running & not bring down VeniceController

[controller] VeniceHelixAdmin: only initialise RepushOrchestrator & C…

759c2ae

…ompactionManager if log compaction is enabled

[controller] setup test controller in TestLogCompactionService

8a9dc9f

- created TestRepushOrchestratorImpl, a dummy RepushOrchestrator implementation for testing

lusong64 reviewed Dec 16, 2024

View reviewed changes

...controller/src/main/java/com/linkedin/venice/controller/logcompaction/CompactionManager.java Outdated Show resolved Hide resolved

Whitney Deng added 6 commits December 17, 2024 11:51

[controller] add config for time since last log compaction threshold

87294ca

[controller] ConfigKeys::TIME_SINCE_LAST_LOG_COMPACTION_THRESHOLD_MS …

e6d2bb4

…cont'd: update CompactionManager init

[controller][PR] lambda function instead of Optional

ec5acaa

[controller] added integration test for scheduled log compaction

8a321ca

- add integration test in TestHybrid::testHybridStoreLogCompaction - add Admin::getCompactionManager for testing - change scope of CompactionManager::isCompactionReady

[controller] incomplete integration test for log compaction in TestLo…

c5376ca

…gCompaction - setup incomplete. decided to do test in TestHybrid::testHybridStoreLogCompaction instead

[controller] todo to extract TestHybrid::testHybridStoreLogCompaction…

3a677e5

… to TestLogCompactionService

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[controller] log compaction #1282

[controller] log compaction #1282

WhitneyDeng commented Nov 5, 2024

mynameborat left a comment

WhitneyDeng commented Dec 6, 2024

WhitneyDeng commented Dec 6, 2024

[controller] log compaction #1282

Are you sure you want to change the base?

[controller] log compaction #1282

Conversation

WhitneyDeng commented Nov 5, 2024

Summary

How was this PR tested?

Does this PR introduce any user-facing changes?

mynameborat left a comment

Choose a reason for hiding this comment

WhitneyDeng commented Dec 6, 2024

WhitneyDeng commented Dec 6, 2024