Repository navigation
Distributes compaction coordination across managers - #6324
Conversation
For an overview of how this works see the javadoc in CompactionCoordinator.
| sleepTime = maxSleepTime; | ||
| } | ||
|
|
||
| UtilWaitThread.sleep(sleepTime); |
There was a problem hiding this comment.
We might need a property here like the TABLE_SUSPEND_DURATION, but for Coordinators. There is a case where we don't want to be too responsive to a Manager going away and coming back. Bouncing the Manager sometimes clears up some state issues. I would think that we only want to recompute the group assignments if the number of compaction groups has changed or the number of managers has changed, and then only after some suspend duration.
There was a problem hiding this comment.
That makes sense. For our ITs we want this to be really responsive, but on a real system having it take its time when there is volatility would probably be good.
…n/CoordinatorLocationsFactory.java Co-authored-by: Dave Marion <dlmarion@apache.org>
revert changes in 9af4fdd as it broke clearing uneeded compaction jobs. CompactionPriorityQueueMetricsIT.testCompactionQueueClearedWhenNotNeeded() was failing
|
All ITs are passing for b54ed2a |
…ction/coordinator/CompactionCoordinator.java Co-authored-by: Dave Marion <dlmarion@apache.org>
…ction/coordinator/CompactionCoordinator.java Co-authored-by: Dave Marion <dlmarion@apache.org>
…ction/coordinator/CompactionCoordinator.java Co-authored-by: Dave Marion <dlmarion@apache.org>
…ction/coordinator/CompactionCoordinator.java Co-authored-by: Dave Marion <dlmarion@apache.org>
…ction/CompactionJobClient.java Co-authored-by: Dave Marion <dlmarion@apache.org>
…ction/CompactionJobClient.java Co-authored-by: Dave Marion <dlmarion@apache.org>
…pactionIT.java Co-authored-by: Dave Marion <dlmarion@apache.org>
|
I kicked off a full IT build and there are some remaining comments with suggested verbiage changes to comments (feel free to ignore those). Everything looks good. |
…ompactor.java Co-authored-by: Dave Marion <dlmarion@apache.org>
For an overview of how this works see the javadoc in CompactionCoordinator.