Adding subshard work items on lease expiry

[AndreKurait]

Description

Continuation on #1160

Changes since #1160

• Added LeaseExpirationTest that refreshes after every bulk put to synthetically generate more segments for the test case.
• Added Binary search to more efficiently find the starting document segment
• Modified initial segment sorting with error logging on equality

---

• Added code to finish and create remainder work item when a lease expires.

Behavior changes is as follows:

• Updates the LuceneDocumentsReader to sort the segments and emit docs in sequence, and skip until passed in startingDocId. Note: Docs within a segment are still read in parallel, just emitted in sequence once aggregated together.
• Updates the RfsLuceneDocument to contain the luceneDocId (segmentBaseDoc + docId)
• Updates the DocumentReindexer to emit a flux of the latest sequential docId processed.
• Updates the DocumentsRunner to plum context from work item progress and cancellation to the LeaseEnd
• Updates the OpenSearchWorkCoordinator to rename numAttempts to nextAcquisitionLeaseExponent (incrementing script version from poc -> 2.0)
• Updates exitOnLeaseTimeout to handle cancelling document reindexing work, and creating successor work item based on progress checkpoint and shard work timing. Logic below.

The lease time increase logic has changed. Behavior is as follows:

• If worker did not have enough time to process any docs, time is doubled for next run.
• else
  • If worker spent more than 10% of time downloading/extracting the shard, double the lease time for the next run
  • else if worker spent less then 2.5% of time downloading/extracting the shard, half the lease time for next run
  • else keep lease time same for successive run

Added E2E test as follows:

• Create docs with workload generator, set up toxiproxy and leases to ensure will take 8 40 second leases to finish shard with checkpoints. Verify exit codes and docs migrated

Issues Resolved

• https://opensearch.atlassian.net/browse/MIGRATIONS-2128

Testing

Tested in AWS and added new E2E test around the scenario

Check List

• New functionality includes testing
  • All tests pass, including unit test, integration test and doctest
• New functionality has been documented
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 80.30%. Comparing base (06ae3ca) to head (9010fcf).
Report is 11 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #1198      +/-   ##
============================================
- Coverage     80.52%   80.30%   -0.23%     
- Complexity     3072     3101      +29     
============================================
  Files           421      423       +2     
  Lines         15629    15850     +221     
  Branches       1057     1072      +15     
============================================
+ Hits          12585    12728     +143     
- Misses         2396     2474      +78     
  Partials        648      648              

Flag	Coverage Δ
unittests	80.30% <ø> (-0.23%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[peternied]

With the holidays coming up - lets hold off on merging this until after we've got a fuller crew in house.

[mikaylathompson]

@AndreKurait do we still use sorting based on segment name?