Test parallel snapshotting feature of debezium

[subkanthi]

<h2 id="optional_parallel_snapshots" style="box-sizing: border-box; font-family: Helvetica, Arial, sans-serif; font-weight: normal; line-height: 1.1; color: rgb(101, 101, 101); margin-top: 22px; margin-bottom: 11px; font-size: 32px; overflow-wrap: break-word; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(255, 255, 255); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">Optional parallel snapshots</h2><div class="sectionbody" style="box-sizing: border-box; color: rgb(101, 101, 101); font-family: &quot;Helvetica Neue&quot;, Helvetica, Arial, sans-serif; font-size: 15px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(255, 255, 255); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><div class="paragraph" style="box-sizing: border-box;"><p style="box-sizing: border-box; margin: 0px 0px 11px; overflow-wrap: break-word;">Debezium’s relational database initial snapshot process has always been single-threaded. This limitation primarily stems from the complexities of ensuring data consistency across multiple transactions.</p></div><div class="paragraph" style="box-sizing: border-box;"><p style="box-sizing: border-box; margin: 0px 0px 11px; overflow-wrap: break-word;">Starting in Debezium 2.2, we’re adding a new and initially optional way to utilize multiple threads to perform consistent database snapshot for a connector. This implementation uses these multiple threads to execute table-level snapshots in parallel.</p></div><div class="paragraph" style="box-sizing: border-box;"><p style="box-sizing: border-box; margin: 0px 0px 11px; overflow-wrap: break-word;">In order to take advantage of this new feature, specify<span> </span><code style="box-sizing: border-box; font-family: Consolas, &quot;Andale Mono WT&quot;, &quot;Andale Mono&quot;, &quot;Lucida Console&quot;, &quot;Lucida Sans Typewriter&quot;, &quot;DejaVu Sans Mono&quot;, &quot;Bitstream Vera Sans Mono&quot;, &quot;Liberation Mono&quot;, &quot;Nimbus Mono L&quot;, Monaco, &quot;Courier New&quot;, Courier, monospace; font-size: 13.5px; padding: 2px 4px; color: rgb(199, 37, 78); background-color: rgb(249, 242, 244); border-radius: 4px;">snapshot.max.threads</code><span> </span>in your connector’s configuration and when this property has a value greater than<span> </span><code style="box-sizing: border-box; font-family: Consolas, &quot;Andale Mono WT&quot;, &quot;Andale Mono&quot;, &quot;Lucida Console&quot;, &quot;Lucida Sans Typewriter&quot;, &quot;DejaVu Sans Mono&quot;, &quot;Bitstream Vera Sans Mono&quot;, &quot;Liberation Mono&quot;, &quot;Nimbus Mono L&quot;, Monaco, &quot;Courier New&quot;, Courier, monospace; font-size: 13.5px; padding: 2px 4px; color: rgb(199, 37, 78); background-color: rgb(249, 242, 244); border-radius: 4px;">1</code>, parallel snapshots will be used.</p></div><div class="listingblock" style="box-sizing: border-box;"><div class="title" style="box-sizing: border-box; color: red; font-style: italic; line-height: unset; margin-bottom: -20px;">Example configuration using parallel snapshots</div><div class="content" style="box-sizing: border-box;"><pre class="CodeRay highlight" style="box-sizing: border-box; overflow: auto; font-family: Consolas, &quot;Andale Mono WT&quot;, &quot;Andale Mono&quot;, &quot;Lucida Console&quot;, &quot;Lucida Sans Typewriter&quot;, &quot;DejaVu Sans Mono&quot;, &quot;Bitstream Vera Sans Mono&quot;, &quot;Liberation Mono&quot;, &quot;Nimbus Mono L&quot;, Monaco, &quot;Courier New&quot;, Courier, monospace; font-size: 14px; display: block; padding: 8.46875px; margin: 1.5em 0px; line-height: 1.5; color: rgb(101, 101, 101); word-break: break-all; overflow-wrap: break-word; background: rgb(247, 247, 248); border: 1px solid rgb(223, 223, 223); border-radius: 4px; white-space: pre;"><code data-lang="properties" class="hljs" style="box-sizing: border-box; font-family: Consolas, &quot;Andale Mono WT&quot;, &quot;Andale Mono&quot;, &quot;Lucida Console&quot;, &quot;Lucida Sans Typewriter&quot;, &quot;DejaVu Sans Mono&quot;, &quot;Bitstream Vera Sans Mono&quot;, &quot;Liberation Mono&quot;, &quot;Nimbus Mono L&quot;, Monaco, &quot;Courier New&quot;, Courier, monospace; font-size: inherit; padding: 0px; color: rgb(68, 68, 68); background: rgb(240, 240, 240); border-radius: 0px; white-space: pre-wrap; display: block; overflow-x: auto; border-style: none;">snapshot.max.threads=4</code></pre></div></div><div class="paragraph" style="box-sizing: border-box;"><p style="box-sizing: border-box; margin: 0px 0px 11px; overflow-wrap: break-word;">In the example above, if the connector needs to snapshot more than 4 tables, there will be at most 4 tables being snapshot in parallel. When one thread finishes processing a table, it will get a new table to snapshot from the queue and the process continues until all tables have been snapshot.</p></div><div class="admonitionblock note" style="box-sizing: border-box; margin: 10px;">
  | This feature is considered incubating, but we strongly suggest that new connector deployments give this feature a try. We would welcome any and all feedback on how to improve this going forward.
-- | --


</div></div>