v0.12.6

This release includes an upgrade to Beam 2.46.

🚀 Enhancements

• Add fanout for CMS hot key computation in skewed joins (#4752) @RustedBones
• Delay deprecation of old skewed API (#4755) @RustedBones
• Expose cache in scio-grpc (#4734) @RustedBones
• Add magnolify imports to docs for methods using Funnel (#4739) @kellen
• Distribute CMS hot key set as side input (#4724) @RustedBones
• Support Parquet in BQ ExtractOps (#4728) @clairemcginty
• Neo4j parametrized from existing SCollection (#4719) @sumitsu
• Make SeqExampleSCollectionOps API consistent (#4731) @clairemcginty
• Add skewed job methods with topN and top percent (#4718) @RustedBones
• Modernize scalafix rules (#4687) @RustedBones

🐛 Bug Fixes

• Don't forget to fan out (#4740) @kellen
• Don't re-use Configuration objects in Parquet SMB (#4727) @clairemcginty
• apply extendedErrorInfo input parameter (#4723) @farzad-sedghi

🏗️ Build Improvements

• Fork all tests. Default to java 11 (#4748) @RustedBones
• Factorize java options and propagate GCP settings (#4746) @RustedBones
• Add java 17 configuration for integration tests (#4745) @RustedBones
• Remove test matrix for integration test (#4737) @RustedBones
• Setup MiMa filter for planed breaking changes (#4733) @RustedBones

🌱 Dependency Updates

• Update beam to 2.46.0 (#4744) @RustedBones
• Update mysql-socket-factory to 1.11.0 (#4730) @scala-steward
• Update circe-core, circe-generic, ... to 0.14.5 (#4735) @scala-steward
• Bump actions/add-to-project from 0.4.0 to 0.4.1 (#4729) @dependabot
• Update scalafmt-core to 3.7.2 (#4720) @scala-steward
• Update elasticsearch-java to 8.6.2 (#4715) @scala-steward

Contributors to this release

@RustedBones, @clairemcginty, @dependabot, @dependabot[bot], @farzad-sedghi, @kellen, @scala-steward and @sumitsu

v0.12.5

This release adds support for Java 17 and contains coder bug fixes. It also includes Beam 2.45.0 dependency update.

🚀 Enhancements

• Unique tuple tags by @farzad-sedghi in #4711
• Set default JdkAddOpenModules for Java 17 by @clairemcginty in #4710

🐛 Bug Fixes

• Avoid using outputfield cache by @shnapz in #4701
• Force scio coders on all input SCollection by @RustedBones in #4713
• replace assert with com.google.common.base.Verify by @farzad-sedghi in #4716

📗 Documentation

• Explain Hadoop configuration in Parquet.md by @clairemcginty in #4703
• Update site releases by @RustedBones in #4717

🏗️ Build Improvements

• Add Java 17 to GHA setup by @clairemcginty in #4659
• update smb multi join code gen to the latest by @farzad-sedghi in #4702
• Update sbt-scoverage to 2.0.7 by @scala-steward in #4706
• Update sbt-assembly to 2.1.1 by @scala-steward in #4708

🌱 Dependency Updates

• Update beam to version 2.45.0 by @RustedBones in #4712
• Update circe-core, circe-generic, ... to 0.14.4 by @scala-steward in #4705
• Update mysql-socket-factory to 1.10.0 by @scala-steward in #4704

Contributors to this release

@RustedBones, @clairemcginty, @farzad-sedghi, @scala-steward and @shnapz

Full Changelog: v0.12.4...v0.12.5

v0.12.4

🚀 Enhancements

• Set BQ staging max time travel to minimum 2 days by @shnapz in #4697

🐛 Bug Fixes

• Fix SingletonCoder serialization by @RustedBones in #4691
• Unwrap internal SCollection coders to be reused in transforms by @RustedBones in #4690
• Make Parquet-SplittableDoFn opt-in, not opt-out by @clairemcginty in #4700

📗 Documentation

• Update 0.12 migration with lookup do fn by @RustedBones in #4695
• Document ScioIO methods, fixes #3937 by @kellen in #4689
• Update SBF docs, fixes #4528 by @kellen in #4693

🌱 Dependency Updates

• Update scalafmt-core to 3.7.1 by @scala-steward in #4682
• Update sbt-jmh to 0.4.4 by @scala-steward in #4688
• Use slf4j version from beam by @RustedBones in #4696

Full Changelog: v0.12.3...v0.12.4

v0.12.3

🐛 Bug Fixes

• ParquetBucketMetadata should not write null secondary keys by @clairemcginty in #4684

🌱 Dependency Updates

• Update mysql-socket-factory to 1.9.0 by @scala-steward in #4681
• Update elasticsearch-java to 8.6.1 by @scala-steward in #4680
• Update sbt-mdoc to 2.3.7 by @scala-steward in #4685

Full Changelog: v0.12.2...v0.12.3

v0.12.2

This release contains a bugfix and a bump to Beam 2.44.0.

🚀 Enhancements

• scalafix for the rest of the BQ API changes by @farzad-sedghi in #4625
• Improve style of REPL test by @shnapz in #4629

🐛 Bug Fixes

• Fix SMB producer compatibility issues by @clairemcginty in #4667
• Respecting required field of inner case classes in Coder macro by @shnapz in #4645
• fix bq lib import for scalafix by @farzad-sedghi in #4666

📗 Documentation

• Update Parquet docs to add example of case-class predicates by @clairemcginty in #4658

🌱 Dependency Updates

• Update beam to 2.44 by @RustedBones in #4668
• Update featran-core, featran-scio, ... to 0.8.0 by @scala-steward in #4661
• Update elasticsearch-java to 8.6.0 by @scala-steward in #4651
• Update elasticsearch-java to v7.17.8 by @RustedBones in #4649
• Update sbt-bloop to 1.5.6 by @scala-steward in #4631
• Update sbt-assembly to 2.1.0 by @scala-steward in #4632
• Update mysql-socket-factory to 1.8.3 by @scala-steward in #4660
• Update mysql-connector-java to 8.0.32 by @scala-steward in #4662
• Update sbt-tpolecat to 0.4.2 by @scala-steward in #4646
• Update sbt, sbt-dependency-tree to 1.8.2 by @scala-steward in #4647
• Update scalactic to 3.2.15 by @scala-steward in #4648
• Update neo4j-java-driver to 4.4.11 by @scala-steward in #4637
• Update scalafmt-core to 3.7.0 by @scala-steward in #4663
• Update sparkey to 3.2.5 by @scala-steward in #4654
• Update magnolia to 1.1.3 by @scala-steward in #4656
• Update circe-core, circe-generic, ... to 0.14.3 by @scala-steward in #4524
• Update tensorflow to 0.4.2 by @RustedBones in #4669
• Update breeze to 2.1.0 by @RustedBones in #4670
• Update cats-core to 2.9.0 by @RustedBones in #4671
• Update scalatest to 3.2.15 by @RustedBones in #4672
• Update algebra to 2.9.0 by @RustedBones in #4673

Full Changelog: v0.12.1...v0.12.2

v0.11.14

🚀 Enhancements

• Added SA impersonation for BigQuery client (#4616)
• Exposed SMB ExternalSorter tempLocation as System Property (#4618)

🐛 Bug Fixes

• Fix SMB metadata compatibility issues (#4617)

🌱 Dependency Updates

• Update pprint to 0.8.1 (#4613)
• Update sbt-ci-release to 1.5.11 (#4559)
• Update elasticsearch-java to 8.5.2 (#4610)

Full Changelog: v0.11.13...v0.11.14

v0.12.1

This release contains mostly enhancements, dependency upgrades and bug fixes. It includes Beam 2.43.0.

🚀 Enhancements

• PubsubIO: expose more params & fix passing of write params (#4615) @Illedran
• Exposed ExternalSorter tempLocation as System Property (#4618) @shnapz
• Added SA impersonation for BigQuery client (#4616) @shnapz

🐛 Bug Fixes

• Fix consistent casing for jlist (#4611) @RustedBones
• Fix SMB metadata compatibility issues (#4617) @clairemcginty
• Scalafix 0.12.0 bug fixes (#4581) @farzad-sedghi
• Do not wrap already nullable coders (#4603) @RustedBones

📗 Documentation

• Detailed release notes for 0.12.0 release (#4599) @clairemcginty
• update Beam version matrix (#4579) @clairemcginty
• update release manual for scalafix rules (#4582) @farzad-sedghi

🏗️ Build Improvements

• Bump google-github-actions/auth from 0 to 1 (#4594) @dependabot
• Bump google-github-actions/setup-gcloud from 0 to 1 (#4593) @dependabot
• Add manually-triggerable GHA workflow for publishing GH site (#4580) @clairemcginty
• set project ID env for publish-gh-site GHA workflow (#4583) @clairemcginty

🌱 Dependency Updates

• Update beam to v2.43.0 (#4607) @RustedBones
• Update scalafmt-core to 3.6.1 (#4589) @scala-steward
• Update scala-collection-compat to 2.9.0 (#4614) @scala-steward
• Update neo4j-java-driver to 4.4.10 (#4624) @scala-steward
• Update sbt-bloop to 1.5.5 (#4621) @scala-steward
• Update sbt-ci-release to 1.5.11 (#4559) @scala-steward
• Update joda-time to 2.12.2 (#4623) @scala-steward
• Update testcontainers-scala-elasticsearch, ... to 0.40.12 (#4622) @scala-steward
• Update pprint to 0.8.1 (#4613) @scala-steward
• Update elasticsearch-java to 8.5.2 (#4610) @scala-steward
• Update sbt-scalafmt to 2.5.0 (#4601) @scala-steward
• Update sbt-header to 5.9.0 (#4606) @scala-steward
• Update sbt-scoverage to 2.0.6 (#4602) @RustedBones
• Update sbt, sbt-dependency-tree to 1.8.0 (#4600) @scala-steward
• Update sbt-mdoc to 2.3.6 (#4574) @scala-steward
• Update sbt-java-formatter to 0.8.0 (#4562) @scala-steward
• Update scalactic to 3.2.14 (#4573) @scala-steward
• Update mysql-socket-factory to 1.7.2 (#4586) @scala-steward
• Update commons-text to 1.10.0 (#4569) @scala-steward
• Update mysql-connector-java to 8.0.31 (#4566) @scala-steward
• Update sbt-paradox to 0.10.3 (#4561) @scala-steward
• Update sbt-assembly to 2.0.0 (#4558) @scala-steward
• Update cassandra-driver-core to 3.11.3 (#4556) @scala-steward
• Remove scala-xml override (#4612) @RustedBones

Contributors to this release

@Illedran, @RustedBones, @clairemcginty, @dependabot, @dependabot[bot], @farzad-sedghi, @scala-steward and @shnapz

v0.11.13

This release reverts the update of magnolia v1 which was binary incompatible.
Other non-breaking bug-fixes and updates from v0.12.0 are back-ported to this release.

🚀 Enhancements

• Add dependency graph workflow by @kellen in #4467
• make it more clear how to run mdoc by @farzad-sedghi in #4487
• optional project and dataset for mock bq by @farzad-sedghi in #4474
• Cross compile IT tests by @RustedBones in #4491
• support 1-to-n mapping function for transformOverride by @farzad-sedghi in #4484
• Enforce javafmt and headers by @RustedBones in #4520
• Changed processElement to newer annotated style in scio-avro, scio-core by @shnapz in #4511
• Final refactoring of @processElement to bring new style by @shnapz in #4529
• Remove unused jna dependency by @RustedBones in #4432
• Reduce warnings by @RustedBones in #4381
• IT + FAQ cleanup by @kellen in #4448

🐛 Bug Fixes

• Fix example page (#4534) @RustedBones
• Fix NullPointerException in DataflowResult (#4538) @bergman
• Add newline for list by @kellen in #4460
• update run instructions for examples by @farzad-sedghi in #4468
• Fix: Capturing ProcessContext in AsyncLookupDoFn leads to NPE in Runner V2 by @Illedran in #4495
• Fix documentation generation by @RustedBones in #4517
• Fix example page using showdown by @RustedBones in #4533
• Fix google-api-client evicted by mysql-socket-factory 1.7.0 by @RustedBones in #4531
• (bugfix) typed-Parquet sink wasn't using user-supplied Configuration by @clairemcginty in #4434
• Patch startRegex for es 8.3+ by @RustedBones in #4444
• Remove org.apache documentation exclusion by @kellen in #4330
• staging: allow .sbt/boot by @danielnorberg in #4262

🌱 Dependency Updates

• Revert Magnolify to 0.4.8 (#4422) @RustedBones
• Update sbt-bloop to 1.5.4 (#4551) @scala-steward
• Update elasticsearch 7 to 7.17.6 (#4554) @RustedBones
• Update elasticsearch-java to 8.4.3 (#4553) @scala-steward
• Update sbt-scalafix to 0.10.4 (#4552) @scala-steward
• Bump JamesIves/github-pages-deploy-action from 4.4.0 to 4.4.1 (#4539) @dependabot
• Update joda-time to 2.11.1 (#4508) @scala-steward
• Update scala-collection-compat to 2.8.0 by @scala-steward in #4456
• Update sbt, sbt-dependency-tree to 1.7.1 by @scala-steward in #4457
• Update testcontainers-scala-elasticsearch, ... to 0.40.9 by @scala-steward in #4465
• Update mysql-connector-java to 8.0.30 by @scala-steward in #4473
• Update sbt-mdoc to 2.3.3 by @scala-steward in #4475
• Update scala-collection-compat to 2.8.1 by @scala-steward in #4479
• Update sbt-tpolecat to 0.4.1 by @scala-steward in #4472
• Update testcontainers-scala-elasticsearch, ... to 0.40.10 by @scala-steward in #4480
• Update sbt-bloop to 1.5.3 by @scala-steward in #4477
• Update scalatest to 3.2.13 by @scala-steward in #4482
• Update sparkey to 3.2.4 by @spotify-steward in #4408
• Update scalactic to 3.2.13 by @scala-steward in #4481
• Update sbt-avro to 3.4.2 by @scala-steward in #4485
• Update sbt-scoverage to 2.0.2 by @scala-steward in #4493
• Update scalafmt-core to 3.5.9 by @scala-steward in #4499
• Bump JamesIves/github-pages-deploy-action from 4.3.4 to 4.4.0 by @dependabot in #4469
• Update elasticsearch-java to 8.4.1 by @scala-steward in #4510
• Update sbt-protoc to 1.0.6 by @scala-steward in #4507
• Update kantan.csv to 0.7.0 by @scala-steward in #4516
• Update shapeless to 2.3.10 by @scala-steward in #4518
• Update sbt-scalafix to 0.10.2 by @scala-steward in #4521
• Update sbt-mima-plugin to 1.1.1 by @scala-steward in #4523
• Update scalacheck to 1.17.0 by @scala-steward in #4526
• Update scala-compiler, scala-library, ... to 2.12.17 by @scala-steward in #4525
• Sync jackson version with beam's by @RustedBones in #4530
• Update mysql-socket-factory to 1.7.0 by @scala-steward in #4519
• Bump JamesIves/github-pages-deploy-action from 4.3.3 to 4.3.4 by @dependabot in #4431
• Update sbt-scoverage to 2.0.0 by @spotify-steward in #4430
• Update elasticsearch-java to 8.3.1 by @spotify-steward in #4441
• Update sbt-bloop to 1.5.2 by @spotify-steward in #4440
• Update to elasticsearch 7.17.5 by @RustedBones in #4443

Contributors to this release

@RustedBones, @bergman, @clairemcginty, @dependabot, @dependabot[bot], @farzad-sedghi, and @scala-steward

v0.12.0

This release builds on v0.12.0-RC1 and v0.12.0-RC2 and uses Beam 2.41.0.

See our Migration Guide for info on breaking changes.

🚀 Enhancements

v0.12.0

• Concurrently support legacy Parquet reads for 0.12.x (#4555, #4578) @clairemcginty
• Provide java.util.UUID Coder (#4546) @RustedBones
• setup scalafix (#4504) @farzad-sedghi

v0.12.0-RC2

• Update version util w/ migration guide, add developer info by @kellen in #4450
• Add dependency graph workflow by @kellen in #4467
• Migrate to magnolia1 by @RustedBones in #4422
• make it more clear how to run mdoc by @farzad-sedghi in #4487
• optional project and dataset for mock bq by @farzad-sedghi in #4474
• Cross compile IT tests by @RustedBones in #4491
• Improve KV batch API by @RustedBones in #4458
• support 1-to-n mapping function for transformOverride by @farzad-sedghi in #4484
• Update beam-runners-core-construction-java, ... to 2.41.0 by @scala-steward in #4506
• BatchDoFn and sio batch API on SCollection by @RustedBones in #4489
• Enforce javafmt and headers by @RustedBones in #4520
• Do not reference scio coder after materialization by @RustedBones in #4509
• Changed processElement to newer annotated style in scio-avro, scio-core by @shnapz in #4511
• Support Neo4jIO by @turb in #4488
• Implement gRPC lookup API by @RustedBones in #4492
• IO Windowing by @kellen in #4461
• Final refactoring of @processElement to bring new style by @shnapz in #4529
• Migrate from sbt-protobuf to sbt-protoc by @RustedBones in #4483

v0.12.0-RC1

• SMB secondary sort key by @kellen in #4288
• Remove unused jna dependency by @RustedBones in #4432
• Cherry-pick Parquet SplittableDoFn implementation. by @clairemcginty in #4436
• Add granularity option for Parquet-SDF reads by @clairemcginty in #4438
• Reduce coder memory footprint by @RustedBones in #4397
• Reduce warnings by @RustedBones in #4381
• Update beam to 2.40 by @RustedBones in #4433
• Remove deprecated APIs by @kellen in #4446
• Add 0.12.0 version information, deprecations, pubsub IO information by @kellen in #4447
• IT + FAQ cleanup by @kellen in #4448

🐛 Bug Fixes

v0.12.0

• Fix NullPointerException with extendedDebugInfo=true (#4550) @RustedBones
• Fix record coder structural value equality (#4547) @RustedBones
• Detect and retrieve GCP temp location (fixes #4535) (#4536) @turb
• Fix example page (#4534) @RustedBones
• Fix NullPointerException in DataflowResult (#4538) @bergman

v0.12.0-RC2

• Add newline for list by @kellen in #4460
• Fix smb secondary key test by @kellen in #4471
• update run instructions for examples by @farzad-sedghi in #4468
• Fix: Capturing ProcessContext in AsyncLookupDoFn leads to NPE in Runner V2 by @Illedran in #4495
• Propagate internal coder in tap() by @RustedBones in #4500
• Fix documentation generation by @RustedBones in #4517
• Fix example page using showdown by @RustedBones in #4533
• Fix google-api-client evicted by mysql-socket-factory 1.7.0 by @RustedBones in #4531

v0.12.0-RC1

• (bugfix) typed-Parquet sink wasn't using user-supplied Configuration by @clairemcginty in #4434
• Patch startRegex for es 8.3+ by @RustedBones in #4444
• Remove org.apache documentation exclusion by @kellen in #4330
• staging: allow .sbt/boot by @danielnorberg in #4262

🌱 Dependency Updates

v0.12.0

• Update Magnolify to 0.6.2 (#4577) @clairemcginty
• Update sbt-bloop to 1.5.4 (#4551) @scala-steward
• Update elasticsearch 7 to 7.17.6 (#4554) @RustedBones
• Update elasticsearch-java to 8.4.3 (#4553) @scala-steward
• Update sbt-scalafix to 0.10.4 (#4552) @scala-steward
• Bump JamesIves/github-pages-deploy-action from 4.4.0 to 4.4.1 (#4539) @dependabot
• Update joda-time to 2.11.1 (#4508) @scala-steward

v0.12.0-RC2

• Update scala-collection-compat to 2.8.0 by @scala-steward in #4456
• Update sbt, sbt-dependency-tree to 1.7.1 by @scala-steward in #4457
• Update testcontainers-scala-elasticsearch, ... to 0.40.9 by @scala-steward in #4465
• Update mysql-connector-java to 8.0.30 by @scala-steward in #4473
• Update sbt-mdoc to 2.3.3 by @scala-steward in #4475
• Update scala-collection-compat to 2.8.1 by @scala-steward in #4479
• Update sbt-tpolecat to 0.4.1 by @scala-steward in #4472
• Update testcontainers-scala-elasticsearch, ... to 0.40.10 by @scala-steward in #4480
• Update sbt-bloop to 1.5.3 by @scala-steward in #4477
• Update scalatest to 3.2.13 by @scala-steward in #4482
• Update sparkey to 3.2.4 by @spotify-steward in #4408
• Update scalactic to 3.2.13 by @scala-steward in #4481
• Update sbt-avro to 3.4.2 by @scala-steward in #4485
• Update sbt-scoverage to 2.0.2 by @scala-steward in #4493
• Update scalafmt-core to 3.5.9 by @scala-steward in #4499
• Bump JamesIves/github-pages-deploy-action from 4.3.4 to 4.4.0 by @dependabot in #4469
• Update elasticsearch-java to 8.4.1 by @scala-steward in #4510
• Update sbt-protoc to 1.0.6 by @scala-steward in #4507
• Update kantan.csv to 0.7.0 by @scala-steward in #4516
• Update shapeless to 2.3.10 by @scala-steward in #4518
• Update sbt-scalafix to 0.10.2 by @scala-steward in #4521
• Update sbt-mima-plugin to 1.1.1 by @scala-steward in #4523
• Update scalacheck to 1.17.0 by @scala-steward in #4526
• Update scala-compiler, scala-library, ... to 2.12.17 by @scala-steward in #4525
• Sync jackson version with beam's by @RustedBones in #4530
• Update mysql-socket-factory to 1.7.0 by @scala-steward in #4519

v0.12.0-RC1

• Bump JamesIves/github-pages-deploy-action from 4.3.3 to 4.3.4 by @dependabot in #4431
• Update sbt-scoverage to 2.0.0 by @spotify-steward in #4430
• Update elasticsearch-java to 8.3.1 by @spotify-steward in #4441
• Update sbt-bloop to 1.5.2 by @spotify-steward in #4440
• Update to elasticsearch 7.17.5 by @RustedBones in #4443

Contributors to this release

@RustedBones, @bergman, @clairemcginty, @dependabot, @dependabot[bot], @farzad-sedghi, @scala-steward and @turb

v0.11.12

⚠️ This release is binary incompatible. If your project have other dependencies on scio 0.11, skip this release and update to 0.11.13

🐛 Bug Fixes

• Fix NullPointerException with extendedDebugInfo=true (#4550) @RustedBones

Contributors to this release

@RustedBones