allow upgrading keys during re-interpretation #19792
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
remyhaemmerle-da
approved these changes
Aug 15, 2024
paulbrauner-da
force-pushed
the
pb/fix-exercise-by-key-upgrades
branch
from
7fb9c8d
to
4c3e95e
Compare
dylant-da
added a commit
that referenced
this pull request
Oct 3, 2024
* Add support for 'module_prefixes' in bazel macro 'daml_test' (#19011) * update canton to 20240416.11953.0.vf10e7795/2.9.0-snapshot.20240416.11953.0.vf10e7795 in main-2.x (#19016) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * update NOTICES file (#19024) Co-authored-by: Azure Pipelines Daml Build <[email protected]> * ledger api test tool tests for party paging (#18955) * ledger api test tool tests for party management paging * fix the build * exclude incompatible cantons from test * Fix the tool after latest canton upgrade * test paging in http json * reduce the page size * [LF] Freeze 1.16 (#19012) Co-authored-by: Paul Brauner <[email protected]> * Fix crash in JSON API test related to page sizes (#19026) * update canton to 20240417.11957.0.v757c5ff4/2.9.0-snapshot.20240417.11957.0.v757c5ff4 in main-2.x (#19029) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * add support of LF 1.16 in canton (#19031) * add support of LF 1.16 in canton * run-all-tests: true * force all tests run-all-tests: true * [2.9] re-enable Windows testing (#19035) If anyone's been wondering why we've had so few Windows failures lately, this is why. * update NOTICES file (#19044) Co-authored-by: Azure Pipelines Daml Build <[email protected]> * [2.9] remove check-proto workaround for 2.8.3 (#19047) The workaround was in place for the transition period, as 2.8.3 was not in a subfolder. * update canton to 20240418.11962.0.v72284101/2.9.0-snapshot.20240418.11962.0.v72284101 in main-2.x (#19043) Co-authored-by: Azure Pipelines Daml Build <[email protected]> Co-authored-by: Remy <[email protected]> * Simplify paging test case for too large page request (#19055) * update canton to 20240419.11970.0.vc58d8271/2.9.0-snapshot.20240419.11970.0.vc58d8271 in main-2.x (#19056) * update canton to 20240419.11970.0.vc58d8271/2.9.0-snapshot.20240419.11970.0.vc58d8271 in main-2.x tell-slack: canton * Add package-name to exercised event * fix build --------- Co-authored-by: Azure Pipelines Daml Build <[email protected]> Co-authored-by: Simon Maxen <[email protected]> Co-authored-by: Paul Brauner <[email protected]> * update canton to 20240423.11976.0.v8dcd78fe/2.9.0-snapshot.20240423.11976.0.v8dcd78fe in main-2.x (#19086) * update canton to 20240423.11976.0.v8dcd78fe/2.9.0-snapshot.20240423.11976.0.v8dcd78fe in main-2.x tell-slack: canton * fix the config of the test canton sandbox * fix the canton config used by 'daml start' --------- Co-authored-by: Azure Pipelines Daml Build <[email protected]> Co-authored-by: Paul Brauner <[email protected]> * update canton to 20240424.11979.0.v3e84726c/2.9.0-snapshot.20240424.11979.0.v3e84726c in main-2.x (#19093) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * update NOTICES file (#19096) Co-authored-by: Azure Pipelines Daml Build <[email protected]> * update canton to 20240425.11981.0.v037dbecd/2.9.0-snapshot.20240425.11981.0.v037dbecd in main-2.x (#19099) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * update canton to 20240426.11984.0.v4361cdbe/2.9.0-snapshot.20240426.11984.0.v4361cdbe in main-2.x (#19105) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * [2.9] fix compat update script, run it (#19053) * [2.9] fix compat update script, run it * hlint * JSON API: support package name with multiple package ids (#18855) Package name support affects how we use ContractTypeId's (CTIDs), which correspond to template ids and interface ids. A CTID has three pieces: package id, module name, entity name. With the introduction of package names, the "package id" component, may instead contain a package name, which is distinguished by a "#" prefix. In that case, the CTID conceptually refers to multiple package-specific CTIDs, for the relevant template in any/all of the packages of that name, in which the template was defined. In JSON API, these CTIDs are used in various ways: * Create and Exercise commands * The CTID is a part of the input. * We pass that id unchanged through to the ledger, which has its own internal handling of package names. * We also need to interpret the JSON sent to us in the request payload for the contract body or choice arguments. If they use a CTID with a package name, then we need to use the most recent package with that name. The most recent package is the one with the highest version number, and will always be backwards-compatible. * Fetches and Queries * A query may be for multiple CTIDs, which themselves, if they use package names, may correspond to a template across multiple packages. * Query requests can contain a JSON payload specifying a filter. We need to interpret that JSON, again using the most recent package per name * Queries can run with or without a db cache * Without cache (aka "in memory"), we do a call against the ledger and filter the results in memory before returning them * For the ledger call, we can pass the input CTIDs through unchanged. The ledger understands package names. * For the filtering * we'll need to interpret the JSON query filter as per the most recent package for the template * results will come back with a package id in their template id and we'll need to apply filters based on that * With cache, we ask the ledger for an update based on a previous cache offset, then execute a query against the updated cache * In the cache we have the concept of a SurrogateTpId, which is how offsets are tracked. These map to specific templates in specific packages, i.e. use package ids, not names. Offsets are tracked per SurrogateTpId, so to preserve this behaviour we keep populating and querying the cache needs to be in terms of package-id CTIDs, after resolving package names to potentially multiple package ids. * However, also note that we still need to propagate package name information down to the cache layer, as it affects how the contract key hash is calculated. Currently we propagate this package name info down everywhere we use ".expand", but we could probably just allow ContractsFetch to be able to lookup (in-mem) the package name for a package id. * Running a query with cache involves two steps: update cache from ledger, then query cache directly. * Update cache involves checking the current cache offsets, per SurrogateTpId, which means breaking out CTIDs per package. Once we've figured out who needs to be updated, the actualy call against the ledger can be done with the original CTID, but we deal with the reults via per-package CTID * querying against the cache then also needs to deal with the per-package CTIDs * query results will have the package-id-specific CTID in their results, even if the query used a package name In this PR, the main type that we added is the `ContractTypeRef`, which enhances a CTID with information about the package name, all the corresponding package-specific CTIDs and the latest one. There are also some potential behaviour impacts from this change. * Information about new packages is refreshed periodically, but not on every query. So if a new package is uploaded for a given name, JSON API may not be aware of it until the configured expiry time, which is currently hardcoded to 60s. * If there is a very large number of package ids per package name, it may increase memory requirements, and reduce query performance. * update canton to 20240429.11985.0.vce2948f3/2.9.0-snapshot.20240429.11985.0.vce2948f3 in main-2.x (#19111) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * update NOTICES file (#19115) Co-authored-by: Azure Pipelines Daml Build <[email protected]> * Update compat versions 2024 04 30 main 2.x (#19117) * update compat versions (2024-04-30 main-2.x) * possibly fix something --------- Co-authored-by: Azure Pipelines Daml Build <[email protected]> * fix main-2.x conformance tests by explicitly setting protocol-version=5 (#19122) * Try to fix conformance tests by explicitly setting protocol-version=5 run-all-tests: true * run problematic tests --------- Co-authored-by: Gary Verhaegen <[email protected]> * update canton to 20240430.11989.0.vefbe78d0/2.9.0-snapshot.20240430.11989.0.vefbe78d0 in main-2.x (#19125) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * update NOTICES file (#19133) Co-authored-by: Azure Pipelines Daml Build <[email protected]> * Switch more tests to LF 1.16 (#19118) * Fix LF-encoder so it reads metadata (#19140) * [Upgrade] static check that LF versions do not decrease while upgrading (#19070) * [2.9] update release instructions (#19136) * update canton to 20240502.11993.0.v9dba0ecb/2.9.0-snapshot.20240502.11993.0.v9dba0ecb in main-2.x (#19147) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * install.{bat,sh} pass --install-with-custom-version with contents of daml_version.txt (#19143) (#19153) * install.{bat,sh} pass --install-with-custom-version with contents of daml_version.txt * pass flag only if file exists (bash) * install.bat only passes --install-with-custom-version if daml_version.txt exists * daml assistant accepts multiple --install-with-custom-version options, but only uses the last one (if given) --------- Co-authored-by: Moisés Ackerman <[email protected]> * Allow injecting custom logger into engine (#19051) (#19154) Before this change, we couldn't inject a custom logger into the engine. As such, debug output of commands would not be associated to the Canton trace-id, making the feature not very useful. Now, the API is extended such that the custom logger can be easily added. * update canton to 20240503.11997.0.vefb6ebc9/2.9.0-snapshot.20240503.11997.0.vefb6ebc9 in main-2.x (#19155) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * update canton to 20240506.12000.0.vb7a95cce/2.9.0-snapshot.20240506.12000.0.vb7a95cce in main-2.x (#19159) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * update canton to 20240507.12004.0.v51ae606a/2.9.0-snapshot.20240507.12004.0.v51ae606a in main-2.x (#19163) * update canton to 20240507.12004.0.v51ae606a/2.9.0-snapshot.20240507.12004.0.v51ae606a in main-2.x tell-slack: canton * fix canton fixture config --------- Co-authored-by: Azure Pipelines Daml Build <[email protected]> Co-authored-by: Paul Brauner <[email protected]> * JSON API Upgrades: ETX-500 require fully qualified template ids (#19149) - update all instances of ContractTypeId.OptionalPkg to ContractTypeId.RequiredPkg - removed ContractTypeId.OptionalPkg and ContractTypeId.NoPkg type aliases entirely - update tests to use fully qualified template ids, i.e. including a package reference - update integration tests which do json-api queries to include a package reference in template ids We still need to distinguish package name vs package id at the point where we resolve contract type ids, so we keep the logic that we used on missing package id, and do the same thing in the case of a package name. This logic deals with the fact that new packages can get uploaded in the background, and so we need to be able to treat a package name as a dynamically late-bound mapping to specific package. * update canton to 20240509.12007.0.v5868a35d/2.9.0-snapshot.20240509.12007.0.v5868a35d in main-2.x (#19171) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * daml script runner fails sooner if provided ledger host lacks a protocol when using the JSON API (#19175) * update canton to 20240510.12009.0.vacced302/2.9.0-snapshot.20240510.12009.0.vacced302 in main-2.x (#19179) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * [damlc] set upgrade feature to 1.16 (#19183) * update canton to 20240513.12012.0.v71e399c2/2.9.0-snapshot.20240513.12012.0.v71e399c2 in main-2.x (#19186) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * Backport com.daml.lf.engine.script.test.DarUtil (#19188) * update NOTICES file (#19205) Co-authored-by: Azure Pipelines Daml Build <[email protected]> * update canton to 20240514.12014.0.vb429d088/2.9.0-snapshot.20240514.12014.0.vb429d088 in main-2.x (#19201) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * Fix disclosure bug in the presence of upgrades in 2.x (#19199) * Backport pull/19167 * fix integration test * [2.9] restore ability to make releases (#19210) Releases were broken by [#19149] because package IDs include the SDK version in what they hash. [#19149]: https://github.com/digital-asset/daml/pull/19149 * backport [Speedy] recompute contract metadata when using different pkg versions #19206 (#19209) * Restructure daml3-script such that everything is internal. (#19207) * [3.1] nix: remove unused xmlstarlet (#19219) * Daml-Script changes for 2.9 (#19225) * First set of tweaks * Implement package preference in daml3-script (without tests) * Support `daml-script-beta` as well as `daml3-script` * Update sdk/daml-script/daml3/Daml/Script/Internal/Questions/Submit.daml Co-authored-by: Paul Brauner <[email protected]> --------- Co-authored-by: Paul Brauner <[email protected]> * HOTFIX: remove `sdk/` prefix from paths when running `./fmt.sh --diff` (#19237) As a developer, we now run `./fmt.sh --diff` from within the `sdk/` dir, but git still returns a list of changed files with a sdk prefix. This change removes the sdk/ prefix from those paths. Incidentally, we were previously referring to `${changed_files[@]}` as though it were a bash array, but it was a regular scalar variable. This makes it actually an array, which allows us to use bash builtin syntax to strip the prefixes. * update NOTICES file (#19235) Co-authored-by: Azure Pipelines Daml Build <[email protected]> * Fix DAML_SDK_VERSION causing a network lookup (#19231) * Support split release artifactory, improve missing connection errors (#18271) * Add resolution via artifactory, handle HTTP errors * Replace origin with wrapErr in resolveReleaseVersionUnsafe * Handle broken connections (e.g. no internet) * drop comment, fix foldMap id to fold * remove debugging writeFile calls run-all-tests: true * Rename partial field damlPath to damlPathUnsafe and add safe function * support multiple possible installation locations * update alternatives * Wrap CouldNotResolveReleaseVersion into AssistantError in unsafe resolve * notify in install.sh * Fix issue with env var DAML_SDK_VERSION triggering a network call * drop print statements * Fix //daml-assistant:test by using unresolvedVersion where needed * make LF 1.16 stable (#19230) * JSON API: etx-500 add type information into the package ids of template ids (#19211) A template/interface id (`ContractTypeId`) consists of three parts: package, module and entity. So far the package portion has been modeled with a `String`, where that string could represent either a package name or a package id, depending on whether it started with a `"#"` prefix or not, respectively. However there are parts of the code, namely those that deal with the db cache, that are designed to work with contract type ids whose package portion is only a *package id*. The type system was not helping us to ensure this so far. Fortunately the `ContractTypeId` class is parameterised with the type of the package, so this PR replaces all existing instantiations which used `String` packages, with either [PackageId](https://github.com/digital-asset/daml/blob/7ffc096fa857700fe510c41f299c7ce06f5bbb4f/sdk/daml-lf/data/src/main/scala/com/digitalasset/daml/lf/data/Ref.scala#L37) or [PackageRef](https://github.com/digital-asset/daml/blob/7ffc096fa857700fe510c41f299c7ce06f5bbb4f/sdk/daml-lf/data/src/main/scala/com/digitalasset/daml/lf/data/Ref.scala#L292) (which is a sum type which can contain either a `PackageId` or a `PackageName`). The `ContractTypeRef` type contains all the package info we have about a contract type id. Its `original` field now returns a `ContractTypeId[PackageRef]`, whereas its `latestPkgId` returns a `ContractTypeId[PackageId]`. This helps us to ensure that we are passing around the right kind of package specifiers to the parts of the code that need one or the other. There are some type aliases used, that may be useful to help read and understand the changes, which tend to look like ``` type RequiredPkg = CtId[Ref.PackageRef] type RequiredPkgId = CtId[Ref.PackageId] ``` i.e. if it ends with Pkg it uses a `PackgeRef` and if it ends with PkgId uses a `PackageId`. Note also that the `Resolved` terminology is an orthogonal concern which refers to whether this id has been determined to belong to a template vs interface (i.e. it extends `Definite`), or whether that is not yet known. * run compatibility tests on jdk17 (#19249) * run compatibility tests on jdk17 * allow for some differences between .bazelrc and compatiblity/.bazelrc * do not copy .bazelrc from the parent dir * ETX-500: ensure types are always looked up via package id, not package name (#19245) tl;dr: The types exist in packages which are indexed by id, so we need to use package ids when doing type lookups. We load type information from packages, which internally identify themselves by ids. When a request comes in, it will specify a template id, and when decoding we internally use that id to look up the relevant type in the appropriate package, to know how to decode the JSON. However, now template ids can use a package name in as their package specifier instead of a package id, so we need to ensure that when doing the type lookups, we are always using the package id to find the relevant package. Concretely, this involves changing the keys for those lookups from accepting a `ContractTypeId[Ref.PackageRef]` to a `ContractTypeId[Ref.PackageId]`. run-all-tests: true * ETX-452: JSON API: package id selection preference (#19236) Allow passing the optional `packageIdSelectionPreference` to JSON API when sending commands. * update NOTICES file (#19254) Co-authored-by: Azure Pipelines Daml Build <[email protected]> * stop hardcoding lf2 stable packages in daml3-script (#19248) * stop hardcoding lf2 stable packages in daml3-script * avoid re-allocating many identical objects * fix bad search and replace * remove usused import * update NOTICES file (#19263) Co-authored-by: Azure Pipelines Daml Build <[email protected]> * Drop print statements (#19266) * Update daml-script-beta warning (#19280) * rename build-release.ps1 (#19287) With the wrong file name it's never used and Windows builds don't benefit from the shortened release process. * update NOTICES file (#19297) Co-authored-by: Azure Pipelines Daml Build <[email protected]> * ETX-453: Typescript bindings: support package names in commands (#19260) - run the "create + fetch & exercise" tests twice, once using package ids in the template ids, and again using package names - removed some cases where the decoder expected the template id in the response to exactly match the one in the request. Now the request template ids can contain package names, whereas the ones in the response will contain a package id. - Allow filtering tests in the build-and-lint tests with env var `BUILD_AND_LINT_TEST_NAME_PATTERN` Diff best viewed with "Hide whitespace" * update canton to 20240528.12044.0.ve8f70526/2.9.0-snapshot.20240528.12044.0.ve8f70526 in main-2.x (#19294) * update canton to 20240528.12044.0.ve8f70526/2.9.0-snapshot.20240528.12044.0.ve8f70526 in main-2.x tell-slack: canton * fix build * Temporarily disable daml start hot reload test * Some vetting behaviour was fixed * Locally implement https://github.com/DACH-NY/canton/commit/10deea37ce44e6bf6637dada33c5041c68702a16 to fix tests --------- Co-authored-by: Azure Pipelines Daml Build <[email protected]> Co-authored-by: Paul Brauner <[email protected]> Co-authored-by: Samuel Williams <[email protected]> * update NOTICES file (#19301) Co-authored-by: Azure Pipelines Daml Build <[email protected]> * update canton to 20240529.12047.0.ved95d3a1/2.9.0-snapshot.20240529.12047.0.ved95d3a1 in main-2.x (#19299) * update canton to 20240529.12047.0.ved95d3a1/2.9.0-snapshot.20240529.12047.0.ved95d3a1 in main-2.x tell-slack: canton * cherry-pick https://github.com/DACH-NY/canton/pull/19399 --------- Co-authored-by: Azure Pipelines Daml Build <[email protected]> Co-authored-by: Paul Brauner <[email protected]> * make Timestamp parsing consistent across Java versions (#19290) * update NOTICES file (#19306) Co-authored-by: Azure Pipelines Daml Build <[email protected]> * split //daml-lf/valiation/upgrade-tests (#19313) * update NOTICES file (#19310) Co-authored-by: Azure Pipelines Daml Build <[email protected]> * update canton to 20240530.12049.0.v53d78bbc/2.9.0-snapshot.20240530.12049.0.v53d78bbc in main-2.x (#19308) * update canton to 20240530.12049.0.v53d78bbc/2.9.0-snapshot.20240530.12049.0.v53d78bbc in main-2.x tell-slack: canton * sync canton at 4c10769ef9 --------- Co-authored-by: Azure Pipelines Daml Build <[email protected]> Co-authored-by: Paul Brauner <[email protected]> * Multi-ide 2.x Backports (#19312) * Daml ide multi ide (#17345) * Multi-ide refactors (#18885) * Reimplement ProgressToken prefixing to be stateless * Refactor parsing logic to not use interleaveIO * Move most verbose logging behind a flag * Address reviews * Multi-IDE Features for 2.9 (#19040) * Implement dar unpacking * Implement packageless IDE * Hot-reloading logic for daml.yaml, multi-package.yaml, *.dar * Implement initial error recovery logic * Switch logging to log levels Replace window reload with LanguageServer restart * Forward args from multi-ide to sub-ides * Change unpacked dar paths to be the unit-id. Update unpacking logic to shutdown previous IDEs * Remove broken experimental flag * Refactor ide restart logic to not lose event handlers * Log subIDE errors to debug logger live * Windows fixes * First review fixes batch * Use newtypes for many FilePaths * Address Dylan's comments * Refactor how SubIDEs are passed around, reduce times it is dropped * Update diagnostic * Multi-ide structure change handling (#19194) * Better support for package structure changes * Rename some handlers * Split up MultiIde.hs into many files (#19198) * Split up MultiIde.hs into many files * Further split SubIde * Multi-IDE Sdk install managament (#19256) * Tweaks from previous version * Implement Sdk Version management * Address review * Fix compilation issues * Fix issues with 2.x * update canton to 20240531.12055.0.v96e7106e/2.9.0-snapshot.20240531.12055.0.v96e7106e in main-2.x (#19317) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * update compat versions (2024-06-03 main-2.x) (#19322) Co-authored-by: Azure Pipelines Daml Build <[email protected]> * update canton to 20240603.12063.0.v3d4a09eb/2.9.0-snapshot.20240603.12063.0.v3d4a09eb in main-2.x (#19326) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * upgrades check all packages (#19328) * Upgrades validation for every package in dar * fix 1.15 error * transitivity is fine * remove unnecessary parens/braces as pointed out by IJ + use EitherUtil.IfThenET for consistency * add tests for dependency upgrade validation * put back message when skipping validation * address comments * comment to address Tudor's comment --------- Co-authored-by: Samuel Williams <[email protected]> * Upgrade validation compare deps pt1 (#19274) (#19330) * minimal initial changes from upgrade-validation-compare-deps * Add stub function for part 1 compilation Co-authored-by: dylant-da <[email protected]> * do not try to upgrade LF < 1.16 contract (#19315) * update canton to 20240604.12070.0.v106f9174/2.9.0-snapshot.20240604.12070.0.v106f9174 in main-2.x (#19334) * update canton to 20240604.12070.0.v106f9174/2.9.0-snapshot.20240604.12070.0.v106f9174 in main-2.x tell-slack: canton * upgrades check all packages (#19328) --------- Co-authored-by: Samuel Williams <[email protected]> --------- Co-authored-by: Azure Pipelines Daml Build <[email protected]> Co-authored-by: Paul Brauner <[email protected]> Co-authored-by: Samuel Williams <[email protected]> * update canton to 20240605.12074.0.v2a43b22c/2.9.0-snapshot.20240605.12074.0.v2a43b22c in main-2.x (#19339) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * Update docs.daml.com copyright notice (#19337) (#19348) * update canton to 20240606.12076.0.v93b6022d/2.9.0-snapshot.20240606.12076.0.v93b6022d in main-2.x (#19343) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * update canton to 20240607.12077.0.v80e280d7/2.9.0-snapshot.20240607.12077.0.v80e280d7 in main-2.x (#19352) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * Raise size of max inbound msg size - main 2.x (#19358) * Raise the default size of inbound message size in java bindings (#19325) * max inbound buffers in scala client and deafult ledger client * update canton to 20240610.12082.0.vdc01d11d/2.9.0-snapshot.20240610.12082.0.vdc01d11d in main-2.x (#19360) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * update canton to 20240612.12084.0.ve7073f12/2.9.0-snapshot.20240612.12084.0.ve7073f12 in main-2.x (#19370) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * [Compiler] types from the non stable part of stdlib are unserialiazable (#19345) first step to fix #19338 * ignore non serializable types during upgrade validation (#19356) * ignore non-serializable types when validating upgrades * Address comments * Update sdk/daml-lf/validation/src/main/scala/com/digitalasset/daml/lf/validation/Upgrading.scala Co-authored-by: Remy <[email protected]> --------- Co-authored-by: Remy <[email protected]> * [main-2.x][WIP] IDE Ledger support for upgrades (#19013) * Fix typo * Enable contract upgrading in v2 IdeLedgerClient * Handle implictly versioned (non-"exact") commands on the IDE ledger client * daml-script/runner: extract RunnerTestBase from RunnerMainTestBase * [WIP] port 'main' daml-script grpc ledger client upgrades test suite to test ide ledger * Fix various issues with ide-ledger upgrades * Fix CollectAuthority test * Appease buildifier * Allow upgrades to be enabled in daml2-script * Tweaks --------- Co-authored-by: Samuel Williams <[email protected]> * Force that upgrading types must belong to identically-named packages (#19357) * add failing test * Compare package names when comparing identifiers * also compare package versions * also compare qualified names when lf<1.16 * add a new test * Apply Remy's suggestion Co-authored-by: Remy <[email protected]> * make bazel targets clearer --------- Co-authored-by: Remy <[email protected]> * Switch `daml sandbox` to Domain PV6 (#19376) * Switch sandbox pv to 6 * Update canton definition to allow pv6 * Update sdk/daml-assistant/daml-helper/src/DA/Daml/Helper/Util.hs Co-authored-by: Remy <[email protected]> --------- Co-authored-by: Remy <[email protected]> * Backport Dylan's interface upgrade checks (#19380) * backport pull/19054 + some fixes * fix remaining failing tests * simplify bazel file * update canton to 20240613.12086.0.vd203a6f8/2.9.0-snapshot.20240613.12086.0.vd203a6f8 in main-2.x (#19377) * update canton to 20240613.12086.0.vd203a6f8/2.9.0-snapshot.20240613.12086.0.vd203a6f8 in main-2.x tell-slack: canton * PackageUpgradeValidator patch from canton see second commit of https://github.com/DACH-NY/canton/pull/19630 --------- Co-authored-by: Azure Pipelines Daml Build <[email protected]> Co-authored-by: oliverse-da <[email protected]> * update canton to 20240614.12093.0.v2fccc5aa/2.9.0-snapshot.20240614.12093.0.v2fccc5aa in main-2.x (#19383) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * update NOTICES file (#19386) Co-authored-by: Azure Pipelines Daml Build <[email protected]> * Triggers should rely on the ledger to manager max message size (#19359) * Script should rely on the ledger to manager max message size (#19373) * [LF] make LF 1.16 preview (#19391) this partially revert #19230 * Ignore "utility packages" in upgrade checks (#19392) * Add utility package upgrade check * Add a test that uploading the std lib for 2 different LF versions shouldn't fail * Fix tests * Fix invalid package preferences --------- Co-authored-by: Paul Brauner <[email protected]> * Run canton in dev mode in 'daml start' (#19399) * Revert 'disable known-broken json-api ee tests' (#18570) * Revert 'disable known-broken json-api ee tests' This reverts commit fe5f33f93f3d68035877c5b35c17a31646466ae8. run-all-tests: true * Add Foo dependencies * reformat * disable hot reload for LF versions that support upgrades (#19404) * disable hot reload for LF versions that support upgrades * Fix dar decoding --------- Co-authored-by: Samuel Williams <[email protected]> * sync canton code drop at b521d81b5f30ce43fba6d47fd07bce80056e3100 (#19401) * sync canton code drop at db5820a1f1c550ba59f5fb1c0fb0276dd5b318b7 * fix build * fix it harder * fix even more * remove unused dep * sync code drop to b521d81b5f30ce43fba6d47fd07bce80056e3100 * default to preview mode in 'daml start' (#19403) * [2.9] remove gary (#19390) * Fix up NonRepudiation test (#19410) run-all-tests: true * update NIGHTLY_PREFIX (#19416) * sync canton code drop to bbffb4cc3b82614d30a032ef5a7971e3e1b0cf11 (#19418) * sync canton code drop to bbffb4cc3b82614d30a032ef5a7971e3e1b0cf11 * fix build * Pipe GrpcStatus error metadata through to upload dar (#19421) * Pipe GrpcStatus error metadata through to upload dar * Fix canton build file * Make compiler typecheck upgrades by default (#19028) (#19422) Co-authored-by: dylant-da <[email protected]> * upload dry run (#19417) * add a dry-run flag to upload-dar * test that dry-run does not upload * update canton to 20240620.12111.0.v28461729/2.9.0-snapshot.20240620.12111.0.v28461729 in main-2.x (#19425) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * Add a test that dry-run reports validation erros. (#19429) * Ignore 2.8.7 in UpdateVersions for compatibility tests (#19440) * Ignore 2.8.7 in UpdateVersions for compatibility tests, as it uses 2.8.6 artifacts * update canton to 20240624.12118.0.v89fd32f0/2.10.0-snapshot.20240624.12118.0.v89fd32f0 in main-2.x (#19441) * update canton to 20240624.12118.0.v89fd32f0/2.10.0-snapshot.20240624.12118.0.v89fd32f0 in main-2.x tell-slack: canton * preview -> beta --------- Co-authored-by: Azure Pipelines Daml Build <[email protected]> Co-authored-by: Remy Haemmerle <[email protected]> * Fix non-internal deps for split release (#19448) run-full-compat: true * Pipe upload error to dry-run (#19455) * Pipe upload error to dry-run * Update dry-run test to check for reason * Java Codegen: port PR #19381 to `main-2.x` branch (#19454) Generates package name and package version in Java source. * update NOTICES file (#19463) Co-authored-by: Azure Pipelines Daml Build <[email protected]> * update canton to 20240626.12123.0.vb9281105/2.10.0-snapshot.20240626.12123.0.vb9281105 in main-2.x (#19461) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * update canton to 20240627.12126.0.v02972334/2.10.0-snapshot.20240627.12126.0.v02972334 in main-2.x (#19473) * update canton to 20240627.12126.0.v02972334/2.10.0-snapshot.20240627.12126.0.v02972334 in main-2.x tell-slack: canton * make it compile * Another fix --------- Co-authored-by: Azure Pipelines Daml Build <[email protected]> Co-authored-by: Marcin Ziolek <[email protected]> * update canton to 20240628.12129.0.v850d6fb1/2.10.0-snapshot.20240628.12129.0.v850d6fb1 in main-2.x (#19475) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * Minimal ts upgrades support (#19482) * update canton to 20240701.12131.0.v3a2ce61b/2.10.0-snapshot.20240701.12131.0.v3a2ce61b in main-2.x (#19487) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * update canton to 20240702.12133.0.vceeef1dd/2.10.0-snapshot.20240702.12133.0.vceeef1dd in main-2.x (#19494) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * Add pruning offset tests to LAPITT (#19470) * Ignore utility packages in ts codegen (#19490) * update canton to 20240703.12135.0.v7d82e992/2.10.0-snapshot.20240703.12135.0.v7d82e992 in main-2.x (#19502) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * Backport of conformance tests pertaining to the TRANSACTION_NOT_FOUND bugfix (#19467) * Backport conformance tests for bugfix #19707 * Implementation adapted from https://github.com/DACH-NY/canton/pull/19777 * Adapt conformance test to 2.x Ledger API * Added conformance test exclusions * [TS] allows undefined to represent input of Optional (#19505) in order to all ts generated code to upgrade template got by query * Java Codegen - JSON decoding: backport #19499 to main-2.x (#19507) See https://github.com/digital-asset/daml/pull/19499 * Improve resolutuion for KNWON_DAR_VERSION (#19511) * Addtional test multi package with different target (#19512) * added test to query multiple packages with different target to ensure json api behave as expected * formatting * fixed test for compatibility addressed comments * formatting * removing warning from unused identifier * removing warning from unused identifier * text comments modified from feedback * update compat versions (2024-07-08 main-2.x) (#19525) Co-authored-by: Azure Pipelines Daml Build <[email protected]> * update NOTICES file (#19526) Co-authored-by: Azure Pipelines Daml Build <[email protected]> * update canton to 20240704.12141.0.v242206c3/2.10.0-snapshot.20240704.12141.0.v242206c3 in main-2.x (#19513) * update canton to 20240704.12141.0.v242206c3/2.10.0-snapshot.20240704.12141.0.v242206c3 in main-2.x tell-slack: canton * Fixed UpgradesSpecBase.scala. --------- Co-authored-by: Azure Pipelines Daml Build <[email protected]> Co-authored-by: Andreas Triantafyllos <[email protected]> * update compat versions (2024-07-07 main-2.x) (#19523) Co-authored-by: Azure Pipelines Daml Build <[email protected]> * update canton to 20240708.12144.0.v22fb89b5/2.10.0-snapshot.20240708.12144.0.v22fb89b5 in main-2.x (#19535) tell-slack: canton * Backport iface compiler checks 2.x (#19533) * Add back compiler upgrade validation for interfaces * Fix dade-copyright-headers check from 21b4c952e * Prioritize more specific error, improve comments on iface checks * Backport upgrade validation warnings to errors flag 2.x (#19552) * Expand upgrade validation testing (#19123) * Remove duplication from tests * lint * lint * Simplify tests * lint * add more tests for dependency behaviour * track new test files * lint * add test for failure being only emitted in the module that originates it * fix stdout capture for rules_daml * lint * lint * Move CannotUpgradeView into own section * Fix more upgrades gen * Fix build issues with test-common upgrades files * Backport changes from https://github.com/digital-asset/daml/pull/19259 * Add back agreement warning test * Lint * Fix reference from 2.dev to 1.dev * Fix upgrade test files * Revert stdout from daml_compile in daml.bzl * Fix reference in IntegrationTests * Fix trigger tests to ignore interface/template upgrade errors * Only run upgrade checks when building with a supporting LF version * Fix MissingModule test for participant upgrade check tests * Java Codegen: ETX-534 support package names (#19485) * populate package reference into TEMPLATE_ID and add separate PACKAGE_ID field to each generated class run-all-tests: true * consolidate resolution of package ref * more s/TEMPLATE_ID.getPackageId/PACKAGE_ID/ * always use a packageId for interface ids * add TEMPLATE_ID_WITH_PACKAGE_ID field to generated Java classes * fix assertions in conformance tests * generateTemplateIdFields --------- Co-authored-by: = <=> * ETX-534: Java Codegen - `ContractDecoder.fromEvent` can lookup via package name (#19549) This allows the codegen to decode contracts from newer packages * Port key upgrade static checks from 2.9 to 2.x (#19565) * Fix key upgrade validation in 2.9 (#19509) Fix key upgrade validation in 2.9 * Fix key type upgrade check test to a weaker one for compiler-side checks --------- Co-authored-by: Dylan Thinnes <[email protected]> * Increase retries for pruning for pruning offset tests (#19567) * Increase retries for pruning for pruning offset tests (#19571) * update canton to 20240711.12155.0.v50f42c68/2.10.0-snapshot.20240711.12155.0.v50f42c68 in main-2.x (#19574) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * update canton to 20240715.12159.0.v00c76867/2.10.0-snapshot.20240715.12159.0.v00c76867 in main-2.x (#19599) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * update canton to 20240716.12162.0.v31441a3b/2.10.0-snapshot.20240716.12162.0.v31441a3b in main-2.x (#19611) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * update canton to 20240717.12166.0.v7be4b52d/2.10.0-snapshot.20240717.12166.0.v7be4b52d in main-2.x (#19620) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * update canton to 20240718.12169.0.vdcc4c254/2.10.0-snapshot.20240718.12169.0.vdcc4c254 in main-2.x (#19628) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * Port upgrade tests + add interface upgrade tests (#19610) * Remove old tests, port new ones, compiling * Bring IDE ledger tests back in * Tweaks + add Tudor test * Add support for lf-version + depends to upgrades runtime tests, migrate Lf version checks, scaffold interface tests * Fix vetting checks * Add interface tests * Fix test exclusion * Update from self review * Review fixes * Fix various issues with multi-build + multi-ide (#19624) * Fix various issues with multi-build + multi-ide * Revert explicit Github Pull Request support * Backport upgrade checks in ide 2.x (#19626) * Add upgrades errors to IDE (#19605) * Add upgrades errors to IDE inline * Remove reference to pTypecheckUpgrades for pUpgradeInfo * lint * Fix failing tests, including gnarly LSP issue (withMaybeConfig) * lint * Fix more tests, incl. info expected on stderr instead of stdout * Remove commented out diagnosis code * Test case for upgrade diagnostics in LSP * Check deeply in buildDar * Replace upgradeInfo with just upgradeDar in pkgConf, consolidate in opts * Track version1 for upgrades LSP test * lint * lint again * Fix: Use opts for upgradeInfo in execPackage * Define global defaults for upgradeInfo bools, drop yesNoAutoToMaybe * Address TODOs * lint * Handle legacy damlc compiler with no typecheck-upgrades flag * lint * Fix pUpgradeInfo to pUpgradeDar * Default to local compiler only * Fix type mismatch, uiWarnBadInterfaceInstances is now Bool * Move UpgradeInfo back to DA.Daml.Options.Types * Fix tests, add version bounds to warn-bad-interface-instances warnings * update compat versions (2024-07-16 main-2.x) (#19600) Co-authored-by: Azure Pipelines Daml Build <[email protected]> * update compat versions (2024-07-22 main-2.x) (#19644) Co-authored-by: Azure Pipelines Daml Build <[email protected]> Co-authored-by: Tudor Voicu <[email protected]> * update canton to 20240719.12171.0.v2d5ec511/2.10.0-snapshot.20240719.12171.0.v2d5ec511 in main-2.x (#19640) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * Fix goto definition for multi-ide (#19646) * update compat versions (2024-07-23 main-2.x) (#19655) Co-authored-by: Azure Pipelines Daml Build <[email protected]> * update canton to 20240722.12173.0.vd172fbc9/2.10.0-snapshot.20240722.12173.0.vd172fbc9 in main-2.x (#19654) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * Fix error message for datatype changing variety (#19636) (#19658) * Fix error message for datatype changing variety * lint * update canton to 20240723.12177.0.vdf7c4002/2.10.0-snapshot.20240723.12177.0.vdf7c4002 in main-2.x (#19662) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * Implement interface upgrades and package preference support for IDE Ledger (#19660) * Implement interface upgrades and package preference support for IDE Ledger * Add upgrades filters to by name conversion Defer package lookup errors to Update machine Improve FailedCmd errors * update canton to 20240724.12179.0.v56114e19/2.10.0-snapshot.20240724.12179.0.v56114e19 in main-2.x (#19671) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * update canton to 20240725.12181.0.v81e474cf/2.10.0-snapshot.20240725.12181.0.v81e474cf in main-2.x (#19679) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * update NOTICES file (#19694) Co-authored-by: Azure Pipelines Daml Build <[email protected]> * update canton to 20240729.12185.0.vdca47a6a/2.10.0-snapshot.20240729.12185.0.vdca47a6a in main-2.x (#19692) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * update canton to 20240730.12189.0.v949f4bf4/2.10.0-snapshot.20240730.12189.0.v949f4bf4 in main-2.x (#19700) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * update canton to 20240731.12191.0.vdfcbe402/2.10.0-snapshot.20240731.12191.0.vdfcbe402 in main-2.x (#19705) * update canton to 20240731.12191.0.vdfcbe402/2.10.0-snapshot.20240731.12191.0.vdfcbe402 in main-2.x tell-slack: canton * Add missing chimney dependency to community_domain Because of Canton #19734 (PR ##19769) adding new chimney library usages. * fmt --------- Co-authored-by: Azure Pipelines Daml Build <[email protected]> Co-authored-by: Yves Geissbuehler <[email protected]> Co-authored-by: Remy Haemmerle <[email protected]> * Fix VSCode starting daml IDE (#19722) * update NOTICES file (#19730) Co-authored-by: Azure Pipelines Daml Build <[email protected]> * update canton to 20240805.12194.0.va47bd38d/2.10.0-snapshot.20240805.12194.0.va47bd38d in main-2.x (#19716) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * fixed types generated by codegen (#19725) backport main commit - b9fe360 Co-authored-by: Raphael Speyer <[email protected]> * Support env var enterpolation in yaml keys (2.x) (#19751) * Support env var enterpolation in yaml keys in haskell * Support env var enterpolation in yaml keys in scala * Fix goto definition and hover on record constructors (#19748) * Fix goto definition and hover on record constructors * Update to main ghcide * Also compare uploaded package against packages that its dar embeds (#19762) (#19770) * Also compare uploaded package against packages that its dar embeds * log error messages for failed lookups * add flag and instructions for attaching a java debugger to the canton fixture (#19763) (#19768) * add flag and instructions for attaching a java debugger to the canton fixture * revert leftover debugging code * Backport (#19668) [LF] make hashing scheme upgradable friendly (#19771) Co-authored-by: Andreas Lochbihler <[email protected]> * update canton to 20240812.12201.0.v56d704ff/2.10.0-snapshot.20240812.12201.0.v56d704ff in main-2.x (#19764) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * update NOTICES file (#19779) Co-authored-by: Azure Pipelines Daml Build <[email protected]> * update canton to 20240813.12203.0.vbe8c2041/2.10.0-snapshot.20240813.12203.0.vbe8c2041 in main-2.x (#19778) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * Fix GetUser in daml script to populate the party participant map (#19785) * Default to CantonFixtureDontDebug in CantonFixtures (#19783) * allow upgrading key types in the participant (#19780) * allow upgrading key types in the participant * add runtime tests * fix queryByKey with upgrades * disable failing test * allow upgrading keys during re-interpretation (#19792) * fix hashContractInstance broken in #19771 (#19811) * Kill LF 1.16 and introduce LF 1.17 (#19774) * Fix fetch interface and fromInterface (#19676) * Pull interface package when replaying Exercice by interface (#19822) fix DACH-NY/canton#20645 * Fix error reporting in daml build, and improve daml clean messages (#19803) * Fix error reporting in daml build, and improve daml clean messages * Update daml clean prints * Include originating contract packages when calculating party packages (#19793) * Add contractPackages to transaction metadata * Add disclosed contracts into contract packages * Forbid adding new interface instances to the upgraded version of a template (#19817) * Forbid adding new interface instances to the upgraded version of a template * allow adding new instances to new templates * [compiler] forbid adding new interface instances to the upgraded version of a template (#19820) * [compiler] forbid adding new interface instances to the upgraded version of a template * allow adding new instances to new templates * add two more tests * update canton to 20240826.12221.0.v4934f780/2.10.0-snapshot.20240826.12221.0.v4934f780 in main-2.x (#19841) * update canton to 20240826.12221.0.v4934f780/2.10.0-snapshot.20240826.12221.0.v4934f780 in main-2.x tell-slack: canton * Use PV5 when not in dev mode --------- Co-authored-by: Azure Pipelines Daml Build <[email protected]> Co-authored-by: Simon Maxen <[email protected]> * update compat versions (2024-08-28 main-2.x) (#19851) Co-authored-by: Azure Pipelines Daml Build <[email protected]> * check view consistency in fetch/exercise by interface (#19857) * first working version * Add a unit test for view mismatches * rename test case * Refactor tests into upgrades and non-upgrades ones. Run non-upgrade ones on LF 1.17 too. * fix 'should request unknown package' test * Add a test that exceptions thrown by view cannot be caught * prevent view exceptions from being caught in 1.17 * return proper errors instead of crashes * compare views in choice exercises for interfaces * add unit test for exercise_interface * add a check that view errors cannot be caught when exercising an interface * improve comment * disable retro instance conformance test in 1.dev * Apply suggestions from code review Co-authored-by: Remy <[email protected]> * improve ensureImplementsInterface * inline variable used only once --------- Co-authored-by: Remy <[email protected]> * update canton to 20240828.12229.0.v26664579/2.10.0-snapshot.20240828.12229.0.v26664579 in main-2.x (#19855) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * update NOTICES file (#19871) Co-authored-by: Azure Pipelines Daml Build <[email protected]> * update canton to 20240829.12230.0.v343fd7c6/2.10.0-snapshot.20240829.12230.0.v343fd7c6 in main-2.x (#19860) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * Check that keys are upgradeable (2.x) (#19667) Structural equality was not implemented in the end because upgrade rules changed to allow keys to upgraded and because there were bugs in the structural equality implementation for per-module typechecking. Also port 5d01c0d2f9 Fix ts codegen checks order (#19633) --------- Co-authored-by: Samuel Williams <[email protected]> * update canton to 20240902.12232.0.v9f3c465e/2.10.0-snapshot.20240902.12232.0.v9f3c465e in main-2.x (#19878) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * update NOTICES file (#19881) Co-authored-by: Azure Pipelines Daml Build <[email protected]> * update canton to 20240903.12233.0.v64e44fad/2.10.0-snapshot.20240903.12233.0.v64e44fad in main-2.x (#19885) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * update canton to 20240904.12234.0.ve4d66c34/2.10.0-snapshot.20240904.12234.0.ve4d66c34 in main-2.x (#19889) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * update canton to 20240905.12235.0.v8a6436cf/2.10.0-snapshot.20240905.12235.0.v8a6436cf in main-2.x (#19894) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * Disable Multi-IDE for older Daml (#19876) * Disable Multi-IDE for older Daml * Apply suggestion * Add note to version strings * update NOTICES file (#19902) Co-authored-by: Azure Pipelines Daml Build <[email protected]> * Deduplicate warnings (#19893) (#19908) * First attempt to de-dupe warnings * Limit depth of whole-package checks to not overlap with module checks * Drop commented-out lines * update canton to 20240910.12237.0.v1f685566/2.10.0-snapshot.20240910.12237.0.v1f685566 in main-2.x (#19910) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * Switch isInterfaceFetch for interfaceId, put into proto (#19858) * Switch isInterfaceFetch for interfaceId, put into proto * (may revert) Try pipe interfaceId through ActionDescription * Add interface id to fetch replay command Put Action description changes into PV7 * Add interface id to Fetch Node packageIds * Update Security evidence * Fix EngineTest compilation * Update test evidence again * Move change to ViewParticipantDataV5 * Switch new ViewParticipantData to + ActionDescription to Unstable * Add interfaceId to pretty on Fetch and Exercise * update canton to 20240911.12239.0.v0a7b34ce/2.10.0-snapshot.20240911.12239.0.v0a7b34ce in main-2.x (#19916) * update canton to 20240911.12239.0.v0a7b34ce/2.10.0-snapshot.20240911.12239.0.v0a7b34ce in main-2.x tell-slack: canton * Apply canton patch from #19858 --------- Co-authored-by: Azure Pipelines Daml Build <[email protected]> Co-authored-by: Samuel Williams <[email protected]> * UpgradeTestLib: improve nested test trees (#19936) * UpgradeTestLib: improve nested test trees * The test suite prints the full path of each test case, starting from the file name * Added 'subtree' helper * fmt * Factor out initial TestState construction * Bump scalapb version to 0.11.14 (#19922) (#19924) * [Upgrades testing] Add cases for unchanged key type with changed key expression (#19897) * Add test cases for unchanged key type with changed key expression * Add test cases for unchanged key type with changed key expression, with fetchByKey * Add test cases for unchanged key type with changed key expression, with fetch * Mark broken tests, organize test tree * Add test cases for exercise{,ByKey}{,Cmd} * Use 'subtree' * update canton to 20240916.12246.0.vff877fe0/2.10.0-snapshot.20240916.12246.0.vff877fe0 in main-2.x (#19942) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * update canton to 20240917.12247.0.v0e160276/2.10.0-snapshot.20240917.12247.0.v0e160276 in main-2.x (#19951) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * update canton to 20240918.12248.0.vb8b5812f/2.10.0-snapshot.20240918.12248.0.vb8b5812f in main-2.x (#19963) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * update compat versions (2024-09-19 main-2.x) (#19964) Co-authored-by: Azure Pipelines Daml Build <[email protected]> * update canton to 20240919.12249.0.v598115e0/2.10.0-snapshot.20240919.12249.0.v598115e0 in main-2.x (#19974) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * update NOTICES file (#19986) Co-authored-by: Azure Pipelines Daml Build <[email protected]> * update canton to 20240920.12250.0.vf177bc05/2.10.0-snapshot.20240920.12250.0.vf177bc05 in main-2.x (#19983) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * Backport fix assistant 3.x architecture suffixes (#19901) (#19918) * If the target SDK version is >= 3.x, add arch's suffix to download URL * Test * update canton to 20240923.12253.0.v74100321/2.10.0-snapshot.20240923.12253.0.v74100321 in main-2.x (#19992) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * Forward port 19892 19970 to 2.x (#19990) * Make expression upgrade warnings robust against LF version changes & dep upgrade changes (#19892) * Refactor checkType to be flexible for all name checks * Expression equality impl * Remove pkgSupportsUpgrades check temporarily * Add deps with no metadata to upgrade list, allow them to upgrade * Replace `error` with EUpgradeMultiplePackagesWithSameNameAndVersion * Test upgrade behaviour for expressions * lint * Split versions in SucceedsWhenUpgradingLFVersionWithoutExpressionWarning * Track source files for two upgrades tests * Pass UnitId down to properly implement expression/type equality * Fix renamed reference checkUpgrade -> checkPackage * drop commented out code * Address comments on PR 19892 * Add mismatches to exprs (#19970) * Improve alpha checking to let us identify which names mismatched where * Address non-variant comments on 19970 * Add variants for warning messages, unify checker for types & values * Use 1_17 instead of 1_16 in lf version test * update canton to 20240924.12255.0.v4a0f7334/2.10.0-snapshot.20240924.12255.0.v4a0f7334 in main-2.x (#20004) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * Prevent catching contract info exceptions except in create (#19904) * Prevent catching ensure clause exceptions except in create * Add tests * fix keys * some script tests are now fixed * fix maintainers * fix evaluation order tests * remove leftover comments and debugging statements * Apply suggestion Co-authored-by: Remy <[email protected]> --------- Co-authored-by: Remy <[email protected]> * Include all witnesses in contract visibility calculation (#19995) * Include all witnesses in contract visibility calculation * delete dead code (#19988) * Check that controller/observer exceptions are not catchable in dynamic choice exercises (#19999) * also test choice controllers and observers * Check that controller/observer exceptions are not catchable in dynamic choice exercises * Add metadata checks to upgraded dar (#19978) * Add metadata checks to upgraded dar * Write tests, add checks for package/lf version changes * Change diagnostic reporting to explicit * Fix LSP tests * Handle odd CI tmp path * Fix test for windows * Fix formatting from merge * Address comments * Update to main ghcide * update canton to 20240925.12259.0.v89401c74/2.10.0-snapshot.20240925.12259.0.v89401c74 in main-2.x (#20013) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * Refactor typescript extension and implement multi-ide gradle support (#19773) * Refactor typescript extension and implement multi-ide gradle support * Address self review * Small fixess, put gradle support behind flag for early merge * Address review * Improve compatibility of gradle support * Fix dependencies in IDE following #19978 (#20017) * update canton to 20240926.12260.0.v6a0ca308/2.10.0-snapshot.20240926.12260.0.v6a0ca308 in main-2.x (#20019) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * Check for alpha equivalence in data types (#20016) * Add two tests for Phantom type, fix error messages (#20024) * Disallow exceptions repetition for 2.x (#20003) * Disallow upgrading exceptions in 2.x * lint * Replace setUpgradeField with True * update canton to 20240927.12261.0.v0beaa6c4/2.10.0-snapshot.20240927.12261.0.v0beaa6c4 in main-2.x (#20027) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * update NOTICES file (#20030) Co-authored-by: Azure Pipelines Daml Build <[email protected]> * update NOTICES file (#20033) Co-authored-by: Azure Pipelines Daml Build <[email protected]> * JSON API: fail loudly if cache is newer than ledger (#20026) Intended to make it more obvious when an existing cache database is re-used with a new ledger. Previously we used `start >= end` to decide that our cache is up-to-date and we silently fetched nothing. This is a valid decision for the `start == end` case, but we should not expect `start > end` under normal use. This is something we can see if they have started json api with a cache that was populated against a different ledger history, in which case we have no recourse other than :boom: Inspired by https://support.digitalasset.com/s/case/500Pq00000LS2dwIAD/inconsistence-querying-jsonapi * JSON API: downgrade crash to log.debug when (start > end) (#20037) I think we can't crash when the cached offset is greater than the fetched ledger end, because it could occur when interleaving queries for the same templates cause concurrent updates to the cache, while some ledger activity is also going on. For example. * Some query has happened in the past which update the cache to offset O1 * Later Query A and Query B are issued concurrently * Query A asks the participant for the ledger end and gets back offset O2 (> O1) * more movement on the ledger * Query B asks the participant for the ledger end and gets back offset O3 (> O2) * Query B checks the cache to see the last offset for this template id and gets back O1 * Query B requests the transactions between O1 and O3, stores the results into the cache and updates the last offset in the cache to O3 * Query A continues where it left off, and checks the cache for the last offset for this template id, and gets back O3 * Query A sees that O3 (i.e. it's start offset) is greater than O2 (the ledger end it requested from the participant earlier) and :boom: ``` Cache Query A Query B Participant | | | | | |--end?----^------------>| | | | | | |<-end=2---^-------------| | | | | | | |--end?------>| | | | | | | |<-end=3------| | | | | |<--------^---last?--| | | | | | |---------^--last=1->| | | | | | | | |-data(1,3]?->| | | | | | | |<-data=...---| | | | | |<--------^--last=3--| | | | | | |<--last?-| | | | | | | |-last=3->| | | | | | | | (3>2!) | | ``` This downgrades the behaviour just introduced in https://github.com/digital-asset/daml/pull/20026/ so we can still get clues from the logs in the case of an incorrect cache, but it doesn't turn harmless contention into a fatal condition. * update canton to 20240930.12263.0.v66dc328c/2.10.0-snapshot.20240930.12263.0.v66dc328c in main-2.x (#20031) tell-slack: canton Co-authored-by: Azure Pipelines Daml Build <[email protected]> * Port upgrade-check-no-serializability-change (20038) to 2.x (#20039) * Port upgrade-check-no-serializability-change to 2.x * Fix flakiness * Fix Succeeds to FailsWhenMakingTypeUnserializable * Add back metadata check * Improve timeout on //compiler/damlc/tests:upgrades * Drop unnecessary BecameSerializable err, name checkDeleted filter arg Also, add a TODO for WEDependencyHasNoMetadataDespiteUpgradeability * Normalize paths before looking up a virtual resource's language client * lint --------- Co-authored-by: Moisés Ackerman <[email protected]> Co-authored-by: azure-pipelines[bot] <36771401+azure-pipelines[bot]@users.noreply.github.com> Co-authored-by: Azure Pipelines Daml Build <[email protected]> Co-authored-by: mziolekda <[email protected]> Co-authored-by: Remy <[email protected]> Co-authored-by: Paul Brauner <[email protected]> Co-authored-by: Gary Verhaegen <[email protected]> Co-authored-by: Simon Maxen <[email protected]> Co-authored-by: Paul Brauner <[email protected]> Co-authored-by: Raphael Speyer <[email protected]> Co-authored-by: Ratko Veprek <[email protected]> Co-authored-by: Samuel Williams <[email protected]> Co-authored-by: Carl Pulley <[email protected]> Co-authored-by: yves-da <[email protected]> Co-authored-by: oliverse-da <[email protected]> Co-authored-by: Simon Maxen <[email protected]> Co-authored-by: tudor-da <[email protected]> Co-authored-by: Erwin Ramirez <[email protected]> Co-authored-by: Andreas Triantafyllos <[email protected]> Co-authored-by: Marton Nagy <[email protected]> Co-authored-by: Andreas Lochbihler <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Addresses #19782 for main-2.x.