Recursive dependency in generated code?

[tecosaur]

Hello, I"m trying to use this package with the Phenopacket protobuf files (see https://github.com/phenopackets/phenopacket-schema). However, when I cannot work out how to load the generated code to avoid hitting an UndefVarError.

The setup

Artifacts.toml

[phenopackets-schema]
git-tree-sha1 = "918e6d6085a108927146c81cb183ba8cd3c9c37c"

    [[phenopackets-schema.download]]
    url = "https://github.com/phenopackets/phenopacket-schema/archive/refs/tags/2.0.0.tar.gz"
    sha256 = "e24f37ffa9ef502c754b914f6ba8eb725700354e9e23c694c4b9dc9a67b6991b"

[vrs-protobuf-schema]
git-tree-sha1 = "651d4ba4927838bbea7a1e057d258ef5ba33762f"

    [[vrs-protobuf-schema.download]]
    url = "https://github.com/ga4gh/vrs-protobuf/archive/refs/tags/v1.0.0.tar.gz"
    sha256 = "5dcc051e98b78c795eea989c5dadd386052614e210118e5d6b812d6a43150c5b"

build.jl

using Pkg.Artifacts
using ProtoBuf

protospec = joinpath(artifact"phenopackets-schema",
                     "phenopacket-schema-2.0.0",
                     "src", "main", "proto")

# Work around annoying submodule issues, in an idempotent way.
let vrs_protobuf = joinpath(artifact"vrs-protobuf-schema", "vrs-protobuf-1.0.0")
    vrs_submodule = joinpath(artifact"phenopackets-schema", "phenopacket-schema-2.0.0",
                             "src", "vrs-protobuf")
    rm(vrs_submodule)
    symlink(vrs_protobuf, vrs_submodule)
end

protoroot = joinpath("phenopackets", "schema", "v2", "phenopackets.proto")

mkpath("out")
protojl(protoroot, protospec, "out")
include("out/org/org.jl")

Generated files

out.tar.gz

out/
├── google/
│  ├── protobuf/
│  │  ├── any_pb.jl
│  │  ├── protobuf.jl
│  │  └── timestamp_pb.jl
│  └── google.jl
└── org/
   ├── ga4gh/
   │  ├── vrs/
   │  │  ├── v1/
   │  │  │  ├── v1.jl
   │  │  │  └── vrs_pb.jl
   │  │  └── vrs.jl
   │  ├── vrsatile/
   │  │  ├── v1/
   │  │  │  ├── v1.jl
   │  │  │  └── vrsatile_pb.jl
   │  │  └── vrsatile.jl
   │  └── ga4gh.jl
   ├── phenopackets/
   │  ├── schema/
   │  │  ├── v2/
   │  │  │  ├── core/
   │  │  │  │  ├── base_pb.jl
   │  │  │  │  ├── biosample_pb.jl
   │  │  │  │  ├── core.jl
   │  │  │  │  ├── disease_pb.jl
   │  │  │  │  ├── individual_pb.jl
   │  │  │  │  ├── interpretation_pb.jl
   │  │  │  │  ├── measurement_pb.jl
   │  │  │  │  ├── medical_action_pb.jl
   │  │  │  │  ├── meta_data_pb.jl
   │  │  │  │  ├── pedigree_pb.jl
   │  │  │  │  └── phenotypic_feature_pb.jl
   │  │  │  ├── phenopackets_pb.jl
   │  │  │  └── v2.jl
   │  │  └── schema.jl
   │  └── phenopackets.jl
   └── org.jl

Output

ERROR: LoadError: UndefVarError: ga4gh not defined
Stacktrace:
  [1] getproperty(x::Module, f::Symbol)
    @ Base ./Base.jl:31
  [2] top-level scope
    @ ~/.julia/dev/Phenopackets/dev/out/org/phenopackets/schema/v2/core/interpretation_pb.jl:20
  [3] include(mod::Module, _path::String)
    @ Base ./Base.jl:419
  [4] include(x::String)
    @ Main.org.phenopackets.schema.v2.core ~/.julia/dev/Phenopackets/dev/out/org/phenopackets/schema/v2/core/core.jl:1
  [5] top-level scope
    @ ~/.julia/dev/Phenopackets/dev/out/org/phenopackets/schema/v2/core/core.jl:7
  [6] include(mod::Module, _path::String)
    @ Base ./Base.jl:419
  [7] include(x::String)
    @ Main.org.phenopackets.schema.v2 ~/.julia/dev/Phenopackets/dev/out/org/phenopackets/schema/v2/v2.jl:1
  [8] top-level scope
    @ ~/.julia/dev/Phenopackets/dev/out/org/phenopackets/schema/v2/v2.jl:5
  [9] include(mod::Module, _path::String)
    @ Base ./Base.jl:419
 [10] include(x::String)
    @ Main.org.phenopackets.schema ~/.julia/dev/Phenopackets/dev/out/org/phenopackets/schema/schema.jl:1
 [11] top-level scope
    @ ~/.julia/dev/Phenopackets/dev/out/org/phenopackets/schema/schema.jl:3
 [12] include(mod::Module, _path::String)
    @ Base ./Base.jl:419
 [13] include(x::String)
    @ Main.org.phenopackets ~/.julia/dev/Phenopackets/dev/out/org/phenopackets/phenopackets.jl:1
 [14] top-level scope
    @ ~/.julia/dev/Phenopackets/dev/out/org/phenopackets/phenopackets.jl:3
 [15] include(mod::Module, _path::String)
    @ Base ./Base.jl:419
 [16] include(x::String)
    @ Main.org ~/.julia/dev/Phenopackets/dev/out/org/org.jl:1
 [17] top-level scope
    @ ~/.julia/dev/Phenopackets/dev/out/org/org.jl:5
 [18] include(fname::String)
    @ Base.MainInclude ./client.jl:476
 [19] top-level scope
    @ ~/.julia/dev/Phenopackets/dev/build.jl:20
 [20] include(fname::String)
    @ Base.MainInclude ./client.jl:476
 [21] top-level scope
    @ REPL[1]:1
in expression starting at /home/tec/.julia/dev/Phenopackets/dev/out/org/phenopackets/schema/v2/core/interpretation_pb.jl:20
in expression starting at /home/tec/.julia/dev/Phenopackets/dev/out/org/phenopackets/schema/v2/core/core.jl:1
in expression starting at /home/tec/.julia/dev/Phenopackets/dev/out/org/phenopackets/schema/v2/v2.jl:1
in expression starting at /home/tec/.julia/dev/Phenopackets/dev/out/org/phenopackets/schema/schema.jl:1
in expression starting at /home/tec/.julia/dev/Phenopackets/dev/out/org/phenopackets/phenopackets.jl:1
in expression starting at /home/tec/.julia/dev/Phenopackets/dev/out/org/org.jl:1
in expression starting at /home/tec/.julia/dev/Phenopackets/dev/build.jl:20

If I try to include("out/org/ga4gh/ga4gh.jl") I get

WARNING: could not import Main.google into v1
ERROR: LoadError: UndefVarError: google not defined
Stacktrace:
  [1] top-level scope
    @ ~/.julia/dev/Phenopackets/dev/out/org/ga4gh/vrsatile/v1/vrsatile_pb.jl:13

I can solve this issue by running include("out/google/google.jl"), but then if I try include("out/org/ga4gh/ga4gh.jl") again I now see

WARNING: replacing module ga4gh.
ERROR: LoadError: UndefVarError: ga4gh not defined
Stacktrace:
  [1] getproperty(x::Module, f::Symbol)
    @ Base ./Base.jl:31
  [2] top-level scope
    @ ~/.julia/dev/Phenopackets/dev/out/org/ga4gh/vrsatile/v1/vrsatile_pb.jl:229

[Drvi]

Thanks @tecosaur for reporting this issue. You are correct that the root cause is us not being able to handle mutual dependencies in submodules -- org.phenopackets requires types from org.ga4gh and vice versa (in this case, the julia module ga4gh doesn't yet exist as we include phenopackets first).

A quick workaround would be to use the older version of this library which (add [email protected]) which uses dicts to represent fields of protobuf messages and thus doesn't require all proto types to be statically known when defining the julia struct.

As you noticed including sub-packages directly rarely works -- currently, whenever a reference is made to a definition in a different package or submodule, we always reach out to the top of the current package to grab the package name and then we use it to fully qualify the reference. So, as-is, we only support including the top level file which is in your case "out/org/org.jl".

It seems like the default python codec is not happy about this situation either, they seem to do some textual postprocessing of generated proto files.

I'm afraid that to get that the current version of ProtoBuf.jl won't work without a change to the original proto files or a manual rewrite of the generated code. Sorry about that!

I'm going to keep this issue open to discuss different approaches to handle mutual dependencies.