交叉编译Rust项目时，cargo的编译期依赖没有被正确安装 #5156

Yangff · 2024-05-28T01:42:59Z

Xmake 版本

xmake v2.9.2+master.d8818c5b0

操作系统版本和架构

all

描述问题

当前，xmake通过编译一个空的项目来复制cargo解析的rust项目依赖，并将产生的所有编译结果安装到目标路径。
#4049 解决了交叉编译时不能正确拷贝目标平台库文件的问题。
但是，在交叉编译时，rust会将需要在主机运行的编译期依赖，例如过程宏编译为可以在主机平台运行的文件，例如.dlls。这些目标文件不会被存放在 target/arch/release/deps , 而是 target/release/deps.
Cargo会在产生rustc指令时同时将这两个目录放进-L中，从而产生正确的编译结果。
但是，通过add_files添加的rs文件只会在xmake复制后的安装路径中寻找依赖，这使得rustc无法在use对应包的时候提示找不到这个包，实际上，是找不到对应的在主机上运行的依赖。

产生类似如下的错误

PS C:\Users\yangf\Documents\Projects\RE-UE4SS> rustc 
-C debuginfo=2 -C opt-level=3 -C linker=C:\Users\yangf\Documents\Projects\RE-UE4SS/tools/zig/zig-c++ 
--target=x86_64-unknown-linux-gnu 
--edition=2021 
-L dependency=C:\Users\yangf\Documents\Projects\RE-UE4SS\Intermediates\.packages\c\cargo_patternsleuth_bind\latest\ca0d54df84b4464a9d56cf13e471daea\lib 
-C debuginfo=2 
--extern patternsleuth=C:\Users\yangf\Documents\Projects\RE-UE4SS\Intermediates\.packages\c\cargo_patternsleuth_bind\latest\cache\source\target\x86_64-unknown-linux-gnu\release\deps\libpatternsleuth-fe8a45f2bce330fe.rlib 
--crate-type=staticlib -o C:\Users\yangf\Documents\Projects\RE-UE4SS\Binaries\Game__Shipping__Linux\patternsleuth_bind\libpatternsleuth_bind.a deps\first\patternsleuth_bind\src\lib.rs
error[E0463]: can't find crate for `patternsleuth`
 --> deps\first\patternsleuth_bind\src\lib.rs:5:5
  |
5 | use patternsleuth::resolvers::{
  |     ^^^^^^^^^^^^^ can't find crate

error: cannot determine resolution for the macro `impl_collector`
  --> deps\first\patternsleuth_bind\src\lib.rs:19:1
   |
19 | impl_collector! {
   | ^^^^^^^^^^^^^^
   |
   = note: import resolution is stuck, try simplifying macro imports

error[E0463]: can't find crate for `patternsleuth`
   --> deps\first\patternsleuth_bind\src\lib.rs:101:15
    |
101 |     let exe = patternsleuth::process::internal::read_image()?;
    |               ^^^^^^^^^^^^^ can't find crate

error[E0433]: failed to resolve: use of undeclared type `UE4SSResolution`
   --> deps\first\patternsleuth_bind\src\lib.rs:105:34
    |
105 |     let resolution = exe.resolve(UE4SSResolution::resolver())?;
    |                                  ^^^^^^^^^^^^^^^ use of undeclared type `UE4SSResolution`

error: aborting due to 4 previous errors

期待的结果

项目应该正常编译，调用应该为

C:\Users\yangf\Documents\Projects\RE-UE4SS> rustc 
-C debuginfo=2 -C opt-level=3 
-C linker=C:\Users\yangf\Documents\Projects\RE-UE4SS/tools/zig/zig-c++ 
--target=x86_64-unknown-linux-gnu --edition=2021 
-L dependency=C:\Users\yangf\Documents\Projects\RE-UE4SS\Intermediates\.packages\c\cargo_patternsleuth_bind\latest\ca0d54df84b4464a9d56cf13e471daea\lib 
**-L dependency=C:\Users\yangf\Documents\Projects\RE-UE4SS\Intermediates\.packages\c\cargo_patternsleuth_bind\latest\cache\source\target\release\deps**  
-C debuginfo=2 
--extern patternsleuth=C:\Users\yangf\Documents\Projects\RE-UE4SS\Intermediates\.packages\c\cargo_patternsleuth_bind\latest\cache\source\target\x86_64-unknown-linux-gnu\release\deps\libpatternsleuth-fe8a45f2bce330fe.rlib 
--crate-type=staticlib 
-o C:\Users\yangf\Documents\Projects\RE-UE4SS\Binaries\Game__Shipping__Linux\patternsleuth_bind\libpatternsleuth_bind.a deps\first\patternsleuth_bind\src\lib.rs

工程配置

我手上没有合适的复现但是原理上来说

    if target then
        os.vcp(path.join(sourcedir, "target", target, opt.mode == "debug" and "debug" or "release", "deps"), path.join(installdir, "lib"))
        -- copy everything host target to lib
        os.vcp(path.join(sourcedir, "target", opt.mode == "debug" and "debug" or "release", "deps", "*"), path.join(installdir, "lib"))
    else
        os.vcp(path.join(sourcedir, "target", opt.mode == "debug" and "debug" or "release", "deps"), path.join(installdir, "lib"))
    end

加上第二行复制之后就可以正确编译了（我测试也是可以的）。
但是，我不认为将主机依赖复制到安装路径下是正确的行为。。我不大确定额xmake是怎么区分编译期依赖和项目依赖的安装路径的。

另外我也遇到了 #4049 (comment) 这里提到的不会自动添加

add_rcflags("--target=aarch64-unknown-none", {force = true})

的问题，这似乎与我之前编译no_std的时候遇到的问题是一样的，和这里的问题无关，只是如果修复上述代码之后交叉编译还是失败的话可能是这个导致的。

附加信息和错误日志

无

The text was updated successfully, but these errors were encountered:

Issues-translate-bot · 2024-05-28T03:08:14Z

Bot detected the issue body's language is not English, translate it automatically.

Title: When cross-compiling a Rust project, the compile-time dependencies of cargo are not installed correctly.

waruqi · 2024-05-29T00:53:00Z

好的，等下，我先把之前那个 #4929 处理了，再看这个。有点忙

Issues-translate-bot · 2024-05-29T00:53:12Z

Bot detected the issue body's language is not English, translate it automatically.

Okay, wait a minute, I'll take care of the previous #4929 first, and then look at this one. a little busy

Yangff · 2024-05-29T02:26:18Z

https://github.com/Yangff/xmake_rust_macro

我试着弄一个MRE但是我不大确定弄错了什么，在install的时候就失败了, UE4SS那边至少我还能install的上……

PS C:\Users\yangf\Documents\Projects\xmake_rust_macro> xmake f -v -a x86_64-unknown-linux-gnu -c
checking for platform ... windows
checking for Microsoft Visual Studio (x86_64-unknown-linux-gnu) version ... no
checking for clang ... ok
checking for zig ... ok
checking for nim ... no
checking for nim ... no
checking for unzip ... no
checking for 7z ... C:\Users\yangf\xmake\winenv\bin\7z
checking for git ... ok
checking for gzip ... no
checking for cargo::bind ... no
note: install or modify (m) these packages (pass -y to skip confirm)?
in cargo:
  -> cargo::bind latest [runtimes:"MT", cargo_toml:[[C:\Users\yangf\Documents\Projects\xmake_rust_macro\ ..)
please input: y (y/n/m)
y
checking for cargo ... ok
checking for rustc ... ok
cargo build --release -v
   Compiling dep1 v0.1.0 (C:\Users\yangf\Documents\Projects\xmake_rust_macro\dep1)
     Running `C:\Users\yangf\.rustup\toolchains\stable-x86_64-pc-windows-msvc\bin\rustc.exe --crate-name dep1 --edition=2021 C:\Users\yangf\Documents\Projects\xmake_rust_macro\dep1\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --diagnostic-width=119 --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C embed-bitcode=no -C debug-assertions=off -C metadata=18415b935cb2aa0e -C extra-filename=-18415b935cb2aa0e --out-dir C:\Users\yangf\Documents\Projects\xmake_rust_macro\build\.packages\c\cargo_bind\latest\cache\source\target\release\deps -L dependency=C:\Users\yangf\Documents\Projects\xmake_rust_macro\build\.packages\c\cargo_bind\latest\cache\source\target\release\deps --extern proc_macro`
   Compiling bind v0.1.0 (C:\Users\yangf\Documents\Projects\xmake_rust_macro\build\.packages\c\cargo_bind\latest\cache\source)
     Running `C:\Users\yangf\.rustup\toolchains\stable-x86_64-pc-windows-msvc\bin\rustc.exe --crate-name bind --edition=2021 src\main.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --diagnostic-width=119 --crate-type bin --emit=dep-info,link -C opt-level=3 -C embed-bitcode=no -C metadata=83a7cac72036801a -C extra-filename=-83a7cac72036801a --out-dir C:\Users\yangf\Documents\Projects\xmake_rust_macro\build\.packages\c\cargo_bind\latest\cache\source\target\x86_64-unknown-linux-gnu\release\deps --target x86_64-unknown-linux-gnu -C linker=C:\Users\yangf\Documents\Projects\xmake_rust_macro\./zig-cc-rust.bat -L dependency=C:\Users\yangf\Documents\Projects\xmake_rust_macro\build\.packages\c\cargo_bind\latest\cache\source\target\x86_64-unknown-linux-gnu\release\deps -L dependency=C:\Users\yangf\Documents\Projects\xmake_rust_macro\build\.packages\c\cargo_bind\latest\cache\source\target\release\deps --extern dep1=C:\Users\yangf\Documents\Projects\xmake_rust_macro\build\.packages\c\cargo_bind\latest\cache\source\target\release\deps\dep1-18415b935cb2aa0e.dll`
    Finished release [optimized] target(s) in 3.29s
> copy C:\Users\yangf\Documents\Projects\xmake_rust_macro\build\.packages\c\cargo_bind\latest\cache\source\target\x86_64-unknown-linux-gnu\release\deps to C:\Users\yangf\Documents\Projects\xmake_rust_macro\build\.packages\c\cargo_bind\latest\4ec0bfa4a9a74a96b47ec0f81923a92b\lib
checking for cargo::bind ... no

error: fetch cargo::bind-latest failed!
  => install cargo::bind latest .. failed
error: install failed!
PS C:\Users\yangf\Documents\Projects\xmake_rust_macro> ls  C:\Users\yangf\Documents\Projects\xmake_rust_macro\build\.packages\c\cargo_bind\latest\cache\source\target\x86_64-unknown-linux-gnu\release\deps

    Directory: C:\Users\yangf\Documents\Projects\xmake_rust_macro\build\.packages\c\cargo_bind\latest\cache\source\targ
et\x86_64-unknown-linux-gnu\release\deps

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a---           5/28/2024  9:22 PM        3958528 bind-83a7cac72036801a
-a---           5/28/2024  9:22 PM            377 bind-83a7cac72036801a.d

不过只拷贝了目标平台的依赖的问题是一样的，比如下面这些就没有拷贝，至于这个install为啥failed我就不知道了。。

PS C:\Users\yangf\Documents\Projects\xmake_rust_macro> ls C:\Users\yangf\Documents\Projects\xmake_rust_macro\build\.packages\c\cargo_bind\latest\cache\source\target\release\deps\

    Directory:
C:\Users\yangf\Documents\Projects\xmake_rust_macro\build\.packages\c\cargo_bind\latest\cache\source\target\release\deps

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a---           5/28/2024  9:22 PM            496 dep1-18415b935cb2aa0e.d
-a---           5/28/2024  9:22 PM         200192 dep1-18415b935cb2aa0e.dll
-a---           5/28/2024  9:22 PM           1245 dep1-18415b935cb2aa0e.dll.exp
-a---           5/28/2024  9:22 PM           2444 dep1-18415b935cb2aa0e.dll.lib
-a---           5/28/2024  9:22 PM        1404928 dep1-18415b935cb2aa0e.pdb

这里用了zig来提供win/linux交叉编译时候的链接，不然得带上一堆Linux的依赖。不过这个失败在Linux下也是一样的，好像设置了-a他就会走交叉编译。所以复现的话其实可能不需要找另一个平台来编译，直接在本地做交叉编译就行了。

waruqi · 2024-05-31T00:23:15Z

我试着弄一个MRE但是我不大确定弄错了什么，在install的时候就失败了, UE4SS那边至少我还能install的上……

根 add_files("bind/src/main.rs") 路径不对
bind 里面只引用了 dep1 依赖，但是它仅仅在 host 下，不在 arch 下，所以安装完，fetch 找包没找到 dep1.dylib ，路径没找到，所以安装失败了。

Issues-translate-bot · 2024-05-31T00:23:27Z

Bot detected the issue body's language is not English, translate it automatically.

I tried to get an MRE but I'm not sure what I did wrong. It failed during the installation. At least I can install UE4SS...

The root add_files("bind/src/main.rs") path is incorrect
Only dep1 dependency is referenced in bind, but it is only under host, not under arch, so after installation, fetch did not find dep1.dylib when searching for packages, and the path was not found, so the installation failed.

waruqi · 2024-05-31T00:27:06Z

但是，我不认为将主机依赖复制到安装路径下是正确的行为。。我不大确定额xmake是怎么区分编译期依赖和项目依赖的安装路径的。

目前没区分，可以走安装，但是不要安装到 lib 目录下，那个用于目标库依赖查找的。。

可以安装到 lib/host 子目录下。这样，cargo/find_package 找依赖时候，额外吧 lib/host 添加到 linkdirs 里面去返回，应该就可以了

比如这样，可以试试，#5173

不过我没验证过。或者你可以根据这个 patch 调整下试试

Issues-translate-bot · 2024-05-31T00:27:17Z

Bot detected the issue body's language is not English, translate it automatically.

However, I don't think copying the host dependencies to the installation path is the correct behavior. . I'm not sure how xmake distinguishes the installation paths of compile-time dependencies and project dependencies.

There is currently no distinction. You can install it, but do not install it in the lib directory, which is used to find target library dependencies. .

Can be installed into the lib/host subdirectory. In this way, when cargo/find_package looks for dependencies, add lib/host to linkdirs to return it, and it should be fine.

Yangff · 2024-05-31T01:35:45Z

不过我没验证过。或者你可以根据这个 patch 调整下试试

似乎这里不会去找host路径下的文件

    if plat == "macosx" then
        libfiles_native = os.files(path.join(librarydir, "*.dylib"))
        -- I'm not sure what's for macos
        libfiles_native = table.join(libfiles_native, os.files(path.join(librarydir, "host", "*.dylib")))
    elseif plat == "windows" or plat == "mingw" then
        libfiles_native = os.files(path.join(librarydir, "*.lib"))
        libfiles_native = table.join(libfiles_native, os.files(path.join(librarydir, "host", "*.dll")))
    else
        libfiles_native = os.files(path.join(librarydir, "*.so"))
        libfiles_native = table.join(libfiles_native, os.files(path.join(librarydir, "host", "*.so")))
    end

看起来应该这样？我不确定mac下的扩展名是什么，windows应该是.dll

（或者用**?）

Yangff · 2024-05-31T01:46:43Z

另外之前给rcflags加的target好像也有点问题
9656518

就是这个，在链接的时候如果用了rust作为链接器（不大确定这是怎么决定的，可能和 set_kind 有关系？之前没有遇到），它会把rcflags和rcldflags都用上，导致出现这样的编译指令

/usr/sbin/rustc --target=x86_64-unknown-linux-gnu --edition 2018 --target=x86_64-unknown-linux-gnu -L/home/yangff/Project/xmake_rust_macro/build/.packages/c/cargo_bind/latest/73ba6387af7b499b9f49233ed9da0761/lib/host -L dependency=/home/yangff/Project/xmake_rust_macro/build/.packages/c/cargo_bind/latest/73ba6387af7b499b9f49233ed9da0761/lib --extern dep1=/home/yangff/Project/xmake_rust_macro/build/.packages/c/cargo_bind/latest/73ba6387af7b499b9f49233ed9da0761/lib/host/libdep1-63694d91d6e348aa.so --crate-type=bin -o build/linux/x86_64-unknown-linux-gnu/release/bind bind/main.rs
error: @programdir/core/main.lua:329: @programdir/actions/build/main.lua:148: @programdir/modules/async/runjobs.lua:322: @programdir/rules/rust/build/target.lua:67: @programdir/core/sandbox/modules/os.lua:273: error: Option 'target' given more than once

Issues-translate-bot · 2024-05-31T01:55:32Z

Bot detected the issue body's language is not English, translate it automatically.

In addition, there seems to be some problem with the target added to rcflags before.
9656518

That's it. If you use rust as the linker when linking, it will use both rcflags and rcldflags, resulting in such a compilation instruction.

/usr/sbin/rustc --target=x86_64-unknown-linux-gnu --edition 2018 --target=x86_64-unknown-linux-gnu -L/home/yangff/Project/xmake_rust_macro/build/.packages/c/ cargo_bind/latest/73ba6387af7b499b9f49233ed9da0761/lib/host -L dependency=/home/yangff/Project/xmake_rust_macro/build/.packages/c/cargo_bind/latest/73ba6387af7b499b9f49233ed9da0761/lib --extern dep1=/home/yang ff/Project/xmake_rust_macro/ build/.packages/c/cargo_bind/latest/73ba6387af7b499b9f49233ed9da0761/lib/host/libdep1-63694d91d6e348aa.so --crate-type=bin -o build/linux/x86_64-unknown-linux-gnu/release/bind bind/main. rs
error: @programdir/core/main.lua:329: @programdir/actions/build/main.lua:148: @programdir/modules/async/runjobs.lua:322: @programdir/rules/rust/build/target. lua:67: @programdir/core/sandbox/modules/os.lua:273: error: Option 'target' given more than once

waruqi · 2024-05-31T03:28:52Z

似乎这里不会去找host路径下的文件

如果找到，不是要 host 的库也得 link 进去？arch 对的上么。。

Cargo会在产生rustc指令时同时将这两个目录放进-L中，从而产生正确的编译结果

之前你不是说只需要加 -L 么。。刚那个 patch ，应该会加 -L 。。

但是前提是有对应 target 的库先被找到，但是你那个demo，除了一个 dep1 的host 依赖。。bind 本身再也没 target arch 的库依赖了，导致找不到任何库，所以这个 host linkdir 也不会被附带进来

Issues-translate-bot · 2024-05-31T03:29:03Z

Bot detected the issue body's language is not English, translate it automatically.

It seems that files under the host path will not be found here.

If found, doesn’t it mean that the host library also needs to be linked in? Is arch right? .

Cargo will put these two directories into -L at the same time when generating rustc instructions, thereby producing correct compilation results.

Didn't you say before that you only need to add -L? . For the patch just now, -L should be added. .

But the premise is that a library corresponding to the target is found first, but your demo has only one dep1 host dependency. . bind itself no longer depends on the library of target arch, so no library can be found, so the host linkdir will not be included.

Yangff · 2024-05-31T04:27:48Z

如果找到，不是要 host 的库也得 link 进去？arch 对的上么。。

是这样的，xmake提示find package失败，我看了一下它试图找到dep1这个依赖（就是红框这个），然后dep1是编译期的依赖，在lib目录里找不到，然后xmake好像就觉得这个cargo::bind找不到了。

我看了一下Windows下交叉编译的时候cargo生成的命令

cargo build --target x86_64-unknown-linux-gnu --verbose
   Compiling dep1 v0.1.0 (C:\Users\yangf\Documents\Projects\xmake_rust_macro\dep1)
     Running `C:\Users\yangf\.rustup\toolchains\stable-x86_64-pc-windows-msvc\bin\rustc.exe --crate-name dep1 --edition=2021 C:\Users\yangf\Documents\Projects\xmake_rust_macro\dep1\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --diagnostic-width=119 --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C embed-bitcode=no -C debuginfo=2 -C metadata=78bee71deb4270a4 -C extra-filename=-78bee71deb4270a4 --out-dir C:\Users\yangf\Documents\Projects\xmake_rust_macro\bind\target\debug\deps -C incremental=C:\Users\yangf\Documents\Projects\xmake_rust_macro\bind\target\debug\incremental -L dependency=C:\Users\yangf\Documents\Projects\xmake_rust_macro\bind\target\debug\deps --extern proc_macro`
   Compiling bind v0.1.0 (C:\Users\yangf\Documents\Projects\xmake_rust_macro\bind)
     Running `C:\Users\yangf\.rustup\toolchains\stable-x86_64-pc-windows-msvc\bin\rustc.exe --crate-name bind --edition=2021 src\main.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --diagnostic-width=119 --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debuginfo=2 -C metadata=9fee3eb6e8c2dfdd -C extra-filename=-9fee3eb6e8c2dfdd --out-dir C:\Users\yangf\Documents\Projects\xmake_rust_macro\bind\target\x86_64-unknown-linux-gnu\debug\deps --target x86_64-unknown-linux-gnu -C linker=C:\Users\yangf\Documents\Projects\xmake_rust_macro\./zig-cc-rust.bat -C incremental=C:\Users\yangf\Documents\Projects\xmake_rust_macro\bind\target\x86_64-unknown-linux-gnu\debug\incremental -L dependency=C:\Users\yangf\Documents\Projects\xmake_rust_macro\bind\target\x86_64-unknown-linux-gnu\debug\deps -L dependency=C:\Users\yangf\Documents\Projects\xmake_rust_macro\bind\target\debug\deps --extern dep1=C:\Users\yangf\Documents\Projects\xmake_rust_macro\bind\target\debug\deps\dep1-78bee71deb4270a4.dll`

他确实是 --extern dep1=C:\Users\yangf\Documents\Projects\xmake_rust_macro\bind\target\debug\deps\dep1-78bee71deb4270a4.dll 了。

在我把host也加进去搜索路径之后，xmake现在生成的命令也自动包含了这个 --extern ，然后就没有问题了。

之前你不是说只需要加 -L 么。。刚那个 patch ，应该会加 -L 。。

ue4ss那边确实没有把这些host的依赖加入bind里面，不管是cargo的 --extern 还是xmake都没有。可能是因为ue4ss那边的bind项目没有直接用这个依赖？我也不是很确定。总之那边的话确实是加上-L 就可以了。
然后看上面cargo生成的命令，我这个简化的复现应该是既要 -L 也要 --extern dep1 的，rustc直接用host的dll，dep1库的编译类型是proc-macro,这部分代码只是编译器用来当宏运行的，反正是不会被链接进目标文件的所以不会有冲突。

导致找不到任何库，所以这个 host linkdir 也不会被附带进来

对，这就是下面这个错误的原因

cargo::bind
finding bind from cargo ..
libbind-40f96b119342e5df.a
checking for cargo::bind ... no

error: .../modules/private/action/require/impl/actions/install.lua:430: fetch cargo::bind-latest failed!
stack traceback:
    [C]: in function 'error'
    [@programdir/core/base/os.lua:973]: in function 'raiselevel'
    [@programdir/core/sandbox/modules/utils.lua:143]: in function 'assert'
    [.../modules/private/action/require/impl/actions/install.lua:430]:

  => install cargo::bind latest .. failed

所以我给他加进去那个路径了……

Yangff · 2024-05-31T06:53:01Z

我想应该是这样的，ue4ss的项目里syn那些用了proc macro的库是patternsleuth的依赖（甚至是依赖的依赖）。
patternsleuth_bind -> patternsleuth -> syn ...
所以在patternsleuth_bind看来他需要 --extern 的只有 patternsleuth，xmake也就只检查了这一个。但是patternsleuth本身在编译期也还是需要syn这些库的，所以那些依赖不存在的时候就提示没法 use patternsleuth

我的复现简化了这一步，bind直接依赖dep1，所以哪怕把dep1添加到 lib/host 里也提示找不到依赖，因为xmake会检查 bind 最外面一层的依赖，这时候就发现找不到 dep1 了。

Issues-translate-bot · 2024-05-31T06:53:11Z

Bot detected the issue body's language is not English, translate it automatically.

I think it should be like this. In the ue4ss project, syn libraries that use proc macro are dependent on patternsleuth.
patternsleuth_bind -> patternsleuth -> syn ...
So it seems that patternsleuth_bind only needs patternsleuth with --extern, but patternsleuth itself still needs syn libraries during compilation, so when those dependencies do not exist, it will prompt that use patternsleuth cannot be used.

My reproduction simplifies this step. Bind directly depends on dep1, so even if dep1 is added to lib/host, it will prompt that the dependency cannot be found, because xmake will check the outermost layer of dependencies of bind. At this time, it is found that the dependency cannot be found. Less than dep1.

waruqi · 2024-05-31T15:05:15Z

我想应该是这样的，ue4ss的项目里syn那些用了proc macro的库是patternsleuth的依赖（甚至是依赖的依赖）。
patternsleuth_bind -> patternsleuth -> syn ...
所以在patternsleuth_bind看来他需要 --extern 的只有 patternsleuth，xmake也就只检查了这一个。但是patternsleuth本身在编译期也还是需要syn这些库的，所以那些依赖不存在的时候就提示没法 use patternsleuth

那现在这个 patch ，按理应该对 us4ss 仅仅加 -L 可以 work 的，你试试呢，还是不行？

我的复现简化了这一步，bind直接依赖dep1，所以哪怕把dep1添加到 lib/host 里也提示找不到依赖，因为xmake会检查 bind 最外面一层的依赖，这时候就发现找不到 dep1 了。

等于当前没任何 target arch 的依赖库，仅仅只有一个 host dep1 库。。

它要是 link dep1 进去了，岂不是跪了，尽管当前你这个 demo 没去 link。所以我还是不确定是否安装进去参与 link 一定可靠。

waruqi · 2024-05-31T15:09:07Z

另外之前给rcflags加的target好像也有点问题 9656518

就是这个，在链接的时候如果用了rust作为链接器（不大确定这是怎么决定的，可能和 set_kind 有关系？之前没有遇到），它会把rcflags和rcldflags都用上，导致出现这样的编译指令

/usr/sbin/rustc --target=x86_64-unknown-linux-gnu --edition 2018 --target=x86_64-unknown-linux-gnu -L/home/yangff/Project/xmake_rust_macro/build/.packages/c/cargo_bind/latest/73ba6387af7b499b9f49233ed9da0761/lib/host -L dependency=/home/yangff/Project/xmake_rust_macro/build/.packages/c/cargo_bind/latest/73ba6387af7b499b9f49233ed9da0761/lib --extern dep1=/home/yangff/Project/xmake_rust_macro/build/.packages/c/cargo_bind/latest/73ba6387af7b499b9f49233ed9da0761/lib/host/libdep1-63694d91d6e348aa.so --crate-type=bin -o build/linux/x86_64-unknown-linux-gnu/release/bind bind/main.rs
error: @programdir/core/main.lua:329: @programdir/actions/build/main.lua:148: @programdir/modules/async/runjobs.lua:322: @programdir/rules/rust/build/target.lua:67: @programdir/core/sandbox/modules/os.lua:273: error: Option 'target' given more than once

我改了下，再试试，xmake update -s dev

Issues-translate-bot · 2024-05-31T15:12:46Z

Bot detected the issue body's language is not English, translate it automatically.

I think it should be like this. In the ue4ss project, syn libraries that use proc macro are dependencies of patternsleuth (or even dependencies of dependencies).
patternsleuth_bind -> patternsleuth -> syn ...
So it seems that patternsleuth_bind only needs --extern patternsleuth, and xmake only checks this one. However, patternsleuth itself still requires syn libraries during compilation, so when those dependencies do not exist, it will prompt that use patternsleuth cannot be used.

So the current patch should work just by adding -L to us4ss. Have you tried it, or does it still not work?

My reproduction simplifies this step. Bind directly depends on dep1, so even if dep1 is added to lib/host, it will prompt that the dependency cannot be found, because xmake will check the outermost layer of dependencies of bind, and then it will be found that it cannot be found. Arrived at dep1.

It means that there is currently no dependent library of target arch, only one host dep1 library. .

If it had link dep1 in, wouldn't it be a disaster, even though your demo is currently not linked. So I'm still not sure whether installing it and participating in the link must be reliable.

Issues-translate-bot · 2024-05-31T15:36:24Z

Bot detected the issue body's language is not English, translate it automatically.

In addition, there seems to be some problem with the target added to rcflags before 9656518

That’s it. If rust is used as the linker when linking (not sure how this is decided, maybe it has something to do with set_kind? I haven’t encountered it before), it will use both rcflags and rcldflags, resulting in Compilation instructions like this appear

/usr/sbin/rustc --target=x86_64-unknown-linux-gnu --edition 2018 --target=x86_64-unknown-linux-gnu -L/home/yangff/Project/xmake_rust_macro/build/.packages/c /cargo_bind/latest/73ba6387af7b499b9f49233ed9da0761/lib/host -L dependency=/home/yangff/Project/xmake_rust_macro/build/.packages/c/cargo_bind/latest/73ba6387af7b499b9f49233ed9da0761/lib --extern dep1=/home/ yangff/Project/xmake_rust_macro /build/.packages/c/cargo_bind/latest/73ba6387af7b499b9f49233ed9da0761/lib/host/libdep1-63694d91d6e348aa.so --crate-type=bin -o build/linux/x86_64-unknown-linux-gnu/release/bind bind/main .rs
error: @programdir/core/main.lua:329: @programdir/actions/build/main.lua:148: @programdir/modules/async/runjobs.lua:322: @programdir/rules/rust/build/target .lua:67: @programdir/core/sandbox/modules/os.lua:273: error: Option 'target' given more than once

I changed it and tried again, xmake update -s dev

Yangff · 2024-05-31T15:51:07Z

按理应该对 us4ss 仅仅加 -L 可以 work 的，你试试呢，还是不行？

行

PS C:\Users\yangf\Documents\Projects\RE-UE4SS> xmake
[ 15%]: cache compiling.Game__Shipping__Linux deps\third\glad\src\glad.c
[ 15%]: linking.Game__Shipping__Linux libpatternsleuth_bind.a

它要是 link dep1 进去了

对于rust来说只是意味着指定这个crate的搜索路径就算你不加 --extern 只要我 extern crate 了rustc都会去 -L 的路径去找这个crate。

https://doc.rust-lang.org/rustc/command-line-arguments.html#--extern-specify-where-an-external-library-is-located

This flag allows you to pass the name and location for an external crate of a direct dependency. Indirect dependencies (dependencies of dependencies) are located using the -L flag. The given crate name is added to the extern prelude, similar to specifying extern crate within the root module.

是否最后和生成的二进制link是rustc根据它的类型决定的。

https://doc.rust-lang.org/reference/linkage.html

--crate-type=proc-macro, #![crate_type = "proc-macro"] - The output produced is not specified, but if a -L path is provided to it then the compiler will recognize the output artifacts as a macro and it can be loaded for a program. Crates compiled with this crate type must only export procedural macros. The compiler will automatically set the proc_macro configuration option. The crates are always compiled with the same target that the compiler itself was built with. For example, if you are executing the compiler from Linux with an x86_64 CPU, the target will be x86_64-unknown-linux-gnu even if the crate is a dependency of another crate being built for a different target.

Yangff · 2024-05-31T16:00:59Z

我改了下，再试试，xmake update -s dev

这个没问题了

Issues-translate-bot · 2024-05-31T16:01:12Z

Bot detected the issue body's language is not English, translate it automatically.

I changed it and tried again, xmake update -s dev

This is no problem

waruqi · 2024-05-31T16:06:09Z

是否最后和生成的二进制link是rustc根据它的类型决定的。

那就是我即使无脑 copy 安装过去，走 extern + link 也没啥问题咯？如果没啥影响的话，那我就直接把 host 也 copy 过去了。

Issues-translate-bot · 2024-05-31T16:06:42Z

Bot detected the issue body's language is not English, translate it automatically.

Whether the final and generated binary link is rustc is determined based on its type.

That is, even if I install it without a brain copy and use extern + link, there will be no problem? If there is no impact, then I will copy the host directly.

Yangff · 2024-05-31T16:11:07Z

是否最后和生成的二进制link是rustc根据它的类型决定的。

那就是我即使无脑 copy 安装过去，走 extern + link 也没啥问题咯？如果没啥影响的话，那我就直接把 host 也 copy 过去了。

我觉得应该是没有影响的，不过xmake那边检查是否安装成功的时候可能还是要 .so, .dll, .dylib（?可能是这个）都搜索一下？

        libfiles_native = os.files(path.join(librarydir, "*.so")) -- plat = linux
        libfiles_native = table.join(libfiles_native, os.files(path.join(librarydir, "host", "*.so"))) -- host = linux, plat=linux
        libfiles_native = table.join(libfiles_native, os.files(path.join(librarydir, "host", "*.dll"))) -- host = win, plat=linux

就可能是这样的？
如果都复制到lib下也是一样的

Issues-translate-bot · 2024-05-31T16:11:38Z

Bot detected the issue body's language is not English, translate it automatically.

Whether the final and generated binary link is rustc is determined based on its type.

So even if I copy and install it without a brain, there will be no problem if I use extern + link? If there is no impact, then I will copy the host directly.

I think there should be no impact, but xmake may still need to check .so, .dll, .dylib (? maybe this) when checking whether the installation is successful?

        libfiles_native = os.files(path.join(librarydir, "*.so")) -- plat = linux
        libfiles_native = table.join(libfiles_native, os.files(path.join(librarydir, "host", "*.so"))) -- host = linux, plat=linux
        libfiles_native = table.join(libfiles_native, os.files(path.join(librarydir, "host", "*.dll"))) -- host = win, plat=linux

Is that possible?

waruqi · 2024-06-01T14:47:46Z

再试试，#5173

Yangff · 2024-06-01T16:40:45Z

再试试，#5173

Linux 下可以了，win上搜索lib的时候前缀会有另一个问题

Yangff · 2024-06-01T16:46:11Z

名字对不上

这个问题好像和当前这个issue无关，不做交叉编译的时候也存在。

这里我关掉了交叉编译单纯Windows编译Windows的时候一样找不到

Issues-translate-bot · 2024-06-01T16:46:21Z

Bot detected the issue body's language is not English, translate it automatically.

The name doesn’t match

This problem seems to have nothing to do with the current issue, and it also exists when cross-compiling is not done.

Here I turned off cross-compilation and it can’t be found when I compile Windows for Windows alone.

Yangff · 2024-06-01T19:24:07Z

lib前缀这个应该是单独的问题，我看了一下

Windows上使用msvc时dll和staticlib应该是都不带lib前缀的

https://github.com/rust-lang/rust/blob/1.78.0/compiler/rustc_target/src/spec/base/windows_msvc.rs

在Windows上使用gnu时，staticlib是带lib前缀的，而dll还是不带
https://github.com/rust-lang/rust/blob/1.78.0/compiler/rustc_target/src/spec/base/windows_gnu.rs

rlib无论如何都带lib前缀

拿gnu做windows的libc的应该比较少吧，再加上拿staticlib做rust内部依赖的应该就更少了，主要是得判断target的类型，如果没有明确set_arch的话得去获得工具链的默认target有点麻烦的。

dll应该是无论如何都不带lib，所以检查的时候应该可以无脑给dll加上lib来做判断。

Issues-translate-bot · 2024-06-01T19:24:18Z

Bot detected the issue body's language is not English, translate it automatically.

The lib prefix should be a separate issue. I took a look.

When using msvc on Windows, dll and staticlib should not have the lib prefix.

https://github.com/rust-lang/rust/blob/1.78.0/compiler/rustc_target/src/spec/base/windows_msvc.rs

When using gnu, staticlib is prefixed with lib, but dll still does not.
https://github.com/rust-lang/rust/blob/1.78.0/compiler/rustc_target/src/spec/base/windows_gnu.rs

rlib is prefixed with lib anyway

There should be relatively few people who use gnu as libc for windows, and even fewer people use staticlib as rust's internal dependency. The main thing is to determine the target type. If set_arch is not specified, it is a bit difficult to obtain the default target of the tool chain. Troublesome.

The dll should not have lib in any case, so when checking, you should be able to add lib to the dll to make a judgment without thinking.

waruqi · 2024-06-02T13:37:56Z

再试试，xmake update -s dev

Issues-translate-bot · 2024-06-02T14:14:28Z

Bot detected the issue body's language is not English, translate it automatically.

Try again, xmake update -s dev

Yangff · 2024-06-02T18:24:24Z

再试试，xmake update -s dev

这次虽然找到了，但是编译的时候不会加载这个依赖

> copy C:\Users\yangf\Documents\Projects\xmake_rust_macro\build\.packages\c\cargo_bind\latest\cache\source\target\release\deps to C:\Users\yangf\Documents\Projects\xmake_rust_macro\build\.packages\c\cargo_bind\latest\1de83963df6043fe9a5ffe11d6b57aef\lib
finding bind from cargo ..
checking for cargo::bind ... bind latest
{
  version = "latest",
  frameworks = {
    "C:\Users\yangf\Documents\Projects\xmake_rust_macro\build\.packages\c\cargo_bind\latest\1de83963df6043fe9a5ffe11d6b57aef\lib\dep1-18415b935cb2aa0e.dll.lib"
  },
  libfiles = { },
  frameworkdirs = {
    "C:\Users\yangf\Documents\Projects\xmake_rust_macro\build\.packages\c\cargo_bind\latest\1de83963df6043fe9a5ffe11d6b57aef\lib"
  }
}

C:\Users\yangf\Documents\Projects\xmake_rust_macro> xmake -v -D
[ 66%]: linking.release bind.exe
rustc --edition 2018 -L dependency=C:\Users\yangf\Documents\Projects\xmake_rust_macro\build\.packages\c\cargo_bind\latest\1de83963df6043fe9a5ffe11d6b57aef\lib --crate-type=bin -o build\windows\x64\release\bind.exe bind\src\main.rs
error: @programdir\core\main.lua:329: @programdir\actions\build\main.lua:148: @programdir\modules\async\runjobs.lua:322: @programdir\rules\rust\build\target.lua:67: @programdir\core\sandbox\modules\os.lua:273: error[E0463]: can't find crate for `dep1`
 --> bind\src\main.rs:1:1
  |
1 | extern crate dep1;
  | ^^^^^^^^^^^^^^^^^^ can't find crate

PS C:\Users\yangf\Documents\Projects\xmake_rust_macro> ls C:\Users\yangf\Documents\Projects\xmake_rust_macro\build\.packages\c\cargo_bind\latest\1de83963df6043fe9a5ffe11d6b57aef\lib

    Directory: C:\Users\yangf\Documents\Projects\xmake_rust_macro\build\.packages\c\cargo_bind\latest\1de83963df6043fe9
a5ffe11d6b57aef\lib

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a---            6/2/2024  1:08 PM            328 bind-519f6883a4a21a7a.d
-a---            6/2/2024  1:08 PM       10354772 bind-519f6883a4a21a7a.lib
-a---            6/2/2024  1:08 PM            496 dep1-18415b935cb2aa0e.d
-a---            6/2/2024  1:08 PM         200704 dep1-18415b935cb2aa0e.dll
-a---            6/2/2024  1:08 PM           1245 dep1-18415b935cb2aa0e.dll.exp
-a---            6/2/2024  1:08 PM           2444 dep1-18415b935cb2aa0e.dll.lib
-a---            6/2/2024  1:08 PM        1404928 dep1-18415b935cb2aa0e.pdb

查看了一下有两个问题导致的这点

首先，libfiles_native = os.files(path.join(librarydir, "*.lib")) 只会去找 .lib，虽然能找到dep1-xxxxxx.dll.lib，但是rustc需要的是 .dll，因为这里是要把它作为编译器的一部分加载进来的。

其次， C:\Users\yangf\xmake\modules\core\tools\rustc.lua 里匹配 nf_framework 用的正则表达式还是

local cratename = basename:match("lib(.-)%-.-") or basename:match("lib(.+)")

于是匹配不到名字这个依赖项就被跳过了。

强行让他加载.dll并且加了一个fallback之后它能成功编译，就是不知道会不会有其它的问题……

libfiles_native = table.join(libfiles_native, os.files(path.join(librarydir, "*.dll")))

    local libraryname = basename:split('-', {plain = true})[1]
    if cratename then
        return {"--extern", cratename .. "=" .. framework}
    end
    return {"--extern", libraryname .. "=" .. framework}

不过到这影响不是很大了，反正依赖是cp都复制完了，不在xmake里直接import需要proc macro的项目应该就行…… 一般做binding的话应该可能也就用用普通的macro就好了。。

Issues-translate-bot · 2024-06-02T18:24:35Z

Bot detected the issue body's language is not English, translate it automatically.

Try again, xmake update -s dev

Although it was found this time, the dependency will not be loaded during compilation.

> copy C:\Users\yangf\Documents\Projects\xmake_rust_macro\build\.packages\c\cargo_bind\latest\cache\source\target\release\deps to C:\Users\yangf\Documents\Projects\xmake_rust_macro\build \.packages\c\cargo_bind\latest\1de83963df6043fe9a5ffe11d6b57aef\lib
finding bind from cargo ..
checking for cargo::bind ... bind latest
{
  version = "latest",
  frameworks = {
    "C:\Users\yangf\Documents\Projects\xmake_rust_macro\build\.packages\c\cargo_bind\latest\1de83963df6043fe9a5ffe11d6b57aef\lib\dep1-18415b935cb2aa0e.dll.lib"
  },
  libfiles = { },
  frameworkdirs = {
    "C:\Users\yangf\Documents\Projects\xmake_rust_macro\build\.packages\c\cargo_bind\latest\1de83963df6043fe9a5ffe11d6b57aef\lib"
  }
}

C:\Users\yangf\Documents\Projects\xmake_rust_macro> xmake -v -D
[66%]: linking.release bind.exe
rustc --edition 2018 -L dependency=C:\Users\yangf\Documents\Projects\xmake_rust_macro\build\.packages\c\cargo_bind\latest\1de83963df6043fe9a5ffe11d6b57aef\lib --crate-type=bin -o build\windows\x64 \release\bind.exe bind\src\main.rs
error: @programdir\core\main.lua:329: @programdir\actions\build\main.lua:148: @programdir\modules\async\runjobs.lua:322: @programdir\rules\rust\build\target. lua:67: @programdir\core\sandbox\modules\os.lua:273: error[E0463]: can't find crate for `dep1`
 --> bind\src\main.rs:1:1
  |
1 | extern crate dep1;
  | ^^^^^^^^^^^^^^^^^ can't find crate

PS C:\Users\yangf\Documents\Projects\xmake_rust_macro> ls C:\Users\yangf\Documents\Projects\xmake_rust_macro\build\.packages\c\cargo_bind\latest\1de83963df6043fe9a5ffe11d6b57aef\lib

    Directory: C:\Users\yangf\Documents\Projects\xmake_rust_macro\build\.packages\c\cargo_bind\latest\1de83963df6043fe9
a5ffe11d6b57aef\lib

Mode LastWriteTime Length Name
---- ------------- ------ ----
-a--- 6/2/2024 1:08 PM 328 bind-519f6883a4a21a7a.d
-a--- 6/2/2024 1:08 PM 10354772 bind-519f6883a4a21a7a.lib
-a--- 6/2/2024 1:08 PM 496 dep1-18415b935cb2aa0e.d
-a--- 6/2/2024 1:08 PM 200704 dep1-18415b935cb2aa0e.dll
-a--- 6/2/2024 1:08 PM 1245 dep1-18415b935cb2aa0e.dll.exp
-a--- 6/2/2024 1:08 PM 2444 dep1-18415b935cb2aa0e.dll.lib
-a--- 6/2/2024 1:08 PM 1404928 dep1-18415b935cb2aa0e.pdb

After checking, there are two problems causing this.

First of all, libfiles_native = os.files(path.join(librarydir, "*.lib")) will only look for .lib. Although dep1-xxxxxx.dll.lib can be found, rustc needs .dll, because it is loaded as part of the compiler.

Secondly, the regular expression used to match nf_framework in C:\Users\yangf\xmake\modules\core\tools\rustc.lua is still the same

local cratename = basename:match("lib(.-)%-.-") or basename:match("lib(.+)")

Therefore, the dependency that does not match the name is skipped.

waruqi · 2024-06-03T06:44:07Z

再试试那个 cargo branch patch

Yangff · 2024-06-03T07:04:54Z

再试试那个 cargo branch patch

cargo branch 加上 5e4bbe9 这个就可以了

Issues-translate-bot · 2024-06-03T07:16:17Z

Bot detected the issue body's language is not English, translate it automatically.

Try that cargo branch patch again

add host dir for cargo #5156

waruqi · 2024-06-03T07:17:50Z

再试试那个 cargo branch patch

cargo branch 加上 5e4bbe9 这个就可以了

这个在 dev 上，我已经 merge 了。再试试

Issues-translate-bot · 2024-06-03T07:47:29Z

Bot detected the issue body's language is not English, translate it automatically.

Try that cargo branch patch again

cargo branch plus 5e4bbe9 This will do it

Issues-translate-bot · 2024-06-03T07:49:01Z

Bot detected the issue body's language is not English, translate it automatically.

Try that cargo branch patch again

cargo branch plus 5e4bbe9 This will do it

This is on dev and I have merged it. try again

Yangff added the bug label May 28, 2024

waruqi added a commit that referenced this issue May 31, 2024

add host dir for cargo #5156

20e1f9b

waruqi mentioned this issue May 31, 2024

add host dir for cargo #5156 #5173

Merged

waruqi added a commit that referenced this issue Jun 3, 2024

Merge pull request #5173 from xmake-io/cargo

af97444

add host dir for cargo #5156

waruqi closed this as completed Jun 3, 2024

waruqi modified the milestones: todo, v2.9.3 Jun 3, 2024

交叉编译Rust项目时，cargo的编译期依赖没有被正确安装 #5156

交叉编译Rust项目时，cargo的编译期依赖没有被正确安装 #5156

Comments

Yangff commented May 28, 2024 • edited Loading

Xmake 版本

操作系统版本和架构

描述问题

期待的结果

工程配置

附加信息和错误日志

Issues-translate-bot commented May 28, 2024

waruqi commented May 29, 2024

Issues-translate-bot commented May 29, 2024

Yangff commented May 29, 2024 • edited Loading

waruqi commented May 31, 2024

Issues-translate-bot commented May 31, 2024

waruqi commented May 31, 2024 • edited Loading

Issues-translate-bot commented May 31, 2024

Yangff commented May 31, 2024 • edited Loading

Yangff commented May 31, 2024 • edited Loading

Issues-translate-bot commented May 31, 2024

waruqi commented May 31, 2024

Issues-translate-bot commented May 31, 2024

Yangff commented May 31, 2024 • edited Loading

Yangff commented May 31, 2024 • edited Loading

Issues-translate-bot commented May 31, 2024

waruqi commented May 31, 2024

waruqi commented May 31, 2024

Issues-translate-bot commented May 31, 2024

Issues-translate-bot commented May 31, 2024

Yangff commented May 31, 2024

Yangff commented May 31, 2024

Issues-translate-bot commented May 31, 2024

waruqi commented May 31, 2024

Issues-translate-bot commented May 31, 2024

Yangff commented May 31, 2024 • edited Loading

Issues-translate-bot commented May 31, 2024

waruqi commented Jun 1, 2024

Yangff commented Jun 1, 2024 • edited Loading

Yangff commented Jun 1, 2024

Issues-translate-bot commented Jun 1, 2024

Yangff commented Jun 1, 2024 • edited Loading

Issues-translate-bot commented Jun 1, 2024

waruqi commented Jun 2, 2024

Issues-translate-bot commented Jun 2, 2024

Yangff commented Jun 2, 2024 • edited Loading

Issues-translate-bot commented Jun 2, 2024

waruqi commented Jun 3, 2024

Yangff commented Jun 3, 2024

Issues-translate-bot commented Jun 3, 2024

waruqi commented Jun 3, 2024

Issues-translate-bot commented Jun 3, 2024

Issues-translate-bot commented Jun 3, 2024

Yangff commented May 28, 2024 •

edited

Loading

Yangff commented May 29, 2024 •

edited

Loading

waruqi commented May 31, 2024 •

edited

Loading

Yangff commented May 31, 2024 •

edited

Loading

Yangff commented May 31, 2024 •

edited

Loading

Yangff commented May 31, 2024 •

edited

Loading

Yangff commented May 31, 2024 •

edited

Loading

Yangff commented May 31, 2024 •

edited

Loading

Yangff commented Jun 1, 2024 •

edited

Loading

Yangff commented Jun 1, 2024 •

edited

Loading

Yangff commented Jun 2, 2024 •

edited

Loading