Add support for new build layout in 24.1 #384

jerboaa · 2024-01-08T17:15:44Z

The new location is, for example, linux-amd64/glibc/liblibchelper.a when it used to be amd64/liblibchelper.a.

This also removes an MX 5.313 conditional which is useless in master of mandrel-packaging which is supposed to build latest graal/master (needing mx 6+).

This fixes build issues seen in CI such as this one:
https://github.com/graalvm/mandrel/actions/runs/7442144672/job/20245208110#step:8:475

jerboaa · 2024-01-08T19:32:02Z

Mac OS build fails with:

Downloading NINJA from ['https://github.com/ninja-build/ninja/releases/download/v1.10.2/ninja-mac.zip']
JvmFuncsFallbacksBuildTask svm-jvmfuncs-fallback-builder: Failed due to error: 1
Error: Process completed with exit code 1.

I have no idea how to fix that and don't have a mac build machine. @zakkak probably something for you?

I'm looking at the windows build failure...

jerboaa · 2024-01-08T19:43:53Z

Windows build failure seems to be:

2024-01-08T17:34:40.6448804Z ninja -j 1 -v 
2024-01-08T17:34:40.6449054Z [6876: started subprocess 4660: ['ninja', '-j', '1', '-v']]
2024-01-08T17:34:40.6449800Z ninja: error: mkdir(D_/a/mandrel-packaging/mandrel-packaging/mandrel/substratevm/mxbuild/jdk23/svm-jvmfuncs-fallback-builder): No such file or directory
2024-01-08T17:34:40.6451788Z [1/3] cl -nologo -showIncludes -ID:\a\mandrel-packaging\mandrel-packaging\mandrel\substratevm\src\com.oracle.svm.native.jvm.windows\src -ID:\a\mandrel-packaging\mandrel-packaging\jdk\include -ID:\a\mandrel-packaging\mandrel-packaging\jdk\include\win32  -MD -Zi -O2 -DJNIEXPORT= -c D:\a\mandrel-packaging\mandrel-packaging\mandrel\substratevm\src\com.oracle.svm.native.jvm.windows\src\JvmFuncs.c -Fosrc\JvmFuncs.obj
2024-01-08T17:34:40.6451882Z ninja: build stopped: .
2024-01-08T17:34:40.6452335Z Building com.oracle.svm.native.jvm.windows_windows-amd64-default with Ninja: Failed due to error: 1

jerboaa · 2024-01-08T20:04:34Z

Windows build failure seems to be:

2024-01-08T17:34:40.6448804Z ninja -j 1 -v 
2024-01-08T17:34:40.6449054Z [6876: started subprocess 4660: ['ninja', '-j', '1', '-v']]
2024-01-08T17:34:40.6449800Z ninja: error: mkdir(D_/a/mandrel-packaging/mandrel-packaging/mandrel/substratevm/mxbuild/jdk23/svm-jvmfuncs-fallback-builder): No such file or directory
2024-01-08T17:34:40.6451788Z [1/3] cl -nologo -showIncludes -ID:\a\mandrel-packaging\mandrel-packaging\mandrel\substratevm\src\com.oracle.svm.native.jvm.windows\src -ID:\a\mandrel-packaging\mandrel-packaging\jdk\include -ID:\a\mandrel-packaging\mandrel-packaging\jdk\include\win32  -MD -Zi -O2 -DJNIEXPORT= -c D:\a\mandrel-packaging\mandrel-packaging\mandrel\substratevm\src\com.oracle.svm.native.jvm.windows\src\JvmFuncs.c -Fosrc\JvmFuncs.obj
2024-01-08T17:34:40.6451882Z ninja: build stopped: .
2024-01-08T17:34:40.6452335Z Building com.oracle.svm.native.jvm.windows_windows-amd64-default with Ninja: Failed due to error: 1

or maybe not. A different run has:

Caused by: java.nio.file.NoSuchFileException: D:\a\mandrel\mandrel\mandrel\substratevm\mxbuild\windows-amd64\com.oracle.svm.native.libchelper\windows-amd64\glibc\libchelper.lib
	at java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:85)
	at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103)
	at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:108)
	at java.base/sun.nio.fs.WindowsFileCopy.copy(WindowsFileCopy.java:108)
	at java.base/sun.nio.fs.WindowsFileSystemProvider.copy(WindowsFileSystemProvider.java:282)
	at java.base/java.nio.file.Files.copy(Files.java:1305)
	at FileSystem.copy(build.java:1606)
	at build.main(build.java:173)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at jdk.compiler/com.sun.tools.javac.launcher.SourceLauncher.execute(SourceLauncher.java:264)
	at jdk.compiler/com.sun.tools.javac.launcher.SourceLauncher.run(SourceLauncher.java:153)
	at jdk.compiler/com.sun.tools.javac.launcher.SourceLauncher.main(SourceLauncher.java:78)

jerboaa · 2024-01-08T20:32:32Z

graalvm/mx@f49c56f seems to be the mx commit which caused this new build failure.

jerboaa · 2024-01-08T21:01:09Z

Not sure what's wrong with the windows PR tester build, but it built fine on windows here:
https://github.com/graalvm/mandrel/actions/runs/7453024302/job/20277533723#step:8:595

zakkak · 2024-01-09T08:04:53Z

I have no idea how to fix that and don't have a mac build machine. @zakkak probably something for you?

On it.

zakkak · 2024-01-09T08:59:29Z

The MacOS issue is caused by adoptium/temurin-build#3602, note however that even after manually renaming the folder the build still fails because in darwin mx uses default in the static lib paths, just like it does in windows.

I also see some formatting issues (tabs vs spaces) causing indentation inconsistencies.

zakkak · 2024-01-09T09:16:07Z

or maybe not. A different run has:

Caused by: java.nio.file.NoSuchFileException: D:\a\mandrel\mandrel\mandrel\substratevm\mxbuild\windows-amd64\com.oracle.svm.native.libchelper\windows-amd64\glibc\libchelper.lib

This looks like a build using master instead of your branch/patch.

The actual issue seems to indeed be some failure to create a directory.

Not sure what's wrong with the windows PR tester build, but it built fine on windows here:
https://github.com/graalvm/mandrel/actions/runs/7453024302/job/20277533723#step:8:595

I see that you use a different mandrel-packaging commit in the passing run (294f441 instead of cec4782) the only difference however appears to be the exclusion of --verbose in the latter 🤷, see 294f441...cec4782

jerboaa · 2024-01-09T09:19:20Z

The MacOS issue is caused by adoptium/temurin-build#3602, note however that even after manually renaming the folder the build still fails because in darwin mx uses default in the static lib paths, just like it does in windows.

Thanks. I'll fix it.

I also see some formatting issues (tabs vs spaces) causing indentation inconsistencies.

OK. I'll fix those.

jerboaa · 2024-01-09T09:23:12Z

or maybe not. A different run has:
Caused by: java.nio.file.NoSuchFileException: D:\a\mandrel\mandrel\mandrel\substratevm\mxbuild\windows-amd64\com.oracle.svm.native.libchelper\windows-amd64\glibc\libchelper.lib

This looks like a build using master instead of your branch/patch.

OK. Ignore that then.

The actual issue seems to indeed be some failure to create a directory.

Yes, the failing build without --verbose (which is too noisy to see the build failure clearly) is:

(ninja: error: mkdir(D_/a/mandrel-packaging/mandrel-packaging/mandrel/substratevm/mxbuild/jdk23/svm-jvmfuncs-fallback-builder): No such file or directory

[1/3] CC src\JvmFuncs.obj

ninja: build stopped: .

, -2)
Building com.oracle.svm.native.jvm.windows_windows-amd64-default with Ninja: Failed due to error: 1
java.net.SocketException: Connection reset
java.net.SocketException: Connection reset

Not sure what's wrong with the windows PR tester build, but it built fine on windows here:
https://github.com/graalvm/mandrel/actions/runs/7453024302/job/20277533723#step:8:595

I see that you use a different mandrel-packaging commit in the passing run (294f441 instead of cec4782) the only difference however appears to be the exclusion of --verbose in the latter 🤷, see 294f441...cec4782

Yes. Only --verbose is the difference. But somehow the manual workflow doesn't have the mkdir issue, while the PR tester has.

jerboaa · 2024-01-09T10:07:44Z

The MacOS issue is caused by adoptium/temurin-build#3602, note however that even after manually renaming the folder

Do you have a patch for the rename until adoptium/temurin-build#3602 is fixed?

zakkak · 2024-01-09T10:14:07Z

Do you have a patch for the rename until adoptium/temurin-build#3602 is fixed?

No, I only tested it locally by doing the rename by hand.

jerboaa · 2024-01-09T11:34:27Z

Mac build is OK now. https://github.com/graalvm/mandrel/actions/runs/7460492845/job/20298676477 is a run verifying the windows build (which looks good so far). Something is strange on the PR tester, but I'm not sure what it is so I'll stop there and let somebody with a windows box look into it. I'll clean up the commits, re-test and would then like to merge so as to unbreak the windows build in CI. @zakkak How does that sound?

zakkak · 2024-01-09T12:01:37Z

Sounds good. FWIW, some differences I see in the mandrel-packaging CI vs the mandrel CI:

mandrel-packaging uses python 3.10 while mandrel uses 3.8
mandrel-packaging checks out mx master, while mandrel checks out the version defined in graal/common.json

The new location is, for example, linux-amd64/glibc/liblibchelper.a when it used to be amd64/liblibchelper.a This also removes an MX 5.313 conditional which is useless in master of mandrel-packaging which is supposed to build latest graal/master (needing mx 6+).

jerboaa · 2024-01-09T13:19:04Z

@zakkak Please review, thanks!

zakkak

LGTM, thanks for taking care of this @jerboaa

jerboaa · 2024-01-09T13:44:22Z

Sounds good. FWIW, some differences I see in the mandrel-packaging CI vs the mandrel CI:
1. mandrel-packaging uses python 3.10 while mandrel uses 3.8

2. mandrel-packaging checks out mx `master`, while mandrel checks out the version defined in `graal/common.json`

FWIW, https://github.com/jerboaa/graal/actions/runs/7461810722/job/20302721226 is a build with python 3.10.11 (as the PR tester uses). It built fine, so that's not it.

jerboaa · 2024-01-12T10:26:40Z

mandrel-packaging checks out mx master, while mandrel checks out the version defined in graal/common.json

We now also have PR tester runs with mx version from graal/common.json and that shows the same windows issue. That's not it either.

jerboaa requested a review from zakkak January 8, 2024 17:15

oracle-contributor-agreement bot added the OCA Verified All contributors have signed the Oracle Contributor Agreement. label Jan 8, 2024

jerboaa force-pushed the fix_liblibchelper branch from de8f1c6 to 294f441 Compare January 8, 2024 20:48

zakkak mentioned this pull request Jan 9, 2024

Wrong folder name in published MacOS x64 builds of static libraries for 23-EA adoptium/temurin-build#3602

Closed

This was referenced Jan 9, 2024

[CI] Quarkus main fails with Java 23 mandrel build of graal/master on Linux graalvm/mandrel#652

Closed

[CI] Quarkus main fails with Java 23 mandrel build of graal/master on Windows graalvm/mandrel#653

Closed

jerboaa added 4 commits January 9, 2024 14:13

Move to JDK 23 for building

8ad133f

Fix Mac build (uses "default") and tabs vs. spaces

979d97d

Work around MacOS X static libs issue

5780fef

jerboaa force-pushed the fix_liblibchelper branch from 7ad23bd to 5780fef Compare January 9, 2024 13:15

zakkak approved these changes Jan 9, 2024

View reviewed changes

zakkak merged commit e88c8b3 into graalvm:master Jan 9, 2024

jerboaa mentioned this pull request Jan 11, 2024

Fix destination directory for Linux c libs #385

Merged

This was referenced Jan 11, 2024

[PR tester] Windows build fails - failing to create a directory in ninja build #386

Closed

[24.0] Backport: Introduce --disable-debuginfo-stripping option #390

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for new build layout in 24.1 #384

Add support for new build layout in 24.1 #384

jerboaa commented Jan 8, 2024

jerboaa commented Jan 8, 2024

jerboaa commented Jan 8, 2024

jerboaa commented Jan 8, 2024

jerboaa commented Jan 8, 2024

jerboaa commented Jan 8, 2024

zakkak commented Jan 9, 2024

zakkak commented Jan 9, 2024

zakkak commented Jan 9, 2024

jerboaa commented Jan 9, 2024

jerboaa commented Jan 9, 2024

jerboaa commented Jan 9, 2024

zakkak commented Jan 9, 2024

jerboaa commented Jan 9, 2024

zakkak commented Jan 9, 2024

jerboaa commented Jan 9, 2024

zakkak left a comment

jerboaa commented Jan 9, 2024

jerboaa commented Jan 12, 2024

Add support for new build layout in 24.1 #384

Add support for new build layout in 24.1 #384

Conversation

jerboaa commented Jan 8, 2024

jerboaa commented Jan 8, 2024

jerboaa commented Jan 8, 2024

jerboaa commented Jan 8, 2024

jerboaa commented Jan 8, 2024

jerboaa commented Jan 8, 2024

zakkak commented Jan 9, 2024

zakkak commented Jan 9, 2024

zakkak commented Jan 9, 2024

jerboaa commented Jan 9, 2024

jerboaa commented Jan 9, 2024

jerboaa commented Jan 9, 2024

zakkak commented Jan 9, 2024

jerboaa commented Jan 9, 2024

zakkak commented Jan 9, 2024

jerboaa commented Jan 9, 2024

zakkak left a comment

Choose a reason for hiding this comment

jerboaa commented Jan 9, 2024

jerboaa commented Jan 12, 2024