Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

protobuf: Bump version/add depends on abseil-cpp #25566

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

vidplace7
Copy link
Contributor

@vidplace7 vidplace7 commented Dec 16, 2024

Maintainer: @neheb, adding myself as a maintainer as well.
Compile tested:

Description:
Update protobuf and add Python bindings (this can be split into another PR if necessary).

Note that protobuf versions >= v22 require abseil-cpp.
https://protobuf.dev/support/migration/#abseil

Depends on:

Marking as draft for now, pending abseil-cpp review.

@vidplace7 vidplace7 force-pushed the protobuf_update branch 8 times, most recently from 7c1321d to 385334f Compare December 17, 2024 04:45
@vidplace7
Copy link
Contributor Author

The setup.py that we actually distribute in our source packages is located in python/dist/setup.py. It is not possible to build this setup.py directly from the GitHub repo or GitHub release tarball, because it depends on the file layout of our Python source package (as distributed on PyPI). The Python source package uses a layout that pulls together all of the things Python needs:

Python Source Package Path GitHub Repo Path Description
setup.py python/dist/setup.py
google/protobuf/* python/google/protobuf/* pure Python sources
python/* python/* C extension sources
utf8_range/* third_party/utf8_range C UTF-8 Validation Library
upb/* upb/upb/* C Protobuf Library

Users who want to build their own Python packages should build from our source package on PyPI, not from our GitHub repo or our GitHub release tarball.

It is also possible to build our source package from GitHub using the following command (this requires Bazel):

$ bazel build //python/dist:source_wheel

PiperOrigin-RevId: 603162788


Found this message deep in the protobuf commits.

@BKPepe seems like Python bindings should be built from PyPI sources. I will remove the Python bindings from this change for now.

I've not seen examples of other packages that bundle a cmake package an a pypi package in the same Makefile. I'd imagine the python bindings will need to be split up into it's own package?

@vidplace7 vidplace7 changed the title protobuf: Bump version and add Python bindings protobuf: Bump version/add depends on abseil-cpp Dec 17, 2024
@vidplace7 vidplace7 force-pushed the protobuf_update branch 3 times, most recently from a9eec0a to 4cbf179 Compare December 17, 2024 06:54
@vortexilation
Copy link
Contributor

vortexilation commented Dec 19, 2024

How you would overcome the protobuf-c & node compatibility?.
Protobuf-c currently only supports up to protobuf 29.1?.

Node compilation issue with abesil-cpp.

[EDIT]
Linking error (glibc, x86_64):

[338/342] Linking CXX executable protoc-gen-upb_minitable-29.1.0
[339/342] Creating executable symlink protoc-gen-upb_minitable
[340/342] Building CXX object CMakeFiles/protoc.dir/src/google/protobuf/compiler/main.cc.o
[341/342] Linking CXX executable protoc-29.1.0
FAILED: protoc-29.1.0 
/usr/bin/ld: /home/user/works/openwrt/staging_dir/hostpkg/lib/libabsl_status.a(status.cc.o):(.rodata+0xe0): multiple definition of `absl::lts_20240722::Status::kMovedFromString'; libprotoc.a(command_line_interface.cc.o):(.rodata._ZN4absl12lts_202407226Status16kMovedFromStringE[_ZN4absl12lts_202407226Status16kMovedFromStringE]+0x0): first defined here

Linking error fixed by :

CMAKE_HOST_OPTIONS += \
	-DABSL_PROPAGATE_CXX_STD=ON \
	-DCMAKE_CXX_STANDARD=17 \
	-DABSL_ENABLE_INSTALL=ON \
	-DABSL_USE_GOOGLETEST_HEAD=OFF

CMAKE_OPTIONS += \
	-DABSL_PROPAGATE_CXX_STD=ON \
	-DCMAKE_CXX_STANDARD=17 \
	-DABSL_ENABLE_INSTALL=ON \
	-DABSL_USE_GOOGLETEST_HEAD=OFF

The issue is with CMAKE_CXX_STANDARD, taken from Microsoft vcpkg.

[EDIT3]
Just re-checking, protobuf-c seems able to compile with protobuf 29.1 , need specifically the "google-protobuf-26-fixes_plus_pull-714" branch :

PKG_NAME:=libprotobuf-c
PKG_VERSION:=1.5.0
PKG_RELEASE:=1

PKG_SOURCE_PROTO:=git
PKG_SOURCE_URL:=https://github.com/protobuf-c/protobuf-c.git
PKG_MIRROR_HASH:=4a4f291be3f0f15ff0a7d1dc874fc4ecc907bd1838a9adea6d193ee3de7a4dc1
PKG_SOURCE_DATE:=2024-12-21
PKG_SOURCE_VERSION:=86a89d1e758557e5df60cf47ed98df837ed93428

@vidplace7
Copy link
Contributor Author

vidplace7 commented Dec 19, 2024

How you would overcome the protobuf-c & node compatibility?. Protobuf-c currently only supports up to protobuf 26.

Node compilation issue with abesil-cpp.

My suggestion is to pinning the protobuf version with 25.3.

This seems like a wise move, so that protobuf-c isn't broken. Thanks for linking that discussion!
For nodejs... this is going to require further investigation.

[EDIT] Linking error (glibc, x86_64):

[338/342] Linking CXX executable protoc-gen-upb_minitable-29.1.0
[339/342] Creating executable symlink protoc-gen-upb_minitable
[340/342] Building CXX object CMakeFiles/protoc.dir/src/google/protobuf/compiler/main.cc.o
[341/342] Linking CXX executable protoc-29.1.0
FAILED: protoc-29.1.0 
/usr/bin/ld: /home/user/works/openwrt/staging_dir/hostpkg/lib/libabsl_status.a(status.cc.o):(.rodata+0xe0): multiple definition of `absl::lts_20240722::Status::kMovedFromString'; libprotoc.a(command_line_interface.cc.o):(.rodata._ZN4absl12lts_202407226Status16kMovedFromStringE[_ZN4absl12lts_202407226Status16kMovedFromStringE]+0x0): first defined here

Linking error fixed by :

CMAKE_HOST_OPTIONS += \
	-DABSL_PROPAGATE_CXX_STD=ON \
	-DCMAKE_CXX_STANDARD=17 \
	-DABSL_ENABLE_INSTALL=ON \
	-DABSL_USE_GOOGLETEST_HEAD=OFF

CMAKE_OPTIONS += \
	-DABSL_PROPAGATE_CXX_STD=ON \
	-DCMAKE_CXX_STANDARD=17 \
	-DABSL_ENABLE_INSTALL=ON \
	-DABSL_USE_GOOGLETEST_HEAD=OFF

The issue is with CMAKE_CXX_STANDARD, taken from Microsoft vcpkg.

I drew inspiration from OpenEmbedded and buildroot for the abseil-cpp packaging / patching.

Wouldn't setting the CXXSTD to 17 preclude any targets that don't support C++17 (but do support 14)? Seems Microsoft is addressing that concern with some conditional logic, from your example.

set(ABSL_USE_CXX17_OPTION "")
if("cxx17" IN_LIST FEATURES)
    set(ABSL_USE_CXX17_OPTION "-DCMAKE_CXX_STANDARD=17")
endif()

A similar approach may be warranted here.

@vortexilation
Copy link
Contributor

Sorry about my previous info, protobuf-c seems compile with protobuf 29.1, but I am not sure either package which are depend on protobuf and protobuf-c will also compile. For example netdata.

For nodejs... this is going to require further investigation.

Yes, nodejs is tough one to fix. My temp workaround is move back with version 20.18.1 LTS which doesn't use abseil-cpp at all.

Wouldn't setting the CXXSTD to 17 preclude any targets that don't support C++17 (but do support 14)? Seems Microsoft is addressing that concern with some conditional logic, from your example.

In my build environment (glibc, targeting x86_64), compilation will still succeed without :

	-DABSL_PROPAGATE_CXX_STD=ON \
	-DCMAKE_CXX_STANDARD=17 \

The question is, is it default into 17 or 14. More likely or probably into 17 by default as with 14 I am getting those linking errors.

@vortexilation
Copy link
Contributor

vortexilation commented Dec 19, 2024

Really weird, now node (version 22.12.0 LTS) doesn't segmentation fault anymore when compiling node-yarn!. Patch for node still needed for compilation.

Is there any reason on why you don't use absl_DIR?
What do you think of :

CMAKE_HOST_OPTIONS += \
	-Dprotobuf_ABSL_PROVIDER=package \
	-Dprotobuf_BUILD_PROTOC_BINARIES=ON \
	-Dprotobuf_BUILD_TESTS=OFF \
	-Dprotobuf_BUILD_EXAMPLES=OFF \
	-DBUILD_SHARED_LIBS=OFF \
	-Dabsl_DIR=$(STAGING_DIR_HOSTPKG)/lib/cmake/absl \
	-DCMAKE_INSTALL_LIBDIR=lib

CMAKE_OPTIONS += \
	-Dprotobuf_ABSL_PROVIDER=package \
	-Dprotobuf_BUILD_PROTOC_BINARIES=ON \
	-Dprotobuf_BUILD_TESTS=OFF \
	-Dprotobuf_BUILD_EXAMPLES=OFF \
	-Dprotobuf_WITH_ZLIB=ON \
	-Dabsl_DIR=$(STAGING_DIR)/usr/lib/cmake/absl \
	-DBUILD_SHARED_LIBS=ON

[EDIT]
Netdata 2.0.3 compiled fine.

+# PROPERTY INSTALL_RPATH "$ORIGIN/../${CMAKE_INSTALL_LIBDIR}")
elseif (APPLE)
set_property(TARGET ${binary}
PROPERTY INSTALL_RPATH "@loader_path/../lib")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can this patch be cleaned up?
or upstreamed?

maybe it's fine, but i'm a bit lazy (right now) to try to understand it :)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added that. IIRC it causes linking to OS libraries instead of OpenWrt ones.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, as of d24229e this should not be needed.

@tiagogaspar8
Copy link
Contributor

This is not compiling on my side with the folloing error:

[337/342] : && /mnt/openwrt/ss_mod/staging_dir/host/bin/g++ -O2 -I/mnt/openwrt/ss_mod/staging_dir/host/include -I/mnt/openwrt/ss_mod/staging_dir/hostpkg/include -I/mnt/openwrt/ss_mod/staging_dir/target-aarch64_cortex-a53_musl/host/include -DNDEBUG -L/mnt/openwrt/ss_mod/staging_dir/host/lib -L/mnt/openwrt/ss_mod/staging_dir/hostpkg/lib -L/mnt/openwrt/ss_mod/staging_dir/target-aarch64_cortex-a53_musl/host/lib    -Wl,--dependency-file=CMakeFiles/protoc.dir/link.d CMakeFiles/protoc.dir/src/google/protobuf/compiler/main.cc.o -o protoc-29.1.0  libprotoc.a  libprotobuf.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_base.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_cord.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_die_if_null.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_hash.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_log_initialize.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_log_globals.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_log_severity.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_random_distributions.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_status.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_statusor.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_strings.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_synchronization.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_time.a  /mnt/openwrt/ss_mod/staging_dir/host/lib/libz.a  third_party/utf8_range/libutf8_validity.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_log_internal_check_op.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_leak_check.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_log_internal_conditions.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_log_internal_message.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_log_internal_nullguard.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_examine_stack.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_log_internal_format.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_log_internal_proto.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_log_internal_log_sink_set.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_log_sink.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_log_entry.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_flags_internal.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_flags_marshalling.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_flags_reflection.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_flags_config.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_flags_program_name.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_flags_private_handle_accessor.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_flags_commandlineflag.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_flags_commandlineflag_internal.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_log_internal_globals.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_log_globals.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_vlog_config_internal.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_log_internal_fnmatch.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_raw_hash_set.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_hash.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_city.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_low_level_hash.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_hashtablez_sampler.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_random_distributions.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_random_seed_sequences.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_random_internal_pool_urbg.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_random_internal_randen.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_random_internal_randen_hwaes.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_random_internal_randen_hwaes_impl.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_random_internal_randen_slow.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_random_internal_platform.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_random_internal_seed_material.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_random_seed_gen_exception.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_status.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_cord.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_cordz_info.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_cord_internal.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_cordz_functions.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_exponential_biased.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_cordz_handle.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_crc_cord_state.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_crc32c.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_crc_internal.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_crc_cpu_detect.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_bad_optional_access.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_strerror.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_str_format_internal.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_synchronization.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_stacktrace.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_symbolize.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_debugging_internal.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_demangle_internal.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_demangle_rust.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_decode_rust_punycode.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_utf8_for_code_point.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_graphcycles_internal.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_kernel_timeout_internal.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_malloc_internal.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_time.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_strings.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_strings_internal.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_string_view.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_base.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_spinlock_wait.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_throw_delegate.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_int128.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_civil_time.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_time_zone.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_bad_variant_access.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_raw_logging_internal.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_log_severity.a && :
FAILED: protoc-29.1.0 
: && /mnt/openwrt/ss_mod/staging_dir/host/bin/g++ -O2 -I/mnt/openwrt/ss_mod/staging_dir/host/include -I/mnt/openwrt/ss_mod/staging_dir/hostpkg/include -I/mnt/openwrt/ss_mod/staging_dir/target-aarch64_cortex-a53_musl/host/include -DNDEBUG -L/mnt/openwrt/ss_mod/staging_dir/host/lib -L/mnt/openwrt/ss_mod/staging_dir/hostpkg/lib -L/mnt/openwrt/ss_mod/staging_dir/target-aarch64_cortex-a53_musl/host/lib    -Wl,--dependency-file=CMakeFiles/protoc.dir/link.d CMakeFiles/protoc.dir/src/google/protobuf/compiler/main.cc.o -o protoc-29.1.0  libprotoc.a  libprotobuf.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_base.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_cord.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_die_if_null.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_hash.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_log_initialize.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_log_globals.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_log_severity.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_random_distributions.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_status.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_statusor.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_strings.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_synchronization.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_time.a  /mnt/openwrt/ss_mod/staging_dir/host/lib/libz.a  third_party/utf8_range/libutf8_validity.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_log_internal_check_op.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_leak_check.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_log_internal_conditions.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_log_internal_message.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_log_internal_nullguard.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_examine_stack.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_log_internal_format.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_log_internal_proto.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_log_internal_log_sink_set.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_log_sink.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_log_entry.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_flags_internal.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_flags_marshalling.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_flags_reflection.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_flags_config.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_flags_program_name.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_flags_private_handle_accessor.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_flags_commandlineflag.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_flags_commandlineflag_internal.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_log_internal_globals.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_log_globals.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_vlog_config_internal.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_log_internal_fnmatch.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_raw_hash_set.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_hash.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_city.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_low_level_hash.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_hashtablez_sampler.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_random_distributions.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_random_seed_sequences.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_random_internal_pool_urbg.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_random_internal_randen.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_random_internal_randen_hwaes.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_random_internal_randen_hwaes_impl.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_random_internal_randen_slow.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_random_internal_platform.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_random_internal_seed_material.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_random_seed_gen_exception.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_status.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_cord.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_cordz_info.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_cord_internal.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_cordz_functions.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_exponential_biased.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_cordz_handle.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_crc_cord_state.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_crc32c.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_crc_internal.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_crc_cpu_detect.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_bad_optional_access.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_strerror.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_str_format_internal.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_synchronization.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_stacktrace.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_symbolize.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_debugging_internal.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_demangle_internal.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_demangle_rust.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_decode_rust_punycode.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_utf8_for_code_point.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_graphcycles_internal.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_kernel_timeout_internal.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_malloc_internal.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_time.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_strings.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_strings_internal.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_string_view.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_base.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_spinlock_wait.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_throw_delegate.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_int128.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_civil_time.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_time_zone.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_bad_variant_access.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_raw_logging_internal.a  /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_log_severity.a && :
/usr/bin/ld: /mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_status.a(status.cc.o):(.rodata+0x120): multiple definition of `absl::lts_20240722::Status::kMovedFromString'; libprotoc.a(command_line_interface.cc.o):(.rodata._ZN4absl12lts_202407226Status16kMovedFromStringE[_ZN4absl12lts_202407226Status16kMovedFromStringE]+0x0): first defined here
collect2: error: ld returned 1 exit status

@vortexilation
Copy link
Contributor

@tiagogaspar8

You could try my finding above.

@tiagogaspar8
Copy link
Contributor

tiagogaspar8 commented Dec 27, 2024

no dice @vortexilation, the error states:
/mnt/openwrt/ss_mod/staging_dir/hostpkg/lib/libabsl_status.a(status.cc.o):(.rodata+0x120): multiple definition of absl::lts_20240722::Status::kMovedFromString'; libprotoc.a(command_line_interface.cc.o):(.rodata._ZN4absl12lts_202407226Status16kMovedFromStringE[_ZN4absl12lts_202407226Status16kMovedFromStringE]+0x0): first defined here`
so it seems to be something else, not sure how this is compiling in actions though.

EDIT

Nvm it works, now I'm battling another error:

make[3]: Leaving directory '/mnt/openwrt/ss_mod/package/utils/usbmode'
time: package/utils/usbmode/compile#0.31#0.21#0.51
make[2]: Leaving directory '/mnt/openwrt/ss_mod'
make[2]: Entering directory '/mnt/openwrt/ss_mod'
rm -rf /mnt/openwrt/ss_mod/staging_dir/packages/mvebu
mkdir -p /mnt/openwrt/ss_mod/staging_dir/packages/mvebu
ln -s /mnt/openwrt/ss_mod/bin/targets/mvebu/cortexa9/packages/*.apk /mnt/openwrt/ss_mod/staging_dir/packages/mvebu; ln -s /mnt/openwrt/ss_mod/bin/packages/arm_cortex-a9_neon/base/*.apk /mnt/openwrt/ss_mod/staging_dir/packages/mvebu; ln -s /mnt/openwrt/ss_mod/bin/packages/arm_cortex-a9_neon/luci/*.apk /mnt/openwrt/ss_mod/staging_dir/packages/mvebu; ln -s /mnt/openwrt/ss_mod/bin/packages/arm_cortex-a9_neon/packages/*.apk /mnt/openwrt/ss_mod/staging_dir/packages/mvebu;   
(cd /mnt/openwrt/ss_mod/staging_dir/packages/mvebu && /mnt/openwrt/ss_mod/staging_dir/host/bin/apk mkndx \
		--root /mnt/openwrt/ss_mod \
		--keys-dir /mnt/openwrt/ss_mod \
		--sign /mnt/openwrt/ss_mod/private-key.pem \
		--output packages.adb \
		*.apk; \
)
Index has 376 packages (of which 376 are new)
find /mnt/openwrt/ss_mod/staging_dir/target-arm_cortex-a9+neon_musl_eabi/root-mvebu -type d | xargs -r chmod 0755
rm -rf /mnt/openwrt/ss_mod/build_dir/target-arm_cortex-a9+neon_musl_eabi/root-mvebu /mnt/openwrt/ss_mod/build_dir/target-arm_cortex-a9+neon_musl_eabi/root.orig-mvebu
mkdir -p /mnt/openwrt/ss_mod/build_dir/target-arm_cortex-a9+neon_musl_eabi/root-mvebu/tmp
IPKG_INSTROOT=/mnt/openwrt/ss_mod/build_dir/target-arm_cortex-a9+neon_musl_eabi/root-mvebu /mnt/openwrt/ss_mod/staging_dir/host/bin/fakeroot /mnt/openwrt/ss_mod/staging_dir/host/bin/apk --root /mnt/openwrt/ss_mod/build_dir/target-arm_cortex-a9+neon_musl_eabi/root-mvebu --keys-dir /mnt/openwrt/ss_mod --no-logfile --preserve-env add --no-cache --initdb --no-scripts --arch arm_cortex-a9_neon \
	--repositories-file /dev/null --repository file:///mnt/openwrt/ss_mod/staging_dir/packages/mvebu/packages.adb \
	$(cat /mnt/openwrt/ss_mod/tmp/apk_install_list)
ERROR: unable to select packages:
  abseil-cpp (no such package):
    required by: protobuf-lite-29.1-r1[abseil-cpp] protobuf-29.1-r1[abseil-cpp]

@vortexilation
Copy link
Contributor

@tiagogaspar8
Yes, I was also experienced the same thing, somehow abseil-cpp need install clause for it being generated an apk package,
add this in abseil-cpp Makefile :

TARGET_CFLAGS += $(FPIC)

define Package/abseil-cpp/install
	$(INSTALL_DIR) $(1)/usr/lib
	$(CP) $(PKG_INSTALL_DIR)/usr/lib/* $(1)/usr/lib/
endef

@tiagogaspar8
Copy link
Contributor

Damn, you always know how to fix stuff @vortexilation ahahah

I can now compile everything and I tested netdata's claim and it works, I'll update my PR, yet it needs abseil and protobuf fixed, curious to see the outcome of this PR.

@vortexilation
Copy link
Contributor

@tiagogaspar8
Thank you, I am just trying my best hehe.

protobuf-c need a PR also, perhaps someone might wanted to open a PR?.

PKG_NAME:=libprotobuf-c
PKG_VERSION:=1.5.0
PKG_RELEASE:=1

PKG_SOURCE_PROTO:=git
PKG_SOURCE_URL:=https://github.com/protobuf-c/protobuf-c.git
PKG_MIRROR_HASH:=4a4f291be3f0f15ff0a7d1dc874fc4ecc907bd1838a9adea6d193ee3de7a4dc1
PKG_SOURCE_DATE:=2024-12-21
PKG_SOURCE_VERSION:=86a89d1e758557e5df60cf47ed98df837ed93428
#PKG_BUILD_DIR:=$(BUILD_DIR)/protobuf-c-$(PKG_VERSION)
#HOST_BUILD_DIR:=$(BUILD_DIR_HOST)/protobuf-c-$(PKG_VERSION)

Based on gentoo's commit from here , libprotobuf-c also needs this patch,
001-revert-protect-against-being-invoked-against-edition-syntax-proto-files.patch :

diff -Naur a/protoc-c/c_generator.h b/protoc-c/c_generator.h
--- a/protoc-c/c_generator.h
+++ b/protoc-c/c_generator.h
@@ -93,12 +93,6 @@
                 const std::string& parameter,
                 OutputDirectory* output_directory,
                 std::string* error) const;
-
-#if GOOGLE_PROTOBUF_VERSION >= 5026000
-  uint64_t GetSupportedFeatures() const { return CodeGenerator::FEATURE_SUPPORTS_EDITIONS; }
-  Edition GetMinimumEdition() const { return Edition::EDITION_PROTO2; }
-  Edition GetMaximumEdition() const { return Edition::EDITION_PROTO3; }
-#endif
 };
 
 }  // namespace c

Tested with compiling frr package with this PR + my changes.

@tiagogaspar8
Copy link
Contributor

I think the next steps would be:

  1. Fixing Abseil (probably with a new pull req)
  2. Merging this protobuf pull request
  3. Open a new pull for the protoc-c
  4. Merge netdata (finaly 😭 )

Now, regarding Abseil, is the fix you provided the correct fix for this issue @vortexilation?
I ask this because I can open the pull for both abseil and protoc, but I want to be sure these are the correct fixes, especially for abseil.

@vortexilation
Copy link
Contributor

Good idea about the steps.

I am sure for libprotobuf-c is the correct one, even though it's from test branch. I am not sure if using test branch is allowed in OpenWrt ?. Also there are no other choice other than using test branch.

Perhaps @neheb & @vidplace7 may comment on my proposed fixes.

Also please be warned, mostly I am only compile tested not run-tested or testing thoroughly the functionalities of aforementioned compilation result.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants