Protobuf is unbuildable without --enable_runfiles

[freeform-andre]

Trying to compile protobuf with Bazel on windows machines that do not have runfiles enabled will lead to errors like the one linked below. I highly suspect the behavior is caused by the _virtual_includes directory being in a different form when runfiles are enabled vs when runfiles are not.

What version of protobuf and what language are you using?
Version: v28.3
Language: C++/Java/Python/C#/Ruby/PHP/Objective-C/Javascript

What operating system (Linux, Windows, ...) and version?
Windows 10

What runtime / compiler are you using (e.g., python version or gcc version)
MSVC

What did you do?
Steps to reproduce the behavior:

1. Setup a Windows machine (ideally without symlinks enabled)
2. Clone the Protobuf repository
3. Run bazel --nowindows_enable_symlinks build //:protobuf --noenable_runfiles
4. See error

What did you expect to see
Successful builds.

What did you see instead?

ERROR: C:/bzl/jhxuwc6s/external/com_google_protobuf/src/google/protobuf/util/BUILD.bazel:11:11: Compiling src/google/protobuf/util/delimited_message_util.cc [for tool] failed: (Exit 2): cl.exe failed: error executing CppCompile command (from target @@com_google_protobuf//src/google/protobuf/util:delimited_message_util) C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\bin\HostX64\x64\cl.exe /nologo /DCOMPILER_MSVC /DNOMINMAX /D_WIN32_WINNT=0x0601 /D_CRT_SECURE_NO_DEPRECATE ... (remaining 60 arguments skipped)
bazel-out/win-opt-exec/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/protobuf_lite\google/protobuf/message_lite.h(863): error C2039: 'StrongReferenceToType': is not a member of 'google::protobuf::internal'
bazel-out/win-opt-exec/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/protobuf_lite\google/protobuf/message_lite.h(67): note: see declaration of 'google::protobuf::internal'
bazel-out/win-opt-exec/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/protobuf_lite\google/protobuf/message_lite.h(1099): error C2039: 'StrongReferenceToType': is not a member of 'google::protobuf::internal'
bazel-out/win-opt-exec/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/protobuf_lite\google/protobuf/message_lite.h(1075): note: see declaration of 'google::protobuf::internal'

The full command line command:

C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.29.30133/bin/HostX64/x64/cl.exe
    /nologo
    /DCOMPILER_MSVC
    /DNOMINMAX
    /D_WIN32_WINNT=0x0601
    /D_CRT_SECURE_NO_DEPRECATE
    /D_CRT_SECURE_NO_WARNINGS
    /bigobj
    /Zm500
    /EHsc
    /wd4351
    /wd4291
    /wd4250
    /wd4996
    /Iexternal/com_google_protobuf
    /Ibazel-out/win-opt-exec/bin/external/com_google_protobuf
    /Iexternal/com_google_absl
    /Ibazel-out/win-opt-exec/bin/external/com_google_absl
    /Ibazel-out/win-opt-exec/bin/external/com_google_protobuf/src/google/protobuf/util/_virtual_includes/delimited_message_util
    /Ibazel-out/win-opt-exec/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/protobuf_lite
    /Ibazel-out/win-opt-exec/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/arena
    /Ibazel-out/win-opt-exec/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/arena_align
    /Ibazel-out/win-opt-exec/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/port
    /Ibazel-out/win-opt-exec/bin/external/com_google_protobuf/src/google/protobuf/stubs/_virtual_includes/lite
    /Ibazel-out/win-opt-exec/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/arena_allocation_policy
    /Ibazel-out/win-opt-exec/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/arena_cleanup
    /Ibazel-out/win-opt-exec/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/string_block
    /Ibazel-out/win-opt-exec/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/internal_visibility
    /Ibazel-out/win-opt-exec/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/varint_shuffle
    /Ibazel-out/win-opt-exec/bin/external/com_google_protobuf/src/google/protobuf/io/_virtual_includes/io
    /Ibazel-out/win-opt-exec/bin/external/com_google_protobuf/src/google/protobuf/io/_virtual_includes/io_win32
    /Ibazel-out/win-opt-exec/bin/external/com_google_protobuf/third_party/utf8_range/_virtual_includes/utf8_validity
    /Ibazel-out/win-opt-exec/bin/external/com_google_protobuf/third_party/utf8_range/_virtual_includes/utf8_range
    /showIncludes
    /O2
    /Oy-
    /DNDEBUG
    /wd4117
    -D__DATE__="redacted"
    -D__TIMESTAMP__="redacted"
    -D__TIME__="redacted"
    /Gy
    /Gw
    /std:c++20
    /wd4018
    /wd4100
    /wd4244
    /wd4267
    /DUNICODE
    /wd4065
    /wd4146
    /wd4244
    /wd4251
    /wd4267
    /wd4305
    /wd4307
    /wd4309
    /wd4334
    /wd4355
    /wd4506
    /wd4800
    /wd4996
    /wd4100
    /Fobazel-out/win-opt-exec/bin/external/com_google_protobuf/src/google/protobuf/util/_objs/delimited_message_util/delimited_message_util.obj
    /c
    external/com_google_protobuf/src/google/protobuf/util/delimited_message_util.cc

Make sure you include information that can help us debug (full error message, exception listing, stack trace, logs).

Anything else we should know about your project / environment

[acozzette]

My understanding is that unfortunately Bazel support for Windows is not great at the moment. We have had a lot of problems with MSVC cl.exe not being able to handle file paths longer than 256 characters, and Bazel seems to like producing really long file paths. It's possible that that is related to this problem you ran into. We would be open to ideas on how to improve our Bazel setup for Windows, though.

[freeform-andre]

My understanding is that unfortunately Bazel support for Windows is not great at the moment. We have had a lot of problems with MSVC cl.exe not being able to handle file paths longer than 256 characters, and Bazel seems to like producing really long file paths. It's possible that that is related to this problem you ran into. We would be open to ideas on how to improve our Bazel setup for Windows, though.

Yeah, windows support has been incredibly painful. I actually think this is a bug within Bazel and not necessarily protobuf, but it manifests the most with protobuf. My current thought is the behavior of _virtual_includes changes with and without runfiles in such a way that it confuses MSVC cl.exe. A lot of this is uncharted territory for me so culd use some help from more informed folks.

[freeform-andre]

I've hit another issue which appears to stem from _virtual_includes:

ERROR: undeclared inclusion(s) in rule '@@com_google_protobuf//src/google/protobuf/json:lexer':
this rule is missing dependency declarations for the following files included by 'src/google/protobuf/json/internal/lexer.cc':
  'bazel-out/win-opt-exec/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/protobuf/google/protobuf/descriptor.h'
  'bazel-out/win-opt-exec/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/protobuf_lite/google/protobuf/descriptor_lite.h'

This is not the same issue as the command line being too long as the full path to this file would the following 198 characters:

C://bzl/********/execroot/********************/bazel-out/win-opt-exec/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/protobuf_lite/google/protobuf/descriptor_lite.h

Full command line:

C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.29.30133/bin/HostX64/x64/cl.exe
/nologo
/DCOMPILER_MSVC
/DNOMINMAX
/D_WIN32_WINNT=0x0601
/D_CRT_SECURE_NO_DEPRECATE
/D_CRT_SECURE_NO_WARNINGS
/bigobj
/Zm500
/EHsc
/wd4351
/wd4291
/wd4250
/wd4996
/Iexternal/com_google_protobuf
/Ibazel-out/win-opt-exec/bin/external/com_google_protobuf
/Iexternal/com_google_absl
/Ibazel-out/win-opt-exec/bin/external/com_google_absl
/Iexternal/utf8_range
/Ibazel-out/win-opt-exec/bin/external/utf8_range
/Ibazel-out/win-opt-exec/bin/external/com_google_protobuf/src/google/protobuf/json/_virtual_includes/lexer
/Ibazel-out/win-opt-exec/bin/external/com_google_protobuf/src/google/protobuf/json/_virtual_includes/message_path
/Ibazel-out/win-opt-exec/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/protobuf
/Ibazel-out/win-opt-exec/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/protobuf_nowkt
/Ibazel-out/win-opt-exec/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/protobuf_lite
/Ibazel-out/win-opt-exec/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/arena
/Ibazel-out/win-opt-exec/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/arena_align
/Ibazel-out/win-opt-exec/bin/external/com_google_protobuf/src/google/protobuf/stubs/_virtual_includes/lite
/Ibazel-out/win-opt-exec/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/port_def
/Ibazel-out/win-opt-exec/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/arena_allocation_policy
/Ibazel-out/win-opt-exec/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/arena_cleanup
/Ibazel-out/win-opt-exec/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/string_block
/Ibazel-out/win-opt-exec/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/internal_visibility
/Ibazel-out/win-opt-exec/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/varint_shuffle
/Ibazel-out/win-opt-exec/bin/external/com_google_protobuf/src/google/protobuf/io/_virtual_includes/io
/Ibazel-out/win-opt-exec/bin/external/com_google_protobuf/src/google/protobuf/io/_virtual_includes/io_win32
/Ibazel-out/win-opt-exec/bin/external/com_google_protobuf/src/google/protobuf/io/_virtual_includes/gzip_stream
/Ibazel-out/win-opt-exec/bin/external/com_google_protobuf/src/google/protobuf/stubs/_virtual_includes/stubs
/Ibazel-out/win-opt-exec/bin/external/com_google_protobuf/src/google/protobuf/io/_virtual_includes/printer
/Ibazel-out/win-opt-exec/bin/external/com_google_protobuf/src/google/protobuf/io/_virtual_includes/zero_copy_sink
/Ibazel-out/win-opt-exec/bin/external/com_google_protobuf/src/google/protobuf/io/_virtual_includes/tokenizer
/Ibazel-out/win-opt-exec/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/wkt_proto
/Ibazel-out/win-opt-exec/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/any_proto
/Ibazel-out/win-opt-exec/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/api_proto
/Ibazel-out/win-opt-exec/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/source_context_proto
/Ibazel-out/win-opt-exec/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/type_proto
/Ibazel-out/win-opt-exec/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/duration_proto
/Ibazel-out/win-opt-exec/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/empty_proto
/Ibazel-out/win-opt-exec/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/field_mask_proto
/Ibazel-out/win-opt-exec/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/struct_proto
/Ibazel-out/win-opt-exec/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/timestamp_proto
/Ibazel-out/win-opt-exec/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/wrappers_proto
/Ibazel-out/win-opt-exec/bin/external/com_google_protobuf/src/google/protobuf/json/_virtual_includes/zero_copy_buffered_stream
/showIncludes
/O2
/Oy-
/DNDEBUG
/wd4117
-D__DATE__="redacted"
-D__TIMESTAMP__="redacted"
-D__TIME__="redacted"
/Gy
/Gw
/std:c++20
/wd4018
/wd4100
/wd4244
/wd4267
/DUNICODE
/wd4065
/wd4146
/wd4244
/wd4251
/wd4267
/wd4305
/wd4307
/wd4309
/wd4334
/wd4355
/wd4506
/wd4800
/wd4996
/wd4100
/Fobazel-out/win-opt-exec/bin/external/com_google_protobuf/src/google/protobuf/json/_objs/lexer/lexer.obj
/c
external/com_google_protobuf/src/google/protobuf/json/internal/lexer.cc