Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ASAN: heap over-read near rmw::impl::cpp:parse_key_value #184

Closed
oneattosecond opened this issue Sep 10, 2019 · 9 comments
Closed

ASAN: heap over-read near rmw::impl::cpp:parse_key_value #184

oneattosecond opened this issue Sep 10, 2019 · 9 comments
Labels
more-information-needed Further information is required

Comments

@oneattosecond
Copy link

oneattosecond commented Sep 10, 2019 •
edited
Loading

Bug report

Seeing a heap over-read reported by ASAN when running cartographer or navigation on MacOS (compiled with ASAN enabled)

Required Info:

  • Operating System:
    • MacOS 10.16.4
  • Installation type:
    • compiled from source
  • Version or commit hash:
    • latest dashing .repos file
  • DDS implementation:
    • Fast-RTPS
  • Client library (if applicable):
    • reproduces in cartographer and navigation modules, multiple nodes

Steps to reproduce issue

  1. enable ASAN by adding below to cartographer and navigation CMakeLists.txt files

    set (CMAKE_CXX_FLAGS_DEBUG "${CMAKE_CXX_FLAGS_DEBUG} -fno-omit-frame-pointer -fsanitize=address")
    set (CMAKE_LINKER_FLAGS_DEBUG "${CMAKE_LINKER_FLAGS_DEBUG} -fno-omit-frame-pointer -fsanitize=address")

  2. recompile cartographer, navigation (didn't try recompiling all of ROS with ASAN, not sure if that's a mistake)

    colcon build --cmake-args -DBUILD_TESTING=OFF -DCMAKE_BUILD_TYPE=Debug -DFORCE_DEBUG_BUILD=True --symlink-install

  3. run cartographer, navigation as normal

  4. observe ASAN flagging 8-byte heap over-read

Expected behavior

No traps from ASAN

Actual behavior

ASAN flags over-read and dies

Additional information

I've reviewed a few different nodes, and they all have some form of this stack trace, where a node (multiple of them are killed by ASAN) is doing something that invokes rmw::impl::cpp:parse_key_value()

[occupancy_grid_node-2] =================================================================
[occupancy_grid_node-2] ==37032==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6020000042d5 at pc 0x0001034d46df bp 0x70000b8f0c30 sp 0x70000b8f03e0
[occupancy_grid_node-2] READ of size 8 at 0x6020000042d5 thread T3
[occupancy_grid_node-2] [WARN] [occupancy_grid_node]: submap_slices and last_frame_id is empty
[occupancy_grid_node-2] [WARN] [occupancy_grid_node]: submap_slices and last_frame_id is empty
[occupancy_grid_node-2] [WARN] [occupancy_grid_node]: submap_slices and last_frame_id is empty
[occupancy_grid_node-2] [WARN] [occupancy_grid_node]: submap_slices and last_frame_id is empty
[occupancy_grid_node-2] 0 0x1034d46de in __asan_memcpy (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x596de)
[occupancy_grid_node-2] [WARN] [occupancy_grid_node]: submap_slices and last_frame_id is empty
[occupancy_grid_node-2] [WARN] [occupancy_grid_node]: submap_slices and last_frame_id is empty
[occupancy_grid_node-2] 1 0x100906a8c in std::__1::enable_if<((std::__1::integral_constant<bool, true>::value) || (!(__has_construct<std::__1::allocator, bool*, bool>::value))) && (is_trivially_move_constructible::value), void>::type std::__1::allocator_traits<std::__1::allocator >::__construct_backward(std::__1::allocator&, bool*, bool*, bool*&) memory:1699
[occupancy_grid_node-2] 2 0x10090550a in std::__1::vector<unsigned char, std::__1::allocator >::__swap_out_circular_buffer(std::__1::__split_buffer<unsigned char, std::__1::allocator&>&) vector:931
[occupancy_grid_node-2] 3 0x101aee75a in void std::__1::vector<unsigned char, std::__1::allocator >::__push_back_slow_path<unsigned char const&>(unsigned char const&&&) (librclcpp.dylib:x86_64+0xdc75a)
[occupancy_grid_node-2] 4 0x111460d93 in std::__1::vector<unsigned char, std::__1::allocator >::push_back(unsigned char const&) (librmw_fastrtps_shared_cpp.dylib:x86_64+0x21d93)
[occupancy_grid_node-2] 5 0x11146005a in rmw::impl::cpp::parse_key_value(std::__1::vector<unsigned char, std::__1::allocator >) (librmw_fastrtps_shared_cpp.dylib:x86_64+0x2105a)
[occupancy_grid_node-2] 6 0x11145d855 in ParticipantListener::onParticipantDiscovery(eprosima::fastrtps::Participant*, eprosima::fastrtps::rtps::ParticipantDiscoveryInfo&&) (librmw_fastrtps_shared_cpp.dylib:x86_64+0x1e855)
[occupancy_grid_node-2] 7 0x103057aa6 in eprosima::fastrtps::rtps::PDPSimpleListener::onNewCacheChangeAdded(eprosima::fastrtps::rtps::RTPSReader*, eprosima::fastrtps::rtps::CacheChange_t const*) (libfastrtps.1.dylib:x86_64+0x1c8aa6)
[occupancy_grid_node-2] 8 0x102ee419b in eprosima::fastrtps::rtps::StatelessReader::change_received(eprosima::fastrtps::rtps::CacheChange_t*) (libfastrtps.1.dylib:x86_64+0x5519b)
[occupancy_grid_node-2] 9 0x102ee4372 in eprosima::fastrtps::rtps::StatelessReader::processDataMsg(eprosima::fastrtps::rtps::CacheChange_t*) (libfastrtps.1.dylib:x86_64+0x55372)
[occupancy_grid_node-2] 10 0x102ef3a8c in eprosima::fastrtps::rtps::MessageReceiver::proc_Submsg_Data(eprosima::fastrtps::rtps::CDRMessage_t*, eprosima::fastrtps::rtps::SubmessageHeader_t*) (libfastrtps.1.dylib:x86_64+0x64a8c)
[occupancy_grid_node-2] 11 0x102ef1869 in eprosima::fastrtps::rtps::MessageReceiver::processCDRMsg(eprosima::fastrtps::rtps::Locator_t const&, eprosima::fastrtps::rtps::CDRMessage_t*) (libfastrtps.1.dylib:x86_64+0x62869)
[occupancy_grid_node-2] 12 0x102efaaac in eprosima::fastrtps::rtps::ReceiverResource::OnDataReceived(unsigned char const*, unsigned int, eprosima::fastrtps::rtps::Locator_t const&, eprosima::fastrtps::rtps::Locator_t const&) (libfastrtps.1.dylib:x86_64+0x6baac)
[occupancy_grid_node-2] 13 0x102f57a91 in eprosima::fastrtps::rtps::UDPTransportInterface::perform_listen_operation(eprosima::fastrtps::rtps::UDPChannelResource*, eprosima::fastrtps::rtps::Locator_t) (libfastrtps.1.dylib:x86_64+0xc8a91)
[occupancy_grid_node-2] 14 0x102f5b402 in void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_deletestd::__1::__thread_struct >, void (eprosima::fastrtps::rtps::UDPTransportInterface::)(eprosima::fastrtps::rtps::UDPChannelResource, eprosima::fastrtps::rtps::Locator_t), eprosima::fastrtps::rtps::UDPTransportInterface*, eprosima::fastrtps::rtps::UDPChannelResource*, eprosima::fastrtps::rtps::Locator_t> >(void*) (libfastrtps.1.dylib:x86_64+0xcc402)
[occupancy_grid_node-2] 15 0x7fff5c05a2ea in _pthread_body (libsystem_pthread.dylib:x86_64+0x32ea)
[occupancy_grid_node-2] 16 0x7fff5c05d248 in _pthread_start (libsystem_pthread.dylib:x86_64+0x6248)
[occupancy_grid_node-2] 17 0x7fff5c05940c in thread_start (libsystem_pthread.dylib:x86_64+0x240c)
[occupancy_grid_node-2]
[occupancy_grid_node-2] 0x6020000042d8 is located 0 bytes to the right of 8-byte region [0x6020000042d0,0x6020000042d8)
[occupancy_grid_node-2] allocated by thread T3 here:
[occupancy_grid_node-2] 0 0x1034e3b92 in wrap__Znwm (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x68b92)
[occupancy_grid_node-2] 1 0x101ac4e88 in std::__1::__libcpp_allocate(unsigned long, unsigned long) (librclcpp.dylib:x86_64+0xb2e88)
[occupancy_grid_node-2] 2 0x101aca084 in std::__1::allocator::allocate(unsigned long, void const*) (librclcpp.dylib:x86_64+0xb8084)
[occupancy_grid_node-2] 3 0x101ac9ec0 in std::__1::allocator_traits<std::__1::allocator >::allocate(std::__1::allocator&, unsigned long) (librclcpp.dylib:x86_64+0xb7ec0)
[occupancy_grid_node-2] 4 0x101aedf89 in std::__1::__split_buffer<unsigned char, std::__1::allocator&>::__split_buffer(unsigned long, unsigned long, std::__1::allocator&) (librclcpp.dylib:x86_64+0xdbf89)
[occupancy_grid_node-2] 5 0x101aede0c in std::__1::__split_buffer<unsigned char, std::__1::allocator&>::__split_buffer(unsigned long, unsigned long, std::__1::allocator&) (librclcpp.dylib:x86_64+0xdbe0c)
[occupancy_grid_node-2] 6 0x101aee70c in void std::__1::vector<unsigned char, std::__1::allocator >::__push_back_slow_path<unsigned char const&>(unsigned char const&&&) (librclcpp.dylib:x86_64+0xdc70c)
[occupancy_grid_node-2] 7 0x111460d93 in std::__1::vector<unsigned char, std::__1::allocator >::push_back(unsigned char const&) (librmw_fastrtps_shared_cpp.dylib:x86_64+0x21d93)
[occupancy_grid_node-2] 8 0x11146005a in rmw::impl::cpp::parse_key_value(std::__1::vector<unsigned char, std::__1::allocator >) (librmw_fastrtps_shared_cpp.dylib:x86_64+0x2105a)
[occupancy_grid_node-2] 9 0x11145d855 in ParticipantListener::onParticipantDiscovery(eprosima::fastrtps::Participant*, eprosima::fastrtps::rtps::ParticipantDiscoveryInfo&&) (librmw_fastrtps_shared_cpp.dylib:x86_64+0x1e855)
[occupancy_grid_node-2] [WARN] [occupancy_grid_node]: submap_slices and last_frame_id is empty
[occupancy_grid_node-2] 10 0x103057aa6 in eprosima::fastrtps::rtps::PDPSimpleListener::onNewCacheChangeAdded(eprosima::fastrtps::rtps::RTPSReader*, eprosima::fastrtps::rtps::CacheChange_t const*) (libfastrtps.1.dylib:x86_64+0x1c8aa6)
[occupancy_grid_node-2] 11 0x102ee419b in eprosima::fastrtps::rtps::StatelessReader::change_received(eprosima::fastrtps::rtps::CacheChange_t*) (libfastrtps.1.dylib:x86_64+0x5519b)
[occupancy_grid_node-2] 12 0x102ee4372 in eprosima::fastrtps::rtps::StatelessReader::processDataMsg(eprosima::fastrtps::rtps::CacheChange_t*) (libfastrtps.1.dylib:x86_64+0x55372)
[occupancy_grid_node-2] 13 0x102ef3a8c in eprosima::fastrtps::rtps::MessageReceiver::proc_Submsg_Data(eprosima::fastrtps::rtps::CDRMessage_t*, eprosima::fastrtps::rtps::SubmessageHeader_t*) (libfastrtps.1.dylib:x86_64+0x64a8c)
[occupancy_grid_node-2] 14 0x102ef1869 in eprosima::fastrtps::rtps::MessageReceiver::processCDRMsg(eprosima::fastrtps::rtps::Locator_t const&, eprosima::fastrtps::rtps::CDRMessage_t*) (libfastrtps.1.dylib:x86_64+0x62869)
[occupancy_grid_node-2] 15 0x102efaaac in eprosima::fastrtps::rtps::ReceiverResource::OnDataReceived(unsigned char const*, unsigned int, eprosima::fastrtps::rtps::Locator_t const&, eprosima::fastrtps::rtps::Locator_t const&) (libfastrtps.1.dylib:x86_64+0x6baac)
[occupancy_grid_node-2] 16 0x102f57a91 in eprosima::fastrtps::rtps::UDPTransportInterface::perform_listen_operation(eprosima::fastrtps::rtps::UDPChannelResource*, eprosima::fastrtps::rtps::Locator_t) (libfastrtps.1.dylib:x86_64+0xc8a91)
[occupancy_grid_node-2] 17 0x102f5b402 in void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_deletestd::__1::__thread_struct >, void (eprosima::fastrtps::rtps::UDPTransportInterface::)(eprosima::fastrtps::rtps::UDPChannelResource, eprosima::fastrtps::rtps::Locator_t), eprosima::fastrtps::rtps::UDPTransportInterface*, eprosima::fastrtps::rtps::UDPChannelResource*, eprosima::fastrtps::rtps::Locator_t> >(void*) (libfastrtps.1.dylib:x86_64+0xcc402)
[occupancy_grid_node-2] 18 0x7fff5c05a2ea in _pthread_body (libsystem_pthread.dylib:x86_64+0x32ea)
[occupancy_grid_node-2] 19 0x7fff5c05d248 in _pthread_start (libsystem_pthread.dylib:x86_64+0x6248)
[occupancy_grid_node-2] 20 0x7fff5c05940c in thread_start (libsystem_pthread.dylib:x86_64+0x240c)
[occupancy_grid_node-2]
[occupancy_grid_node-2] Thread T3 created by T0 here:
[occupancy_grid_node-2] 0 0x1034cf02d in wrap_pthread_create (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x5402d)
[occupancy_grid_node-2] 1 0x102f5b30d in std::__1::thread::thread<void (eprosima::fastrtps::rtps::UDPTransportInterface::)(eprosima::fastrtps::rtps::UDPChannelResource, eprosima::fastrtps::rtps::Locator_t), eprosima::fastrtps::rtps::UDPTransportInterface*, eprosima::fastrtps::rtps::UDPChannelResource*&, eprosima::fastrtps::rtps::Locator_t const&, void>(void (eprosima::fastrtps::rtps::UDPTransportInterface::&&)(eprosima::fastrtps::rtps::UDPChannelResource, eprosima::fastrtps::rtps::Locator_t), eprosima::fastrtps::rtps::UDPTransportInterface*&&, eprosima::fastrtps::rtps::UDPChannelResource*&&&, eprosima::fastrtps::rtps::Locator_t const&&&) (libfastrtps.1.dylib:x86_64+0xcc30d)
[occupancy_grid_node-2] 2 0x102f578e6 in eprosima::fastrtps::rtps::UDPTransportInterface::CreateInputChannelResource(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, eprosima::fastrtps::rtps::Locator_t const&, bool, unsigned int, eprosima::fastrtps::rtps::TransportReceiverInterface*) (libfastrtps.1.dylib:x86_64+0xc88e6)
[occupancy_grid_node-2] 3 0x102f5748a in eprosima::fastrtps::rtps::UDPTransportInterface::OpenAndBindInputSockets(eprosima::fastrtps::rtps::Locator_t const&, eprosima::fastrtps::rtps::TransportReceiverInterface*, bool, unsigned int) (libfastrtps.1.dylib:x86_64+0xc848a)
[occupancy_grid_node-2] 4 0x102f40671 in eprosima::fastrtps::rtps::UDPv4Transport::OpenInputChannel(eprosima::fastrtps::rtps::Locator_t const&, eprosima::fastrtps::rtps::TransportReceiverInterface*, unsigned int) (libfastrtps.1.dylib:x86_64+0xb1671)
[occupancy_grid_node-2] 5 0x102efa214 in eprosima::fastrtps::rtps::ReceiverResource::ReceiverResource(eprosima::fastrtps::rtps::TransportInterface&, eprosima::fastrtps::rtps::Locator_t const&, unsigned int) (libfastrtps.1.dylib:x86_64+0x6b214)
[occupancy_grid_node-2] 6 0x102ef85f9 in eprosima::fastrtps::rtps::NetworkFactory::BuildReceiverResources(eprosima::fastrtps::rtps::Locator_t&, unsigned int, std::__1::vector<std::__1::shared_ptreprosima::fastrtps::rtps::ReceiverResource, std::__1::allocator<std::__1::shared_ptreprosima::fastrtps::rtps::ReceiverResource > >&) (libfastrtps.1.dylib:x86_64+0x695f9)
[occupancy_grid_node-2] 7 0x102efde05 in eprosima::fastrtps::rtps::RTPSParticipantImpl::createReceiverResources(eprosima::fastrtps::rtps::LocatorList_t&, bool) (libfastrtps.1.dylib:x86_64+0x6ee05)
[occupancy_grid_node-2] 8 0x102efd0e7 in eprosima::fastrtps::rtps::RTPSParticipantImpl::RTPSParticipantImpl(eprosima::fastrtps::rtps::RTPSParticipantAttributes const&, eprosima::fastrtps::rtps::GuidPrefix_t const&, eprosima::fastrtps::rtps::RTPSParticipant*, eprosima::fastrtps::rtps::RTPSParticipantListener*) (libfastrtps.1.dylib:x86_64+0x6e0e7)
[occupancy_grid_node-2] 9 0x102f05a2e in eprosima::fastrtps::rtps::RTPSDomain::createParticipant(eprosima::fastrtps::rtps::RTPSParticipantAttributes const&, eprosima::fastrtps::rtps::RTPSParticipantListener*) (libfastrtps.1.dylib:x86_64+0x76a2e)
[occupancy_grid_node-2] 10 0x102f08882 in eprosima::fastrtps::Domain::createParticipant(eprosima::fastrtps::ParticipantAttributes const&, eprosima::fastrtps::ParticipantListener*) (libfastrtps.1.dylib:x86_64+0x79882)
[occupancy_grid_node-2] 11 0x11145c087 in rmw_fastrtps_shared_cpp::create_node(char const*, char const*, char const*, eprosima::fastrtps::ParticipantAttributes) (librmw_fastrtps_shared_cpp.dylib:x86_64+0x1d087)
[occupancy_grid_node-2] 12 0x11145cf07 in rmw_fastrtps_shared_cpp::__rmw_create_node(char const*, char const*, char const*, unsigned long, rmw_node_security_options_t const*) (librmw_fastrtps_shared_cpp.dylib:x86_64+0x1df07)
[occupancy_grid_node-2] 13 0x1113f2de3 in rmw_create_node (librmw_fastrtps_cpp.dylib:x86_64+0x1cde3)
[occupancy_grid_node-2] 14 0x1021f37fc in rmw_create_node (librmw_implementation.dylib:x86_64+0x47fc)
[occupancy_grid_node-2] 15 0x102077874 in rcl_node_init (librcl.dylib:x86_64+0x19874)
[occupancy_grid_node-2] 16 0x101aa2c75 in rclcpp::node_interfaces::NodeBase::NodeBase(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, std::__1::shared_ptrrclcpp::Context, rcl_node_options_t const&, bool) (librclcpp.dylib:x86_64+0x90c75)
[occupancy_grid_node-2] 17 0x101aa37fd in rclcpp::node_interfaces::NodeBase::NodeBase(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, std::__1::shared_ptrrclcpp::Context, rcl_node_options_t const&, bool) (librclcpp.dylib:x86_64+0x917fd)
[occupancy_grid_node-2] 18 0x101a87440 in rclcpp::Node::Node(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, rclcpp::NodeOptions const&) (librclcpp.dylib:x86_64+0x75440)
[occupancy_grid_node-2] 19 0x101a872b2 in rclcpp::Node::Node(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, rclcpp::NodeOptions const&) (librclcpp.dylib:x86_64+0x752b2)
[occupancy_grid_node-2] 20 0x10083da58 in cartographer_ros::(anonymous namespace)::OccupancyGridNode::OccupancyGridNode(double, double) occupancy_grid_node_main.cc:56
[occupancy_grid_node-2] 21 0x10083d458 in cartographer_ros::(anonymous namespace)::OccupancyGridNode::OccupancyGridNode(double, double) occupancy_grid_node_main.cc:58
[occupancy_grid_node-2] 22 0x10083d311 in std::__1::__compressed_pair_elem<cartographer_ros::(anonymous namespace)::OccupancyGridNode, 1, false>::__compressed_pair_elem<double&, double&, 0ul, 1ul>(std::__1::piecewise_construct_t, std::__1::tuple<double&, double&>, std::__1::__tuple_indices<0ul, 1ul>) memory:2153
[occupancy_grid_node-2] 23 0x10083cec3 in std::__1::__compressed_pair<std::__1::allocator<cartographer_ros::(anonymous namespace)::OccupancyGridNode>, cartographer_ros::(anonymous namespace)::OccupancyGridNode>::__compressed_pair<std::__1::allocator<cartographer_ros::(anonymous namespace)::OccupancyGridNode>&, double&, double&>(std::__1::piecewise_construct_t, std::__1::tuple<std::__1::allocator<cartographer_ros::(anonymous namespace)::OccupancyGridNode>&>, std::__1::tuple<double&, double&>) memory:2255
[occupancy_grid_node-2] 24 0x10083c4f3 in std::__1::__compressed_pair<std::__1::allocator<cartographer_ros::(anonymous namespace)::OccupancyGridNode>, cartographer_ros::(anonymous namespace)::OccupancyGridNode>::__compressed_pair<std::__1::allocator<cartographer_ros::(anonymous namespace)::OccupancyGridNode>&, double&, double&>(std::__1::piecewise_construct_t, std::__1::tuple<std::__1::allocator<cartographer_ros::(anonymous namespace)::OccupancyGridNode>&>, std::__1::tuple<double&, double&>) memory:2256
[occupancy_grid_node-2] 25 0x10083bd35 in std::__1::__shared_ptr_emplace<cartographer_ros::(anonymous namespace)::OccupancyGridNode, std::__1::allocator<cartographer_ros::(anonymous namespace)::OccupancyGridNode> >::__shared_ptr_emplace<double&, double&>(std::__1::allocator<cartographer_ros::(anonymous namespace)::OccupancyGridNode>, double&&&, double&&&) memory:3668
[occupancy_grid_node-2] 26 0x10083b244 in std::__1::__shared_ptr_emplace<cartographer_ros::(anonymous namespace)::OccupancyGridNode, std::__1::allocator<cartographer_ros::(anonymous namespace)::OccupancyGridNode> >::__shared_ptr_emplace<double&, double&>(std::__1::allocator<cartographer_ros::(anonymous namespace)::OccupancyGridNode>, double&&&, double&&&) memory:3669
[occupancy_grid_node-2] 27 0x10083adb6 in std::__1::shared_ptr<cartographer_ros::(anonymous namespace)::OccupancyGridNode> std::__1::shared_ptr<cartographer_ros::(anonymous namespace)::OccupancyGridNode>::make_shared<double&, double&>(double&&&, double&&&) memory:4327
[occupancy_grid_node-2] 28 0x10082b9a3 in std::__1::enable_if<!(is_array<cartographer_ros::(anonymous namespace)::OccupancyGridNode>::value), std::__1::shared_ptr<cartographer_ros::(anonymous namespace)::OccupancyGridNode> >::type std::__1::make_shared<cartographer_ros::(anonymous namespace)::OccupancyGridNode, double&, double&>(double&&&, double&&&) memory:4706
[occupancy_grid_node-2] 29 0x10082b659 in main occupancy_grid_node_main.cc:193
[occupancy_grid_node-2] 30 0x7fff5be663d4 in start (libdyld.dylib:x86_64+0x163d4)
[occupancy_grid_node-2]
[occupancy_grid_node-2] SUMMARY: AddressSanitizer: heap-buffer-overflow (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x596de) in __asan_memcpy
[occupancy_grid_node-2] Shadow bytes around the buggy address:
[occupancy_grid_node-2] 0x1c0400000800: fa fa 00 00 fa fa 00 00 fa fa 00 00 fa fa 00 fa
[occupancy_grid_node-2] 0x1c0400000810: fa fa 00 00 fa fa 00 00 fa fa 00 fa fa fa 00 00
[occupancy_grid_node-2] 0x1c0400000820: fa fa 00 00 fa fa 00 fa fa fa 00 00 fa fa 00 00
[occupancy_grid_node-2] 0x1c0400000830: fa fa 00 fa fa fa 00 00 fa fa 00 00 fa fa 00 fa
[occupancy_grid_node-2] 0x1c0400000840: fa fa 00 00 fa fa 00 00 fa fa 00 fa fa fa fd fa
[occupancy_grid_node-2] =>0x1c0400000850: fa fa fd fa fa fa fd fa fa fa[05]fa fa fa 00 00
[occupancy_grid_node-2] 0x1c0400000860: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
[occupancy_grid_node-2] 0x1c0400000870: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
[occupancy_grid_node-2] 0x1c0400000880: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
[occupancy_grid_node-2] 0x1c0400000890: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
[occupancy_grid_node-2] 0x1c04000008a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
[occupancy_grid_node-2] Shadow byte legend (one shadow byte represents 8 application bytes):
[occupancy_grid_node-2] Addressable: 00
[occupancy_grid_node-2] Partially addressable: 01 02 03 04 05 06 07
[occupancy_grid_node-2] Heap left redzone: fa
[occupancy_grid_node-2] Freed heap region: fd
[occupancy_grid_node-2] Stack left redzone: f1
[occupancy_grid_node-2] Stack mid redzone: f2
[occupancy_grid_node-2] Stack right redzone: f3
[occupancy_grid_node-2] Stack after return: f5
[occupancy_grid_node-2] Stack use after scope: f8
[occupancy_grid_node-2] Global redzone: f9
[occupancy_grid_node-2] Global init order: f6
[occupancy_grid_node-2] Poisoned by user: f7
[occupancy_grid_node-2] Container overflow: fc
[occupancy_grid_node-2] Array cookie: ac
[occupancy_grid_node-2] Intra object redzone: bb
[occupancy_grid_node-2] ASan internal: fe
[occupancy_grid_node-2] Left alloca redzone: ca
[occupancy_grid_node-2] Right alloca redzone: cb
[occupancy_grid_node-2] Shadow gap: cc
[occupancy_grid_node-2] ==37032==ABORTING
@clalancette
Copy link
Contributor

I'm inclined to believe that this is a problem with certain key/value pairs, though I haven't been able to reproduce it myself. The thing that would make this the easiest for us to track down would be to compile ROS 2 from source, inserting this code:

  for (uint8_t u8 : kv) {
    std::cerr << u8;
  }
  std::cerr << std::endl;

at this line: https://github.com/ros2/rmw/blob/master/rmw/include/rmw/impl/cpp/key_value.hpp#L34 (you'll also have to add a #include <iostream> at the top of the file). With that in place, when you run your system you should get a bunch of prints that look something like:

name=talker;namespace=/;

If you could post those here, we can probably track down what is going on. Thanks!

@clalancette clalancette self-assigned this Sep 11, 2019
@oneattosecond
Copy link
Author

cool I'll try that today, time willing.

@oneattosecond
Copy link
Author

with TB3 teleop node running:

[occupancy_grid_node-2] name=teleop_keyboard;namespace=/;
[occupancy_grid_node-2] =================================================================
[occupancy_grid_node-2] ==58014==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6020000052d5 at pc 0x00010fd4a6df bp 0x7000002ddc00 sp 0x7000002dd3b0
[occupancy_grid_node-2] READ of size 8 at 0x6020000052d5 thread T3

without teleop running:

[occupancy_grid_node-2] name=rviz2;namespace=/;
[occupancy_grid_node-2] name=launch_ros;namespace=/;
[occupancy_grid_node-2] =================================================================
[occupancy_grid_node-2] ==58033==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6020000045d5 at pc 0x0001038ea6df bp 0x700002fa1c00 sp 0x700002fa13b0
[occupancy_grid_node-2] READ of size 8 at 0x6020000045d5 thread T3

@oneattosecond
Copy link
Author

on navigation launch, few different signatures

[world_model-4] name=bt_navigator;namespace=/;
[world_model-4] =================================================================
[world_model-4] ==58132==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6020000052f5 at pc 0x00010ff896df bp 0x700005fe2c00 sp 0x700005fe23b0
[world_model-4] READ of size 8 at 0x6020000052f5 thread T3

[lifecycle_manager-1] name=map_server;namespace=/;
[lifecycle_manager-1] =================================================================
[lifecycle_manager-1] ==58129==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6020000042f5 at pc 0x0001072e56df bp 0x700001c82c00 sp 0x700001c823b0
[lifecycle_manager-1] READ of size 8 at 0x6020000042f5 thread T3

[recoveries_node-7] name=launch_ros;namespace=/;
[recoveries_node-7] =================================================================
[recoveries_node-7] ==58135==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6020000052f5 at pc 0x00010720d6df bp 0x70000306bc00 sp 0x70000306b3b0
[recoveries_node-7] READ of size 8 at 0x6020000052f5 thread T3

[dwb_controller-5] name=rviz2;namespace=/;
[dwb_controller-5] name=launch_ros;namespace=/;
[dwb_controller-5] =================================================================
[dwb_controller-5] ==58133==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6020000055f5 at pc 0x00011097e6df bp 0x70000dba0c00 sp 0x70000dba03b0
[dwb_controller-5] READ of size 8 at 0x6020000055f5 thread T3

@clalancette
Copy link
Contributor

Thanks for the data.

But it looks like I was wrong about being able to track it down. I have been unable to reproduce this with a simple test program on macOS (even using the data you provided). I've also not been able to reproduce on Linux with the talker/listener built with ASAN.

So this probably needs deeper investigation. Unfortunately I probably won't have time in the next couple of weeks or so to look into it. If you have some time, it would be worthwhile to rebuild the entire ROS 2 stack from latest source with ASAN (you can use the colcon mixin for that), and then see if that gives you more detailed information.

@clalancette clalancette removed their assignment Sep 12, 2019
@oneattosecond
Copy link
Author

If you have some time, it would be worthwhile to rebuild the entire ROS 2 stack from latest source with ASAN (you can use the colcon mixin for that)

Sorry, not following, can you provide some example command to run? I've been enabling ASAN in CMakeLists.txt per project, not sure how to enable it from command line with colcon...

@clalancette
Copy link
Contributor

You should be able to do something like the following:

sudo apt-get install python3-colcon-mixin # or use pip on macOS
colcon mixin add default https://raw.githubusercontent.com/colcon/colcon-mixin-repository/master/index.yaml
colcon mixin update default
colcon build --event-handlers console_direct+ --mixin asan-gcc

It will then add -fsanitize=address to all CFLAGS and CXXFLAGS for all packages it builds.

@oneattosecond
Copy link
Author

dang, ros2 isn't launching and google search isn't revealing any working tips...

==48967==ERROR: Interceptors are not working. This may be because AddressSanitizer is loaded too late (e.g. via dlopen)

@mabelzhang mabelzhang added the more-information-needed Further information is required label Sep 19, 2019
@clalancette
Copy link
Contributor

This is an old issue, and Dashing is now End-of-Life. We've also fixed a lot of bugs in this area. So I'm going to close this one out. If you run across this again with a newer ROS distribution, please feel free to open another issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
more-information-needed Further information is required
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@clalancette @mabelzhang @oneattosecond