WIP/RFC: stack scrubbing

[yuyichao]

This is a rewrite of @carnaval 's previous work in #10725. The main differences are,

1. (As suggested by Oscar) Instead of messing with stack data, we fill the dead object that might be referenced by the stack with garbage (0xff ATM) so that we won't have any data corruption.

2. Split out the logic for conservative pool scan (i.e. figure out if a pointer might be refering a GC object in the pool)

   This function is useful for a number of other things including conservative stack scan (either for GC root conservative stack scanning? #11714, or for pinning conservative stack scanning? #11714 (comment)), ccall sanitizer RFC: ccall sanitizer #12173 and gc debugging in general.

[carnaval]

can this go into gcdebug.c ?

[yuyichao]

I prefer to keep it here for now since it's potentially useful for the GC itself as well.

[carnaval]

if we need it in this form we'll move it back I think, gc.c is already messy enough not to add dead functions

[yuyichao]

k. sure..

[carnaval]

sorry forgot to review this. Could you try to see if it works by removing a useful root on purpose somewhere ?

other than the minor comment about files, if it works it's good to go

[yuyichao]

Suggestions implemented.

It's very hard to come up with a case that only reliably triggers an error with this PR mainly because it doesn't increase GC frequency. I believe it will make debugging GC easier because it make a missing GC root fail fast instead of generating some wierd error much later.

The following script at least show that it works.

@noinline ptr1() = ccall(:jl_astaggedvalue, Ptr{UInt}, (Any,),
                         Ref{UInt}(0x12345))

@noinline ptr2() = ccall(:jl_astaggedvalue, Ptr{UInt}, (Any,),
                         Ref{UInt}(0x12345)) + sizeof(Int) * 2

@noinline function load1(ptr)
    i1 = unsafe_load(ptr)
    i2 = unsafe_load(ptr + sizeof(Int))
    ptr, i1, i2
end

@noinline function load2(ptr)
    i1 = unsafe_load(ptr - 2 * sizeof(Int))
    i2 = unsafe_load(ptr - sizeof(Int))
    ptr, i1, i2
end

function f1()
    ptr = ptr1()
    gc()
    load1(ptr)
end

function f2()
    ptr = ptr2()
    gc()
    load2(ptr)
end

println(f1())
println(f2())

Output on master

yuyichao% julia -f scrub_test.jl 
(Ptr{UInt64} @0x00007f3cf28563c8,0x00007f3cefe47df0,0x0000000000012345)
(Ptr{UInt64} @0x00007f3cf2673e80,0x00007f3cf2673aa8,0x0000000000012345)

Output on this branch with the build option enabled.

yuyichao% ./julia -f scrub_test.jl
(Ptr{UInt64} @0x00007fa422460468,0xffffffffffffffff,0xffffffffffffffff)
(Ptr{UInt64} @0x00007fa4221b3330,0x00007fa4221b2f58,0x0000000000012345)

The output clearly shows that the scrubing only happens when the pointer is stored on the stack.

Will merge when the CI is green again.

[tkelman]

For MSVC, you need DLLEXPORT to be consistent between the prototype and the implementation. It should either be on both or neither. Which should this be?

[yuyichao]

Export for debugging purpose THX.

[yuyichao]

I'm not by my computer now, will fix it later. Feel free to do it if you need to.

[tkelman]

It's okay, just came up in #12056. We don't do the MSVC build on CI right now, so just trying to remind people who touch the C code of some of its constraints so hopefully it breaks a little less often.

[yuyichao]

@tkelman Should be fixed by 24f6647 . I'll push to master after the branch travis passes.

[yuyichao]

@tkelman Merged. Please let me know if there's still an issue.