Random segfaults on Linux

[reinux]

I'm not sure I want to make an issue about this just yet when I can't figure out exactly how to repo it, but I'm getting random segfaults, usually several seconds after I first use any API call, including seemingly inconsequential things like Access.GetStorjVersion.

It might have something to do with my using F# or ASP.NET. Happens on either of my computers running NixOS. On the other hand, F# Interactive and C# on their own seem not to cause any issues.

Are there any combinations of libraries that are known to cause issues?

[TopperDEL]

I do not have any info about problems with the library like the ones you have - but that does not mean there aren't any. They might just not have reached me.
There might be a problem related to Go used in the binaries. uplink-c itself is written in Go. To use it from C#, I created a c-library out of it (adding some wrapper-magic using SWIG) which contains the whole Go-runtime. And then I call into it from C# using P/Invoke. Unfortunately with this approach there are many possible hickups.

Finding the root-cause of such segfaults is tricky and I'm not very educated helping here. I also did not use uplink.NET in Linux (NixOS is Linux, right?) very often, so I don't have any personal experience with it.

What you could try, though:

• try to get as many information from the segfaults as possible. I may not help, how - but if you have a stacktrace or the exact segfault message, that might help finding the root cause.
• as I assume the error comes from the included go-runtime, you may try to use an older version of uplink.NET. Try any of the 2.10., 2.9. versions and see if the error persists. If not, then I might have a deeper look into the changes between the working and the failing version. I regularly update uplink.NET and bring it to the latest Go-runtime - so if there happens to be a version working better, we might check what changes within Go might lead to the issue.
• try to search for the error message in golang-issue-list. Maybe there is already something mentioned regarding NixOS

It is very likely that this is a problem within Go. The C#-part of uplink.NET is quite trivial and is mostly about converting data-structs between C/Go und .Net.

I hope that helps you!

[reinux]

Thanks for the help!

I'll see if I can get some more error data somehow... Downgrading to 2.9 actually gave me a stack overflow error, but not in the usual .NET exception format, so that's a good hint that it's coming from Go side.

Aside from that, it happens frequently but at different points during execution -- usually within a few seconds when it does. I think I need to learn how to use gdb or something so I can debug the coredumps, but I'm not too familiar with either that or with Go.

Would you happen to be allocating memory C# side? Something that tends to go wrong a lot of times when porting low-level code from Windows is that a lot of Linux/Unix environments don't actually zero out memory when you allocate, so if the Go code is expecting to read zeroes, it could be getting random bytes instead.

[TopperDEL]

This should be the relevant part for downloading stuff:

uplink.net/uplink.NET/uplink.NET/Models/DownloadOperation.cs

Line 146 in 759c0a0

fixed (byte* arrayPtr = part)

And this for uploading:

uplink.net/uplink.NET/uplink.NET/Models/UploadOperation.cs

Line 172 in 759c0a0

fixed (byte* arrayPtr = bytesToUpload)

Can you check whether the crash happens during up/download or also during bucket-listings? If it does not occur between listing, but during up/download, this might be a good start.

[reinux]

It actually happens a few seconds after I make any API call, e.g. Access.GetStorjVersion, but anything really. When it doesn't happen, on the other hand, I can usually get an upload or download done successfully -- but it changes from build to build.

That's usually a good indication of some kind of memory access bug, but...

I'm drilling into the calls from that method, and you're right, there isn't much of anything too interesting interesting on the C# side:

https://github.com/TopperDEL/uplink.net/blob/759c0a00b50cb48a6c543189cf96dbb23f375c68/uplink.NET/uplink.NET/SWIG-Generated/storj_uplink.cs#L508C1-L511C4

https://github.com/TopperDEL/uplink.net/blob/759c0a00b50cb48a6c543189cf96dbb23f375c68/uplink.NET/uplink.NET/SWIG-Generated/storj_uplinkPINVOKE.cs#L1560C1-L1561C66

And in the .i file, which I'm guessing is some kind of template code generated by SWIG, the implementation is just a simple C function:

https://github.com/TopperDEL/uplink.net/blob/759c0a00b50cb48a6c543189cf96dbb23f375c68/SWIG/storj_uplink_second.i#L119C1-L122C2

Given that there isn't much going on, I'm not too sure where to look.

Maybe there's some SWIG initialization stuff that's allocating, upon the first invocation, but I'm not seeing any particularly suspicious open issues on the SWIG repo.

Hmm....

[TopperDEL]

Hm, both Methods GetStorjVersion() and GetGoVersion() should not yet lead to these issues - I would first expect them when we really call into the Go-part of the binaries, which fires up the Go-runtime/GarbaceCollection and such. But it may be that the DLL itself has some initialization going.

In this case we really need to get a better dump. You may try to build a little Go-Hello-World on your system and try to find out if that's making problems, too. But to really understand what's happening, we need a stacktrace/error message from the segfault.

[reinux]

I'll probably have to revisit this sometime when things are a little less hectic... I did try in F# Interactive and in a straight C# app, and I wasn't able to reproudce the issue in either of those.