Past low key bug in GNU linker makes note enumeration unreliable

[sevaa]

Please take a look at eliben/pyelftools#591. It contains a couple of ELF files on which constructing an pwnlib.elf.elf.ELF object crashes with an exception that originates within pyelftools.

Internally, the crash happens when pwnlib tries to enumerate the PT_NOTE type segments in an ELF file. Both binaries in that issue contain a PT_NOTE segment header record that is malformed - it doesn't point at the actual notes in the binary, it points at bogus data.

Pyelftools, which provides the note parsing logic there, is not entirely blameless - for one thing, zero length note name and description are legitimate (if pointless in practice) and should parse without throwing exceptions. We will fix that. The problem from the pwntools' standpoint, however, is that the real notes in the binary are not found.

I've found some evidence on the big wide net that this malformed PT_NOTE segment could be found in numerous, unrelated binaries. I am convinced that the generation of this bogus PT_NOTE segment was a bug in the GNU linker that didn't manifest prominently, went unnoticed for a while, and was quietly fixed since. But the malformed binaries remain, floating around Linux distros' repositories.

In the meantime, I'm trying to get to the history of it here.

May I suggest that pwntools enumerates note sections instead of segments?

[Arusekk]

I don't think so. Pwntools is trying to adhere to what the loader does, and it seems the loader does not look at sections at all. In most cases it does not even know the section names! (section headers and strings being placed in a file typically after .data).

As you have probably guessed by grepping, pwntools iterates notes for two reasons. One, examining core dumps (kernel bugs tend to live shorter than coreutils bugs, and are often backported as security fixes unlike in coreutils; core dumps also live shorter). This works and is probably safer than iterating note sections I think. And two, mimicking the loader to infer whether or not there are some properties to be activated. I think pyelftools should also stick to what the loader typically does (including ignoring incorrect notes; maybe some ignore_bogus flag at least).

[sevaa]

OK, up to you. I don't know if the lead behind pyelftools (@eliben) will agree to the introduction of an ignore_bogus flag; we normally avoid bug compatibility hacks.

Fully mimicking what the loader does is way beyond the scope. Were segment parsing logic aware of the possibility of segment overlap, we'd have to build up the entire memory layout of the process to determine the physical edges of what's readable and what's not, and that, in turn, would have to bring page boundaries into consideration (in flat x86 and elsewhere the granularity of memory protection is a page, after all), and suddenly it becomes CPU and OS dependent (ELF files are used outside of Linux). Too hairy for comfort.

In fact, I'm not sure what does the loader do if segments overlap. How does would it treat, for example, conflicting protection flags? This is no longer documented behavior of the loader, this is implementation details of the loader; mimicking those is a much trickier proposition.

Do you happen to know what does the loader need notes for? I was under the impression that the loader pretty much ignores them.

[Arusekk]

The relevant piece of code from pwntools:

    def iter_properties(self):
        """
        Yields:
            All the GNU properties in the PT_NOTE segments.  Each result is a dictionary-
            like object with ``pr_type``, ``pr_datasz``, and ``pr_data`` fields.
        """
        for note in self.iter_notes():
            if note.n_type ≢ 'NT_GNU_PROPERTY_TYPE_0':
                continue
            for prop in note.n_desc:
                yield prop

...and then the properties are parsed to find whether IBT and SHSTK (indirect branch tracking & shadow stack) are enabled or not. Arguably minor loss of functionality if broken.

Grepping for NT_GNU_PROPERTY yields results both in linux and glibc; grepping for gnu.property only finds assembly directives and linker scripts. Grepping for SHT_NOTE in glibc yields only the definition, and in Linux is used only in the context of vdso/modules/tooling (perceived as a binutils 'bug'); PT_NOTE gives many many results. There is also a newer PT_GNU_PROPERTY, but not yet implemented in pwntools (nor pyelftools I think). Maybe pyelftools would like to unify it somehow.

I believe the kernel especially never bothers to even read section descriptors; the loader might do it, but not for notes.
I believe sections are in general only used for linking relocatables, and for any 'production' use there are segments (like the dynamic segment). I can do some further research if needed, but I think nothing surprising will turn up.