Opcodes.md

Pickle Opcode Documentation

Each opcode implemented in Pickle is documented below. The following items are included:

• The hex and (optionally) character representation of the opcode byte
• The protocol it was introduced in
• The official description found in pickletools.py source code
• Notes from the author (me!) expanding upon functionality, dissecting edge cases, or explaining discrepancies between various implementations
• Examples of how that opcode can modify the stack/metastack/memo
• Collapsible sections with the Python 3.11 source code and links for all 3 implementations (pickle.py, _pickle.c, and pickletools)
  • Note that not all opcodes have pickletools sections; this is because pickletools only parses opcode arguments found in the pickle bytestream and doesn't actually run the instruction, so any opcodes that don't use opcode arguments don't have a corresponding section.
  • The code snippets are from Python 3.11 and may not accurately reflect all versions of Python.

Summary Table

A quick overview of key information is located here, the thorough documentation is found underneath.

Opcode Name	Opcode Hex Value	Protocol	Modifies Stack	Modifies Metastack	Modifies Memo	Calls find_class()
INT	0x49	0	✅	❌	❌	❌
BININT	0x4a	1	✅	❌	❌	❌
BININT1	0x4b	1	✅	❌	❌	❌
BININT2	0x4d	1	✅	❌	❌	❌
LONG	0x4c	0	✅	❌	❌	❌
LONG1	0x8a	2	✅	❌	❌	❌
LONG4	0x8b	2	✅	❌	❌	❌
STRING	0x53	0	✅	❌	❌	❌
BINSTRING	0x54	1	✅	❌	❌	❌
SHORT_BINSTRING	0x55	1	✅	❌	❌	❌
BINBYTES	0x42	3	✅	❌	❌	❌
SHORT_BINBYTES	0x43	3	✅	❌	❌	❌
BINBYTES8	0x8e	4	✅	❌	❌	❌
BYTEARRAY8	0x96	5	✅	❌	❌	❌
NEXT_BUFFER	0x97	5	✅	❌	❌	❌
READONLY_BUFFER	0x98	5	❌	❌	❌	❌
NONE	0x4e	2	✅	❌	❌	❌
NEWTRUE	0x88	2	✅	❌	❌	❌
NEWFALSE	0x89	2	✅	❌	❌	❌
UNICODE	0x56	0	✅	❌	❌	❌
SHORT_BINUNICODE	0x8c	4	✅	❌	❌	❌
BINUNICODE	0x58	1	✅	❌	❌	❌
BINUNICODE8	0x8d	4	✅	❌	❌	❌
FLOAT	0x46	0	✅	❌	❌	❌
BINFLOAT	0x47	1	✅	❌	❌	❌
EMPTY_LIST	0x5d	1	✅	❌	❌	❌
APPEND	0x61	0	✅	❌	❌	❌
APPENDS	0x65	1	✅	✅	❌	❌
LIST	0x6c	0	✅	✅	❌	❌
EMPTY_TUPLE	0x29	1	✅	❌	❌	❌
TUPLE	0x74	0	✅	✅	❌	❌
TUPLE1	0x85	2	✅	❌	❌	❌
TUPLE2	0x86	2	✅	❌	❌	❌
TUPLE3	0x87	2	✅	❌	❌	❌
EMPTY_DICT	0x7d	1	✅	❌	❌	❌
DICT	0x64	0	✅	✅	❌	❌
SETITEM	0x73	0	✅	❌	❌	❌
SETITEMS	0x75	1	✅	✅	❌	❌
EMPTY_SET	0x8f	4	✅	❌	❌	❌
ADDITEMS	0x90	4	✅	✅	❌	❌
FROZENSET	0x91	4	✅	✅	❌	❌
POP	0x30	0	✅	✅	❌	❌
DUP	0x32	0	✅	❌	❌	❌
MARK	0x28	0	✅	✅	❌	❌
POP_MARK	0x31	0	✅	✅	❌	❌
GET	0x67	0	✅	❌	❌	❌
BINGET	0x68	1	✅	❌	❌	❌
LONG_BINGET	0x6a	1	✅	❌	❌	❌
PUT	0x70	0	❌	❌	✅	❌
BINPUT	0x71	1	❌	❌	✅	❌
LONG_BINPUT	0x72	1	❌	❌	✅	❌
MEMOIZE	0x94	4	❌	❌	✅	❌
EXT1	0x82	2	✅	❌	❌	✅
EXT2	0x83	2	✅	❌	❌	✅
EXT4	0x84	2	✅	❌	❌	✅
GLOBAL	0x63	0	✅	❌	❌	✅
STACK_GLOBAL	0x93	4	✅	❌	❌	✅
REDUCE	0x52	0	✅	❌	❌	❌
BUILD	0x62	0	✅	❌	❌	❌
INST	0x69	0	✅	✅	❌	✅
OBJ	0x6f	1	✅	✅	❌	❌
NEWOBJ	0x81	2	✅	❌	❌	❌
NEWOBJ_EX	0x92	4	✅	❌	❌	❌
PROTO	0x80	2	❌	❌	❌	❌
STOP	0x2e	0	❌	❌	❌	❌
FRAME	0x95	4	❌	❌	❌	❌
PERSID	0x50	0	✅	❌	❌	❌
BINPERSID	0x51	1	✅	❌	❌	❌

Opcodes

INT - Push an integer or bool

• Opcode - 0x49 ('I')
• Protocol introduced - 0

Official Description:

The argument is a newline-terminated decimal literal string.

The intent may have been that this always fit in a short Python int, but INT can be generated in pickles written on a 64-bit box that require a Python long on a 32-bit box. The difference between this and LONG then is that INT skips a trailing 'L', and produces a short int whenever possible.

Another difference is due to that, when bool was introduced as a distinct type in 2.3, builtin names True and False were also added to 2.2.2, mapping to ints 1 and 0. For compatibility in both directions, True gets pickled as INT + "I01\\n", and False as INT + "I00\\n". Leading zeroes are never produced for a genuine integer. The 2.3 (and later) unpicklers special-case these and return bool instead; earlier unpicklers ignore the leading "0" and return the int.

Author's Note:

Any amount of non-newline whitespace before or after the digits can be inserted as int() automatically strips whitespace. Underscores (that aren't leading or trailing) can also be placed in the string and won't affect the value. A negative (-) or positive (+) sign can precede the numbers.

The documentation for this opcode according to pickletools states the argument is a decimal literal string, and the source code uses the function call int(s) when disassembling. However, the actual source code for pickle uses the call int(data, 0) (and _pickle.c also uses base 0), meaning that other formats can be used such as hexadecimal. This means the last example will be unpickled correctly, but will cause the pickletools disassembly to throw an exception.

A discrepancy exists in handling input like b'I 0\n'. pickle and pickletools specifically check for 00\n and 01\n to become False and True (respectively), but _pickle.c checks whether the length is 3 and strtolower() gives 0 or 1. Therefore, b'I 0\n' is 0 to pickle and pickletools, but False to _pickle.c. Note that any non-newline whitespace can be used here instead of a space.

A discrepancy also exists in handling strings with multiple 0s prepended. For example, b'I001\n' resolves as 1 for _pickle.c and pickletools, but fails for pickle. This is because the int(..., 0) function with the base 0 argument fails on any string input with prepended zeros. _pickle.c uses strtol and pickletools doesn't specify base 0, which is why they don't fail.

In Python's C implementation, PyLong_FromString stops at a null byte, which means a payload like b'I1\x00anything\n' would read everything up until the newline (1\x00anything\n) but then only process the data up until the null byte. This means _pickle.c would return 1, but pickle.py and pickletools would error out. This also means that literally anything but a newline can be placed in between the null byte and the newline and it won't get processed.

Example(s):

Before stack : []
Pickle bytes : b'I00\n' (len 4)
After stack  : [False]

Before stack : []
Pickle bytes : b'I01\n' (len 4)
After stack  : [True]

Before stack : []
Pickle bytes : b'I1337\n' (len 6)
After stack  : [1337]

Before stack : []
Pickle bytes : b'I0x1337\n' (len 8)
After stack  : [4919]

Before stack : []
Pickle bytes : b'I  0x_133_7 \n' (len 13)
After stack  : [4919]

Source Code (link)

def load_int(self):
    data = self.readline()
    if data == FALSE[1:]:
        val = False
    elif data == TRUE[1:]:
        val = True
    else:
        val = int(data, 0)
    self.append(val)

Pickletools Code (link)

def read_decimalnl_short(f):
    s = read_stringnl(f, decode=False, stripquotes=False)

    # There's a hack for True and False here.
    if s == b"00":
        return False
    elif s == b"01":
        return True

    return int(s)

_pickle.c Code (link)

static int
load_int(UnpicklerObject *self)
{
    PyObject *value;
    char *endptr, *s;
    Py_ssize_t len;
    long x;

    if ((len = _Unpickler_Readline(self, &s)) < 0)
        return -1;
    if (len < 2)
        return bad_readline();

    errno = 0;
    /* XXX: Should the base argument of strtol() be explicitly set to 10?
       XXX(avassalotti): Should this uses PyOS_strtol()? */
    x = strtol(s, &endptr, 0);

    if (errno || (*endptr != '\n' && *endptr != '\0')) {
        /* Hm, maybe we've got something long.  Let's try reading
         * it as a Python int object. */
        errno = 0;
        /* XXX: Same thing about the base here. */
        value = PyLong_FromString(s, NULL, 0);
        if (value == NULL) {
            PyErr_SetString(PyExc_ValueError,
                            "could not convert string to int");
            return -1;
        }
    }
    else {
        if (len == 3 && (x == 0 || x == 1)) {
            if ((value = PyBool_FromLong(x)) == NULL)
                return -1;
        }
        else {
            if ((value = PyLong_FromLong(x)) == NULL)
                return -1;
        }
    }

    PDATA_PUSH(self->stack, value, -1);
    return 0;
}

BININT - Push a four-byte signed integer

• Opcode - 0x4a ('J')
• Protocol introduced - 1

Official Description:

This handles the full range of Python (short) integers on a 32-bit box, directly as binary bytes (1 for the opcode and 4 for the integer). If the integer is non-negative and fits in 1 or 2 bytes, pickling via BININT1 or BININT2 saves space.

Author's Note:

The data read in is a four-byte signed integer, little-endian, using 2's complement.

Example(s):

Before stack : []
Pickle bytes : b'J\x01\x00\x00\x00' (len 5)
After stack  : [1]

Before stack : []
Pickle bytes : b'J\x00\x00\x00\x0f' (len 5)
After stack  : [251658240]

Before stack : []
Pickle bytes : b'J\x00\x00\x00\xff' (len 5)
After stack  : [-16777216]

Source Code (link)

def load_binint(self):
    self.append(unpack('<i', self.read(4))[0])

Pickletools Code (link)

def read_int4(f):
    data = f.read(4)
    if len(data) == 4:
        return _unpack("<i", data)[0]
    raise ValueError("not enough data in stream to read int4")

_pickle.c Code (link)

static int
load_binint(UnpicklerObject *self)
{
    char *s;

    if (_Unpickler_Read(self, &s, 4) < 0)
        return -1;

    return load_binintx(self, s, 4);
}

BININT1 - Push a one-byte unsigned integer

• Opcode - 0x4b ('K')
• Protocol introduced - 1

Official Description:

This is a space optimization for pickling very small non-negative ints, in range(256).

Example(s):

Before stack : []
Pickle bytes : b'K\x01' (len 2)
After stack  : [1]

Before stack : []
Pickle bytes : b'K\xff' (len 2)
After stack  : [255]

Source Code (link)

def load_binint1(self):
    self.append(self.read(1)[0])

Pickletools Code (link)

def read_uint1(f):
    data = f.read(1)
    if data:
        return data[0]
    raise ValueError("not enough data in stream to read uint1")

_pickle.c Code (link)

static int
load_binint1(UnpicklerObject *self)
{
    char *s;

    if (_Unpickler_Read(self, &s, 1) < 0)
        return -1;

    return load_binintx(self, s, 1);
}

BININT2 - Push a two-byte unsigned integer

• Opcode - 0x4d ('M')
• Protocol introduced - 1

Official Description:

This is a space optimization for pickling small positive ints, in range(256, 2**16). Integers in range(256) can also be pickled via BININT2, but BININT1 instead saves a byte.

Author's Note:

The two-byte unsigned integer uses little-endian encoding.

Example(s):

Before stack : []
Pickle bytes : b'M\x01\x00' (len 3)
After stack  : [1]

Before stack : []
Pickle bytes : b'M\x00\xff' (len 3)
After stack  : [65280]

Source Code (link)

def load_binint2(self):
    self.append(unpack('<H', self.read(2))[0])

Pickletools Code (link)

def read_uint2(f):
    data = f.read(2)
    if len(data) == 2:
        return _unpack("<H", data)[0]
    raise ValueError("not enough data in stream to read uint2")

_pickle.c Code (link)

static int
load_binint2(UnpicklerObject *self)
{
    char *s;

    if (_Unpickler_Read(self, &s, 2) < 0)
        return -1;

    return load_binintx(self, s, 2);
}

LONG - Push a long integer

• Opcode - 0x4c ('L')
• Protocol introduced - 0

Official Description:

The same as INT, except that the literal ends with 'L', and always unpickles to a Python long. There doesn't seem a real purpose to the trailing 'L'.

Note that LONG takes time quadratic in the number of digits when unpickling (this is simply due to the nature of decimal->binary conversion). Proto 2 added linear-time (in C; still quadratic-time in Python) LONG1 and LONG4 opcodes.

Author's Note:

Including an L is not required for this opcode (to maintain backwards compatibility).

Just like INT, any amount of non-newline whitespace before or after the digits can be inserted as int() automatically strips whitespace. Underscores (that aren't leading or trailing) can also be placed in the string and won't affect the value. A negative (-) or positive (+) sign can precede the numbers.

In additional, LONG is also vulnerable to the same discrepancy when parsing non-decimal formatted strings (like hexadecimal, octal, or binary). When it comes to parsing strings with multiple 0s prepended (like 0001), both pickle and _pickle.c will error out as they both use base 0; pickletools will continue to show the number as 1.

Just like INT, the discrepancy exists where placing a null byte inside the string causes _pickle.c to only process characters for the string up until the null byte (but still reads from the buffer up until the newline). This means a payload like b'L1\x00anything\n' would cause _pickle.c to return 1, while pickle.py and pickletools error out. This also means that literally anything but a newline can be placed in between the null byte and the newline and it won't get processed.

Example(s):

Before stack : []
Pickle bytes : b'L1337L\n' (len 7)
After stack  : [1337]

Before stack : []
Pickle bytes : b'L0x1337L\n' (len 9)
After stack  : [4919]

Before stack : []
Pickle bytes : b'L1337\n' (len 6)
After stack  : [1337]

Before stack : []
Pickle bytes : b'L  0x_133_7 L\n' (len 14)
After stack  : [4919]

Source Code (link)

def load_long(self):
    val = self.readline()[:-1]
    if val and val[-1] == b'L'[0]:
        val = val[:-1]
    self.append(int(val, 0))

Pickletools Code (link)

def read_decimalnl_long(f):
    s = read_stringnl(f, decode=False, stripquotes=False)
    if s[-1:] == b'L':
        s = s[:-1]
    return int(s)

_pickle.c Code (link)

static int
load_long(UnpicklerObject *self)
{
    PyObject *value;
    char *s = NULL;
    Py_ssize_t len;

    if ((len = _Unpickler_Readline(self, &s)) < 0)
        return -1;
    if (len < 2)
        return bad_readline();

    /* s[len-2] will usually be 'L' (and s[len-1] is '\n'); we need to remove
       the 'L' before calling PyLong_FromString.  In order to maintain
       compatibility with Python 3.0.0, we don't actually *require*
       the 'L' to be present. */
    if (s[len-2] == 'L')
        s[len-2] = '\0';
    /* XXX: Should the base argument explicitly set to 10? */
    value = PyLong_FromString(s, NULL, 0);
    if (value == NULL)
        return -1;

    PDATA_PUSH(self->stack, value, -1);
    return 0;
}

LONG1 - Long integer using one-byte length

• Opcode - 0x8a
• Protocol introduced - 2

Official Description:

A more efficient encoding of a Python long; the long1 encoding says it all.

This first reads one byte as an unsigned size, then reads that many bytes and interprets them as a little-endian 2's-complement long. If the size is 0, that's taken as a shortcut for the long 0L.

Author's Note:

Note that the _pickle.c implementation uses the same function for the LONG1 and LONG4 opcodes, but the size parameter is passed in as 1 and 4 (respectively).

While the size parameter is unsigned, the long value itself may be signed.

Example(s):

Before stack : []
Pickle bytes : b'\x8a\x00' (len 2)
After stack  : [0]

Before stack : []
Pickle bytes : b'\x8a\x02\xff\x00' (len 4)
After stack  : [255]

Before stack : []
Pickle bytes : b'\x8a\x02\x00\x80' (len 4)
After stack  : [-32768]

Source Code (link)

def load_long1(self):
    n = self.read(1)[0]
    data = self.read(n)
    self.append(decode_long(data))

Pickletools Code (link)

def read_long1(f):
    n = read_uint1(f)
    data = f.read(n)
    if len(data) != n:
        raise ValueError("not enough data in stream to read long1")
    return decode_long(data)

_pickle.c Code (link)

static int
load_counted_long(UnpicklerObject *self, int size)
{
    PyObject *value;
    char *nbytes;
    char *pdata;

    assert(size == 1 || size == 4);
    if (_Unpickler_Read(self, &nbytes, size) < 0)
        return -1;

    size = calc_binint(nbytes, size);
    if (size < 0) {
        PickleState *st = _Pickle_GetGlobalState();
        /* Corrupt or hostile pickle -- we never write one like this */
        PyErr_SetString(st->UnpicklingError,
                        "LONG pickle has negative byte count");
        return -1;
    }

    if (size == 0)
        value = PyLong_FromLong(0L);
    else {
        /* Read the raw little-endian bytes and convert. */
        if (_Unpickler_Read(self, &pdata, size) < 0)
            return -1;
        value = _PyLong_FromByteArray((unsigned char *)pdata, (size_t)size,
                                      1 /* little endian */ , 1 /* signed */ );
    }
    if (value == NULL)
        return -1;
    PDATA_PUSH(self->stack, value, -1);
    return 0;
}

LONG4 - Long integer using four-byte length

• Opcode - 0x8b
• Protocol introduced - 2

Official Description:

A more efficient encoding of a Python long; the long4 encoding says it all.

This first reads four bytes as a signed size (but requires the size to be >= 0), then reads that many bytes and interprets them as a little-endian 2's-complement long. If the size is 0, that's taken as a shortcut for the int 0, although LONG1 should really be used then instead (and in any case where number of bytes < 256).

Author's Note:

Note that the _pickle.c implementation uses the same function for the LONG1 and LONG4 opcodes, but the size parameter is passed in as 1 and 4 (respectively).

Example(s):

Before stack : []
Pickle bytes : b'\x8b\x00\x00\x00\x00' (len 5)
After stack  : [0]

Before stack : []
Pickle bytes : b'\x8b\x02\x00\x00\x00\xff\x7f' (len 7)
After stack  : [32767]

Before stack : []
Pickle bytes : b'\x8b\x02\x00\x00\x00\x00\xff' (len 7)
After stack  : [-256]

Source Code (link)

def load_long4(self):
    n, = unpack('<i', self.read(4))
    if n < 0:
        # Corrupt or hostile pickle -- we never write one like this
        raise UnpicklingError("LONG pickle has negative byte count")
    data = self.read(n)
    self.append(decode_long(data))

Pickletools Code (link)

def read_long4(f):
    n = read_int4(f)
    if n < 0:
        raise ValueError("long4 byte count < 0: %d" % n)
    data = f.read(n)
    if len(data) != n:
        raise ValueError("not enough data in stream to read long4")
    return decode_long(data)

_pickle.c Code (link)

static int
load_counted_long(UnpicklerObject *self, int size)
{
    PyObject *value;
    char *nbytes;
    char *pdata;

    assert(size == 1 || size == 4);
    if (_Unpickler_Read(self, &nbytes, size) < 0)
        return -1;

    size = calc_binint(nbytes, size);
    if (size < 0) {
        PickleState *st = _Pickle_GetGlobalState();
        /* Corrupt or hostile pickle -- we never write one like this */
        PyErr_SetString(st->UnpicklingError,
                        "LONG pickle has negative byte count");
        return -1;
    }

    if (size == 0)
        value = PyLong_FromLong(0L);
    else {
        /* Read the raw little-endian bytes and convert. */
        if (_Unpickler_Read(self, &pdata, size) < 0)
            return -1;
        value = _PyLong_FromByteArray((unsigned char *)pdata, (size_t)size,
                                      1 /* little endian */ , 1 /* signed */ );
    }
    if (value == NULL)
        return -1;
    PDATA_PUSH(self->stack, value, -1);
    return 0;
}

STRING - A newline-terminated string

• Opcode - 0x53 ('S')
• Protocol introduced - 0

Official Description:

The argument is a repr-style string, with bracketing quote characters, and perhaps embedded escapes. The argument extends until the next newline character. These are usually decoded into a str instance using the encoding given to the Unpickler constructor, or the default, ASCII. If the encoding given was bytes however, they will be decoded as bytes object instead.

Author's Note:

All strings must be wrapped in double quotes (") or single quotes ('), which are then stripped. Embedded escapes are simplified, such as \\n --> \n.

The pickle library supports decoding as bytes/latin-1/etc (must be passed in as an argument when initializing the Unpickler), but pickletools will assume the value is decoded as ASCII. This means a string with non-ASCII/Unicode bytes like '\x99' would be processed by pickle/_pickle.c successfully, but error out in pickletools. This is because pickletools doesn't seem to support user-specified string encodings, but always assumes ASCII. Note that the default encoding for pickle and _pickle.c is ASCII.

Example(s):

Before stack : []
Pickle bytes : b'S"abcd"\n' (len 8)
After stack  : ['abcd']

Before stack : []
Pickle bytes : b'S"ab\\ncd"\n' (len 10)
After stack  : ['ab\ncd']

Before stack : []
Pickle bytes : b'S\'abcd\'\n' (len 8)
After stack  : ['abcd']

(with Unpickler.encoding == "bytes")
Before stack : []
Pickle bytes : b'S"abcd"\n' (len 8)
After stack  : [b'abcd']

(with Unpickler.encoding == "bytes")
Before stack : []
Pickle bytes : b'S"ab\x99cd"\n' (len 9)
After stack  : [b'ab\x99cd']

Source Code (link)

def load_string(self):
    data = self.readline()[:-1]
    # Strip outermost quotes
    if len(data) >= 2 and data[0] == data[-1] and data[0] in b'"\'':
        data = data[1:-1]
    else:
        raise UnpicklingError("the STRING opcode argument must be quoted")
    self.append(self._decode_string(codecs.escape_decode(data)[0]))

Pickletools Code (link)

def read_stringnl(f, decode=True, stripquotes=True):
    data = f.readline()
    if not data.endswith(b'\n'):
        raise ValueError("no newline found when trying to read stringnl")
    data = data[:-1]    # lose the newline

    if stripquotes:
        for q in (b'"', b"'"):
            if data.startswith(q):
                if not data.endswith(q):
                    raise ValueError("strinq quote %r not found at both "
                                     "ends of %r" % (q, data))
                data = data[1:-1]
                break
        else:
            raise ValueError("no string quotes around %r" % data)

    if decode:
        data = codecs.escape_decode(data)[0].decode("ascii")
    return data

_pickle.c Code (link)

static int
load_string(UnpicklerObject *self)
{
    PyObject *bytes;
    PyObject *obj;
    Py_ssize_t len;
    char *s, *p;

    if ((len = _Unpickler_Readline(self, &s)) < 0)
        return -1;
    /* Strip the newline */
    len--;
    /* Strip outermost quotes */
    if (len >= 2 && s[0] == s[len - 1] && (s[0] == '\'' || s[0] == '"')) {
        p = s + 1;
        len -= 2;
    }
    else {
        PickleState *st = _Pickle_GetGlobalState();
        PyErr_SetString(st->UnpicklingError,
                        "the STRING opcode argument must be quoted");
        return -1;
    }
    assert(len >= 0);

    /* Use the PyBytes API to decode the string, since that is what is used
       to encode, and then coerce the result to Unicode. */
    bytes = PyBytes_DecodeEscape(p, len, NULL, 0, NULL);
    if (bytes == NULL)
        return -1;

    /* Leave the Python 2.x strings as bytes if the *encoding* given to the
       Unpickler was 'bytes'. Otherwise, convert them to unicode. */
    if (strcmp(self->encoding, "bytes") == 0) {
        obj = bytes;
    }
    else {
        obj = PyUnicode_FromEncodedObject(bytes, self->encoding, self->errors);
        Py_DECREF(bytes);
        if (obj == NULL) {
            return -1;
        }
    }

    PDATA_PUSH(self->stack, obj, -1);
    return 0;
}

BINSTRING - Push a Python string object

• Opcode - 0x54 ('T')
• Protocol introduced - 1

Official Description:

There are two arguments: the first is a 4-byte little-endian signed int giving the number of bytes in the string, and the second is that many bytes, which are taken literally as the string content. These are usually decoded into a str instance using the encoding given to the Unpickler constructor, or the default, 'ASCII'. If the encoding given was 'bytes' however, they will be decoded as bytes object instead.

Author's Note:

Because pickletools doesn't support custom encodings, it makes assumptions about the encoding of strings introduced through opcodes like this. STRING assumes an ASCII encoding since that's the default, but this one uses latin-1. If the pickle is supposed to be encoded using another encoding, this will appear wrong in pickletools.

A bug exists in _pickle.c's parsing of BINSTRING, causing a discrepancy between how pickle/pickletools and _pickle.c calculate the length of the string in 64-bit systems. pickle and pickletools specifically use a 32-bit signed number for the length, meaning 0x80000000 is -2147483648. However, _pickle.c feeds the bytes into calc_binsize and the result is casted to a Py_ssize_t variable, which is a 64-bit signed data type (on 64-bit systems). Therefore, 0x80000000 is treated as a large positive number instead of a negative number. Note that in LONG4 (where similar functionality exists) this bug doesn't exist because the output from calc_binsize is a signed integer.

Note that the _pickle.c implementation uses the same function for the BINSTRING and SHORT_BINSTRING opcodes, but the size parameter is passed in as 4 and 1 (respectively).

Example(s):

Before stack : []
Pickle bytes : b'T\x04\x00\x00\x001234' (len 9)
After stack  : ['1234']

Before stack : []
Pickle bytes : b'T\x07\x00\x00\x00123\n456' (len 12)
After stack  : ['123\n456']

(with Unpickler.encoding == "latin-1")
Before stack : []
Pickle bytes : b'T\x03\x00\x00\x00a\x99b' (len 8)
After stack  : ['a\x99b']

Source Code (link)

def load_binstring(self):
    # Deprecated BINSTRING uses signed 32-bit length
    len, = unpack('<i', self.read(4))
    if len < 0:
        raise UnpicklingError("BINSTRING pickle has negative byte count")
    data = self.read(len)
    self.append(self._decode_string(data))

Pickletools Code (link)

def read_string4(f):
    n = read_int4(f)
    if n < 0:
        raise ValueError("string4 byte count < 0: %d" % n)
    data = f.read(n)
    if len(data) == n:
        return data.decode("latin-1")
    raise ValueError("expected %d bytes in a string4, but only %d remain" %
                     (n, len(data)))

_pickle.c Code (link)

static int
load_counted_binstring(UnpicklerObject *self, int nbytes)
{
    PyObject *obj;
    Py_ssize_t size;
    char *s;

    if (_Unpickler_Read(self, &s, nbytes) < 0)
        return -1;

    size = calc_binsize(s, nbytes);
    if (size < 0) {
        PickleState *st = _Pickle_GetGlobalState();
        PyErr_Format(st->UnpicklingError,
                     "BINSTRING exceeds system's maximum size of %zd bytes",
                     PY_SSIZE_T_MAX);
        return -1;
    }

    if (_Unpickler_Read(self, &s, size) < 0)
        return -1;

    /* Convert Python 2.x strings to bytes if the *encoding* given to the
       Unpickler was 'bytes'. Otherwise, convert them to unicode. */
    if (strcmp(self->encoding, "bytes") == 0) {
        obj = PyBytes_FromStringAndSize(s, size);
    }
    else {
        obj = PyUnicode_Decode(s, size, self->encoding, self->errors);
    }
    if (obj == NULL) {
        return -1;
    }

    PDATA_PUSH(self->stack, obj, -1);
    return 0;
}

SHORT_BINSTRING - Push a Python string object

• Opcode - 0x55 ('U')
• Protocol introduced - 1

Official Description:

There are two arguments: the first is a 1-byte unsigned int giving the number of bytes in the string, and the second is that many bytes, which are taken literally as the string content. These are usually decoded into a str instance using the encoding given to the Unpickler constructor. or the default, 'ASCII'. If the encoding given was 'bytes' however, they will be decoded as bytes object instead.

Author's Note:

Because pickletools doesn't support custom encodings, it makes assumptions about the encoding of strings introduced through opcodes like this. STRING assumes an ASCII encoding since that's the default, but this one uses latin-1. If the pickle is supposed to be encoded using another encoding, this will appear wrong in pickletools.

Note that the _pickle.c implementation uses the same function for the BINSTRING and SHORT_BINSTRING opcodes, but the size parameter is passed in as 4 and 1 (respectively).

Example(s):

Before stack : []
Pickle bytes : b'U\x041234' (len 6)
After stack  : ['1234']

Before stack : []
Pickle bytes : b'U\x07123\n456' (len 9)
After stack  : ['123\n456']

(with Unpickler.encoding == "latin-1")
Before stack : []
Pickle bytes : b'U\x03a\x99b' (len 5)
After stack  : ['a\x99b']

Source Code (link)

def load_short_binstring(self):
    len = self.read(1)[0]
    data = self.read(len)
    self.append(self._decode_string(data))

Pickletools Code (link)

def read_string1(f):
    n = read_uint1(f)
    assert n >= 0
    data = f.read(n)
    if len(data) == n:
        return data.decode("latin-1")
    raise ValueError("expected %d bytes in a string1, but only %d remain" %
                     (n, len(data)))

_pickle.c Code (link)

static int
load_counted_binstring(UnpicklerObject *self, int nbytes)
{
    PyObject *obj;
    Py_ssize_t size;
    char *s;

    if (_Unpickler_Read(self, &s, nbytes) < 0)
        return -1;

    size = calc_binsize(s, nbytes);
    if (size < 0) {
        PickleState *st = _Pickle_GetGlobalState();
        PyErr_Format(st->UnpicklingError,
                     "BINSTRING exceeds system's maximum size of %zd bytes",
                     PY_SSIZE_T_MAX);
        return -1;
    }

    if (_Unpickler_Read(self, &s, size) < 0)
        return -1;

    /* Convert Python 2.x strings to bytes if the *encoding* given to the
       Unpickler was 'bytes'. Otherwise, convert them to unicode. */
    if (strcmp(self->encoding, "bytes") == 0) {
        obj = PyBytes_FromStringAndSize(s, size);
    }
    else {
        obj = PyUnicode_Decode(s, size, self->encoding, self->errors);
    }
    if (obj == NULL) {
        return -1;
    }

    PDATA_PUSH(self->stack, obj, -1);
    return 0;
}

BINBYTES - Push a Python bytes object

• Opcode - 0x42 ('B')
• Protocol introduced - 3

Official Description:

There are two arguments: the first is a 4-byte little-endian unsigned int giving the number of bytes, and the second is that many bytes, which are taken literally as the bytes content.

Author's Note:

Note that the _pickle.c implementation uses the same function for the BINBYTES, SHORT_BINBYTES, and BINBYTES8 opcodes, but the size parameter is passed in as 4, 1, and 8 (respectively).

Example(s):

Before stack : []
Pickle bytes : b'B\x04\x00\x00\x001234' (len 9)
After stack  : [b'1234']

Before stack : []
Pickle bytes : b'B\x07\x00\x00\x00123\n456' (len 12)
After stack  : [b'123\n456']

Before stack : []
Pickle bytes : b'B\x00\x00\x00\x00' (len 5)
After stack  : [b'']

Source Code (link)

def load_binbytes(self):
    len, = unpack('<I', self.read(4))
    if len > maxsize:
        raise UnpicklingError("BINBYTES exceeds system's maximum size "
                                "of %d bytes" % maxsize)
    self.append(self.read(len))

Pickletools Code (link)

def read_bytes4(f):
    n = read_uint4(f)
    assert n >= 0
    if n > sys.maxsize:
        raise ValueError("bytes4 byte count > sys.maxsize: %d" % n)
    data = f.read(n)
    if len(data) == n:
        return data
    raise ValueError("expected %d bytes in a bytes4, but only %d remain" %
                     (n, len(data)))

_pickle.c Code (link)

static int
load_counted_binbytes(UnpicklerObject *self, int nbytes)
{
    PyObject *bytes;
    Py_ssize_t size;
    char *s;

    if (_Unpickler_Read(self, &s, nbytes) < 0)
        return -1;

    size = calc_binsize(s, nbytes);
    if (size < 0) {
        PyErr_Format(PyExc_OverflowError,
                     "BINBYTES exceeds system's maximum size of %zd bytes",
                     PY_SSIZE_T_MAX);
        return -1;
    }

    bytes = PyBytes_FromStringAndSize(NULL, size);
    if (bytes == NULL)
        return -1;
    if (_Unpickler_ReadInto(self, PyBytes_AS_STRING(bytes), size) < 0) {
        Py_DECREF(bytes);
        return -1;
    }

    PDATA_PUSH(self->stack, bytes, -1);
    return 0;
}

SHORT_BINBYTES - Push a Python bytes object

• Opcode - 0x43 ('C')
• Protocol introduced - 3

Official Description:

There are two arguments: the first is a 1-byte unsigned int giving the number of bytes, and the second is that many bytes, which are taken literally as the string content.

Author's Note:

Note that the _pickle.c implementation uses the same function for the BINBYTES, SHORT_BINBYTES, and BINBYTES8 opcodes, but the size parameter is passed in as 4, 1, and 8 (respectively).

Example(s):

Before stack : []
Pickle bytes : b'C\x041234' (len 6)
After stack  : [b'1234']

Before stack : []
Pickle bytes : b'C\x07123\n456' (len 9)
After stack  : [b'123\n456']

Before stack : []
Pickle bytes : b'C\x00' (len 2)
After stack  : [b'']

Source Code (link)

def load_short_binbytes(self):
    len = self.read(1)[0]
    self.append(self.read(len))

Pickletools Code (link)

def read_bytes1(f):
    n = read_uint1(f)
    assert n >= 0
    data = f.read(n)
    if len(data) == n:
        return data
    raise ValueError("expected %d bytes in a bytes1, but only %d remain" %
                     (n, len(data)))

_pickle.c Code (link)

static int
load_counted_binbytes(UnpicklerObject *self, int nbytes)
{
    PyObject *bytes;
    Py_ssize_t size;
    char *s;

    if (_Unpickler_Read(self, &s, nbytes) < 0)
        return -1;

    size = calc_binsize(s, nbytes);
    if (size < 0) {
        PyErr_Format(PyExc_OverflowError,
                     "BINBYTES exceeds system's maximum size of %zd bytes",
                     PY_SSIZE_T_MAX);
        return -1;
    }

    bytes = PyBytes_FromStringAndSize(NULL, size);
    if (bytes == NULL)
        return -1;
    if (_Unpickler_ReadInto(self, PyBytes_AS_STRING(bytes), size) < 0) {
        Py_DECREF(bytes);
        return -1;
    }

    PDATA_PUSH(self->stack, bytes, -1);
    return 0;
}

BINBYTES8 - Push a Python bytes object

• Opcode - 0x8e
• Protocol introduced - 4

Official Description:

There are two arguments: the first is an 8-byte unsigned int giving the number of bytes in the string, and the second is that many bytes, which are taken literally as the string content.

Author's Note:

Note that the _pickle.c implementation uses the same function for the BINBYTES, SHORT_BINBYTES, and BINBYTES8 opcodes, but the size parameter is passed in as 4, 1, and 8 (respectively).

Example(s):

Before stack : []
Pickle bytes : b'\x8e\x04\x00\x00\x00\x00\x00\x00\x001234' (len 13)
After stack  : [b'1234']

Before stack : []
Pickle bytes : b'\x8e\x07\x00\x00\x00\x00\x00\x00\x00123\n456' (len 16)
After stack  : [b'123\n456']

Before stack : []
Pickle bytes : b'\x8e\x00\x00\x00\x00\x00\x00\x00\x00' (len 9)
After stack  : [b'']

Source Code (link)

def load_binbytes8(self):
    len, = unpack('<Q', self.read(8))
    if len > maxsize:
        raise UnpicklingError("BINBYTES8 exceeds system's maximum size "
                                "of %d bytes" % maxsize)
    self.append(self.read(len))

Pickletools Code (link)

def read_bytes8(f):
    n = read_uint8(f)
    assert n >= 0
    if n > sys.maxsize:
        raise ValueError("bytes8 byte count > sys.maxsize: %d" % n)
    data = f.read(n)
    if len(data) == n:
        return data
    raise ValueError("expected %d bytes in a bytes8, but only %d remain" %
                     (n, len(data)))

_pickle.c Code (link)

static int
load_counted_binbytes(UnpicklerObject *self, int nbytes)
{
    PyObject *bytes;
    Py_ssize_t size;
    char *s;

    if (_Unpickler_Read(self, &s, nbytes) < 0)
        return -1;

    size = calc_binsize(s, nbytes);
    if (size < 0) {
        PyErr_Format(PyExc_OverflowError,
                     "BINBYTES exceeds system's maximum size of %zd bytes",
                     PY_SSIZE_T_MAX);
        return -1;
    }

    bytes = PyBytes_FromStringAndSize(NULL, size);
    if (bytes == NULL)
        return -1;
    if (_Unpickler_ReadInto(self, PyBytes_AS_STRING(bytes), size) < 0) {
        Py_DECREF(bytes);
        return -1;
    }

    PDATA_PUSH(self->stack, bytes, -1);
    return 0;
}

BYTEARRAY8 - Push a Python bytearray object

• Opcode - 0x96
• Protocol introduced - 5

Official Description:

There are two arguments: the first is an 8-byte unsigned int giving the number of bytes in the bytearray, and the second is that many bytes, which are taken literally as the bytearray content.

Author's Note:

Note that BYTEARRAY8 is practically identical to BINBYTES8, except a bytearray is returned instead of a bytestring. The main difference between the two is that a bytestring is immutable but a bytearray is not.

Example(s):

Before stack : []
Pickle bytes : b'\x96\x04\x00\x00\x00\x00\x00\x00\x001234' (len 13)
After stack  : [bytearray(b'1234')]

Before stack : []
Pickle bytes : b'\x96\x07\x00\x00\x00\x00\x00\x00\x00123\n456' (len 16)
After stack  : [bytearray(b'123\n456')]

Before stack : []
Pickle bytes : b'\x96\x00\x00\x00\x00\x00\x00\x00\x00' (len 9)
After stack  : [bytearray(b'')]

Source Code (link)

def load_bytearray8(self):
    len, = unpack('<Q', self.read(8))
    if len > maxsize:
        raise UnpicklingError("BYTEARRAY8 exceeds system's maximum size "
                                "of %d bytes" % maxsize)
    b = bytearray(len)
    self.readinto(b)
    self.append(b)

Pickletools Code (link)

def read_bytearray8(f):
    n = read_uint8(f)
    assert n >= 0
    if n > sys.maxsize:
        raise ValueError("bytearray8 byte count > sys.maxsize: %d" % n)
    data = f.read(n)
    if len(data) == n:
        return bytearray(data)
    raise ValueError("expected %d bytes in a bytearray8, but only %d remain" %
                     (n, len(data)))

_pickle.c Code (link)

static int
load_counted_bytearray(UnpicklerObject *self)
{
    PyObject *bytearray;
    Py_ssize_t size;
    char *s;

    if (_Unpickler_Read(self, &s, 8) < 0) {
        return -1;
    }

    size = calc_binsize(s, 8);
    if (size < 0) {
        PyErr_Format(PyExc_OverflowError,
                     "BYTEARRAY8 exceeds system's maximum size of %zd bytes",
                     PY_SSIZE_T_MAX);
        return -1;
    }

    bytearray = PyByteArray_FromStringAndSize(NULL, size);
    if (bytearray == NULL) {
        return -1;
    }
    if (_Unpickler_ReadInto(self, PyByteArray_AS_STRING(bytearray), size) < 0) {
        Py_DECREF(bytearray);
        return -1;
    }

    PDATA_PUSH(self->stack, bytearray, -1);
    return 0;
}

NEXT_BUFFER - Push an out-of-band buffer object

• Opcode - 0x97
• Protocol introduced - 5

Author's Note:

When deserializing a pickle, you can pass in iterables in the buffer argument like pickle.loads(p, buffers=[b'a']). When NEXT_BUFFER is called, the next element in the buffer iterable is pushed onto the stack. Note that this variable cannot be controlled in the pickle bytes, but must be passed in at runtime during the unpickling process.

Note that the elements of the iterable can be of any type, meaning it could potentially push anything to the stack including functions and classes.

Example(s):

(with buffers=b'a')
Before stack : []
Pickle bytes : b'\x97' (len 1)
After stack  : [97]

(with buffers=[b'a'])
Before stack : []
Pickle bytes : b'\x97' (len 1)
After stack  : [b'a']

(with buffers='a')
Before stack : []
Pickle bytes : b'\x97' (len 1)
After stack  : ['a']

(with buffers=[print])
Before stack : []
Pickle bytes : b'\x97' (len 1)
After stack  : [<built-in function print>]

Source Code (link)

def load_next_buffer(self):
    if self._buffers is None:
        raise UnpicklingError("pickle stream refers to out-of-band data "
                                "but no *buffers* argument was given")
    try:
        buf = next(self._buffers)
    except StopIteration:
        raise UnpicklingError("not enough out-of-band buffers")
    self.append(buf)

_pickle.c Code (link)

static int
load_next_buffer(UnpicklerObject *self)
{
    if (self->buffers == NULL) {
        PickleState *st = _Pickle_GetGlobalState();
        PyErr_SetString(st->UnpicklingError,
                        "pickle stream refers to out-of-band data "
                        "but no *buffers* argument was given");
        return -1;
    }
    PyObject *buf = PyIter_Next(self->buffers);
    if (buf == NULL) {
        if (!PyErr_Occurred()) {
            PickleState *st = _Pickle_GetGlobalState();
            PyErr_SetString(st->UnpicklingError,
                            "not enough out-of-band buffers");
        }
        return -1;
    }

    PDATA_PUSH(self->stack, buf, -1);
    return 0;
}

READONLY_BUFFER - Make an out-of-band buffer object read-only

• Opcode - 0x98
• Protocol introduced - 5

Author's Note:

This opcode assumes the top value on the stack is a buffer object. It removes the top value, sets it to read-only, then pushes the immutable result back on the stack. This means that the stack does not visibly change, however the underlying buffer object is slightly different. Note that the only built-in types that work with the memoryview() function are bytes and bytearray. Since bytes are not mutable anyway, this pretty much only prevents bytearrays from not changing during the unpickling process.

If the top value on the stack is not a valid buffer object, the unpickling process will error out.

Also note that if another reference to the buffer object still exists, then the "read-only" buffer CAN be changed. See the following code:

# initialize data
data = bytearray(b'example')
view = memoryview(data)
ro_view = view.toreadonly()

data[0] = 99 # this changes the first element for data, view, AND ro_view
view[0] = 98 # this changes the first element for data, view, AND ro_view
ro_view[0] = 97 # this will throw a TypeError

This is because data, view, and ro_view are all different objects pointing to the same data. However, the ro_view variable (acting like an interface) doesn't allow changes, but the same underlying data can be changed through the other variables.

In unpickling, references to an object can be created using the DUP opcode or saving the object in the memo.

Also, since the bytearray object is replaced, the value on the stack will change datatype from <class 'bytearray'> to <class 'memoryview'>.

Example(s):

Before stack : [b'abcd']
Pickle bytes : b'\x98' (len 1)
After stack  : [b'abcd']
(no visible change to stack)

Source Code (link)

def load_readonly_buffer(self):
    buf = self.stack[-1]
    with memoryview(buf) as m:
        if not m.readonly:
            self.stack[-1] = m.toreadonly()

_pickle.c Code (link)

static int
load_readonly_buffer(UnpicklerObject *self)
{
    Py_ssize_t len = Py_SIZE(self->stack);
    if (len <= self->stack->fence) {
        return Pdata_stack_underflow(self->stack);
    }

    PyObject *obj = self->stack->data[len - 1];
    PyObject *view = PyMemoryView_FromObject(obj);
    if (view == NULL) {
        return -1;
    }
    if (!PyMemoryView_GET_BUFFER(view)->readonly) {
        /* Original object is writable */
        PyMemoryView_GET_BUFFER(view)->readonly = 1;
        self->stack->data[len - 1] = view;
        Py_DECREF(obj);
    }
    else {
        /* Original object is read-only, no need to replace it */
        Py_DECREF(view);
    }
    return 0;
}

NONE - Push None on the stack

• Opcode - 0x4e ('N')
• Protocol introduced - 2

Example(s):

Before stack : []
Pickle bytes : b'N' (len 1)
After stack  : [None]

Source Code (link)

def load_none(self):
    self.append(None)

_pickle.c Code (link)

static int
load_none(UnpicklerObject *self)
{
    PDATA_APPEND(self->stack, Py_None, -1);
    return 0;
}

NEWTRUE - Push True onto the stack

• Opcode - 0x88
• Protocol introduced - 2

Author's Note:

Note that the _pickle.c implementation uses the same function for the NEWTRUE and NEWFALSE opcodes, but the boolean parameter is passed in as Py_True and Py_False (respectively).

Example(s):

Before stack : []
Pickle bytes : b'\x88' (len 1)
After stack  : [True]

Source Code (link)

def load_true(self):
    self.append(True)

_pickle.c Code (link)

static int
load_bool(UnpicklerObject *self, PyObject *boolean)
{
    assert(boolean == Py_True || boolean == Py_False);
    PDATA_APPEND(self->stack, boolean, -1);
    return 0;
}

NEWFALSE - Push False onto the stack

• Opcode - 0x89
• Protocol introduced - 2

Author's Note:

Note that the _pickle.c implementation uses the same function for the NEWTRUE and NEWFALSE opcodes, but the boolean parameter is passed in as Py_True and Py_False (respectively).

Example(s):

Before stack : []
Pickle bytes : b'\x89' (len 1)
After stack  : [False]

Source Code (link)

def load_false(self):
    self.append(False)

_pickle.c Code (link)

static int
load_bool(UnpicklerObject *self, PyObject *boolean)
{
    assert(boolean == Py_True || boolean == Py_False);
    PDATA_APPEND(self->stack, boolean, -1);
    return 0;
}

UNICODE - Push a Python Unicode string object

• Opcode - 0x56 ('V')
• Protocol introduced - 0

Official Description:

The argument is a raw-unicode-escape encoding of a Unicode string, and so may contain embedded escape sequences. The argument extends until the next newline character.

Author's Note:

This opcode pretty much only still exists for backwards compatibility. Python2 handled unicode strings in a really complicated matter, but a lot of it was simplified in Python3. For more specific details, see this documentation.

Example(s):

Before stack : []
Pickle bytes : b'Vab\n' (len 4)
After stack  : ['ab']

Before stack : []
Pickle bytes : b'Va\\x20b\n' (len 8)
After stack  : ['a\\x20b']

Before stack : []
Pickle bytes : b'Va\\u0020b\n' (len 10)
After stack  : ['a b']

Before stack : []
Pickle bytes : b'Va\\\\u0020b\n' (len 11)
After stack  : ['a\\\\u0020']

Source Code (link)

def load_unicode(self):
    self.append(str(self.readline()[:-1], 'raw-unicode-escape'))

Pickletools Code (link)

def read_unicodestringnl(f):
    data = f.readline()
    if not data.endswith(b'\n'):
        raise ValueError("no newline found when trying to read "
                         "unicodestringnl")
    data = data[:-1]    # lose the newline
    return str(data, 'raw-unicode-escape')

_pickle.c Code (link)

static int
load_unicode(UnpicklerObject *self)
{
    PyObject *str;
    Py_ssize_t len;
    char *s = NULL;

    if ((len = _Unpickler_Readline(self, &s)) < 0)
        return -1;
    if (len < 1)
        return bad_readline();

    str = PyUnicode_DecodeRawUnicodeEscape(s, len - 1, NULL);
    if (str == NULL)
        return -1;

    PDATA_PUSH(self->stack, str, -1);
    return 0;
}

SHORT_BINUNICODE - Push a Python Unicode string object

• Opcode - 0x8c
• Protocol introduced - 4

Official Description:

There are two arguments: the first is a 1-byte little-endian signed int giving the number of bytes in the string. The second is that many bytes, and is the UTF-8 encoding of the Unicode string.

Author's Note:

This opcode assumes utf-8 encoding with the surrogatepass option. To see more about surrogatepass, see this link and this link.

Note that the _pickle.c implementation uses the same function for the SHORT_BINUNICODE, BINUNICODE, and BINUNICODE8 opcodes, but the nbytes parameter is passed in as 1, 4, and 8 (respectively).

Example(s):

Before stack : []
Pickle bytes : b'\x8c\x041234' (len 6)
After stack  : ['1234']

Before stack : []
Pickle bytes : b'\x8c\x07123\n456' (len 9)
After stack  : ['123\n456']

Before stack : []
Pickle bytes : b'\x8c\x02\xc2\x99' (len 4)
After stack  : ['\x99']

Source Code (link)

def load_short_binunicode(self):
    len = self.read(1)[0]
    self.append(str(self.read(len), 'utf-8', 'surrogatepass'))

Pickletools Code (link)

def read_unicodestring1(f):
    n = read_uint1(f)
    assert n >= 0
    data = f.read(n)
    if len(data) == n:
        return str(data, 'utf-8', 'surrogatepass')
    raise ValueError("expected %d bytes in a unicodestring1, but only %d "
                     "remain" % (n, len(data)))

_pickle.c Code (link)

static int
load_counted_binunicode(UnpicklerObject *self, int nbytes)
{
    PyObject *str;
    Py_ssize_t size;
    char *s;

    if (_Unpickler_Read(self, &s, nbytes) < 0)
        return -1;

    size = calc_binsize(s, nbytes);
    if (size < 0) {
        PyErr_Format(PyExc_OverflowError,
                     "BINUNICODE exceeds system's maximum size of %zd bytes",
                     PY_SSIZE_T_MAX);
        return -1;
    }

    if (_Unpickler_Read(self, &s, size) < 0)
        return -1;

    str = PyUnicode_DecodeUTF8(s, size, "surrogatepass");
    if (str == NULL)
        return -1;

    PDATA_PUSH(self->stack, str, -1);
    return 0;
}

BINUNICODE - Push a Python Unicode string object

• Opcode - 0x58 ('X')
• Protocol introduced - 1

Official Description:

There are two arguments: the first is a 4-byte little-endian unsigned int giving the number of bytes in the string. The second is that many bytes, and is the UTF-8 encoding of the Unicode string.

Author's Note:

This opcode assumes utf-8 encoding with the surrogatepass option. To see more about surrogatepass, see this link and this link.

Note that the _pickle.c implementation uses the same function for the SHORT_BINUNICODE, BINUNICODE, and BINUNICODE8 opcodes, but the nbytes parameter is passed in as 1, 4, and 8 (respectively).

Example(s):

Before stack : []
Pickle bytes : b'X\x04\x00\x00\x001234' (len 9)
After stack  : ['1234']

Before stack : []
Pickle bytes : b'X\x07\x00\x00\x00123\n456' (len 12)
After stack  : ['123\n456']

Before stack : []
Pickle bytes : b'X\x02\x00\x00\x00\xc2\x99' (len 7)
After stack  : ['\x99']

Source Code (link)

def load_binunicode(self):
    len, = unpack('<I', self.read(4))
    if len > maxsize:
        raise UnpicklingError("BINUNICODE exceeds system's maximum size "
                                "of %d bytes" % maxsize)
    self.append(str(self.read(len), 'utf-8', 'surrogatepass'))

Pickletools Code (link)

def read_unicodestring4(f):
    n = read_uint4(f)
    assert n >= 0
    if n > sys.maxsize:
        raise ValueError("unicodestring4 byte count > sys.maxsize: %d" % n)
    data = f.read(n)
    if len(data) == n:
        return str(data, 'utf-8', 'surrogatepass')
    raise ValueError("expected %d bytes in a unicodestring4, but only %d "
                     "remain" % (n, len(data)))

_pickle.c Code (link)

static int
load_counted_binunicode(UnpicklerObject *self, int nbytes)
{
    PyObject *str;
    Py_ssize_t size;
    char *s;

    if (_Unpickler_Read(self, &s, nbytes) < 0)
        return -1;

    size = calc_binsize(s, nbytes);
    if (size < 0) {
        PyErr_Format(PyExc_OverflowError,
                     "BINUNICODE exceeds system's maximum size of %zd bytes",
                     PY_SSIZE_T_MAX);
        return -1;
    }

    if (_Unpickler_Read(self, &s, size) < 0)
        return -1;

    str = PyUnicode_DecodeUTF8(s, size, "surrogatepass");
    if (str == NULL)
        return -1;

    PDATA_PUSH(self->stack, str, -1);
    return 0;
}

BINUNICODE8 - Push a Python Unicode string object

• Opcode - 0x8d
• Protocol introduced - 4

Official Description:

There are two arguments: the first is an 8-byte little-endian signed int giving the number of bytes in the string. The second is that many bytes, and is the UTF-8 encoding of the Unicode string.

Author's Note:

This opcode assumes utf-8 encoding with the surrogatepass option. To see more about surrogatepass, see this link and this link.

Note that the description says the 8-byte length field is signed, however all 3 implementations treat it as an unsigned integer. However, all signed 8-byte longs are > 0x7fffffffffffffff, which is greater than sys.maxsize, and all implementations check if the number is larger than sys.maxsize. Therefore, it is not explicitly treated as a signed integer but is functionally signed with the restriction >= 0.

Note that the _pickle.c implementation uses the same function for the SHORT_BINUNICODE, BINUNICODE, and BINUNICODE8 opcodes, but the nbytes parameter is passed in as 1, 4, and 8 (respectively).

Example(s):

Before stack : []
Pickle bytes : b'\x8d\x04\x00\x00\x00\x00\x00\x00\x001234' (len 13)
After stack  : ['1234']

Before stack : []
Pickle bytes : b'\x8d\x07\x00\x00\x00\x00\x00\x00\x00123\n456' (len 16)
After stack  : ['123\n456']

Before stack : []
Pickle bytes : b'\x8d\x02\x00\x00\x00\x00\x00\x00\x00\xc2\x99' (len 11)
After stack  : ['\x99']

Source Code (link)

def load_binunicode8(self):
    len, = unpack('<Q', self.read(8))
    if len > maxsize:
        raise UnpicklingError("BINUNICODE8 exceeds system's maximum size "
                                "of %d bytes" % maxsize)
    self.append(str(self.read(len), 'utf-8', 'surrogatepass'))

Pickletools Code (link)

def read_unicodestring8(f):
    n = read_uint8(f)
    assert n >= 0
    if n > sys.maxsize:
        raise ValueError("unicodestring8 byte count > sys.maxsize: %d" % n)
    data = f.read(n)
    if len(data) == n:
        return str(data, 'utf-8', 'surrogatepass')
    raise ValueError("expected %d bytes in a unicodestring8, but only %d "
                     "remain" % (n, len(data)))

_pickle.c Code (link)

static int
load_counted_binunicode(UnpicklerObject *self, int nbytes)
{
    PyObject *str;
    Py_ssize_t size;
    char *s;

    if (_Unpickler_Read(self, &s, nbytes) < 0)
        return -1;

    size = calc_binsize(s, nbytes);
    if (size < 0) {
        PyErr_Format(PyExc_OverflowError,
                     "BINUNICODE exceeds system's maximum size of %zd bytes",
                     PY_SSIZE_T_MAX);
        return -1;
    }

    if (_Unpickler_Read(self, &s, size) < 0)
        return -1;

    str = PyUnicode_DecodeUTF8(s, size, "surrogatepass");
    if (str == NULL)
        return -1;

    PDATA_PUSH(self->stack, str, -1);
    return 0;
}

FLOAT - Newline-terminated decimal float literal

• Opcode - 0x46 ('F')
• Protocol introduced - 0

Official Description:

The argument is repr(a_float), and in general requires 17 significant digits for roundtrip conversion to be an identity (this is so for IEEE-754 double precision values, which is what Python float maps to on most boxes).

In general, FLOAT cannot be used to transport infinities, NaNs, or minus zero across boxes (or even on a single box, if the platform C library can't read the strings it produces for such things -- Windows is like that), but may do less damage than BINFLOAT on boxes with greater precision or dynamic range than IEEE-754 double.

Example(s):

Before stack : []
Pickle bytes : b'F1\n' (len 3)
After stack  : [1.0]

Before stack : []
Pickle bytes : b'F-1337\n' (len 7)
After stack  : [-1337.0]

Before stack : []
Pickle bytes : b'F9999999999999999\n' (len 18)
After stack  : [1e+16]

Before stack : []
Pickle bytes : b'F1.9999999999999995\n' (len 20)
After stack  : [1.9999999999999996]

Before stack : []
Pickle bytes : b'F1.99999999999999999\n' (len 21)
After stack  : [2.0]

Source Code (link)

def load_float(self):
    self.append(float(self.readline()[:-1]))

Pickletools Code (link)

def read_floatnl(f):
    s = read_stringnl(f, decode=False, stripquotes=False)
    return float(s)

_pickle.c Code (link)

static int
load_float(UnpicklerObject *self)
{
    PyObject *value;
    char *endptr, *s;
    Py_ssize_t len;
    double d;

    if ((len = _Unpickler_Readline(self, &s)) < 0)
        return -1;
    if (len < 2)
        return bad_readline();

    errno = 0;
    d = PyOS_string_to_double(s, &endptr, PyExc_OverflowError);
    if (d == -1.0 && PyErr_Occurred())
        return -1;
    if ((endptr[0] != '\n') && (endptr[0] != '\0')) {
        PyErr_SetString(PyExc_ValueError, "could not convert string to float");
        return -1;
    }
    value = PyFloat_FromDouble(d);
    if (value == NULL)
        return -1;

    PDATA_PUSH(self->stack, value, -1);
    return 0;
}

BINFLOAT - An 8-byte binary representation of a float, big-endian

• Opcode - 0x47 ('G')
• Protocol introduced - 1

Official Description:

This generally requires less than half the space of FLOAT encoding. In general, BINFLOAT cannot be used to transport infinities, NaNs, or minus zero, raises an exception if the exponent exceeds the range of an IEEE-754 double, and retains no more than 53 bits of precision (if there are more than that, "add a half and chop" rounding is used to cut it back to 53 significant bits).

The format is unique to Python, and shared with the struct module (format string '>d') "in theory" (the struct and pickle implementations don't share the code -- they should). It's strongly related to the IEEE-754 double format, and, in normal cases, is in fact identical to the big-endian 754 double format. On other boxes the dynamic range is limited to that of a 754 double, and "add a half and chop" rounding is used to reduce the precision to 53 bits. However, even on a 754 box, infinities, NaNs, and minus zero may not be handled correctly (may not survive roundtrip pickling intact).

Example(s):

Before stack : []
Pickle bytes : b'G\x00\x00\x00\x00\x00\x00\x00\x00' (len 9)
After stack  : [0.0]

Before stack : []
Pickle bytes : b'G\x80\x00\x00\x00\x00\x00\x00\x00' (len 9)
After stack  : [-0.0]

Before stack : []
Pickle bytes : b'G\xf0\x00\x00\x00\x00\x00\x00\x00' (len 9)
After stack  : [-3.105036184601418e+231]

Before stack : []
Pickle bytes : b'G\x70\x00\x00\x00\x00\x00\x00\x00' (len 9)
After stack  : [3.105036184601418e+231]

Before stack : []
Pickle bytes : b'G\x00\x00\x00\x00\x00\x00\x00\x01' (len 9)
After stack  : [5e-324]

Source Code (link)

def load_binfloat(self):
    self.append(unpack('>d', self.read(8))[0])

Pickletools Code (link)

def read_float8(f):
    data = f.read(8)
    if len(data) == 8:
        return _unpack(">d", data)[0]
    raise ValueError("not enough data in stream to read float8")

_pickle.c Code (link)

static int
load_binfloat(UnpicklerObject *self)
{
    PyObject *value;
    double x;
    char *s;

    if (_Unpickler_Read(self, &s, 8) < 0)
        return -1;

    x = PyFloat_Unpack8(s, 0);
    if (x == -1.0 && PyErr_Occurred())
        return -1;

    if ((value = PyFloat_FromDouble(x)) == NULL)
        return -1;

    PDATA_PUSH(self->stack, value, -1);
    return 0;
}

EMPTY_LIST - Push an empty list

• Opcode - 0x5d (']')
• Protocol introduced - 1

Example(s):

Before stack : []
Pickle bytes : b']' (len 1)
After stack  : [[]]

Source Code (link)

def load_empty_list(self):
    self.append([])

_pickle.c Code (link)

static int
load_empty_list(UnpicklerObject *self)
{
    PyObject *list;

    if ((list = PyList_New(0)) == NULL)
        return -1;
    PDATA_PUSH(self->stack, list, -1);
    return 0;
}

APPEND - Append an object to a list

• Opcode - 0x61 ('a')
• Protocol introduced - 0

Official Description:

Stack before:  ... pylist anyobject
Stack after:   ... pylist+[anyobject]

Although pylist is really extended in-place.

Example(s):

Before stack : [[1,2,3], 4]
Pickle bytes : b'a' (len 1)
After stack  : [[1,2,3,4]]

Before stack : [bytearray(b'abc'), 0x64]
Pickle bytes : b'a' (len 1)
After stack  : [bytearray(b'abcd')]

Source Code (link)

def load_append(self):
    stack = self.stack
    value = stack.pop()
    list = stack[-1]
    list.append(value)

_pickle.c Code (link)

static int
load_append(UnpicklerObject *self)
{
    if (Py_SIZE(self->stack) - 1 <= self->stack->fence)
        return Pdata_stack_underflow(self->stack);
    return do_append(self, Py_SIZE(self->stack) - 1);
}

APPENDS - Extend a list by a slice of stack objects

• Opcode - 0x65 ('e')
• Protocol introduced - 1

Official Description:

Stack before:  ... pylist markobject stackslice
Stack after:   ... pylist+stackslice

Although pylist is really extended in-place.

Author's Note:

The APPEND opcode is used to add a single object to the end of a list; APPENDS is a more efficient approach of adding several items to the end of a list. If one were to add 4 and 5 to the end of the list [1, 2, 3] using APPEND, they would have to:

• Push 4 on the stack
• Append 4 to the list
• Push 5 on the stack
• Append 5 to the list

If one were to add 4 and 5 using APPENDS instead, they would only have to:

• Use MARK opcode
• Push 4 on the stack
• Push 5 on the stack
• Append everything from the MARK opcode onwards to the list

The default behavior of pickle and _pickle.c is to try to use the .extend() attribute FIRST and the .append() attribute if that fails. As example, to add [4,5] to [1,2,3], they would try [1,2,3].extend([4,5]) and if that fails (aka .extend() doesn't exist) then [1,2,3].append(x) for [4,5].

Note that it's valid for no objects to be placed on the stack after the MARK object, in which case the list doesn't change. Interestingly, this actually leads to a discrepancy in how pickle and _pickle.c treat this opcode. If (for example) an integer is used instead of a list in this case, obviously 1.append() is invalid. pickle checks that the object on the stack has a .extend or .append attribute (regardless of how many objects come after MARK), and if it doesn't an error is thrown. However, _pickle.c looks to see how many objects come after MARK and if it's 0, it continues processing the remaining pickle bytes. Therefore, pickle bytes like b'K\x01(e.' (push 1, MARK, APPENDS, STOP) will error out in pickle but not in _pickle.c.

Example(s):

Before stack     : [4, 5]
Before metastack : [[[1,2,3]]]

Pickle bytes     : b'e' (len 1)

After stack      : [[1,2,3,4,5]]
After metastack  : []

Before stack     : [0x64, 0x65]
Before metastack : [[bytearray(b'abc')]]

Pickle bytes     : b'e' (len 1)

After stack      : [bytearray(b'abcde')]
After metastack  : []

Source Code (link)

def load_appends(self):
    items = self.pop_mark()
    list_obj = self.stack[-1]
    try:
        extend = list_obj.extend
    except AttributeError:
        pass
    else:
        extend(items)
        return
    # Even if the PEP 307 requires extend() and append() methods,
    # fall back on append() if the object has no extend() method
    # for backward compatibility.
    append = list_obj.append
    for item in items:
        append(item)

_pickle.c Code (link)

static int
load_appends(UnpicklerObject *self)
{
    Py_ssize_t i = marker(self);
    if (i < 0)
        return -1;
    return do_append(self, i);
}

LIST - Build a list out of the topmost stack slice, after markobject

• Opcode - 0x6c ('l')
• Protocol introduced - 0

Official Description:

All the stack entries following the topmost markobject are placed into a single Python list, which single list object replaces all of the stack from the topmost markobject onward. For example,

Stack before: ... markobject 1 2 3 'abc'
Stack after:  ... [1, 2, 3, 'abc']

Example(s):

Before stack     : [4, 5]
Before metastack : [['abc']]

Pickle bytes     : b'l' (len 1)

After stack      : ['abc', [4,5]]
After metastack  : []

Before stack     : [b'hello', 1337]
Before metastack : [[<built-in function print>]]

Pickle bytes     : b'l' (len 1)

After stack      : [<built-in function print>, [b'hello',1337]]
After metastack  : []

Source Code (link)

def load_list(self):
    items = self.pop_mark()
    self.append(items)

_pickle.c Code (link)

static int
load_list(UnpicklerObject *self)
{
    PyObject *list;
    Py_ssize_t i;

    if ((i = marker(self)) < 0)
        return -1;

    list = Pdata_poplist(self->stack, i);
    if (list == NULL)
        return -1;
    PDATA_PUSH(self->stack, list, -1);
    return 0;
}

EMPTY_TUPLE - Push an empty tuple

• Opcode - 0x29 (')')
• Protocol introduced - 1

Author's Note:

Note that the _pickle.c implementation uses the same function for the EMPTY_TUPLE, TUPLE1, TUPLE2, and TUPLE3 opcodes, but the len parameter is passed in as 0, 1, 2, and 3 (respectively).

Example(s):

Before stack : []
Pickle bytes : b')' (len 1)
After stack  : [()]

Source Code (link)

def load_empty_tuple(self):
    self.append(())

_pickle.c Code (link)

static int
load_counted_tuple(UnpicklerObject *self, Py_ssize_t len)
{
    PyObject *tuple;

    if (Py_SIZE(self->stack) < len)
        return Pdata_stack_underflow(self->stack);

    tuple = Pdata_poptuple(self->stack, Py_SIZE(self->stack) - len);
    if (tuple == NULL)
        return -1;
    PDATA_PUSH(self->stack, tuple, -1);
    return 0;
}

TUPLE - Build a tuple out of the topmost stack slice, after markobject

• Opcode - 0x74 ('t')
• Protocol introduced - 0

Official Description:

All the stack entries following the topmost markobject are placed into a single Python tuple, which single tuple object replaces all of the stack from the topmost markobject onward. For example,

Stack before: ... markobject 1 2 3 'abc'
Stack after:  ... (1, 2, 3, 'abc')

Example(s):

Before stack     : [4, 5]
Before metastack : [['abc']]

Pickle bytes     : b't' (len 1)

After stack      : ['abc', (4,5)]
After metastack  : []

Before stack     : [b'hello', 1337]
Before metastack : [[<built-in function print>]]

Pickle bytes     : b't' (len 1)

After stack      : [<built-in function print>, (b'hello',1337)]
After metastack  : []

Source Code (link)

def load_tuple(self):
    items = self.pop_mark()
    self.append(tuple(items))

_pickle.c Code (link)

static int
load_tuple(UnpicklerObject *self)
{
    Py_ssize_t i;

    if ((i = marker(self)) < 0)
        return -1;

    return load_counted_tuple(self, Py_SIZE(self->stack) - i);
}

TUPLE1 - Build a one-tuple out of the topmost item on the stack

• Opcode - 0x85
• Protocol introduced - 2

Official Description:

This code pops one value off the stack and pushes a tuple of length 1 whose one item is that value back onto it. In other words:

stack[-1] = tuple(stack[-1:])

Author's Note:

Note that the _pickle.c implementation uses the same function for the EMPTY_TUPLE, TUPLE1, TUPLE2, and TUPLE3 opcodes, but the len parameter is passed in as 0, 1, 2, and 3 (respectively).

Example(s):

Before stack : [1, 2, 3, 4]
Pickle bytes : b'\x85' (len 1)
After stack  : [1, 2, 3, (4,)]

Source Code (link)

def load_tuple1(self):
    self.stack[-1] = (self.stack[-1],)

_pickle.c Code (link)

static int
load_counted_tuple(UnpicklerObject *self, Py_ssize_t len)
{
    PyObject *tuple;

    if (Py_SIZE(self->stack) < len)
        return Pdata_stack_underflow(self->stack);

    tuple = Pdata_poptuple(self->stack, Py_SIZE(self->stack) - len);
    if (tuple == NULL)
        return -1;
    PDATA_PUSH(self->stack, tuple, -1);
    return 0;
}

TUPLE2 - Build a two-tuple out of the topmost item on the stack

• Opcode - 0x86
• Protocol introduced - 2

Official Description:

This code pops two values off the stack and pushes a tuple of length 2 whose items are those values back onto it. In other words:

stack[-2:] = [tuple(stack[-2:])]

Author's Note:

Note that the _pickle.c implementation uses the same function for the EMPTY_TUPLE, TUPLE1, TUPLE2, and TUPLE3 opcodes, but the len parameter is passed in as 0, 1, 2, and 3 (respectively).

Example(s):

Before stack : [1, 2, 3, 4]
Pickle bytes : b'\x86' (len 1)
After stack  : [1, 2, (3, 4)]

Source Code (link)

def load_tuple2(self):
    self.stack[-2:] = [(self.stack[-2], self.stack[-1])]

_pickle.c Code (link)

static int
load_counted_tuple(UnpicklerObject *self, Py_ssize_t len)
{
    PyObject *tuple;

    if (Py_SIZE(self->stack) < len)
        return Pdata_stack_underflow(self->stack);

    tuple = Pdata_poptuple(self->stack, Py_SIZE(self->stack) - len);
    if (tuple == NULL)
        return -1;
    PDATA_PUSH(self->stack, tuple, -1);
    return 0;
}

TUPLE3 - Build a three-tuple out of the topmost item on the stack

• Opcode - 0x87
• Protocol introduced - 2

Official Description:

This code pops three values off the stack and pushes a tuple of length 3 whose items are those values back onto it. In other words:

stack[-3:] = [tuple(stack[-3:])]

Author's Note:

Note that the _pickle.c implementation uses the same function for the EMPTY_TUPLE, TUPLE1, TUPLE2, and TUPLE3 opcodes, but the len parameter is passed in as 0, 1, 2, and 3 (respectively).

Example(s):

Before stack : [1, 2, 3, 4]
Pickle bytes : b'\x87' (len 1)
After stack  : [1, (2, 3, 4)]

Source Code (link)

def load_tuple3(self):
    self.stack[-3:] = [(self.stack[-3], self.stack[-2], self.stack[-1])]

_pickle.c Code (link)

static int
load_counted_tuple(UnpicklerObject *self, Py_ssize_t len)
{
    PyObject *tuple;

    if (Py_SIZE(self->stack) < len)
        return Pdata_stack_underflow(self->stack);

    tuple = Pdata_poptuple(self->stack, Py_SIZE(self->stack) - len);
    if (tuple == NULL)
        return -1;
    PDATA_PUSH(self->stack, tuple, -1);
    return 0;
}

EMPTY_DICT - Push an empty dict

• Opcode - 0x7d ('}')
• Protocol introduced - 1

Example(s):

Before stack : []
Pickle bytes : b'}' (len 1)
After stack  : [{}]

Source Code (link)

def load_empty_dictionary(self):
    self.append({})

_pickle.c Code (link)

static int
load_empty_dict(UnpicklerObject *self)
{
    PyObject *dict;

    if ((dict = PyDict_New()) == NULL)
        return -1;
    PDATA_PUSH(self->stack, dict, -1);
    return 0;
}

DICT - Build a dict out of the topmost stack slice, after markobject

• Opcode - 0x64 ('d')
• Protocol introduced - 0

Official Description:

All the stack entries following the topmost markobject are placed into a single Python dict, which single dict object replaces all of the stack from the topmost markobject onward. The stack slice alternates key, value, key, value, .... For example,

Stack before: ... markobject 1 2 3 'abc'
Stack after:  ... {1: 2, 3: 'abc'}

Example(s):

Before stack     : [1, 2, 3, 4]
Before metastack : [['abc']]

Pickle bytes     : b'd' (len 1)

After stack      : ['abc', {1: 2, 3: 4}]
After metastack  : []

Before stack     : [b'hello', 1337, 'testing', 69420]
Before metastack : [[<built-in function print>]]

Pickle bytes     : b'd' (len 1)

After stack      : [<built-in function print>, {b'hello': 1337, 'testing': 69420}]
After metastack  : []

Source Code (link)

def load_dict(self):
    items = self.pop_mark()
    d = {items[i]: items[i+1]
            for i in range(0, len(items), 2)}
    self.append(d)

_pickle.c Code (link)

static int
load_dict(UnpicklerObject *self)
{
    PyObject *dict, *key, *value;
    Py_ssize_t i, j, k;

    if ((i = marker(self)) < 0)
        return -1;
    j = Py_SIZE(self->stack);

    if ((dict = PyDict_New()) == NULL)
        return -1;

    if ((j - i) % 2 != 0) {
        PickleState *st = _Pickle_GetGlobalState();
        PyErr_SetString(st->UnpicklingError, "odd number of items for DICT");
        Py_DECREF(dict);
        return -1;
    }

    for (k = i + 1; k < j; k += 2) {
        key = self->stack->data[k - 1];
        value = self->stack->data[k];
        if (PyDict_SetItem(dict, key, value) < 0) {
            Py_DECREF(dict);
            return -1;
        }
    }
    Pdata_clear(self->stack, i);
    PDATA_PUSH(self->stack, dict, -1);
    return 0;
}

SETITEM - Add a key+value pair to an existing dict

• Opcode - 0x73 ('s')
• Protocol introduced - 0

Official Description:

Stack before:  ... pydict key value
Stack after:   ... pydict

Where pydict has been modified via pydict[key] = value.

Author's Note:

Although the official description states a dictionary is updated using this opcode, technically any object that supports item assignment (ie has a __setitem__ attribute) can be updated here, such as a list or bytearray.

Example(s):

Before stack : [{'a':1, 'b':2}, 'c', 3]
Pickle bytes : b's' (len 1)
After stack  : [{'a':1, 'b':2, 'c':3}]

Before stack : [[13,37,420], 2, 69]
Pickle bytes : b's' (len 1)
After stack  : [[13,37,69]]

Before stack : [bytearray(b'abcf'), 3, 100]
Pickle bytes : b's' (len 1)
After stack  : [bytearray(b'abcd')]

Source Code (link)

def load_setitem(self):
    stack = self.stack
    value = stack.pop()
    key = stack.pop()
    dict = stack[-1]
    dict[key] = value

_pickle.c Code (link)

static int
load_setitem(UnpicklerObject *self)
{
    return do_setitems(self, Py_SIZE(self->stack) - 2);
}

SETITEMS - Add an arbitrary number of key+value pairs to an existing dict

• Opcode - 0x75 ('u')
• Protocol introduced - 1

Official Description:

The slice of the stack following the topmost markobject is taken as an alternating sequence of keys and values, added to the dict immediately under the topmost markobject. Everything at and after the topmost markobject is popped, leaving the mutated dict at the top of the stack.

Stack before:  ... pydict markobject key_1 value_1 ... key_n value_n
Stack after:   ... pydict

Where pydict has been modified via pydict[key_i] = value_i for i in 1, 2, ..., n, and in that order.

Author's Note:

Although the official description states a dictionary is updated using this opcode, technically any object that supports item assignment (ie has a __setitem__ attribute) can be updated here, such as a list or bytearray.

The SETITEM opcode is used to add a single entry to a dictionary; SETITEMS is a more efficient approach of adding several entries to a dictionary. If one were to add the entries 'b':2 and 'c':3 to an empty dictionary using SETITEM, they would have to:

• Push 'b' on the stack
• Push 2 on the stack
• SETITEM
• Push 'c' on the stack
• Push 3 on the stack
• SETITEM

If one were to add 'b':2 and 'c':3 using SETITEMS instead, they would only have to:

• Use MARK opcode
• Push 'b' on the stack
• Push 2 on the stack
• Push 'c' on the stack
• Push 3 on the stack
• Append everything from the MARK opcode onwards to the dictionary

Example(s):

Before stack     : ['c', 3, 'd', 4]
Before metastack : [[{'a':1, 'b':2}]]

Pickle bytes     : b'u' (len 1)

After stack      : [{'a':1, 'b':2, 'c':3, 'd':4}]
After metastack  : []

Before stack     : [2, 69, 1, 337]
Before metastack : [[[13,37,420]]]

Pickle bytes     : b'u' (len 1)

After stack      : [[13,337,69]]
After metastack  : []

Before stack     : [3, 100, 2, 98]
Before metastack : [[bytearray(b'abcf')]]

Pickle bytes     : b'u' (len 1)

After stack      : [bytearray(b'abbd')]
After metastack  : []

Source Code (link)

def load_setitems(self):
    items = self.pop_mark()
    dict = self.stack[-1]
    for i in range(0, len(items), 2):
        dict[items[i]] = items[i + 1]

_pickle.c Code (link)

static int
load_setitems(UnpicklerObject *self)
{
    Py_ssize_t i = marker(self);
    if (i < 0)
        return -1;
    return do_setitems(self, i);
}

EMPTY_SET - Push an empty set

• Opcode - 0x8f
• Protocol introduced - 4

Example(s):

Before stack : []
Pickle bytes : b'\x8f' (len 1)
After stack  : [set()]

Source Code (link)

def load_empty_set(self):
    self.append(set())

_pickle.c Code (link)

static int
load_empty_set(UnpicklerObject *self)
{
    PyObject *set;

    if ((set = PySet_New(NULL)) == NULL)
        return -1;
    PDATA_PUSH(self->stack, set, -1);
    return 0;
}

ADDITEMS - Add an arbitrary number of items to an existing set

• Opcode - 0x90
• Protocol introduced - 4

Official Description:

The slice of the stack following the topmost markobject is taken as a sequence of items, added to the set immediately under the topmost markobject. Everything at and after the topmost markobject is popped, leaving the mutated set at the top of the stack.

Stack before:  ... pyset markobject item_1 ... item_n
Stack after:   ... pyset

Where pyset has been modified via pyset.add(item_i) = item_i for i in 1, 2, ..., n, and in that order.

Author's Note:

Although the official description states a set is updated using this opcode, technically any object with a .add() attribute can be updated here.

Note that it's valid for no objects to be placed on the stack after the MARK object, in which case the set doesn't change. Interestingly, this actually leads to a discrepancy in how pickle and _pickle.c treat this opcode. If (for example) an integer is used instead of a set in this case, obviously 1.add() is invalid. pickle checks that the object on the stack has a .add attribute (regardless of how many objects come after MARK), and if it doesn't an error is thrown. However, _pickle.c looks to see how many objects come after MARK and if it's 0, it continues processing the remaining pickle bytes. Therefore, pickle bytes like b'K\x01(\x90.' (push 1, MARK, ADDITEMS, STOP) will error out in pickle but not in _pickle.c.

Example(s):

Before stack     : [4, 5]
Before metastack : [[{1, 2, 3}]]

Pickle bytes     : b'\x90' (len 1)

After stack      : [{1, 2, 3, 4, 5}]
After metastack  : []

Source Code (link)

def load_additems(self):
    items = self.pop_mark()
    set_obj = self.stack[-1]
    if isinstance(set_obj, set):
        set_obj.update(items)
    else:
        add = set_obj.add
        for item in items:
            add(item)

_pickle.c Code (link)

static int
load_additems(UnpicklerObject *self)
{
    PyObject *set;
    Py_ssize_t mark, len, i;

    mark =  marker(self);
    if (mark < 0)
        return -1;
    len = Py_SIZE(self->stack);
    if (mark > len || mark <= self->stack->fence)
        return Pdata_stack_underflow(self->stack);
    if (len == mark)  /* nothing to do */
        return 0;

    set = self->stack->data[mark - 1];

    if (PySet_Check(set)) {
        PyObject *items;
        int status;

        items = Pdata_poptuple(self->stack, mark);
        if (items == NULL)
            return -1;

        status = _PySet_Update(set, items);
        Py_DECREF(items);
        return status;
    }
    else {
        PyObject *add_func;

        add_func = PyObject_GetAttr(set, &_Py_ID(add));
        if (add_func == NULL)
            return -1;
        for (i = mark; i < len; i++) {
            PyObject *result;
            PyObject *item;

            item = self->stack->data[i];
            result = _Pickle_FastCall(add_func, item);
            if (result == NULL) {
                Pdata_clear(self->stack, i + 1);
                Py_SET_SIZE(self->stack, mark);
                return -1;
            }
            Py_DECREF(result);
        }
        Py_SET_SIZE(self->stack, mark);
    }

    return 0;
}

FROZENSET - Build a frozenset out of the topmost slice, after markobject

• Opcode - 0x91
• Protocol introduced - 4

Official Description:

All the stack entries following the topmost markobject are placed into a single Python frozenset, which single frozenset object replaces all of the stack from the topmost markobject onward. For example,

Stack before: ... markobject 1 2 3
Stack after:  ... frozenset({1, 2, 3})

Example(s):

Before stack     : [1, 2, 3]
Before metastack : [['asdf', None]]

Pickle bytes     : b'\x91' (len 1)

After stack      : ['asdf', None, frozenset({1, 2, 3})]
After metastack  : []

Source Code (link)

def load_frozenset(self):
    items = self.pop_mark()
    self.append(frozenset(items))

_pickle.c Code (link)

static int
load_frozenset(UnpicklerObject *self)
{
    PyObject *items;
    PyObject *frozenset;
    Py_ssize_t i;

    if ((i = marker(self)) < 0)
        return -1;

    items = Pdata_poptuple(self->stack, i);
    if (items == NULL)
        return -1;

    frozenset = PyFrozenSet_New(items);
    Py_DECREF(items);
    if (frozenset == NULL)
        return -1;

    PDATA_PUSH(self->stack, frozenset, -1);
    return 0;
}

POP - Discard the top stack item, shrinking the stack by one item

• Opcode - 0x30 ('0')
• Protocol introduced - 0

Author's Note:

The POP operation will not only discard the top stack item, but if the stack has no items in it AND the metastack is not empty, then it will act like POP_MARK.

Example(s):

Before stack     : [1, 2]
Pickle bytes     : b'0' (len 1)
After stack      : [1]

Before stack     : []
Before metastack : [['asdf', None]]

Pickle bytes     : b'0' (len 1)

After stack      : ['asdf', None]
After metastack  : []

Source Code (link)

def load_pop(self):
    if self.stack:
        del self.stack[-1]
    else:
        self.pop_mark()

_pickle.c Code (link)

static int
load_pop(UnpicklerObject *self)
{
    Py_ssize_t len = Py_SIZE(self->stack);

    /* Note that we split the (pickle.py) stack into two stacks,
     * an object stack and a mark stack. We have to be clever and
     * pop the right one. We do this by looking at the top of the
     * mark stack first, and only signalling a stack underflow if
     * the object stack is empty and the mark stack doesn't match
     * our expectations.
     */
    if (self->num_marks > 0 && self->marks[self->num_marks - 1] == len) {
        self->num_marks--;
        self->stack->mark_set = self->num_marks != 0;
        self->stack->fence = self->num_marks ?
                self->marks[self->num_marks - 1] : 0;
    } else if (len <= self->stack->fence)
        return Pdata_stack_underflow(self->stack);
    else {
        len--;
        Py_DECREF(self->stack->data[len]);
        Py_SET_SIZE(self->stack, len);
    }
    return 0;
}

DUP - Push the top stack item onto the stack again, duplicating it

• Opcode - 0x32 ('2')
• Protocol introduced - 0

Example(s):

Before stack     : [1]
Pickle bytes     : b'2' (len 1)
After stack      : [1, 1]

Source Code (link)

def load_dup(self):
    self.append(self.stack[-1])

_pickle.c Code (link)

static int
load_dup(UnpicklerObject *self)
{
    PyObject *last;
    Py_ssize_t len = Py_SIZE(self->stack);

    if (len <= self->stack->fence)
        return Pdata_stack_underflow(self->stack);
    last = self->stack->data[len - 1];
    PDATA_APPEND(self->stack, last, -1);
    return 0;
}

MARK - Push markobject onto the stack

• Opcode - 0x28 ('(')
• Protocol introduced - 0

Official Description:

markobject is a unique object, used by other opcodes to identify a region of the stack containing a variable number of objects for them to work on. See markobject.doc for more detail.

Author's Note:

I am not sure where to find this markobject.doc that is being referenced.

When the MARK opcode is encounted in pickle.py, all current objects in the stack are pushed (as a list) onto the end of the metastack, and the stack is set to an empty dictionary.

Example(s):

Before stack     : ['asdf', None]
Before metastack : []

Pickle bytes     : b'(' (len 1)

After stack      : []
After metastack  : [['asdf', None]]

Source Code (link)

def load_mark(self):
    self.metastack.append(self.stack)
    self.stack = []
    self.append = self.stack.append

_pickle.c Code (link)

static int
load_mark(UnpicklerObject *self)
{

    /* Note that we split the (pickle.py) stack into two stacks, an
     * object stack and a mark stack. Here we push a mark onto the
     * mark stack.
     */

    if (self->num_marks >= self->marks_size) {
        size_t alloc = ((size_t)self->num_marks << 1) + 20;
        Py_ssize_t *marks_new = self->marks;
        PyMem_RESIZE(marks_new, Py_ssize_t, alloc);
        if (marks_new == NULL) {
            PyErr_NoMemory();
            return -1;
        }
        self->marks = marks_new;
        self->marks_size = (Py_ssize_t)alloc;
    }

    self->stack->mark_set = 1;
    self->marks[self->num_marks++] = self->stack->fence = Py_SIZE(self->stack);

    return 0;
}

POP_MARK - Pop all the stack objects at and above the topmost markobject

• Opcode - 0x31 ('1')
• Protocol introduced - 0

Official Description:

When an opcode using a variable number of stack objects is done, POP_MARK is used to remove those objects, and to remove the markobject that delimited their starting position on the stack.

Author's Note:

In pickle.py, POP_MARK clears the stack and pops the last element of the metastack into the current stack.

Example(s):

Before stack     : [1, 2, 3, 4]
Before metastack : [['asdf', None]]

Pickle bytes     : b'1' (len 1)

After stack      : ['asdf', None]
After metastack  : []

Source Code (link)

def load_pop_mark(self):
    self.pop_mark()

_pickle.c Code (link)

static int
load_pop_mark(UnpicklerObject *self)
{
    Py_ssize_t i;

    if ((i = marker(self)) < 0)
        return -1;

    Pdata_clear(self->stack, i);

    return 0;
}

GET - Read an object from the memo and push it on the stack

• Opcode - 0x67 ('g')
• Protocol introduced - 0

Official Description:

The index of the memo object to push is given by the newline-terminated decimal string following. BINGET and LONG_BINGET are space-optimized versions.

Author's Note:

Any amount of non-newline whitespace before or after the digits can be inserted as int() automatically strips whitespace. Underscores (that aren't leading or trailing) can also be placed in the string and won't affect the value. A negative (-) or positive (+) sign can precede the numbers, however all memo-setting opcodes only allow positive numbers, therefore the memo shouldn't have any negative keys.

Although the memo is a dictionary, all implementations ensure the key provided is a decimal literal string, which is converted to a signed integer. Unlike the INT opcode, all implementations correctly treat the number as explicitly having a base of 10.

While it doesn't lead to any unexpected behavior, it's possible to set the bytes to g01\n, which pickletools will treat as True instead of 1 (borrowing code from INT). Since True and 1 will always pull the same index of the memo, this should be easy to understand. This is also possible with g00\n and False.

Example(s):

Before stack : []
Before memo  : {1: 1337}

Pickle bytes : b'g1\n' (len 3)

After stack  : [1337]
After memo   : {1: 1337}

Before stack : []
Before memo  : {0: 'asdf'}

Pickle bytes : b'g00\n' (len 4)

After stack  : ['asdf']
After memo   : {0: 'asdf'}

Source Code (link)

def load_get(self):
    i = int(self.readline()[:-1])
    try:
        self.append(self.memo[i])
    except KeyError:
        msg = f'Memo value not found at index {i}'
        raise UnpicklingError(msg) from None

Pickletools Code (link)

def read_decimalnl_short(f):
    s = read_stringnl(f, decode=False, stripquotes=False)

    # There's a hack for True and False here.
    if s == b"00":
        return False
    elif s == b"01":
        return True

    return int(s)

_pickle.c Code (link)

static int
load_get(UnpicklerObject *self)
{
    PyObject *key, *value;
    Py_ssize_t idx;
    Py_ssize_t len;
    char *s;

    if ((len = _Unpickler_Readline(self, &s)) < 0)
        return -1;
    if (len < 2)
        return bad_readline();

    key = PyLong_FromString(s, NULL, 10);
    if (key == NULL)
        return -1;
    idx = PyLong_AsSsize_t(key);
    if (idx == -1 && PyErr_Occurred()) {
        Py_DECREF(key);
        return -1;
    }

    value = _Unpickler_MemoGet(self, idx);
    if (value == NULL) {
        if (!PyErr_Occurred()) {
           PickleState *st = _Pickle_GetGlobalState();
           PyErr_Format(st->UnpicklingError, "Memo value not found at index %ld", idx);
        }
        Py_DECREF(key);
        return -1;
    }
    Py_DECREF(key);

    PDATA_APPEND(self->stack, value, -1);
    return 0;
}

BINGET - Read an object from the memo and push it on the stack

• Opcode - 0x68 ('h')
• Protocol introduced - 1

Official Description:

The index of the memo object to push is given by the 1-byte unsigned integer following.

Example(s):

Before stack : []
Before memo  : {1: 1337}

Pickle bytes : b'h\x01' (len 2)

After stack  : [1337]
After memo   : {1: 1337}

Before stack : []
Before memo  : {255: 'asdf'}

Pickle bytes : b'h\xff' (len 2)

After stack  : ['asdf']
After memo   : {255: 'asdf'}

Source Code (link)

def load_binget(self):
    i = self.read(1)[0]
    try:
        self.append(self.memo[i])
    except KeyError as exc:
        msg = f'Memo value not found at index {i}'
        raise UnpicklingError(msg) from None

Pickletools Code (link)

def read_uint1(f):
    data = f.read(1)
    if data:
        return data[0]
    raise ValueError("not enough data in stream to read uint1")

_pickle.c Code (link)

static int
load_binget(UnpicklerObject *self)
{
    PyObject *value;
    Py_ssize_t idx;
    char *s;

    if (_Unpickler_Read(self, &s, 1) < 0)
        return -1;

    idx = Py_CHARMASK(s[0]);

    value = _Unpickler_MemoGet(self, idx);
    if (value == NULL) {
        PyObject *key = PyLong_FromSsize_t(idx);
        if (key != NULL) {
            PickleState *st = _Pickle_GetGlobalState();
            PyErr_Format(st->UnpicklingError, "Memo value not found at index %ld", idx);
            Py_DECREF(key);
        }
        return -1;
    }

    PDATA_APPEND(self->stack, value, -1);
    return 0;
}

LONG_BINGET - Read an object from the memo and push it on the stack

• Opcode - 0x6a ('j')
• Protocol introduced - 1

Official Description:

The index of the memo object to push is given by the 4-byte unsigned little-endian integer following.

Author's Note:

While LONG_BINGET can theoretically pull from memo indexes up to 0xffffffff, the C implementation of pickle will throw a memory error when trying to store a value at a memo index around 0x10000000 (using any type of PUT opcode). The Python implementation of pickle will store (and retrieve) values up to 0xffffffff just fine.

Example(s):

Before stack : []
Before memo  : {1: 1337}

Pickle bytes : b'j\x01\x00\x00\x00' (len 5)

After stack  : [1337]
After memo   : {1: 1337}

Before stack : []
Before memo  : {4026531840: 'asdf'}

Pickle bytes : b'j\x00\x00\x00\xf0' (len 5)

After stack  : ['asdf']
After memo   : {4026531840: 'asdf'}

Source Code (link)

def load_long_binget(self):
    i, = unpack('<I', self.read(4))
    try:
        self.append(self.memo[i])
    except KeyError as exc:
        msg = f'Memo value not found at index {i}'
        raise UnpicklingError(msg) from None

Pickletools Code (link)

def read_uint4(f):
    data = f.read(4)
    if len(data) == 4:
        return _unpack("<I", data)[0]
    raise ValueError("not enough data in stream to read uint4")

_pickle.c Code (link)

static int
load_long_binget(UnpicklerObject *self)
{
    PyObject *value;
    Py_ssize_t idx;
    char *s;

    if (_Unpickler_Read(self, &s, 4) < 0)
        return -1;

    idx = calc_binsize(s, 4);

    value = _Unpickler_MemoGet(self, idx);
    if (value == NULL) {
        PyObject *key = PyLong_FromSsize_t(idx);
        if (key != NULL) {
            PickleState *st = _Pickle_GetGlobalState();
            PyErr_Format(st->UnpicklingError, "Memo value not found at index %ld", idx);
            Py_DECREF(key);
        }
        return -1;
    }

    PDATA_APPEND(self->stack, value, -1);
    return 0;
}

PUT - Store the stack top into the memo, the stack is not popped

• Opcode - 0x70 ('p')
• Protocol introduced - 0

Official Description:

The index of the memo location to write into is given by the newline-terminated decimal string following. BINPUT and LONG_BINPUT are space-optimized versions.

Author's Note:

Any amount of non-newline whitespace before or after the digits can be inserted as int() automatically strips whitespace. Underscores (that aren't leading or trailing) can also be placed in the string and won't affect the value. A negative (-) or positive (+) sign can precede the numbers, but all negative numbers are rejected.

Although the memo is a dictionary, all implementations ensure the key provided is a decimal literal string, which is converted to a signed integer. Unlike the INT opcode, all implementations correctly treat the number as explicitly having a base of 10.

While it doesn't lead to any unexpected behavior, it's possible to set the bytes to g01\n, which pickletools will treat as True instead of 1 (borrowing code from INT). Since True and 1 will always pull the same index of the memo, this should be easy to understand. This is also possible with g00\n and False.

While _pickle.c and pickle.py only accept positive integers or 0 as valid memo indices, pickletools will not throw any errors for negative PUT arguments.

In _pickle.c, if the index in the memo is larger than the current size of the memo, it resizes the memo to be twice as large as the index provided. For example, if the memo has space for 20 elements and you try to set something at index 22, the memo will be resized to have space for 44 elements. Therefore, if you pass in a large number such as 0x20000000 (536870912), it will be so large a memory error is thrown since there isn't enough space for a memo that large.

Also in _pickle.c, the provided number is first turned into a PyLong, then casted to Py_ssize_t, which is a signed version of the OS's size_t datatype. Therefore, if you provide a number greater than 0x7fffffffffffffff, you'll get an error that it can't be converted.

If a null byte is placed anywhere in the argument for PUT, the C implementation ignores everything including and after the null byte, while pickletools and pickle.py throw an error.

Pickletools checks that the memo hasn't already been set at that index and errors if it is, while neither C or Python implementations do.

Example(s):

Before stack : [1337]
Before memo  : {}

Pickle bytes : b'p1\n' (len 3)

After stack  : [1337]
After memo   : {1: 1337}

Before stack : ['asdf']
Before memo  : {}

Pickle bytes : b'p00\n' (len 4)

After stack  : ['asdf']
After memo   : {0: 'asdf'}

Source Code (link)

def load_put(self):
    i = int(self.readline()[:-1])
    if i < 0:
        raise ValueError("negative PUT argument")
    self.memo[i] = self.stack[-1]

Pickletools Code (link)

def read_decimalnl_short(f):
    s = read_stringnl(f, decode=False, stripquotes=False)

    # There's a hack for True and False here.
    if s == b"00":
        return False
    elif s == b"01":
        return True

    return int(s)

_pickle.c Code (link)

static int
load_put(UnpicklerObject *self)
{
    PyObject *key, *value;
    Py_ssize_t idx;
    Py_ssize_t len;
    char *s = NULL;

    if ((len = _Unpickler_Readline(self, &s)) < 0)
        return -1;
    if (len < 2)
        return bad_readline();
    if (Py_SIZE(self->stack) <= self->stack->fence)
        return Pdata_stack_underflow(self->stack);
    value = self->stack->data[Py_SIZE(self->stack) - 1];

    key = PyLong_FromString(s, NULL, 10);
    if (key == NULL)
        return -1;
    idx = PyLong_AsSsize_t(key);
    Py_DECREF(key);
    if (idx < 0) {
        if (!PyErr_Occurred())
            PyErr_SetString(PyExc_ValueError,
                            "negative PUT argument");
        return -1;
    }

    return _Unpickler_MemoPut(self, idx, value);
}

BINPUT - Store the stack top into the memo, the stack is not popped

• Opcode - 0x71 ('q')
• Protocol introduced - 1

Official Description:

The index of the memo location to write into is given by the 1-byte unsigned integer following.

Author's Note:

Just like PUT, pickletools checks that the memo hasn't already been set at that index, while neither C or Python implementations do.

Example(s):

Before stack : [1337]
Before memo  : {}

Pickle bytes : b'q\x01' (len 2)

After stack  : [1337]
After memo   : {1: 1337}

Before stack : ['asdf']
Before memo  : {}

Pickle bytes : b'q\xff' (len 2)

After stack  : ['asdf']
After memo   : {255: 'asdf'}

Source Code (link)

def load_binput(self):
    i = self.read(1)[0]
    if i < 0:
        raise ValueError("negative BINPUT argument")
    self.memo[i] = self.stack[-1]

Pickletools Code (link)

def read_uint1(f):
    data = f.read(1)
    if data:
        return data[0]
    raise ValueError("not enough data in stream to read uint1")

_pickle.c Code (link)

static int
load_binput(UnpicklerObject *self)
{
    PyObject *value;
    Py_ssize_t idx;
    char *s;

    if (_Unpickler_Read(self, &s, 1) < 0)
        return -1;

    if (Py_SIZE(self->stack) <= self->stack->fence)
        return Pdata_stack_underflow(self->stack);
    value = self->stack->data[Py_SIZE(self->stack) - 1];

    idx = Py_CHARMASK(s[0]);

    return _Unpickler_MemoPut(self, idx, value);
}

LONG_BINPUT - Store the stack top into the memo, the stack is not popped

• Opcode - 0x72 ('r')
• Protocol introduced - 1

Official Description:

The index of the memo location to write into is given by the 4-byte unsigned little-endian integer following.

Author's Note:

While LONG_BINGET can theoretically pull from memo indexes up to 0xffffffff, the C implementation of pickle will throw a memory error when trying to store a value at a memo index around 0x10000000. The Python implementation of pickle will store values up to 0xffffffff just fine.

Just like PUT, pickletools checks that the memo hasn't already been set at that index, while neither C or Python implementations do.

Example(s):

Before stack : [1337]
Before memo  : {}

Pickle bytes : b'r\x01\x00\x00\x00' (len 5)

After stack  : [1337]
After memo   : {1: 1337}

Before stack : ['asdf']
Before memo  : {}

Pickle bytes : b'r\x00\x00\x00\xf0' (len 5)

After stack  : ['asdf']
After memo   : {4026531840: 'asdf'}

Source Code (link)

def load_long_binput(self):
    i, = unpack('<I', self.read(4))
    if i > maxsize:
        raise ValueError("negative LONG_BINPUT argument")
    self.memo[i] = self.stack[-1]

Pickletools Code (link)

def read_uint4(f):
    data = f.read(4)
    if len(data) == 4:
        return _unpack("<I", data)[0]
    raise ValueError("not enough data in stream to read uint4")

_pickle.c Code (link)

static int
load_long_binput(UnpicklerObject *self)
{
    PyObject *value;
    Py_ssize_t idx;
    char *s;

    if (_Unpickler_Read(self, &s, 4) < 0)
        return -1;

    if (Py_SIZE(self->stack) <= self->stack->fence)
        return Pdata_stack_underflow(self->stack);
    value = self->stack->data[Py_SIZE(self->stack) - 1];

    idx = calc_binsize(s, 4);
    if (idx < 0) {
        PyErr_SetString(PyExc_ValueError,
                        "negative LONG_BINPUT argument");
        return -1;
    }

    return _Unpickler_MemoPut(self, idx, value);
}

MEMOIZE - Store the stack top into the memo, the stack is not popped

• Opcode - 0x94
• Protocol introduced - 4

Official Description:

The index of the memo location to write is the number of elements currently present in the memo.

Author's Note:

Since all other PUT-like variables specify the index, it's easy to have the memo contain indexes that don't start at 0, which is the assumption MEMOIZE makes. This means that if the memo is set to {0: 'a', 1: 'b', 3:'c'} and MEMOIZE is run, then memo[3] will be replaced.

Just like PUT, pickletools checks that the memo hasn't already been set at that index, while neither C or Python implementations do.

Example(s):

Before stack : [1337]
Before memo  : {}

Pickle bytes : b'\x94' (len 1)

After stack  : [1337]
After memo   : {0: 1337}

Before stack : ['asdf']
Before memo  : {0: 'test'}

Pickle bytes : b'\x94' (len 1)

After stack  : ['asdf']
After memo   : {0: 'test', 1: 'asdf'}

Source Code (link)

def load_memoize(self):
    memo = self.memo
    memo[len(memo)] = self.stack[-1]

_pickle.c Code (link)

static int
load_memoize(UnpicklerObject *self)
{
    PyObject *value;

    if (Py_SIZE(self->stack) <= self->stack->fence)
        return Pdata_stack_underflow(self->stack);
    value = self->stack->data[Py_SIZE(self->stack) - 1];

    return _Unpickler_MemoPut(self, self->memo_len, value);
}

EXT1 - Extension code

• Opcode - 0x82
• Protocol introduced - 2

Official Description:

This code and the similar EXT2 and EXT4 allow using a registry of popular objects that are pickled by name, typically classes. It is envisioned that through a global negotiation and registration process, third parties can set up a mapping between ints and object names.

In order to guarantee pickle interchangeability, the extension code registry ought to be global, although a range of codes may be reserved for private use.

EXT1 has a 1-byte integer argument. This is used to index into the extension registry, and the object at that index is pushed on the stack.

Author's Note:

Code like copyreg.add_extension("builtins", "str", 1) must be ran before the pickle deserialization process occurs to register an object, and find_class() is ran during the unpickling process to actually recover the object definition. This means that RestrictedUnpicklers with a custom find_class() will still be able to filter what callables are added to the stack. See the main README.md for more information on how the extension registry works.

Note that the _pickle.c implementation uses the same function for the EXT1, EXT2, and EXT4 opcodes, but the size parameter is passed in as 1, 2, and 4 (respectively).

Example(s):

Before stack   : []
Before ext_reg : {('builtins', 'str'): 1}

Pickle bytes : b'\x82\x01' (len 2)

After stack    : [<class 'str'>]
After ext_reg  : {('builtins', 'str'): 1}

Source Code (link)

def load_ext1(self):
    code = self.read(1)[0]
    self.get_extension(code)

Pickletools Code (link)

def read_uint1(f):
    data = f.read(1)
    if data:
        return data[0]
    raise ValueError("not enough data in stream to read uint1")

_pickle.c Code (link)

static int
load_extension(UnpicklerObject *self, int nbytes)
{
    char *codebytes;            /* the nbytes bytes after the opcode */
    long code;                  /* calc_binint returns long */
    PyObject *py_code;          /* code as a Python int */
    PyObject *obj;              /* the object to push */
    PyObject *pair;             /* (module_name, class_name) */
    PyObject *module_name, *class_name;
    PickleState *st = _Pickle_GetGlobalState();

    assert(nbytes == 1 || nbytes == 2 || nbytes == 4);
    if (_Unpickler_Read(self, &codebytes, nbytes) < 0)
        return -1;
    code = calc_binint(codebytes, nbytes);
    if (code <= 0) {            /* note that 0 is forbidden */
        /* Corrupt or hostile pickle. */
        PyErr_SetString(st->UnpicklingError, "EXT specifies code <= 0");
        return -1;
    }

    /* Look for the code in the cache. */
    py_code = PyLong_FromLong(code);
    if (py_code == NULL)
        return -1;
    obj = PyDict_GetItemWithError(st->extension_cache, py_code);
    if (obj != NULL) {
        /* Bingo. */
        Py_DECREF(py_code);
        PDATA_APPEND(self->stack, obj, -1);
        return 0;
    }
    if (PyErr_Occurred()) {
        Py_DECREF(py_code);
        return -1;
    }

    /* Look up the (module_name, class_name) pair. */
    pair = PyDict_GetItemWithError(st->inverted_registry, py_code);
    if (pair == NULL) {
        Py_DECREF(py_code);
        if (!PyErr_Occurred()) {
            PyErr_Format(PyExc_ValueError, "unregistered extension "
                         "code %ld", code);
        }
        return -1;
    }
    /* Since the extension registry is manipulable via Python code,
     * confirm that pair is really a 2-tuple of strings.
     */
    if (!PyTuple_Check(pair) || PyTuple_Size(pair) != 2) {
        goto error;
    }

    module_name = PyTuple_GET_ITEM(pair, 0);
    if (!PyUnicode_Check(module_name)) {
        goto error;
    }

    class_name = PyTuple_GET_ITEM(pair, 1);
    if (!PyUnicode_Check(class_name)) {
        goto error;
    }

    /* Load the object. */
    obj = find_class(self, module_name, class_name);
    if (obj == NULL) {
        Py_DECREF(py_code);
        return -1;
    }
    /* Cache code -> obj. */
    code = PyDict_SetItem(st->extension_cache, py_code, obj);
    Py_DECREF(py_code);
    if (code < 0) {
        Py_DECREF(obj);
        return -1;
    }
    PDATA_PUSH(self->stack, obj, -1);
    return 0;

error:
    Py_DECREF(py_code);
    PyErr_Format(PyExc_ValueError, "_inverted_registry[%ld] "
                 "isn't a 2-tuple of strings", code);
    return -1;
}

EXT2 - Extension code

• Opcode - 0x83
• Protocol introduced - 2

Official Description:

See EXT1. EXT2 has a two-byte integer argument.

Author's Note:

Code like copyreg.add_extension("builtins", "str", 1) must be ran before the pickle deserialization process occurs to register an object, and find_class() is ran during the unpickling process to actually recover the object definition. This means that RestrictedUnpicklers with a custom find_class() will still be able to filter what callables are added to the stack. See the main README.md for more information on how the extension registry works.

Note that the _pickle.c implementation uses the same function for the EXT1, EXT2, and EXT4 opcodes, but the size parameter is passed in as 1, 2, and 4 (respectively).

Example(s):

Before stack   : []
Before ext_reg : {('builtins', 'str'): 1}

Pickle bytes : b'\x83\x01\x00' (len 3)

After stack    : [<class 'str'>]
After ext_reg  : {('builtins', 'str'): 1}

Source Code (link)

def load_ext2(self):
    code, = unpack('<H', self.read(2))
    self.get_extension(code)

Pickletools Code (link)

def read_uint2(f):
    data = f.read(2)
    if len(data) == 2:
        return _unpack("<H", data)[0]
    raise ValueError("not enough data in stream to read uint2")

_pickle.c Code (link)

static int
load_extension(UnpicklerObject *self, int nbytes)
{
    char *codebytes;            /* the nbytes bytes after the opcode */
    long code;                  /* calc_binint returns long */
    PyObject *py_code;          /* code as a Python int */
    PyObject *obj;              /* the object to push */
    PyObject *pair;             /* (module_name, class_name) */
    PyObject *module_name, *class_name;
    PickleState *st = _Pickle_GetGlobalState();

    assert(nbytes == 1 || nbytes == 2 || nbytes == 4);
    if (_Unpickler_Read(self, &codebytes, nbytes) < 0)
        return -1;
    code = calc_binint(codebytes, nbytes);
    if (code <= 0) {            /* note that 0 is forbidden */
        /* Corrupt or hostile pickle. */
        PyErr_SetString(st->UnpicklingError, "EXT specifies code <= 0");
        return -1;
    }

    /* Look for the code in the cache. */
    py_code = PyLong_FromLong(code);
    if (py_code == NULL)
        return -1;
    obj = PyDict_GetItemWithError(st->extension_cache, py_code);
    if (obj != NULL) {
        /* Bingo. */
        Py_DECREF(py_code);
        PDATA_APPEND(self->stack, obj, -1);
        return 0;
    }
    if (PyErr_Occurred()) {
        Py_DECREF(py_code);
        return -1;
    }

    /* Look up the (module_name, class_name) pair. */
    pair = PyDict_GetItemWithError(st->inverted_registry, py_code);
    if (pair == NULL) {
        Py_DECREF(py_code);
        if (!PyErr_Occurred()) {
            PyErr_Format(PyExc_ValueError, "unregistered extension "
                         "code %ld", code);
        }
        return -1;
    }
    /* Since the extension registry is manipulable via Python code,
     * confirm that pair is really a 2-tuple of strings.
     */
    if (!PyTuple_Check(pair) || PyTuple_Size(pair) != 2) {
        goto error;
    }

    module_name = PyTuple_GET_ITEM(pair, 0);
    if (!PyUnicode_Check(module_name)) {
        goto error;
    }

    class_name = PyTuple_GET_ITEM(pair, 1);
    if (!PyUnicode_Check(class_name)) {
        goto error;
    }

    /* Load the object. */
    obj = find_class(self, module_name, class_name);
    if (obj == NULL) {
        Py_DECREF(py_code);
        return -1;
    }
    /* Cache code -> obj. */
    code = PyDict_SetItem(st->extension_cache, py_code, obj);
    Py_DECREF(py_code);
    if (code < 0) {
        Py_DECREF(obj);
        return -1;
    }
    PDATA_PUSH(self->stack, obj, -1);
    return 0;

error:
    Py_DECREF(py_code);
    PyErr_Format(PyExc_ValueError, "_inverted_registry[%ld] "
                 "isn't a 2-tuple of strings", code);
    return -1;
}

EXT4 - Extension code

• Opcode - 0x84
• Protocol introduced - 2

Official Description:

See EXT1. EXT4 has a four-byte integer argument.

Author's Note:

Code like copyreg.add_extension("builtins", "str", 1) must be ran before the pickle deserialization process occurs to register an object, and find_class() is ran during the unpickling process to actually recover the object definition. This means that RestrictedUnpicklers with a custom find_class() will still be able to filter what callables are added to the stack. See the main README.md for more information on how the extension registry works.

Note that the _pickle.c implementation uses the same function for the EXT1, EXT2, and EXT4 opcodes, but the size parameter is passed in as 1, 2, and 4 (respectively).

Since the argument to EXT4 is a signed integer, it's possible to specify a negative index. However, the get_extension() function checks if the argument is negative and throws an error. However, pickletools does not error out if the argument is negative. Note that it's also not possible to register an object with a negative code through the copyreg.add_extension() function.

Example(s):

Before stack   : []
Before ext_reg : {('builtins', 'str'): 1}

Pickle bytes : b'\x84\x01\x00\x00\x00' (len 5)

After stack    : [<class 'str'>]
After ext_reg  : {('builtins', 'str'): 1}

Source Code (link)

def load_ext4(self):
    code, = unpack('<i', self.read(4))
    self.get_extension(code)

Pickletools Code (link)

def read_int4(f):
    data = f.read(4)
    if len(data) == 4:
        return _unpack("<i", data)[0]
    raise ValueError("not enough data in stream to read int4")

_pickle.c Code (link)

static int
load_extension(UnpicklerObject *self, int nbytes)
{
    char *codebytes;            /* the nbytes bytes after the opcode */
    long code;                  /* calc_binint returns long */
    PyObject *py_code;          /* code as a Python int */
    PyObject *obj;              /* the object to push */
    PyObject *pair;             /* (module_name, class_name) */
    PyObject *module_name, *class_name;
    PickleState *st = _Pickle_GetGlobalState();

    assert(nbytes == 1 || nbytes == 2 || nbytes == 4);
    if (_Unpickler_Read(self, &codebytes, nbytes) < 0)
        return -1;
    code = calc_binint(codebytes, nbytes);
    if (code <= 0) {            /* note that 0 is forbidden */
        /* Corrupt or hostile pickle. */
        PyErr_SetString(st->UnpicklingError, "EXT specifies code <= 0");
        return -1;
    }

    /* Look for the code in the cache. */
    py_code = PyLong_FromLong(code);
    if (py_code == NULL)
        return -1;
    obj = PyDict_GetItemWithError(st->extension_cache, py_code);
    if (obj != NULL) {
        /* Bingo. */
        Py_DECREF(py_code);
        PDATA_APPEND(self->stack, obj, -1);
        return 0;
    }
    if (PyErr_Occurred()) {
        Py_DECREF(py_code);
        return -1;
    }

    /* Look up the (module_name, class_name) pair. */
    pair = PyDict_GetItemWithError(st->inverted_registry, py_code);
    if (pair == NULL) {
        Py_DECREF(py_code);
        if (!PyErr_Occurred()) {
            PyErr_Format(PyExc_ValueError, "unregistered extension "
                         "code %ld", code);
        }
        return -1;
    }
    /* Since the extension registry is manipulable via Python code,
     * confirm that pair is really a 2-tuple of strings.
     */
    if (!PyTuple_Check(pair) || PyTuple_Size(pair) != 2) {
        goto error;
    }

    module_name = PyTuple_GET_ITEM(pair, 0);
    if (!PyUnicode_Check(module_name)) {
        goto error;
    }

    class_name = PyTuple_GET_ITEM(pair, 1);
    if (!PyUnicode_Check(class_name)) {
        goto error;
    }

    /* Load the object. */
    obj = find_class(self, module_name, class_name);
    if (obj == NULL) {
        Py_DECREF(py_code);
        return -1;
    }
    /* Cache code -> obj. */
    code = PyDict_SetItem(st->extension_cache, py_code, obj);
    Py_DECREF(py_code);
    if (code < 0) {
        Py_DECREF(obj);
        return -1;
    }
    PDATA_PUSH(self->stack, obj, -1);
    return 0;

error:
    Py_DECREF(py_code);
    PyErr_Format(PyExc_ValueError, "_inverted_registry[%ld] "
                 "isn't a 2-tuple of strings", code);
    return -1;
}

GLOBAL - Push a global object (module.attr) on the stack

• Opcode - 0x63 ('c')
• Protocol introduced - 0

Official Description:

Two newline-terminated strings follow the GLOBAL opcode. The first is taken as a module name, and the second as a class name. The class object module.class is pushed on the stack. More accurately, the object returned by self.find_class(module, class) is pushed on the stack, so unpickling subclasses can override this form of lookup.

Author's Note:

See the find_class() section below for more information about how classes are imported in through unpickling.

Example(s):

Before stack : []

Pickle bytes : b'cbuiltins\nstr\n' (len 14)

After stack  : [<class 'str'>]

Source Code (link)

def load_global(self):
    module = self.readline()[:-1].decode("utf-8")
    name = self.readline()[:-1].decode("utf-8")
    klass = self.find_class(module, name)
    self.append(klass)

Pickletools Code (link)

def read_stringnl_noescape_pair(f):
    return "%s %s" % (read_stringnl_noescape(f), read_stringnl_noescape(f))

_pickle.c Code (link)

static int
load_global(UnpicklerObject *self)
{
    PyObject *global = NULL;
    PyObject *module_name;
    PyObject *global_name;
    Py_ssize_t len;
    char *s;

    if ((len = _Unpickler_Readline(self, &s)) < 0)
        return -1;
    if (len < 2)
        return bad_readline();
    module_name = PyUnicode_DecodeUTF8(s, len - 1, "strict");
    if (!module_name)
        return -1;

    if ((len = _Unpickler_Readline(self, &s)) >= 0) {
        if (len < 2) {
            Py_DECREF(module_name);
            return bad_readline();
        }
        global_name = PyUnicode_DecodeUTF8(s, len - 1, "strict");
        if (global_name) {
            global = find_class(self, module_name, global_name);
            Py_DECREF(global_name);
        }
    }
    Py_DECREF(module_name);

    if (global == NULL)
        return -1;
    PDATA_PUSH(self->stack, global, -1);
    return 0;
}

STACK_GLOBAL - Push a global object (module.attr) on the stack

• Opcode - 0x93
• Protocol introduced - 4

Author's Note:

This opcode is the same as GLOBAL except the module and name variables are popped from the top of the stack. See the find_class() section below for more information about how classes are imported in through unpickling.

Example(s):

Before stack : ['builtins', 'str']

Pickle bytes : b'\x93' (len 1)

After stack  : [<class 'str'>]

Source Code (link)

def load_stack_global(self):
    name = self.stack.pop()
    module = self.stack.pop()
    if type(name) is not str or type(module) is not str:
        raise UnpicklingError("STACK_GLOBAL requires str")
    self.append(self.find_class(module, name))

_pickle.c Code (link)

static int
load_stack_global(UnpicklerObject *self)
{
    PyObject *global;
    PyObject *module_name;
    PyObject *global_name;

    PDATA_POP(self->stack, global_name);
    if (global_name == NULL) {
        return -1;
    }
    PDATA_POP(self->stack, module_name);
    if (module_name == NULL) {
        Py_DECREF(global_name);
        return -1;
    }
    if (!PyUnicode_CheckExact(module_name) ||
        !PyUnicode_CheckExact(global_name))
    {
        PickleState *st = _Pickle_GetGlobalState();
        PyErr_SetString(st->UnpicklingError, "STACK_GLOBAL requires str");
        Py_DECREF(global_name);
        Py_DECREF(module_name);
        return -1;
    }
    global = find_class(self, module_name, global_name);
    Py_DECREF(global_name);
    Py_DECREF(module_name);
    if (global == NULL)
        return -1;
    PDATA_PUSH(self->stack, global, -1);
    return 0;
}

REDUCE - Push an object built from a callable and an argument tuple

• Opcode - 0x52 ('R')
• Protocol introduced - 0

Official Description:

The opcode is named to remind of the __reduce__() method.

Stack before: ... callable pytuple
Stack after:  ... callable(*pytuple)

The callable and the argument tuple are the first two items returned by a __reduce__ method. Applying the callable to the argtuple is supposed to reproduce the original object, or at least get it started. If the __reduce__ method returns a 3-tuple, the last component is an argument to be passed to the object's __setstate__, and then the REDUCE opcode is followed by code to create setstate's argument, and then a BUILD opcode to apply __setstate__ to that argument.

If not isinstance(callable, type), REDUCE complains unless the callable has been registered with the copyreg module's safe_constructors dict, or the callable has a magic '__safe_for_unpickling__' attribute with a True value. I'm not sure why it does this, but I've sure seen this complaint often enough when I didn't want to.

Author's Note:

The entire last paragraph of the official description is outdated since it was written in 2002 and then the whole __safe_for_unpickling__ thing was removed in PEP 307 released in 2003 (so you can ignore it all).

Example(s):

Before stack : [<class 'str'>, (1,)]

Pickle bytes : b'R' (len 1)

After stack  : ['1']

Before stack : [<built-in function system>, ('whoami',)]

Pickle bytes : b'R' (len 1)

After stack  : [0]

Source Code (link)

def load_reduce(self):
    stack = self.stack
    args = stack.pop()
    func = stack[-1]
    stack[-1] = func(*args)

_pickle.c Code (link)

static int
load_reduce(UnpicklerObject *self)
{
    PyObject *callable = NULL;
    PyObject *argtup = NULL;
    PyObject *obj = NULL;

    PDATA_POP(self->stack, argtup);
    if (argtup == NULL)
        return -1;
    PDATA_POP(self->stack, callable);
    if (callable) {
        obj = PyObject_CallObject(callable, argtup);
        Py_DECREF(callable);
    }
    Py_DECREF(argtup);

    if (obj == NULL)
        return -1;

    PDATA_PUSH(self->stack, obj, -1);
    return 0;
}

BUILD - Finish building an object, via __setstate__ or dict update

• Opcode - 0x62 ('b')
• Protocol introduced - 0

Official Description:

Stack before: ... anyobject argument
Stack after:  ... anyobject

where anyobject may have been mutated, as follows:

If the object has a __setstate__ method, anyobject.__setstate__(argument) is called.

Else the argument must be a dict, the object must have a __dict__, and the object is updated via anyobject.__dict__.update(argument)

Author's Note:

To clarify, the BUILD opcode removes the argument tuple from the stack but leaves the (possibly modified) object.

When using the __setstate__() functionality of BUILD, a class definition (eg. from GLOBAL or STACK_GLOBAL) or an instance of a class (eg. from INST or OBJ) may be used as the object. However, when using the inst.__dict__[key] = value functionality (see example 3 below), only a class instance may be used because a class definition's __dict__ attribute is read-only. The slotstate setattr() functionality can be used on either a class definition or instance of a class.

Note that this BUILD opcode provides several primitives, such as calling the __setstate__() attribute, modifying the __dict__ attribute, or setting arbitrary attributes through setattr().

Examples may reference these custom Python classes:

class X:
    def __setstate__(self, arg):
        print('Hello', arg[0])

class Y:
    def __setstate__(arg):
        print('Hello', arg[0])

class Z:
    pass

Example(s):

Before stack : [<__main__.X object at 0x7fb5e2d11bb0>, ('world!',)]

Pickle bytes : b'b' (len 1)

After stack  : [<__main__.X object at 0x7fb5e2d11bb0>]
Printed      : 'Hello world!'

Before stack : [<class '__main__.Y'>, ('world!',)]

Pickle bytes : b'b' (len 1)

After stack  : [<class '__main__.Y'>]
Printed      : 'Hello world!'

Before stack : [<__main__.Z object at 0x7fb5e2d11bb0>, {'asdf': 1, 'fdsa': 2}]

Pickle bytes : b'b' (len 1)

After stack  : [<__main__.Z object at 0x7fb5e2d11bb0>]

__dict__ attribute of returned class instantiation = {'asdf': 1, 'fdsa': 2}

Before stack : [<class '__main__.Z'>, (None, {'asdf': 1})]

Pickle bytes : b'b' (len 1)

After stack  : [<class '__main__.Z'>]

asdf attribute of returned class = 1

Before stack : [<__main__.Z object at 0x7fb5e2d11bb0>, (None, {'asdf': 1})]

Pickle bytes : b'b' (len 1)

After stack  : [<__main__.Z object at 0x7fb5e2d11bb0>]

asdf attribute of returned class instantiation = 1

Source Code (link)

def load_build(self):
    stack = self.stack
    state = stack.pop()
    inst = stack[-1]
    setstate = getattr(inst, "__setstate__", None)
    if setstate is not None:
        setstate(state)
        return
    slotstate = None
    if isinstance(state, tuple) and len(state) == 2:
        state, slotstate = state
    if state:
        inst_dict = inst.__dict__
        intern = sys.intern
        for k, v in state.items():
            if type(k) is str:
                inst_dict[intern(k)] = v
            else:
                inst_dict[k] = v
    if slotstate:
        for k, v in slotstate.items():
            setattr(inst, k, v)

_pickle.c Code (link)

static int
load_build(UnpicklerObject *self)
{
    PyObject *state, *inst, *slotstate;
    PyObject *setstate;
    int status = 0;

    /* Stack is ... instance, state.  We want to leave instance at
     * the stack top, possibly mutated via instance.__setstate__(state).
     */
    if (Py_SIZE(self->stack) - 2 < self->stack->fence)
        return Pdata_stack_underflow(self->stack);

    PDATA_POP(self->stack, state);
    if (state == NULL)
        return -1;

    inst = self->stack->data[Py_SIZE(self->stack) - 1];

    if (_PyObject_LookupAttr(inst, &_Py_ID(__setstate__), &setstate) < 0) {
        Py_DECREF(state);
        return -1;
    }
    if (setstate != NULL) {
        PyObject *result;

        /* The explicit __setstate__ is responsible for everything. */
        result = _Pickle_FastCall(setstate, state);
        Py_DECREF(setstate);
        if (result == NULL)
            return -1;
        Py_DECREF(result);
        return 0;
    }

    /* A default __setstate__.  First see whether state embeds a
     * slot state dict too (a proto 2 addition).
     */
    if (PyTuple_Check(state) && PyTuple_GET_SIZE(state) == 2) {
        PyObject *tmp = state;

        state = PyTuple_GET_ITEM(tmp, 0);
        slotstate = PyTuple_GET_ITEM(tmp, 1);
        Py_INCREF(state);
        Py_INCREF(slotstate);
        Py_DECREF(tmp);
    }
    else
        slotstate = NULL;

    /* Set inst.__dict__ from the state dict (if any). */
    if (state != Py_None) {
        PyObject *dict;
        PyObject *d_key, *d_value;
        Py_ssize_t i;

        if (!PyDict_Check(state)) {
            PickleState *st = _Pickle_GetGlobalState();
            PyErr_SetString(st->UnpicklingError, "state is not a dictionary");
            goto error;
        }
        dict = PyObject_GetAttr(inst, &_Py_ID(__dict__));
        if (dict == NULL)
            goto error;

        i = 0;
        while (PyDict_Next(state, &i, &d_key, &d_value)) {
            /* normally the keys for instance attributes are
               interned.  we should try to do that here. */
            Py_INCREF(d_key);
            if (PyUnicode_CheckExact(d_key))
                PyUnicode_InternInPlace(&d_key);
            if (PyObject_SetItem(dict, d_key, d_value) < 0) {
                Py_DECREF(d_key);
                goto error;
            }
            Py_DECREF(d_key);
        }
        Py_DECREF(dict);
    }

    /* Also set instance attributes from the slotstate dict (if any). */
    if (slotstate != NULL) {
        PyObject *d_key, *d_value;
        Py_ssize_t i;

        if (!PyDict_Check(slotstate)) {
            PickleState *st = _Pickle_GetGlobalState();
            PyErr_SetString(st->UnpicklingError,
                            "slot state is not a dictionary");
            goto error;
        }
        i = 0;
        while (PyDict_Next(slotstate, &i, &d_key, &d_value)) {
            if (PyObject_SetAttr(inst, d_key, d_value) < 0)
                goto error;
        }
    }

    if (0) {
  error:
        status = -1;
    }

    Py_DECREF(state);
    Py_XDECREF(slotstate);
    return status;
}

INST - Build a class instance

• Opcode - 0x69 ('i')
• Protocol introduced - 0

Official Description:

This is the protocol 0 version of protocol 1's OBJ opcode. INST is followed by two newline-terminated strings, giving a module and class name, just as for the GLOBAL opcode (and see GLOBAL for more details about that). self.find_class(module, name) is used to get a class object.

In addition, all the objects on the stack following the topmost markobject are gathered into a tuple and popped (along with the topmost markobject), just as for the TUPLE opcode.

Now it gets complicated. If all of these are true:

• The argtuple is empty (markobject was at the top of the stack at the start)
• The class object does not have a __getinitargs__ attribute

Then we want to create an old-style class instance without invoking its __init__() method (pickle has waffled on this over the years; not calling __init__() is current wisdom). In this case, an instance of an old-style dummy class is created, and then we try to rebind its __class__ attribute to the desired class object. If this succeeds, the new instance object is pushed on the stack, and we're done.

Else (the argtuple is not empty, it's not an old-style class object, or the class object does have a __getinitargs__ attribute), the code first insists that the class object have a __safe_for_unpickling__ attribute. Unlike as for the __safe_for_unpickling__ check in REDUCE, it doesn't matter whether this attribute has a true or false value, it only matters whether it exists (XXX this is a bug). If __safe_for_unpickling__ doesn't exist, UnpicklingError is raised.

Else (the class object does have a __safe_for_unpickling__ attr), the class object obtained from INST's arguments is applied to the argtuple obtained from the stack, and the resulting instance object is pushed on the stack.

Note - checks for __safe_for_unpickling__ went away in Python 2.3

Note - the distinction between old-style and new-style classes does not make sense in Python 3

Author's Notes:

While this opcode is meant to create an instance of a Python class, it can also be used like the REDUCE opcode, where it calls an arbitrary function (found through find_class()) with a tuple of arguments. See the third example below.

INST only supports classes with ASCII names, which is intentional. Opcodes like GLOBAL (which also get a module through find_class()) use UTF-8 encoding; however, since INST is only supported by older protocols in Python 2.x, it's intended to only accept ASCII names.

Examples may reference these custom Python classes:

class X:
    pass

class Y:
    def __init__(self, args):
        self.args = args

Example(s):

Before stack     : []
Before metastack : [[1, 2, 'asdf']]

Pickle bytes     : b'i__main__\nX\n' (len 12)

After stack      : [1, 2, 'asdf', <__main__.X object at 0x7f4103d4bfe0>]
Before metastack : []

Before stack     : [12]
Before metastack : [[1, 2, 'asdf']]

Pickle bytes     : b'i__main__\nY\n' (len 12)

After stack      : [1, 2, 'asdf', <__main__.Y object at 0x7f4103d4bf50>]
Before metastack : []

Before stack     : [255]
Before metastack : [[1, 2, 'asdf']]

Pickle bytes     : b'ibuiltins\nhex\n' (len 14)

After stack      : [1, 2, 'asdf', '0xff']
Before metastack : []

Source Code (link)

def load_inst(self):
    module = self.readline()[:-1].decode("ascii")
    name = self.readline()[:-1].decode("ascii")
    klass = self.find_class(module, name)
    self._instantiate(klass, self.pop_mark())

Pickletools Code (link)

def read_stringnl_noescape_pair(f):
    return "%s %s" % (read_stringnl_noescape(f), read_stringnl_noescape(f))

_pickle.c Code (link)

static int
load_inst(UnpicklerObject *self)
{
    PyObject *cls = NULL;
    PyObject *args = NULL;
    PyObject *obj = NULL;
    PyObject *module_name;
    PyObject *class_name;
    Py_ssize_t len;
    Py_ssize_t i;
    char *s;

    if ((i = marker(self)) < 0)
        return -1;
    if ((len = _Unpickler_Readline(self, &s)) < 0)
        return -1;
    if (len < 2)
        return bad_readline();

    /* Here it is safe to use PyUnicode_DecodeASCII(), even though non-ASCII
       identifiers are permitted in Python 3.0, since the INST opcode is only
       supported by older protocols on Python 2.x. */
    module_name = PyUnicode_DecodeASCII(s, len - 1, "strict");
    if (module_name == NULL)
        return -1;

    if ((len = _Unpickler_Readline(self, &s)) >= 0) {
        if (len < 2) {
            Py_DECREF(module_name);
            return bad_readline();
        }
        class_name = PyUnicode_DecodeASCII(s, len - 1, "strict");
        if (class_name != NULL) {
            cls = find_class(self, module_name, class_name);
            Py_DECREF(class_name);
        }
    }
    Py_DECREF(module_name);

    if (cls == NULL)
        return -1;

    if ((args = Pdata_poptuple(self->stack, i)) != NULL) {
        obj = instantiate(cls, args);
        Py_DECREF(args);
    }
    Py_DECREF(cls);

    if (obj == NULL)
        return -1;

    PDATA_PUSH(self->stack, obj, -1);
    return 0;
}

OBJ - Build a class instance

• Opcode - 0x6f ('o')
• Protocol introduced - 1

Official Description:

This is the protocol 1 version of protocol 0's INST opcode, and is very much like it. The major difference is that the class object is taken off the stack, allowing it to be retrieved from the memo repeatedly if several instances of the same class are created. This can be much more efficient (in both time and space) than repeatedly embedding the module and class names in INST opcodes.

Unlike INST, OBJ takes no arguments from the opcode stream. Instead the class object is taken off the stack, immediately above the topmost markobject:

Stack before: ... markobject classobject stackslice
Stack after:  ... new_instance_object

As for INST, the remainder of the stack above the markobject is gathered into an argument tuple, and then the logic seems identical, except that no __safe_for_unpickling__ check is done (XXX this is a bug). See INST for the gory details.

Note - In Python 2.3, INST and OBJ are identical except for how they get the class object. That was always the intent; the implementations had diverged for accidental reasons

Author's Notes:

While this opcode is meant to create an instance of a Python class, it can also be used like the REDUCE opcode, where it calls an arbitrary function with a tuple of arguments. See the third example below.

Examples may reference these custom Python classes:

class X:
    pass

class Y:
    def __init__(self, args):
        self.args = args

Example(s):

Before stack     : [<class '__main__.X'>]
Before metastack : [[1, 2, 'asdf']]

Pickle bytes     : b'o' (len 1)

After stack      : [1, 2, 'asdf', <__main__.X object at 0x7f4103d4bfe0>]
Before metastack : []

Before stack     : [<class '__main__.Y'>, 12]
Before metastack : [[1, 2, 'asdf']]

Pickle bytes     : b'o' (len 1)

After stack      : [1, 2, 'asdf', <__main__.Y object at 0x7f4103d4bf50>]
Before metastack : []

Before stack     : [<built-in function hex>, 255]
Before metastack : [[1, 2, 'asdf']]

Pickle bytes     : b'o' (len 1)

After stack      : [1, 2, 'asdf', '0xff']
Before metastack : []

Source Code (link)

def load_obj(self):
    args = self.pop_mark()
    cls = args.pop(0)
    self._instantiate(cls, args)

_pickle.c Code (link)

static int
load_obj(UnpicklerObject *self)
{
    PyObject *cls, *args, *obj = NULL;
    Py_ssize_t i;

    if ((i = marker(self)) < 0)
        return -1;

    if (Py_SIZE(self->stack) - i < 1)
        return Pdata_stack_underflow(self->stack);

    args = Pdata_poptuple(self->stack, i + 1);
    if (args == NULL)
        return -1;

    PDATA_POP(self->stack, cls);
    if (cls) {
        obj = instantiate(cls, args);
        Py_DECREF(cls);
    }
    Py_DECREF(args);
    if (obj == NULL)
        return -1;

    PDATA_PUSH(self->stack, obj, -1);
    return 0;
}

NEWOBJ - Build an object instance

• Opcode - 0x81
• Protocol introduced - 2

Official Description:

The stack before should be thought of as containing a class object followed by an argument tuple (the tuple being the stack top). Call these cls and args. They are popped off the stack, and the value returned by cls.__new__(cls, *args) is pushed back onto the stack.

Author's Notes:

In pickle.py, the syntax used for creating a new object instance is cls.__new__(cls, *args). Since *args is used, args must be an iterator, but can be any type of iterator. _pickle.c specifically checks that args are a tuple, so if you make args a list (for example), it will work in pickle.py but not _pickle.c.

Note that the _pickle.c implementation uses the same function for the NEWOBJ and NEWOBJ_EX opcodes, but the use_kwargs parameter is passed in as 0 and 1 (respectively).

Examples may reference these custom Python classes:

class X:
    pass

class Y:
    def __init__(self, args):
        self.args = args

Example(s):

Before stack     : [1, 'asdf', <class '__main__.X'>, ()]

Pickle bytes     : b'\x81' (len 1)

After stack      : [1, 'asdf', <__main__.X object at 0x7f4103d4bfe0>]

Before stack     : [1, 'asdf', <class '__main__.Y'>, (12,)]

Pickle bytes     : b'\x81' (len 1)

After stack      : [1, 'asdf', <__main__.Y object at 0x7f4103d4bf50>]

Source Code (link)

def load_newobj(self):
    args = self.stack.pop()
    cls = self.stack.pop()
    obj = cls.__new__(cls, *args)
    self.append(obj)

_pickle.c Code (link)

static int
load_newobj(UnpicklerObject *self, int use_kwargs)
{
    PyObject *cls, *args, *kwargs = NULL;
    PyObject *obj;

    /* Stack is ... cls args [kwargs], and we want to call
     * cls.__new__(cls, *args, **kwargs).
     */
    if (use_kwargs) {
        PDATA_POP(self->stack, kwargs);
        if (kwargs == NULL) {
            return -1;
        }
    }
    PDATA_POP(self->stack, args);
    if (args == NULL) {
        Py_XDECREF(kwargs);
        return -1;
    }
    PDATA_POP(self->stack, cls);
    if (cls == NULL) {
        Py_XDECREF(kwargs);
        Py_DECREF(args);
        return -1;
    }

    if (!PyType_Check(cls)) {
        newobj_unpickling_error("%s class argument must be a type, not %.200s",
                                use_kwargs, cls);
        goto error;
    }
    if (((PyTypeObject *)cls)->tp_new == NULL) {
        newobj_unpickling_error("%s class argument '%.200s' doesn't have __new__",
                                use_kwargs, cls);
        goto error;
    }
    if (!PyTuple_Check(args)) {
        newobj_unpickling_error("%s args argument must be a tuple, not %.200s",
                                use_kwargs, args);
        goto error;
    }
    if (use_kwargs && !PyDict_Check(kwargs)) {
        newobj_unpickling_error("%s kwargs argument must be a dict, not %.200s",
                                use_kwargs, kwargs);
        goto error;
    }

    obj = ((PyTypeObject *)cls)->tp_new((PyTypeObject *)cls, args, kwargs);
    if (obj == NULL) {
        goto error;
    }
    Py_XDECREF(kwargs);
    Py_DECREF(args);
    Py_DECREF(cls);
    PDATA_PUSH(self->stack, obj, -1);
    return 0;

error:
    Py_XDECREF(kwargs);
    Py_DECREF(args);
    Py_DECREF(cls);
    return -1;
}

NEWOBJ_EX - Build an object instance

• Opcode - 0x92
• Protocol introduced - 4

Official Description:

The stack before should be thought of as containing a class object followed by an argument tuple and by a keyword argument dict (the dict being the stack top). Call these cls and args. They are popped off the stack, and the value returned by cls.__new__(cls, *args, *kwargs) is pushed back onto the stack.

Author's Notes:

In pickle.py, the syntax used for creating a new object instance is cls.__new__(cls, *args, **kwargs). Since *args is used, args must be an iterator, but can be any type of iterator. _pickle.c specifically checks that args are a tuple, so if you make args a list (for example), it will work in pickle.py but not _pickle.c.

Note that the _pickle.c implementation uses the same function for the NEWOBJ and NEWOBJ_EX opcodes, but the use_kwargs parameter is passed in as 0 and 1 (respectively).

Examples may reference these custom Python classes:

class X:
    pass

class Y:
    def __init__(self, args):
        self.args = args

Example(s):

Before stack     : [1, 'asdf', <class '__main__.X'>, (), {}]

Pickle bytes     : b'\x92' (len 1)

After stack      : [1, 'asdf', <__main__.X object at 0x7f4103d4bfe0>]

Before stack     : [1, 'asdf', <class '__main__.Y'>, (12,), {}]

Pickle bytes     : b'\x92' (len 1)

After stack      : [1, 'asdf', <__main__.Y object at 0x7f4103d4bf50>]

Source Code (link)

def load_newobj_ex(self):
    kwargs = self.stack.pop()
    args = self.stack.pop()
    cls = self.stack.pop()
    obj = cls.__new__(cls, *args, **kwargs)
    self.append(obj)

_pickle.c Code (link)

static int
load_newobj(UnpicklerObject *self, int use_kwargs)
{
    PyObject *cls, *args, *kwargs = NULL;
    PyObject *obj;

    /* Stack is ... cls args [kwargs], and we want to call
     * cls.__new__(cls, *args, **kwargs).
     */
    if (use_kwargs) {
        PDATA_POP(self->stack, kwargs);
        if (kwargs == NULL) {
            return -1;
        }
    }
    PDATA_POP(self->stack, args);
    if (args == NULL) {
        Py_XDECREF(kwargs);
        return -1;
    }
    PDATA_POP(self->stack, cls);
    if (cls == NULL) {
        Py_XDECREF(kwargs);
        Py_DECREF(args);
        return -1;
    }

    if (!PyType_Check(cls)) {
        newobj_unpickling_error("%s class argument must be a type, not %.200s",
                                use_kwargs, cls);
        goto error;
    }
    if (((PyTypeObject *)cls)->tp_new == NULL) {
        newobj_unpickling_error("%s class argument '%.200s' doesn't have __new__",
                                use_kwargs, cls);
        goto error;
    }
    if (!PyTuple_Check(args)) {
        newobj_unpickling_error("%s args argument must be a tuple, not %.200s",
                                use_kwargs, args);
        goto error;
    }
    if (use_kwargs && !PyDict_Check(kwargs)) {
        newobj_unpickling_error("%s kwargs argument must be a dict, not %.200s",
                                use_kwargs, kwargs);
        goto error;
    }

    obj = ((PyTypeObject *)cls)->tp_new((PyTypeObject *)cls, args, kwargs);
    if (obj == NULL) {
        goto error;
    }
    Py_XDECREF(kwargs);
    Py_DECREF(args);
    Py_DECREF(cls);
    PDATA_PUSH(self->stack, obj, -1);
    return 0;

error:
    Py_XDECREF(kwargs);
    Py_DECREF(args);
    Py_DECREF(cls);
    return -1;
}

PROTO - Protocol version indicator

• Opcode - 0x80
• Protocol introduced - 2

Official Description:

For protocol 2 and above, a pickle must start with this opcode. The argument is the protocol version, an int in range(2, 256).

Author's Notes:

The official description states that the protocol needs to be in range(2,256) but the actual source code ensures it's between 0 and 5 (inclusive). If PROTO is not included in the bytestream, the default protocol is 0. The pickling process for modern Python versions will always include the PROTO opcode (and always as the first instruction), but the position of the PROTO opcode is not enforced.

The only place proto is used in the unpickling process is in find_class().

Example(s):

Before stack : []

Pickle bytes : b'\x80\x05' (len 2)

After stack  : []

Source Code (link)

def load_proto(self):
    proto = self.read(1)[0]
    if not 0 <= proto <= HIGHEST_PROTOCOL:
        raise ValueError("unsupported pickle protocol: %d" % proto)
    self.proto = proto

Pickletools Code (link)

def read_uint1(f):
    data = f.read(1)
    if data:
        return data[0]
    raise ValueError("not enough data in stream to read uint1")

_pickle.c Code (link)

static int
load_proto(UnpicklerObject *self)
{
    char *s;
    int i;

    if (_Unpickler_Read(self, &s, 1) < 0)
        return -1;

    i = (unsigned char)s[0];
    if (i <= HIGHEST_PROTOCOL) {
        self->proto = i;
        return 0;
    }

    PyErr_Format(PyExc_ValueError, "unsupported pickle protocol: %d", i);
    return -1;
}

STOP - Stop the unpickling machine

• Opcode - 0x2e ('.')
• Protocol introduced - 0

Official Description:

Every pickle ends with this opcode. The object at the top of the stack is popped, and that's the result of unpickling. The stack should be empty then.

Author's Notes:

All pickle implementations require that the STOP opcode be present, or it will error out. Any bytes in the unpickling bytestream after STOP are ignored.

If no objects are present on the stack when STOP is encountered, it will error out. Pickletools checks that the stack is empty after popping off the return value, but the other implementations don't. Pickletools also doesn't like ending the unpickling process with a MARK object on the stack, but the other implementations don't care.

Example(s):

Before stack : []

Pickle bytes : b'.' (len 1)

After stack  : []

Source Code (link)

def load_stop(self):
    value = self.stack.pop()
    raise _Stop(value)

FRAME - Indicate the beginning of a new frame

• Opcode - 0x95
• Protocol introduced - 4

Official Description:

The unpickler may use this opcode to safely prefetch data from its underlying stream.

Author's Notes:

Pickle framing is very undocumented, and other than the source code and PEP 3154, there's hardly anything on it. Framing acts like stream buffering, and exists to prevent several small I/O reads. Several opcodes have some sort of readline() or read(n_bytes) call inside it to pull out the pickle arguments or subsequent data. If the pickle is being read from a file, then each of these function calls results in the operating system reading from a file. Several calls across the unpickling process can lead to several I/O reads, severely slowing down the unpickling process for large pickles.

To provide an alternative, the FRAME opcode will read n bytes from the file into a BytesIO() class so that future readline() or read() calls pull from memory instead of disk. Frames are not required in pickles, and a pickle can use frames for however little or many bytes they would like.

Many discrepancies in the implementation of the FRAME opcode exist between C pickle and Python pickle, mostly because C pickle doesn't really support framing. C pickle will read the 8-byte unsigned frame length argument (n), then check that n bytes exist in the stream and nothing more (no actual buffering). Python pickle reads in the length n, then creates a frame and reads up to n bytes from the pickled stream into the BytesIO() frame. All future reads try to go through the current frame if it exists.

A discrepancy exists because Python pickle will not throw an error if the argument for the FRAME opcode is larger than the number of bytes available in the bytestream; instead, it will pull out as many bytes as it can up to the argument. However, the C pickle implementation explicitly checks for n bytes, and throws an error if that many bytes don't exist in the stream.

Overlapping frames are disallowed in Python pickles, but are not checked in C pickle. Therefore, if a new frame is created, and the pickled bytestream attempts to create a second frame before using all the bytes in the first frame are used up, an error is thrown.

According to PEP 3154, opcodes cannot be split across different frames. Interestingly, the opcode and opcode argument(s) can be in different frames in both implementations, but a multi-byte opcode argument cannot be split across frames in Python pickle (C pickle doesn't care).

Example(s):

Before stack : []

Pickle bytes : b'\x95\xff\xff\xff\xff\xff\xff\xff\x7f' (len 9)

After stack  : []

Before stack : []

Pickle bytes : b'\x95\x01\x00\x00\x00\x00\x00\x00\x00' (len 9)

After stack  : []

Source Code (link)

def load_frame(self):
    frame_size, = unpack('<Q', self.read(8))
    if frame_size > sys.maxsize:
        raise ValueError("frame size > sys.maxsize: %d" % frame_size)
    self._unframer.load_frame(frame_size)

Pickletools Code (link)

def read_uint8(f):
    data = f.read(8)
    if len(data) == 8:
        return _unpack("<Q", data)[0]
    raise ValueError("not enough data in stream to read uint8")

_pickle.c Code (link)

static int
load_frame(UnpicklerObject *self)
{
    char *s;
    Py_ssize_t frame_len;

    if (_Unpickler_Read(self, &s, 8) < 0)
        return -1;

    frame_len = calc_binsize(s, 8);
    if (frame_len < 0) {
        PyErr_Format(PyExc_OverflowError,
                     "FRAME length exceeds system's maximum of %zd bytes",
                     PY_SSIZE_T_MAX);
        return -1;
    }

    if (_Unpickler_Read(self, &s, frame_len) < 0)
        return -1;

    /* Rewind to start of frame */
    self->next_read_idx -= frame_len;
    return 0;
}

PERSID - Push an object identified by a persistent ID

• Opcode - 0x50 ('P')
• Protocol introduced - 0

Official Description:

The pickle module doesn't define what a persistent ID means. PERSID's argument is a newline-terminated str-style (no embedded escapes, no bracketing quote characters) string, which is "the persistent ID". The unpickler passes this string to self.persistent_load(). Whatever object that returns is pushed on the stack. There is no implementation of persistent_load() in Python's unpickler: it must be supplied by an unpickler subclass.

Author's Notes:

This opcode almost functions as a generic custom opcode that can be implemented by developers. You just use P with a custom string, and it runs a custom Python function that should have been defined already by the developer. Presumably, the intended functionality is to have some kind of lookup table where a provided "persistent ID" is used as a key in that table, and the value is returned and then pushed onto the stack.

In the C implementation, self->pers_func is set to NULL and during runtime the Unpickler checks whether it's still NULL or not. In the Python implementation, self.persistent_load() is actually defined, but the function will simply throw a ValueError.

Both implementations require the string be ASCII-decodable.

Example(s):

Before stack : []

Pickle bytes : b'Pcustom_persid\n' (len 15)

After stack  : ['some_custom_value']

Source Code (link)

def load_persid(self):
    try:
        pid = self.readline()[:-1].decode("ascii")
    except UnicodeDecodeError:
        raise UnpicklingError(
            "persistent IDs in protocol 0 must be ASCII strings")
    self.append(self.persistent_load(pid))

Pickletools Code (link)

def read_stringnl(f, decode=True, stripquotes=True):
    data = f.readline()
    if not data.endswith(b'\n'):
        raise ValueError("no newline found when trying to read stringnl")
    data = data[:-1]    # lose the newline

    if stripquotes:
        for q in (b'"', b"'"):
            if data.startswith(q):
                if not data.endswith(q):
                    raise ValueError("strinq quote %r not found at both "
                                     "ends of %r" % (q, data))
                data = data[1:-1]
                break
        else:
            raise ValueError("no string quotes around %r" % data)

    if decode:
        data = codecs.escape_decode(data)[0].decode("ascii")
    return data

_pickle.c Code (link)

static int
load_persid(UnpicklerObject *self)
{
    PyObject *pid, *obj;
    Py_ssize_t len;
    char *s;

    if (self->pers_func) {
        if ((len = _Unpickler_Readline(self, &s)) < 0)
            return -1;
        if (len < 1)
            return bad_readline();

        pid = PyUnicode_DecodeASCII(s, len - 1, "strict");
        if (pid == NULL) {
            if (PyErr_ExceptionMatches(PyExc_UnicodeDecodeError)) {
                PyErr_SetString(_Pickle_GetGlobalState()->UnpicklingError,
                                "persistent IDs in protocol 0 must be "
                                "ASCII strings");
            }
            return -1;
        }

        obj = call_method(self->pers_func, self->pers_func_self, pid);
        Py_DECREF(pid);
        if (obj == NULL)
            return -1;

        PDATA_PUSH(self->stack, obj, -1);
        return 0;
    }
    else {
        PickleState *st = _Pickle_GetGlobalState();
        PyErr_SetString(st->UnpicklingError,
                        "A load persistent id instruction was encountered,\n"
                        "but no persistent_load function was specified.");
        return -1;
    }
}

BINPERSID - Push an object identified by a persistent ID

• Opcode - 0x51 ('Q')
• Protocol introduced - 1

Official Description:

Like PERSID, except the persistent ID is popped off the stack (instead of being a string embedded in the opcode bytestream). The persistent ID is passed to self.persistent_load(), and whatever object that returns is pushed on the stack. See PERSID for more detail.

Example(s):

Before stack : ['custom_persid']

Pickle bytes : b'Q' (len 1)

After stack  : ['some_custom_value']

Source Code (link)

def load_binpersid(self):
    pid = self.stack.pop()
    self.append(self.persistent_load(pid))

_pickle.c Code (link)

static int
load_binpersid(UnpicklerObject *self)
{
    PyObject *pid, *obj;

    if (self->pers_func) {
        PDATA_POP(self->stack, pid);
        if (pid == NULL)
            return -1;

        obj = call_method(self->pers_func, self->pers_func_self, pid);
        Py_DECREF(pid);
        if (obj == NULL)
            return -1;

        PDATA_PUSH(self->stack, obj, -1);
        return 0;
    }
    else {
        PickleState *st = _Pickle_GetGlobalState();
        PyErr_SetString(st->UnpicklingError,
                        "A load persistent id instruction was encountered,\n"
                        "but no persistent_load function was specified.");
        return -1;
    }
}

find_class()

The find_class() function in the _Unpickler class of pickle.py is used to import arbitrary classes and functions during the unpickling process. It takes in a module name and attribute name (both as strings) and pushes the resulting class onto the stack. It first runs __import__() on the module name to import the module (making sure it's located inside sys.modules), then runs either getattr() or _getattribute() on sys.modules[module] with name.

Special behavior exists depending on the prototype version declared using the PROTO opcode. If the version is less than 3, then a special mapping for import and attribute names exist because several commonly-used built-in classes changed from Python 2 to Python 3. It either looks up (module, name) in NAME_MAPPING or module in IMPORT_MAPPING from _compat.pickle.py and replaces module and name with the new one.

If the prototype version is 4 or higher, _getattribute() is used instead of the builtin getattr() function. _getattribute() is a custom Pickle helper function that will continually import attributes of the name argument until there's none left. For example, _getattribute() can resolve the name 'str.format' or 'str.format.__call__' attributes (splitting on periods) so only a single GLOBAL opcode needs to be used and not multiple.

Note that _pickle.c is set up to just call the Python find_class() function so if it's overwritten in a RestrictedUnpickler implementation, the overwritten class is still used.

Source Code (link)

def find_class(self, module, name):
    # Subclasses may override this.
    sys.audit('pickle.find_class', module, name)
    if self.proto < 3 and self.fix_imports:
        if (module, name) in _compat_pickle.NAME_MAPPING:
            module, name = _compat_pickle.NAME_MAPPING[(module, name)]
        elif module in _compat_pickle.IMPORT_MAPPING:
            module = _compat_pickle.IMPORT_MAPPING[module]
    __import__(module, level=0)
    if self.proto >= 4:
        return _getattribute(sys.modules[module], name)[0]
    else:
        return getattr(sys.modules[module], name)

_pickle.c Code (link)

static PyObject *
find_class(UnpicklerObject *self, PyObject *module_name, PyObject *global_name)
{
    return PyObject_CallMethodObjArgs((PyObject *)self, &_Py_ID(find_class),
                                      module_name, global_name, NULL);
}