[BUG] HTTPX Binary upload data unsuported

[07pepa]

I am trying to migrate one library from requests to http and it seems httpx portion have some bugs

folowing casete cant be used with httpx
https://github.com/billdeitrick/pypco/blob/master/tests/cassettes/TestPublicRequestFunctions.test_upload_and_use_file.yaml

because it tries to decode the binary string (that is nonsense)

Error
Traceback (most recent call last):
  File "pypco\pypco\pco.py", line 251, in request_response
    response = self._do_url_managed_request(method, url, payload, upload, **params)
  File "pypco\pypco\pco.py", line 223, in _do_url_managed_request
    return self._do_ratelimit_managed_request(method, url, payload, upload, **params)
  File "pypco\pypco\pco.py", line 184, in _do_ratelimit_managed_request
    response = self._do_timeout_managed_request(method, url, payload, upload, **params)
  File "pypco\pypco\pco.py", line 145, in _do_timeout_managed_request
    return self._do_request(method, url, payload, upload, **params)
  File "pypco\pypco\pco.py", line 110, in _do_request
    response = self.session.request(
  File ".virtualenvs\pypco-EZZlDlNc\lib\site-packages\httpx\_client.py", line 815, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
  File ".virtualenvs\pypco-EZZlDlNc\lib\site-packages\vcr\stubs\httpx_stubs.py", line 168, in _inner_send
    return _sync_vcr_send(cassette, real_send, *args, **kwargs)
  File ".virtualenvs\pypco-EZZlDlNc\lib\site-packages\vcr\stubs\httpx_stubs.py", line 155, in _sync_vcr_send
    vcr_request, response = _shared_vcr_send(cassette, real_send, *args, **kwargs)
  File ".virtualenvs\pypco-EZZlDlNc\lib\site-packages\vcr\stubs\httpx_stubs.py", line 81, in _shared_vcr_send
    vcr_request = _make_vcr_request(real_request, **kwargs)
  File ".virtualenvs\pypco-EZZlDlNc\lib\site-packages\vcr\stubs\httpx_stubs.py", line 72, in _make_vcr_request
    body = httpx_request.read().decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 137: invalid start byte

[llybin]

Probably related: ktosiek/pytest-vcr#46

[07pepa]

indeed it is

[ryanberryhill]

The same thing happens when trying to record a cassette with httpx, where binary request data causes a UnicodeDecodeError. Here's a simple reproduction script that does the same thing successfully both with requests and with httpx outside of a vcr.use_cassette context.

import httpx
import requests
import vcr


requests_response = requests.post('http://example.com', data=b'\xff')
print(requests_response.status_code)

httpx_response = httpx.post('http://example.com', data=b'\xff')
print(httpx_response.status_code)

with vcr.use_cassette('requests.yaml'):
    requests_response = requests.post('http://example.com', data=b'\xff')
    print(requests_response.status_code)

with vcr.use_cassette('httpx.yaml'):
    httpx_response = httpx.post('http://example.com', data=b'\xff')
    print(httpx_response.status_code)

Expected output:

200
200
200
200

Actual output:

200
200
200
Traceback (most recent call last):
  File "/Users/rberryhill/.pyenv/versions/3.8.13/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Users/rberryhill/.pyenv/versions/3.8.13/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/rberryhill/src/vcrbug/bug.py", line 18, in <module>
    httpx_response = httpx.post('http://example.com', data=b'\xff')
  File "/Users/rberryhill/src/vcrbug/venv/lib/python3.8/site-packages/httpx/_api.py", line 304, in post
    return request(
  File "/Users/rberryhill/src/vcrbug/venv/lib/python3.8/site-packages/httpx/_api.py", line 100, in request
    return client.request(
  File "/Users/rberryhill/src/vcrbug/venv/lib/python3.8/site-packages/httpx/_client.py", line 821, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/Users/rberryhill/src/vcrbug/venv/lib/python3.8/site-packages/vcr/stubs/httpx_stubs.py", line 168, in _inner_send
    return _sync_vcr_send(cassette, real_send, *args, **kwargs)
  File "/Users/rberryhill/src/vcrbug/venv/lib/python3.8/site-packages/vcr/stubs/httpx_stubs.py", line 155, in _sync_vcr_send
    vcr_request, response = _shared_vcr_send(cassette, real_send, *args, **kwargs)
  File "/Users/rberryhill/src/vcrbug/venv/lib/python3.8/site-packages/vcr/stubs/httpx_stubs.py", line 81, in _shared_vcr_send
    vcr_request = _make_vcr_request(real_request, **kwargs)
  File "/Users/rberryhill/src/vcrbug/venv/lib/python3.8/site-packages/vcr/stubs/httpx_stubs.py", line 72, in _make_vcr_request
    body = httpx_request.read().decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

This is with python 3.8.13, httpx==0.23.3, vcrpy==4.2.1.

[gotmax23]

Can we use errors="surrogateescape" here?

[76tr75b]

I still get the same error if i upload a file via httpx in a test with vcrpy.

• Python 3.11.8
• vcrpy 6.0.1
• httpx 0.27.0

/usr/local/lib/python3.11/site-packages/statsd/client/timer.py:41: in _wrapped
    return f(*args, **kwargs)
src/common/client/example/client.py:39: in analyse
    result = self._httpx_client.post(
/usr/local/lib/python3.11/site-packages/httpx/_client.py:1145: in post
    return self.request(
/usr/local/lib/python3.11/site-packages/httpx/_client.py:827: in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
/usr/local/lib/python3.11/site-packages/sentry_sdk/integrations/httpx.py:84: in send
    rv = real_send(self, request, **kwargs)
/usr/local/lib/python3.11/site-packages/httpx/_client.py:914: in send
    response = self._send_handling_auth(
/usr/local/lib/python3.11/site-packages/httpx/_client.py:942: in _send_handling_auth
    response = self._send_handling_redirects(
/usr/local/lib/python3.11/site-packages/httpx/_client.py:979: in _send_handling_redirects
    response = self._send_single_request(request)
/usr/local/lib/python3.11/site-packages/vcr/stubs/httpx_stubs.py:184: in _inner_send
    return _sync_vcr_send(cassette, real_send, *args, **kwargs)
/usr/local/lib/python3.11/site-packages/vcr/stubs/httpx_stubs.py:170: in _sync_vcr_send
    vcr_request, response = _shared_vcr_send(cassette, real_send, *args, **kwargs)
/usr/local/lib/python3.11/site-packages/vcr/stubs/httpx_stubs.py:117: in _shared_vcr_send
    vcr_request = _make_vcr_request(real_request, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

httpx_request = <Request('POST', 'http://example.com/upload')>, kwargs = {}

    def _make_vcr_request(httpx_request, **kwargs):
>       body = httpx_request.read().decode("utf-8")
E       UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 161: invalid start byte

[76tr75b]

Ok, my issue is fixed by this PR.

[simonw]

This isssue should not be closed - PR #820 was not accepted, so the problem still occurs - it's not possible to use httpx mode with multipart/form data requests containing binary files.

[simonw]

I worked around this problem in my own project by applying this monkey-patch, using code from #820:

from vcr.request import Request as VcrRequest
from vcr.stubs import httpx_stubs

def _make_vcr_request(httpx_request, **kwargs):
    try:
        body = httpx_request.read().decode("utf-8")
    except UnicodeDecodeError:
        body = httpx_request.read().decode("ISO-8859-1").encode("utf-8")

    uri = str(httpx_request.url)
    headers = dict(httpx_request.headers)
    return VcrRequest(httpx_request.method, uri, body, headers)

httpx_stubs._make_vcr_request = _make_vcr_request

[kevin1024]

Thanks @simonw. I’ll reopen.

Love your blog by the way! Thanks for using VCR.