Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use of SVG generates TypeError: Object of type 'bytes' is not JSON serializable #904

Closed
jmgc opened this issue Oct 28, 2018 · 9 comments · May be fixed by #910
Closed

Use of SVG generates TypeError: Object of type 'bytes' is not JSON serializable #904

jmgc opened this issue Oct 28, 2018 · 9 comments · May be fixed by #910

Comments

@jmgc
Copy link

jmgc commented Oct 28, 2018

When using SVG plots

%config InlineBackend.figure_format = 'svg'

the conversion to latex generate the error:

TypeError: Object of type 'bytes' is not JSON serializable

If I use PDF or PNG

%config InlineBackend.figure_format = 'pdf'

or

%config InlineBackend.figure_format = 'png'

No error is detected. I understand that this is an issue associated with the svg to json conversion. The log is:

jupyter-nbconvert-3.6 --to pdf notebook.ipynb

[NbConvertApp] Converting notebook ProbeArmResults.ipynb to pdf
Traceback (most recent call last):
  File "/opt/local/bin/jupyter-nbconvert-3.6", line 11, in <module>
    load_entry_point('nbconvert==5.4.0', 'console_scripts', 'jupyter-nbconvert')()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/jupyter_core/application.py", line 266, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/traitlets/config/application.py", line 658, in launch_instance
    app.start()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/nbconvert/nbconvertapp.py", line 337, in start
    self.convert_notebooks()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/nbconvert/nbconvertapp.py", line 507, in convert_notebooks
    self.convert_single_notebook(notebook_filename)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/nbconvert/nbconvertapp.py", line 478, in convert_single_notebook
    output, resources = self.export_single_notebook(notebook_filename, resources, input_buffer=input_buffer)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/nbconvert/nbconvertapp.py", line 407, in export_single_notebook
    output, resources = self.exporter.from_filename(notebook_filename, resources=resources)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/nbconvert/exporters/exporter.py", line 178, in from_filename
    return self.from_file(f, resources=resources, **kw)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/nbconvert/exporters/exporter.py", line 196, in from_file
    return self.from_notebook_node(nbformat.read(file_stream, as_version=4), resources=resources, **kw)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/nbconvert/exporters/pdf.py", line 158, in from_notebook_node
    nb, resources=resources, **kw
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/nbconvert/exporters/latex.py", line 83, in from_notebook_node
    return super(LatexExporter, self).from_notebook_node(nb, resources, **kw)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/nbconvert/exporters/templateexporter.py", line 300, in from_notebook_node
    nb_copy, resources = super(TemplateExporter, self).from_notebook_node(nb, resources, **kw)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/nbconvert/exporters/exporter.py", line 138, in from_notebook_node
    nb_copy, resources = self._preprocess(nb_copy, resources)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/nbconvert/exporters/exporter.py", line 315, in _preprocess
    nbc, resc = preprocessor(nbc, resc)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/nbconvert/preprocessors/base.py", line 47, in __call__
    return self.preprocess(nb, resources)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/nbconvert/preprocessors/base.py", line 69, in preprocess
    nb.cells[index], resources = self.preprocess_cell(cell, resources, index)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/nbconvert/preprocessors/extractoutput.py", line 92, in preprocess_cell
    data = json.dumps(data)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/encoder.py", line 180, in default
    o.__class__.__name__)
TypeError: Object of type 'bytes' is not JSON serializable
@sgsaenger
Copy link

sgsaenger commented Nov 6, 2018 •
edited
Loading

There's something really strange going on.
#847 introduced

if (                                                          
    not isinstance(data, text_type)                           
    or mime_type == 'application/json'                        
):                                                            
    # Data is either JSON-like and was parsed into a Python   
    # object according to the spec, or data is for sure       
    # JSON. In the latter case we want to go extra sure that  
    # we enclose a scalar string value into extra quotes by   
    # serializing it properly.                                
    data = json.dumps(data)

assuming all non-string data to be otherwise valid json.
I think this assumption is way too strong, as can be seen here:

  • When using the pdf-backend, the resulting pdf will be stored as a string in the cell.outputs, and will thus pass the not isinstance(data, text_type) check
  • When using the svg-backend, some other part of the code will convert the svg to pdf and store it as bytes, thus not passing the not isinstance(data, text_type) check and triggering the error

I don't know anything about json-data embedded in cells, but i don't think it's reasonable to assume all non-string data should be piped through json.dumps

@sgsaenger sgsaenger mentioned this issue Nov 6, 2018
Open
@sdpython sdpython mentioned this issue Dec 18, 2018
Closed
@GorgiAstro GorgiAstro mentioned this issue Mar 25, 2019
Merged
1 task
@SpencerPark
Copy link

This does actually seem to be in svg2pdf. It is injecting invalid json into a notebook cell outputs. It just manifests itself as an issue with extractoutput.

It uses base64.encodestring which (despite the name) returns a bytes object.

nbconvert/nbconvert/preprocessors/svg2pdf.py

Line 100 in 9a22426

return base64.encodestring(f.read())

@SpencerPark SpencerPark mentioned this issue May 14, 2019
Closed
@MSeal
Copy link
Contributor

MSeal commented May 14, 2019

This should be fixed in the nbconvert 5.5 release already. I believe this is supposed to be returning bytes and not a unicode string.

@MSeal MSeal closed this as completed May 14, 2019
@rpgoldman
Copy link

rpgoldman commented May 24, 2019 •
edited
Loading

@MSeal

This should be fixed in the nbconvert 5.5 release already. I believe this is supposed to be returning bytes and not a unicode string.

FWIW, I am running with nbconvert 5.5.0 and I just got this error a few minutes ago trying to use the command to download as PDF through latex:

[E 12:18:44.434 NotebookApp] 500 GET /nbconvert/pdf/Continuous%20model.ipynb?download=true (::1) 1516.76ms referer=http://localhost:8888/notebooks/Continuous%20model.ipynb
[E 12:19:03.774 NotebookApp] nbconvert failed: Object of type bytes is not JSON serializable
    Traceback (most recent call last):
      File "/Users/rpg/Library/Python/3.7/lib/python/site-packages/notebook/nbconvert/handlers.py", line 130, in get
        resources=resource_dict
      File "/Users/rpg/Library/Python/3.7/lib/python/site-packages/nbconvert/exporters/pdf.py", line 158, in from_notebook_node
        nb, resources=resources, **kw
      File "/Users/rpg/Library/Python/3.7/lib/python/site-packages/nbconvert/exporters/latex.py", line 84, in from_notebook_node
        return super(LatexExporter, self).from_notebook_node(nb, resources, **kw)
      File "/Users/rpg/Library/Python/3.7/lib/python/site-packages/nbconvert/exporters/templateexporter.py", line 300, in from_notebook_node
        nb_copy, resources = super(TemplateExporter, self).from_notebook_node(nb, resources, **kw)
      File "/Users/rpg/Library/Python/3.7/lib/python/site-packages/nbconvert/exporters/exporter.py", line 138, in from_notebook_node
        nb_copy, resources = self._preprocess(nb_copy, resources)
      File "/Users/rpg/Library/Python/3.7/lib/python/site-packages/nbconvert/exporters/exporter.py", line 315, in _preprocess
        nbc, resc = preprocessor(nbc, resc)
      File "/Users/rpg/Library/Python/3.7/lib/python/site-packages/nbconvert/preprocessors/base.py", line 47, in __call__
        return self.preprocess(nb, resources)
      File "/Users/rpg/Library/Python/3.7/lib/python/site-packages/nbconvert/preprocessors/base.py", line 69, in preprocess
        nb.cells[index], resources = self.preprocess_cell(cell, resources, index)
      File "/Users/rpg/Library/Python/3.7/lib/python/site-packages/nbconvert/preprocessors/extractoutput.py", line 92, in preprocess_cell
        data = json.dumps(data)
      File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/__init__.py", line 231, in dumps
        return _default_encoder.encode(obj)
      File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/encoder.py", line 199, in encode
        chunks = self.iterencode(o, _one_shot=True)
      File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/encoder.py", line 257, in iterencode
        return _iterencode(o, 0)
      File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/encoder.py", line 179, in default
        raise TypeError(f'Object of type {o.__class__.__name__} '
    TypeError: Object of type bytes is not JSON serializable
[W 12:19:03.775 NotebookApp] 500 GET /nbconvert/pdf/Continuous%20model.ipynb?download=true (::1): nbconvert failed: Object of type bytes is not JSON serializable
[E 12:19:03.776 NotebookApp] {
      "Host": "localhost:8888",
      "Connection": "keep-alive",
      "Upgrade-Insecure-Requests": "1",
      "Dnt": "1",
      "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36",
      "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3",
      "Referer": "http://localhost:8888/notebooks/Continuous%20model.ipynb",
      "Accept-Encoding": "gzip, deflate, br",
      "Accept-Language": "en-US,en;q=0.9",
      "Cookie": "username-localhost-8889=\"2|1:0|10:1557094549|23:username-localhost-8889|44:MTlkNjYxMmI5ZDdkNGJkYWFlMzJmYzcxMmM1YTUzNzI=|e07c4f8ee41d84dc25632f038c8dae0255876174ee09ae08e42cf547255e593d\"; username-localhost-8890=\"2|1:0|10:1557266878|23:username-localhost-8890|44:YjhkODllOWU2YzYwNGVlYzhmMzZjNjFiNzU1ZjRjMGQ=|b66a62bc2ccb2f3ee8535897c014aa6e4a9fa7d76b4070bcc70d60fd512593f0\"; username-localhost-8888=\"2|1:0|10:1558122572|23:username-localhost-8888|44:YjE2NTlhZDQ2NTJiNDc1MWJlMmI2ZmQ4MjY4NzM3OTI=|00edcbb85aacc7114ec06cb67cdba903843c640ad6e4831008eed32163e0c550\"; _xsrf=2|4b836cd5|6530c5b3448667e1abd65d4d9aa83125|1558716658"
    }

@rpgoldman
Copy link

Update: this occurred for a notebook with an embedded graphic generated by graphviz. When I cleared the cell containing that graphic, the error went away. I'm afraid I don't know what graphviz puts in an image in a notebook, so I don't have any idea how to repair this.

@MSeal
Copy link
Contributor

MSeal commented May 25, 2019

Can you post the minimal notebook to reproduce? (might have to change the extension to post on the issue)

@rpgoldman
Copy link

rpgoldman commented May 27, 2019 •
edited
Loading

@MSeal I'm attaching one that is minimal, but not runnable -- it has a huge dependency tail. But you should be able to open it and test the download as PDF, and if you see the same error as I do, should be able to see what it is in the file that triggers it (and reopen this issue?).
If that isn't enough to diagnose the issue, let me know further, and I will try to get something more minimal. It's tricky, though, because I think it's a graphviz issue and I don't ever use graphviz directly, only through PyMC3.
nbconvert-graphviz-out-error.ipynb.txt

@SpencerPark
Copy link

SpencerPark commented May 27, 2019 •
edited
Loading

I was able to convert your example notebook @rpgoldman, maybe double check that you are indeed using nbconvert 5.5.0? I used

$ pip show nbconvert
Name: nbconvert
Version: 5.5.0
...

It's possible that you have 5.5.0 installed but the server is not picking that one up. Double check the nbconvert version in the shell you are using to run the jupyter server.

I ran jupyter nbconvert nbconvert-graphviz-out-error.ipynb --to latex and then pdflatex nbconvert-graphviz-out-error.tex -interaction=nonstopmode to get the pdf.

@rpgoldman
Copy link

@SpencerPark Sorry -- you were quite right: I was having a problem with my virtual environment so that the nbconvert that I was looking at, and the one that was being run by jupyter were two different things. Thanks for testing that.

I was able to convert your example notebook @rpgoldman, maybe double check that you are indeed using nbconvert 5.5.0? I used

$ pip show nbconvert
Name: nbconvert
Version: 5.5.0
...

It's possible that you have 5.5.0 installed but the server is not picking that one up. Double check the nbconvert version in the shell you are using to run the jupyter server.

I ran jupyter nbconvert nbconvert-graphviz-out-error.ipynb --to latex and then pdflatex nbconvert-graphviz-out-error.tex -interaction=nonstopmode to get the pdf.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
5 participants
@rpgoldman @jmgc @MSeal @sgsaenger @SpencerPark