Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Performance when reading TIFF file with small tiles #8607

Open
zzjjbb opened this issue Dec 17, 2024 · 3 comments · May be fixed by #8609
Open

Performance when reading TIFF file with small tiles #8607

zzjjbb opened this issue Dec 17, 2024 · 3 comments · May be fixed by #8609
Labels
Performance TIFF

Comments

@zzjjbb
Copy link

zzjjbb commented Dec 17, 2024 •
edited
Loading

What did you do?

I use the method Image.open(tiff_filename).load() to read a tiff file from a SMB server.

What did you expect to happen?

25MB data is read from the server during reading.

What actually happened?

It is very slow and downloads hundreds megabytes of data.
Currently, I change to reading the content into memory first and adding a BytesIO wrapper.
raw = f.read(); Image.open(BytesIO(raw)).load()
This can gives me the same result and the speed and network traffic is expected.

My findings

The tiff file is a ~5000x5000 8bit image with no compression. The library first loads the header and gets the data of ~5000 tiles. As the offsets in the tiles list show, each tile has 5KB size. However, pillow reads a much larger size for each tile, which is a const MAXBLOCK = 65536 (bytes) = ~65KB.

Pillow/src/PIL/ImageFile.py

Line 280 in 76919ad

s = read(self.decodermaxblock)

What are your OS, Python and Pillow versions?

  • OS: Windows
  • Python: 3.12.7
  • Pillow: 10.2.0
--------------------------------------------------------------------
Pillow 10.2.0
Python 3.12.7 | packaged by Anaconda, Inc. | (main, Oct  4 2024, 13:17:27) [MSC v.1929 64 bit (AMD64)]
--------------------------------------------------------------------
Python modules loaded from ......\site-packages\PIL
Binary modules loaded from ......\site-packages\PIL
--------------------------------------------------------------------
--- PIL CORE support ok, compiled for 10.2.0
--- TKINTER support ok, loaded 8.6
--- FREETYPE2 support ok, loaded 2.13.2
--- LITTLECMS2 support ok, loaded 2.16
--- WEBP support ok, loaded 1.3.2
--- WEBP Transparency support ok
--- WEBPMUX support ok
--- WEBP Animation support ok
--- JPEG support ok, compiled for libjpeg-turbo 3.0.1
--- OPENJPEG (JPEG2000) support ok, loaded 2.5.0
--- ZLIB (PNG/ZIP) support ok, loaded 1.3
--- LIBTIFF support ok, loaded 4.6.0
*** RAQM (Bidirectional Text) support not installed
*** LIBIMAGEQUANT (Quantization method) support not installed
*** XCB (X protocol) support not installed
--------------------------------------------------------------------
BLP
Extensions: .blp
Features: open, save, encode
--------------------------------------------------------------------
BMP image/bmp
Extensions: .bmp
Features: open, save
--------------------------------------------------------------------
BUFR
Extensions: .bufr
Features: open, save
--------------------------------------------------------------------
CUR
Extensions: .cur
Features: open
--------------------------------------------------------------------
DCX
Extensions: .dcx
Features: open
--------------------------------------------------------------------
DDS
Extensions: .dds
Features: open, save
--------------------------------------------------------------------
DIB image/bmp
Extensions: .dib
Features: open, save
--------------------------------------------------------------------
EPS application/postscript
Extensions: .eps, .ps
Features: open, save
--------------------------------------------------------------------
FITS
Extensions: .fit, .fits
Features: open
--------------------------------------------------------------------
FLI
Extensions: .flc, .fli
Features: open
--------------------------------------------------------------------
FTEX
Extensions: .ftc, .ftu
Features: open
--------------------------------------------------------------------
GBR
Extensions: .gbr
Features: open
--------------------------------------------------------------------
GIF image/gif
Extensions: .gif
Features: open, save, save_all
--------------------------------------------------------------------
GRIB
Extensions: .grib
Features: open, save
--------------------------------------------------------------------
HDF5
Extensions: .h5, .hdf
Features: open, save
--------------------------------------------------------------------
ICNS image/icns
Extensions: .icns
Features: open, save
--------------------------------------------------------------------
ICO image/x-icon
Extensions: .ico
Features: open, save
--------------------------------------------------------------------
IM
Extensions: .im
Features: open, save
--------------------------------------------------------------------
IMT
Features: open
--------------------------------------------------------------------
IPTC
Extensions: .iim
Features: open
--------------------------------------------------------------------
JPEG image/jpeg
Extensions: .jfif, .jpe, .jpeg, .jpg
Features: open, save
--------------------------------------------------------------------
JPEG2000 image/jp2
Extensions: .j2c, .j2k, .jp2, .jpc, .jpf, .jpx
Features: open, save
--------------------------------------------------------------------
MCIDAS
Features: open
--------------------------------------------------------------------
MPEG video/mpeg
Extensions: .mpeg, .mpg
Features: open
--------------------------------------------------------------------
MSP
Extensions: .msp
Features: open, save, decode
--------------------------------------------------------------------
PCD
Extensions: .pcd
Features: open
--------------------------------------------------------------------
PCX image/x-pcx
Extensions: .pcx
Features: open, save
--------------------------------------------------------------------
PIXAR
Extensions: .pxr
Features: open
--------------------------------------------------------------------
PNG image/png
Extensions: .apng, .png
Features: open, save, save_all
--------------------------------------------------------------------
PPM image/x-portable-anymap
Extensions: .pbm, .pgm, .pnm, .ppm
Features: open, save
--------------------------------------------------------------------
PSD image/vnd.adobe.photoshop
Extensions: .psd
Features: open
--------------------------------------------------------------------
QOI
Extensions: .qoi
Features: open
--------------------------------------------------------------------
SGI image/sgi
Extensions: .bw, .rgb, .rgba, .sgi
Features: open, save
--------------------------------------------------------------------
SPIDER
Features: open, save
--------------------------------------------------------------------
SUN
Extensions: .ras
Features: open
--------------------------------------------------------------------
TGA image/x-tga
Extensions: .icb, .tga, .vda, .vst
Features: open, save
--------------------------------------------------------------------
TIFF image/tiff
Extensions: .tif, .tiff
Features: open, save, save_all
--------------------------------------------------------------------
WEBP image/webp
Extensions: .webp
Features: open, save, save_all
--------------------------------------------------------------------
WMF
Extensions: .emf, .wmf
Features: open, save
--------------------------------------------------------------------
XBM image/xbm
Extensions: .xbm
Features: open, save
--------------------------------------------------------------------
XPM image/xpm
Extensions: .xpm
Features: open
--------------------------------------------------------------------
XVTHUMB
Features: open
--------------------------------------------------------------------
@radarhere radarhere added the TIFF label Dec 17, 2024
@radarhere
Copy link
Member

Are you able to provide a copy of the file?

@zzjjbb
Copy link
Author

zzjjbb commented Dec 18, 2024

Are you able to provide a copy of the file?

Thanks for the reminder! I should have it uploaded at first...
IMG07.zip
These tiff files are the output of some industrial camera and the one I upload here is simply some noise. I need to zip it since GitHub do not like tiff files in comment.

@radarhere
Copy link
Member

radarhere commented Dec 19, 2024 •
edited
Loading

Thanks.

So, you're aware that you could work around this by reading the file into memory first.
And you have probably also realised that you could

from PIL import ImageFile
ImageFile.MAXBLOCK = 1024

to reduce the amount of data read in each block.

But instead, you're hoping that Pillow by default will perform smaller read operations.

We could make an assumption that the data for each tile will stop by the start of the next tile. Does #8609 solve your problem?

@radarhere radarhere changed the title Performance issue of reading TIFF file with small tiles Performance when reading TIFF file with small tiles Dec 19, 2024
@radarhere radarhere linked a pull request Dec 19, 2024 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Performance TIFF
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Only read until the offset of the next tile radarhere/Pillow
2 participants
@radarhere @zzjjbb