Feature request: Dewarp before selecting content

[Mister-Teatime]

I'm digitizing books which I'd rather not press flat against a scanner, which are much quicker digitized by means of setting up decent light, lying them flat on the floor and pointing a good camera straight down at them.

This causes quite a bit of warping (particularly with softcovers), which the dewarping algorithm deals with reasonably well.

However, I have to select the pages, their content and in some cases pictures beforehand, and Scantailor assumes that most of these things were rectangular (which they only become afterwards ...). For image maps, I can use polygons instead, for content selection I can adjust content boxes after the fact (iteratively...), but these workarounds take significantly more time than if the contents were nicely aligned, and sometimes the box around content will have corners off-page, despite ample page margins

This often makes me wish I could reverse the order of those steps, and do the dewarping first.
I'm not familiar with the code architecture (yet...?) but I would hope that it must be possible to change the order of processing steps to put dewarping before the content selection stage.
I would imagine that the automatic content detection should also have an easier time with dewarped inputs.

Proposed implementation:

I'm not familiar with Scantailor's code structure, so I can only comment on what would make sense from a user's perspective -- so I accept that some of the following may not be practically realistic, or that there are smarter/faster ways to get a working implementation.

My favourite implementation would be to bundle the dewarping with the deskewing step. It seems to me that the dewarp tool would be able to replace the "deskew" (i.e. rotation) function, as it can represent rotation, too. This might be overkill for users who really just need to rotate the contents a little, but since dewarping is already optional, it could stay optional, either following the rotation step, or optionally replacing it. ("deskew method: nothing | rotate | dewarp")

[zvezdochiot]

@Mister-Teatime say:

deskew method: nothing | rotate | dewarp

Fixed: transform method: nothing [+ deskew [+ deperspective [+ dewarp ]]]

See also https://github.com/tibob/yasw , https://github.com/ImageProcessing-ElectronicPublications/yasw , 4lex4/scantailor-advanced#85 , https://github.com/scantailor/scantailor/issues/317 .

[Mister-Teatime]

nothing | rotate | deperspective | (deperspective+dewarp)

If I get you right, you're suggesting to split the current dewarp tool into two components? That sounds nice to me. It kinda does that already, when manually picking four corners and not adding control points, but I'd quite like the ability to (for example) turn the additional control points on/off (by switching between deperspective and dewarp modes), or picking the page corners by hand, then but leaving dewarp on auto so it can find the control points to make the control cage match the paper edge.
... but I think that would be a separate feature request, and possibly a lot more work.
I'd be quite happy for now if dewarping, as it works at the moment, could happen before content selection, page layout, picture zone selection etc..

[mara004]

I would also consider this very useful, as content could then be selected on the dewarped picture (otherwise you have to estimate how your image will look). It would just make a lot more sense. I tried the experimental branch from the original author (https://github.com/Tulon/scantailor/tree/experimental) which already has dewarp before content selection. Could it be ported to ST Universal?

[trufanov-nok]

Hi,
Let's say it's not in a top 5 of features that I would like to implement right now. I'm not using dewarping in ST at all. It seems to need some work on it and probably Tulon have made it better in his Experimental, but warping still doesn't feel to be finalized. Also I don't expect I can just easely port it. Tulon powered Experimental with Eigen lib and OpenCL which support lacks the other forks. Also it will be a pain to integrate new controls into existing UI/Settings/Hotkeys etc of Universal's GUI. So, unless anyone would like to help me by contributing to code...

Currently I'm leisurely working on 2 other features: ellipse zones (among polylines and rects) and brand new last processing stage that will automatically encode output images into bundled DjVu doc using a so-called "Separated scans method" with minidjvu_mod.exe as an encoder.

[zvezdochiot]

Hi @trufanov-nok .

It would be nice to create a separate utility combining djvu created by minidjvu (BW) and djvu created by c44 and djvm (illustrations).

[trufanov-nok]

Hi, @zvezdochiot
That's doable but lets discuss this when at least ST integrated version will be ready.

[mara004]

@trufanov-nok I understand you don't want to spend time on dewarp if you don't have a use for it yourself, but I think it's a very nice feature and for me, it's the main/only reason to use ScanTailor. I'm digitalizing books, too, and the warping / wrong geometry that I get makes the pages look bad and difficult to read. If I didn't use dewarp, I would have the choice to either remove page content or include a lot of unneeded space in the end result.
I'm not a programmer, but I wonder what involves so much difficulty to just move dewarp from the output stage to before the content selection? That would be quite sufficient for me.
Ellipse zones are a nice feature.
I don't really like djvu, because
a) most related software seems to be outdated/unmaintained (e. g. the only free/libre colour layer segmentation program still is python2 only)
b) even if it may be the superior format regarding image encodings, djvu is a very uncommon, which makes it unsuitable for sharing

[zvezdochiot]

@mara004 say:

e. g. the only free/libre colour layer segmentation program still is python2 only

Do you mean https://github.com/jwilk/didjvu ?

But http://djvu.sourceforge.net/ !:

man djvumake

          cjb2 mymask.pbm mymask.djvu
          djvumake my.djvu Sjbz=mymask.djvu PPM=myimage.ppm

[mara004]

It seems to need some work on it and probably Tulon have made it better in his Experimental, but warping still doesn't feel to be finalized.

Hmm, for me, dewarping always worked very well, except for the problem that it's at the wrong point. (However, I agree that automatic dewarping fails a lot - one has to adjust it manually to achieve good results)

Also I don't expect I can just easely port it. Tulon powered Experimental with Eigen lib and OpenCL which support lacks the other forks.
Also it will be a pain to integrate new controls into existing UI/Settings/Hotkeys etc of Universal's GUI. So, unless anyone would like to help me by contributing to code...

Okay, I see.
However, it wouldn't need to be exactly as it is in experimental - only moving the existing dewarp from the tab in output to a new stage before content selection would be fine for me. Everything else could stay as it is.

[Mister-Teatime]

I'd like to echo what mara said: Although I'd like to have a look at Tulon's new openCL-based dewarp, my request really is "simply" to move the dewarp tool from its current place to before content selection. I've tried and failed a few times to compile the experimental branch on my machine, but I've been working in Python only for so long, and compiling code is a distant memory to me, so I don't really know what I'm doing...

I know that there may be a lot more consequences to changing the order of execution than moving GUI elements, of course, but on the GUI side, I'd imagine that the "dewarp" tab would disappear from the output view and show up in the "Deskew" view, and then all following steps would be performed on the dewarped (or just rotated) version of the scan. There's any number of things which could be updated and improved on the deskewing and dewarping process, but really, this request is only about having at an earlier stage.
I know that there may be a lot more consequences to changing the order of execution than moving GUI elements, of course, so I'm not in a position know how much work this would be...

[mara004]

@Mister-Teatime Sorry for my late response, but if you want to, I can try to help you compile ScanTailor Experimental (on Linux). I too had never done any c++ compiling before and the ScanTailor branches were one of my very first tries. It's actually not that difficult. The only thing which can be a bit tough is to get all dependencies (in particular for OpenCL it depends on your GPU vendor which packages you need to install). I assume you are using a Linux distribution? In case you are on Windows, you don't necessarily need to compile it yourself, you can just use a pre-release binary from https://github.com/Tulon/scantailor/releases
My experience with ST Experimental is somewhat mixed, because on one hand, it's very fast and it has dewarping at the right stage, but on the other hand, it's a little unreliable at times and you should to be prepared for crashes. Sometimes, thumbnails might also not render correctly and dewarping has issues (for instance, when you drag a point, the blue lines might jump around). Overall, I personally find the problems rather minor, though. For pure geometry correction I believe Experimental is the best ST branch. It's a pity nobody maintains and improves it anymore.

[zvezdochiot]

@mara004 say:

dewarping has issues

This is because a flexible mesh is used. A local affine transformation of quadrangles into rectangles would be used, it would be more reliable and simpler.

[mara004]

@Mister-Teatime Another problem with building ST Experimental that I forgot to mention above can be missing QPainterPath includes at some source files. With newer Qt (I believe at least since 5.15), you need to explicitly include it. Previously, that wasn't mandatory. If you get error messages from the compiler related to QPainterPath, then just try adding #include <QPainterPath> to the problematic source files, that should fix it.

[mara004]

@zvezdochiot Okay, interesting. However, I only experienced these dewarping problems in ST Experimental but not in ST Advanced

[Mister-Teatime]

@mara004 Thanks a bunch for the good advice! I did take some time to get back to dealing with Scantailor, but I did, eventually.
Yes, I am using Linux (mostly), although my main machine (SUSE) has recently bitten the dust, so now I'm on a Manjaro laptop, which is a bit less comfortable for compiling work (although that may be due to me using SUSE for almost 20 years now...).

Anyway, turned out I had to install Eigen first, and adding the QPainterPathinclude to two of the .cpp files dealt with the error messages about it. But now I'm getting an error message about the "incorrect use of incomplete type »class QButtonGroup«".
I suspect that this might be because my Qt version is a few years newer than the latest commit to the experimental branch. Have you come across this issue (and if yes, how did you solve it?). It might be the smartest thing to build with an older version of Qt, but I'm unfortunately not sure how I'd even go about doing that ...
Also interesting: the error occurs in filters/page_split/OptionsWidget.cpp and is repeated for lines 49 through 51, which are commented in the source as "workaround for QTBUG-182". That seems to refer to this issue which has not been fixed so far. Unfortunately, I don't think I understand what the code is trying to do there and why that may not work (and why that would have been okay when you compiled it...)

... any smart ideas?

[trufanov-nok]

@Mister-Teatime try to add #include <QButtonGroup>

[Mister-Teatime]

Oh, that was an easy fix! Thanks a lot for the hint! I would not expected that to work, but it did.

The Experimental branch just finished compiling and is running!
Tested it with some (barely warped) pictures, and it's working great.
(I do miss some of the Universal features, though...)
The automatic dewarping is sometimes guessing very wrong, but in most cases I have to adjust manually anyway, so that's not such a problem.

I used that as an opportunity to get accustomed to Git (as a long-time TortoiseHG user). I wonder if I could submit the new includes as a patch/pull request or something, to spare others the trouble of doing that manually ... but I won't have time to work that out before next weekend, I think.

[zvezdochiot]

Hi @trufanov-nok .

Simple dewarping:

• photoquick-examples/tree/main/main/transform/geometry/dewarping
• photoquick-examples/tree/main/main/transform/geometry/un-tilt

[trufanov-nok]

Hi @zvezdochiot
Thanks, I'll check it in a few days.

[mara004]

Simple dewarping:

I viewed the examples, but this does not seem to be what I'm looking for.
I don't think this can compete with ScanTailor's flexible mesh dewarping.

[zvezdochiot]

@mara004 say:

I viewed the examples, but this does not seem to be what I'm looking for.

What you are looking for is implemented in STEX. But for this, @Tulon had to throw a lot out of ST.

[mara004]

What you are looking for is implemented in STEX. But for this, @Tulon had to throw a lot out of ST.

Yeah, I know. STE is the variant I'm using ATM (see above).

For clarity, the removal of some other features is not logically caused by third-stage dewarping, I think it's just that this improvement was committed amidst a deep technical refactoring (eigen, opencl).
When refactoring code, I believe it's common practice to initially remove features to simplify things, only it's a pity STE development stopped in this beta state.

[mara004]

Question though: How does https://github.com/ImageProcessing-ElectronicPublications/scantailor-experimental (A) differ from @Tulon's https://github.com/Tulon/scantailor/tree/experimental (B) ?
I see that (A) is not a GH fork of (B). History and directory layout don't match. Who authored this for which reason, and are there more changes?

[zvezdochiot]

@mara004 say:

are there more changes?

@Tulon STEX (not to be confused with STE) has been out of business for a long time. I took his code, shoved it on the shelves, added the missing includes. As a result, the code is built under Qt5.

[mara004]

Yes, I mean ScanTailor Experimental, not Enhanced.

I took his code, shoved it on the shelves, added the missing includes. As a result, the code is built under Qt5.

Thanks. FYI, I had made a similar private fork/patch some time ago, yours may be better though:
https://github.com/mara004/scantailor/commits/build-experimental
mara004/scantailor@237616f