Add initial SpatiaLite ↔ PostGIS support

[mkgrgis]

+

• PostGIS is the most popular PostgreSQL extension. PostGIS provides many spatial functions and fully compatible with GEOS bytea data formats.
• SpatiaLite is SQLite extension with spatial functions based on original storage model with blob affinity.

No need to add here PostGIS header files, because SpatiaLite has functions for EWKB data transformation.

In this PR:

• Add libspatialite-dev for compilation and tests.
• Add preprocessor define SQLITE_FDW_GIS_ENABLE variable which can be not defined for compilation without spatial data types support.
• PostGIS data types can not be identified by Oid, only by names. Hence bind and convert functions in case of unknown Oids should analyze name of data type.
• Some of complex PostGIS data types intentionally denied by name for dealing with sqlite_fdw because of no similar or translatable conceptions in SpatiaLite.
• Add SRID check during PostGIS→SpatiaLite EWKT transformation. All SpatiaLite spatial data must have SRID according official documentation.
• Add PostgreSQL working with both bytea domains having listed names and real PostGIS data types. If there is no PostGIS SpatiaLite spatial data will transformed to bytea data, binary identical with usual PostGIS data format.
• Some function comments and naming style in sqlite_query.c are improved.
• IMPORT FOREIGN SCHEMA converts formal geometry and geography to similar names if the FDW was compiled with GIS support, otherwise this columns will be bytea in PostgreSQL.
• Improve some details in README.md.
• Improve some debug messages.
• Testcases based or real spatial data near Olympus Mons on ♂ and Ancienne Sylvie/Vieille Sylvie for ♁.

[mkgrgis]

@t-kataym . I am ready to discussing about this PR and testing environment setup. Look like the problems is near Ubuntu libspatialite-dev. Can pgspider team help me with CI testing in this PR?

[son-phamngoc]

Hello @mkgrgis, thank you for your effort.

In my understanding, if we want to enable PostGIS to be able to use its data types, we need to CREATE EXTENSION PostGIS.
However, in your test file (postgis.sql), you do not create it. You just create a data type which is bytea and has the same name as PostGIS.

--Testcase 01:
CREATE DOMAIN geometry AS bytea;
--Testcase 02:
CREATE DOMAIN geography AS bytea;

Is there any reason for this creation? Why didn't you enable PostGIS by creating extension PostGIS?
I also do not see any step to install PostGIS in your PR.

FYI, I tried to enable PostGIS by creating extension, executed your test file and got several error.
I think it does not work well with PostGIS extension.

CREATE EXTENSION postgis;
CREATE DOMAIN raster AS bytea;
--Testcase 31: ERR - raster
INSERT INTO "types_PostGIS" ( "i", gm, gg, r, t ) VALUES (1, decode('0101000020e6100000fd5aa846f9733e406c054d4bacd74d40', 'hex'),  decode('0101000020e6100000fd5aa846f9733e406c054d4bacd74d40', 'hex'),  decode('1223456890', 'hex'), '{"genus": "Rhododendron", "taxon": "Rhododendron ledebourii", "natural": "shrub", "genus:ru": "Рододендрон", "taxon:ru": "Рододендрон Ледебура", "source:taxon": "board"}');
ERROR:  Transformation between GEOS/PostGIS has failed for a value with PostgreSQL data type "public.geometry"
HINT:  In column "gm" incorrect value in 28 bytes

--Testcase 37: OK
SELECT "i", gm, gg, t FROM "types_PostGIS";
ERROR:  Unknown geometry type: 1107488 - Invalid type

--Testcase 38: OK
INSERT INTO "types_PostGIS" ( "i", gm, gg, t ) VALUES (2, decode('0101000020e6100000bf72ce99fe763e40ed4960730ed84d40', 'hex'),  decode('0101000020e6100000bf72ce99fe763e40ed4960730ed84d40', 'hex'), '{"genus": "Rhododendron", "taxon": "Rhododendron ledebourii"}');
ERROR:  Transformation between GEOS/PostGIS has failed for a value with PostgreSQL data type "public.geometry"
HINT:  In column "gm" incorrect value in 28 bytes

[mkgrgis]

Hello, @son-phamngoc ! Thanks for code and tests reading!

In my understanding, if we want to enable PostGIS to be able to use its data types, we need to CREATE EXTENSION PostGIS.

No. My idea of initial experimental geospatial data types support based on bytea representation of PostGIS objects. This means only this blob-affinity values should have sense in PostGIS, should be well-formed. But we don't need to test this.

I also do not see any step to install PostGIS in your PR.

There are some reasons:

1. PostGIS based on GEOS binary or bytea formats. We shouldn't test some PostGIS functions. We can only once manually ensure created binary object is well formed by GEOS and use this result image for tests of SpatiaLite outputs. In my case DBeaver shows normal PostGIS content for a column later changed to PostGIS geometry type.

Is there any reason for this creation? Why didn't you enable PostGIS by creating extension PostGIS?

2. PostGIS is not a part of testing environment. We can do imitation of existing PostGIS - some special data types with domain form and listed names. Our FDW based only on BLOB format and column type name. No PostGIS functions are used, no needed for tests. All tested functions belongs to SpatiaLite.
3. There is no raster support. Any error in case of showed TC 31 is normal. If you want try this PR code with PostGIS in normal style, please comment creating of domains with special names, use automatically created internal PostGIS data types instead. For our FDW this should have the same effect.
4. This PR doesn't implement detailed malformed data diagnostics.

[son-phamngoc]

@mkgrgis Thanks for your answering.

If you want try this PR code with PostGIS in normal style, please comment creating of domains with special names, use automatically created internal PostGIS data types instead. For our FDW this should have the same effect.

I did. I commented out the SQL to create domain, create PostGIS extension, and then got the error in #96 (comment)
I understood that the result of TC31 is normal, but how about TC37 and TC38?
They should be OK, but they returned error instead.
Could you explain more to me?

CREATE EXTENSION postgis;
-- --Testcase 01:
-- CREATE DOMAIN geometry AS bytea;
-- --Testcase 02:
-- CREATE DOMAIN geography AS bytea;
-- --Testcase 03:
-- CREATE DOMAIN addbandarg AS bytea;
-- --Testcase 04:
-- CREATE DOMAIN box2d AS bytea;
-- --Testcase 05:
-- CREATE DOMAIN box3d AS bytea;
-- --Testcase 06:
-- CREATE DOMAIN geometry_dump AS bytea;
-- --Testcase 07:
-- CREATE DOMAIN geomval AS bytea;
-- --Testcase 08:
-- CREATE DOMAIN getfaceedges_returntype AS bytea;
-- --Testcase 09:
-- CREATE DOMAIN rastbandarg AS bytea;
--Testcase 10:
CREATE DOMAIN raster AS bytea;
-- --Testcase 11:
-- CREATE DOMAIN reclassarg AS bytea;
-- --Testcase 12:
-- CREATE DOMAIN summarystats AS bytea;
-- --Testcase 13:
-- CREATE DOMAIN topoelement AS bytea;
-- --Testcase 14:
-- CREATE DOMAIN topoelementarray AS bytea;
-- --Testcase 15:
-- CREATE DOMAIN topogeometry AS bytea;
-- --Testcase 16:
-- CREATE DOMAIN unionarg AS bytea;
-- --Testcase 17:
-- CREATE DOMAIN validatetopology_returntype AS bytea;
...
--Testcase 37: OK
SELECT "i", gm, gg, t FROM "types_PostGIS";
ERROR:  Unknown geometry type: 1107488 - Invalid type
--Testcase 38: OK
INSERT INTO "types_PostGIS" ( "i", gm, gg, t ) VALUES (2, decode('0101000020e6100000bf72ce99fe763e40ed4960730ed84d40', 'hex'),  decode('0101000020e6100000bf72ce99fe763e40ed4960730ed84d40', 'hex'), '{"genus": "Rhododendron", "taxon": "Rhododendron ledebourii"}');
ERROR:  Transformation between GEOS/PostGIS has failed for a value with PostgreSQL data type "public.geometry"
HINT:  In column "gm" incorrect value in 28 bytes

[mkgrgis]

I understood that the result of TC31 is error, but how about TC37 and TC38?
They should be OK, but they returned error instead.

Yes. What about SpatiaLite versions on your system, @son-phamngoc ? How did you get the files?
I use

• libspatialite-dev
• libspatialite7
• libsqlite3-mod-spatialite
  All this packages have 5.0.1-3 version. For our tests only libspatialite-dev with /usr/include/spatialite.h and /usr/include/spatialite/... is used.

[son-phamngoc]

What about SpatiaLite versions on your system, @son-phamngoc ? How did you get the files?

@mkgrgis I used libspatialite-devel version 5.0.0-1. I installed it by command dnf install libspatialite-devel.
This one, I think: https://rhel.pkgs.org/8/epel-x86_64/libspatialite-devel-5.0.0-1.el8.x86_64.rpm.html
FYI, about OS, I'm using Rocky Linux 8.8.

[mkgrgis]

FYI, about OS, I'm using Rocky Linux 8.8.

This should be normal environment. libspatialite-devel is also normal package. I am testing on Debian 12 with such result for the newest PostgreSQL version 16.0 output.txt

Your 5.0.0-1 SpatiaLite doesn't critically differs with my 5.0.1-3. I'll test with DBeaver visualisation and update this message, but no problems expected.

[mkgrgis]

Look like some problem is in listed_datatype for different contexts. In my case filter of data type names works good, in your case, @son-phamngoc , maybe not.

[son-phamngoc]

@mkgrgis
Did you try the same as me: CREATE EXTENSION PostGIS and comment out all domains with special names?

I think I found the problem.
My environment uses PostGIS 3. When printing the tuple, functions of PostGIS is called, and error occurred.

According to your description in README, this PR only works with PostGIS 2. Is that correct?
Could you tell me which functions is called when printtup with PostGIS 2? In my understanding, it should also call functions of PostGIS.

[mkgrgis]

@son-phamngoc , I think such PostGIS functions as geometry_in or geometry(bytea) and geography_in or geography(bytea) can be used in our case for input. geometry_out and geography_out functions are used for text output as usual. All of this functions stored in the schema of PostGIS(CREATE EXTENSION postgis SCHEMA...).
Main idea of my PR is using gaiaFromSpatiaLiteBlobWkbEx↔gaiaToSpatiaLiteBlobWkbEx2 from /libspatialite-5.1.0/src/gaiageo/gg_wkb.c in https://www.gaia-gis.it/gaia-sins/libspatialite-sources/libspatialite-5.1.0.tar.gz
Why I thinks we shouldn't test PostGIS functions? Because after my PR I have normal bytea data for PostGIS. Maybe I have forgotten some SQL conversion marker or context? In case of value::bytea there were no PostGIS 3.0 problems in my manual tests.

[son-phamngoc]

This FDW works with PostGIS 2+ and confirmed with SpatiaLite 5.1.

@mkgrgis
Could you tell me why this PR only works with PostGIS 2+? Is there any reason that it cannot work with PostGIS 3+?

[mkgrgis]

Could you tell me why this PR only works with PostGIS 2+? Is there any reason that it cannot work with PostGIS 3+?

@son-phamngoc, this PR should works also with PostGIS 3+. In PostGIS 2 was implemented current stable internal storage format EWKB. This storage format is used in all of PostGIS 2+ functions including all existed PostGIS 3+ versions.

Now I am trying to test context of using of the format. As data source I use https://osmcode.org/osmium-tool/ as
osmium export -f pg https://www.openstreetmap.org/api/0.6/node/8906857478;
this command gives me
0101000020E6100000FD5AA846F9733E406C054D4BACD74D40 {"genus":"Rhododendron","genus:ru":"Рододендрон","natural":"shrub","source:taxon":"board","taxon":"Rhododendron ledebourii","taxon:ru":"Рододендрон Ледебура"}
In this listing 0101000020E6100000FD5AA846F9733E406C054D4BACD74D40 is my test data.
select st_astext('0101000020E6100000FD5AA846F9733E406C054D4BACD74D40'::geometry) WKT shows POINT(30.4530224 59.6849455), no problems here. select '0101000020E6100000FD5AA846F9733E406C054D4BACD74D40'::geometry rh; is also decodable

I am really confused where is the problem.

[mkgrgis]

@son-phamngoc , some additional context

PostGIS for testing

• Source codes and version matrix https://trac.osgeo.org/postgis/wiki/UsersWikiPostgreSQLPostGIS gives us postgis-3.3.6.tar.gz for PostgreSQL versions from tests of this FDW.
• For Ubuntu environment PostGIS compiling conditions listed in sudo apt-get install libproj-dev libgeos-dev libxml2-dev gettext libjson-c-dev libgdal-dev libsfcgal-dev libprotobuf-c-dev protobuf-c-compiler; command.

SpatiaLite tests

For Ubuntu/Debian there is libsqlite3-mod-spatialite package, for RHEL/Rocky no additional package is needed after libspatialite, see SO question.

In my case SQLite listing with values from my PostGIS/SpatiaLite tests is also correct:

SQLite version 3.40.1 2022-12-28 14:03:47
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
sqlite> select load_extension('mod_spatialite');

sqlite> select ST_AsText(X'0001e6100000bf72ce99fe763e40ed4960730ed84d40bf72ce99fe763e40ed4960730ed84d407c01000000bf72ce99fe763e40ed4960730ed84d40fe');
POINT(30.464822 59.687941)

PostGIS shows some point near, POINT(30.4530224 59.6849455). You can decide about precision yourself.
PostGIS select st_geomfromewkb(decode ('0101000020e6100000bf72ce99fe763e40ed4960730ed84d40', 'hex')) shows the same point as Spatialite above, POINT(30.464822 59.687941).

[son-phamngoc]

Why I thinks we shouldn't test PostGIS functions? Because after my PR I have normal bytea data for PostGIS. Maybe I have forgotten some SQL conversion marker or context? In case of value::bytea there were no PostGIS 3.0 problems in my manual tests.

@mkgrgis
I think if you cast data to bytea, bytea_out function will be called to print data, so possibly, no problem occurs.
But in my case, I do not cast data to bytea. I also did not define DOMAIN geometry AS bytea (As you can see in #96 (comment) ).
I use the original geometry data type of PostGIS, so when SELECT, the function of PostGIS is called to print. As you can see in my image in #96 (comment), function LWGEOM_out is called.

Could you try exactly the same as my example in #96 (comment) and give me the result in your PC (I think a screenshot or attachment of postgis.out result file is good for me)?

Possibly, your inserted data is correct, but sqlite_fdw cannot display geometry data. If so, we need to fix more.

@son-phamngoc, this PR should works also with PostGIS 3+. In PostGIS 2 was implemented current stable internal storage format EWKB. This storage format is used in all of PostGIS 2+ functions including all existed PostGIS 3+ versions.

Do you mean the following description in your README is not correct? and this PR can also work with PostGIS 3+?

Also this foreign data wrapper (FDW) can connect PostgreSQL with [PostGIS](https://www.postgis.net/) to [SpatiaLite](https://www.gaia-gis.it/fossil/libspatialite/index) SQLite database file. This FDW works with PostGIS 2+ and confirmed with SpatiaLite 5.1.

[mkgrgis]

@mkgrgis I think if you cast data to bytea, bytea_out function will be called to print data, so possibly, no problem occurs.

Yes. Now I have understood this is not enough. We should also test PostGIS output for the values.

But in my case, I do not cast data to bytea. <...> I use the original geometry data type of PostGIS, so when SELECT, the function of PostGIS is called to print. As you can see in my image in #96 (comment), function LWGEOM_out is called.

Could you try exactly the same as my example in #96 (comment) and give me the result in your PC (I think a screenshot or attachment of postgis.out result file is good for me)?

Yes. On my real (not testing) system with installed experimental code and PostGIS and postgis_raster the script postgis_extract.txt gives
postgis_extract.out.txt after something like cat 'postgis_extract.sql' | psql -e -d "GIS";

The main point is parse error - invalid geometry. Maybe I doesn't deparse SQLite blob properly for PostGIS geometry and geography input functions.

Possibly, your inserted data is correct, but sqlite_fdw cannot display geometry data. If so, we need to fix more.

Look like you are right after our diagnostics.

Do you mean the following description in your README is not correct? and this PR can also work with PostGIS 3+?

Also this foreign data wrapper (FDW) can connect PostgreSQL with [PostGIS](https://www.postgis.net/) to [SpatiaLite](https://www.gaia-gis.it/fossil/libspatialite/index) SQLite database file. This FDW works with PostGIS 2+ and confirmed with SpatiaLite 5.1.

Yes. This is correct description because of common semantic version rules. Some examplees:

• 2.x means 2.1, 2.0.4, 2.7, 2.9.83 but not 3.0.2, but 2+ means all of listed version numbers.
• PostgreSQL 11+ means 12.14, 17.0, 11.4 and 16.1, but not 10.16.

[son-phamngoc]

@mkgrgis Thanks for your understanding.

Yes. Now I have understood this is not enough. We should also test PostGIS output for the values.

Yes. On my real (not testing) system with installed experimental code and PostGIS and postgis_raster the script postgis_extract.txt gives
postgis_extract.out.txt after something like cat 'postgis_extract.sql' | psql -e -d "GIS";
The main point is parse error - invalid geometry. Maybe I doesn't deparse SQLite blob properly for PostGIS geometry and geography input functions.

Could you investigate and fix this PR, considering above problem?

Yes. This is correct description because of common semantic version rules. Some examplees:

2.x means 2.1, 2.0.4, 2.7, 2.9.83 but not 3.0.2, but 2+ means all of listed version numbers.
PostgreSQL 11+ means 12.14, 17.0, 11.4 and 16.1, but not 10.16.

I understood.

[mkgrgis]

Could you investigate and fix this PR, considering above problem?

Yes, @son-phamngoc . Also I need help with testing environment. How can I enable PostGIS in testing databases? Just unpack PostGIS source to contrib and compile after sudo apt-get install libproj-dev libgeos-dev libxml2-dev gettext libjson-c-dev libgdal-dev libsfcgal-dev libprotobuf-c-dev protobuf-c-compiler -y ? Will this enough?

[mkgrgis]

@son-phamngoc , after current fixed CI and CREATE EXTENSION postgis; in testing postgis.sql I have

ERROR:  extension "postgis" is not available
DETAIL:  Could not open extension control file "/home/runner/work/sqlite_fdw/sqlite_fdw/workdir/postgresql-16.0/tmp_install/usr/local/pgsql/share/extension/postgis.control": No such file or directory.

Have you trying to get available PostGIS on testing PostgreSQL installation?

[mkgrgis]

@son-phamngoc , currently the main problem in CI is invisible postgres.h for unpacked to ./contrib/postgis-VER PostGIS code. I think after this problem solving I'll can CREATE EXTENSION postgis; in sqlite_fdw test scripts. Now -I or includedir was not useful in some tested configurations. Maybe I didn't use it properly.

[son-phamngoc]

Have you trying to get available PostGIS on testing PostgreSQL installation?

Please add the following code to Makefile of sqlite_fdw. After that, you can execute make check with PostGIS.

check: temp-install

temp-install: EXTRA_INSTALL+=contrib/postgis

checkprep: EXTRA_INSTALL+=contrib/postgis

The reason is make check command copies the necessary source code to tmp_install folder to execute test. This test requires contrib/postgis, so you have to specify it in Makefile.

[mkgrgis]

Please add the following code to Makefile of sqlite_fdw. After that, you can execute make check with PostGIS.

Thanks, @son-phamngoc , but this is not enough. Please read the scripts and error. In every PostgreSQL source code directory there is configured PostGIS source code in contrib/postgis now, but something still is wrong.

[mkgrgis]

@son-phamngoc , current code works with both PostGIS and bytea data in PostgreSQL. Please note, SpatiaLite GIS blob doesn't accept any data without SRID. Now I am writing diagnostics and error about missing SRID in EWKB. Deparsing for = operator also is enabled.

[mkgrgis]

@son-phamngoc , now this PR contains universal code for both PostGIS data types and bytea domains and big test datasets. There is no problems with PostGIS compilation, but there are still big problems with make install to PostgreSQL version temporary installation. Many my commands like following was not effective

ti=$(realpath ../../tmp_install);
	BINDIR="$ti/usr/local/pgsql/bin" \
	DOCDIR="$ti/usr/local/pgsql/share/doc" \
	HTMLDIR="$ti/usr/local/pgsql/share/doc" \
	INCLUDEDIR="$ti/usr/local/pgsql/include" \
	PKGINCLUDEDIR="$ti/usr/local/pgsql/include" \
	LIBDIR="$ti/usr/local/pgsql/lib" \
	PKGLIBDIR="$ti/usr/local/pgsql/share/lib" \
	LOCALEDIR="$ti/usr/local/pgsql/share/locale" \
	MANDIR="$ti/usr/local/pgsql/share/man" \
	SHAREDIR="$ti/usr/local/pgsql/share" \
	DESTDIR="$ti" \
	make install

./configure --prefix "$ti" also look like not effective. Could you please help with this problem during CI for PostGIS IO functions testing?

[mkgrgis]

@son-phamngoc , please try to reproduce our CRS destructor problem in current configuration of this PR. I am ready to try detailed debug if the problem is still reproducible.

[son-phamngoc]

@mkgrgis I tried again with Postgres 17.0, Postgis 3.4.2 and the latest source code of sqlite_fdw in this branch. The same problem still occurred.

[mkgrgis]

Thanks, @son-phamngoc ! I'll continue research around of the problem.

[mkgrgis]

@son-phamngoc, please subscribe at OSGeo/PROJ#4361 Maybe PROJ team can help us to pin our memory allocating/freeing problem.

[mkgrgis]

@son-phamngoc , there is answer from libproj team: OSGeo/PROJ#4361 (comment)
I think we have some work for environment group of pgspider team. How do you think?

[mkgrgis]

@son-phamngoc , please check libproj version for SpatiaLite and PostGIS on your system. Is this the same versions or different? See OSGeo/PROJ#4361 (comment).

[son-phamngoc]

@mkgrgis I checked OSGeo/PROJ#4361 (comment) and checked my environment.
I have 2 versions of libproj.
Original version of Rocky Linux 8:

/usr/bin/projinfo
Rel. 6.3.2, May 1st, 2020

Built version:

/usr/local/bin/projinfo
Rel. 9.0.0, March 1st, 2022

I didn't build it with option -DPROJ_RENAME_SYMBOLS or -DPROJ_INTERNAL_CPP_NAMESPACE. I built with the default configuration (using cmake --build .).
Anyway, I will try to build and check again. Please wait.

[mkgrgis]

I have 2 versions of libproj. Original version of Rocky Linux 8:

@son-phamngoc , I think in your case SpatiaLite uses /usr/bin/projinfo Rel. 6.3.2, May 1st, 2020, but PostGIS uses /usr/local/bin/projinfo Rel. 9.0.0, March 1st, 2022. In case of PostgreSQL memory freeing your system calls destructor for some SpatiaLite (?) objects, but C++ namespace and functions is yet overwritten by 9.0.0. Usisng 9.0.0 destructors for object created by 6.3.2 cause error. This is how I understand integration error from OSGeo/PROJ#4361 (comment).
In my environment and in CI Ubuntu scripts SpatiaLite and PostGIS uses equal versions of libproj and projection information, hence we are unable to detect the problem in this cases.

[son-phamngoc]

@mkgrgis Thanks for your support. It seems that's correct. I tried to install a clean environment, and then install only proj version 6.3.2. postgis test worked well without any problem. This is an integration error in my environment.

[son-phamngoc]

@mkgrgis This PR has many changes compared to the 1st review, so I would like to perform the 2nd review for this PR. Is this PR ready for review again?

[mkgrgis]

Is this PR ready for review again?

Yes, @son-phamngoc . I was want to propose this yesterday.

[son-phamngoc]

@mkgrgis Could you check my comments?

[mkgrgis]

@son-phamngoc , I am ready to final review by @t-kataym and merging of this PR as squashed commit. I think in PR merge note we should add thanks list:

1. Alessandro Furieri as SpatiaLite's author and our consultant
2. Even Rouault as OSGeo/PROJ supporter and our consultant about PROJ library versions
3. @son-phamngoc and @t-kataym as pgspider team members and I as author of a primary draft.

[son-phamngoc]

@mkgrgis Thank you for your contribution.
I have no more comment. Please wait for @t-kataym's decision.

[mkgrgis]

@t-kataym , are here any other problems or there was gone internal checks about code quality and productivity?

[t-kataym]

@mkgrgis I confirmed and there is no further comment.
I will merge it. Thank you for your development.