[WIP] Add support for Postgis Geometry data types #130

bettdouglas · 2020-06-08T23:47:11Z

This PR enables support of postgis geometries which is provided via an extension.
PostGIS is an extension that enables performing storage of spatial objects, enables manipulation of the objects as well as doing spatial analytics like finding features which are inside a specific region, e.g find all shops inside New York, find 20 closest shops near me, etc. PostGIS Webiste

This PR hopes to add support for this datatype.
It requires a dependency on dart_jts which has ability to handle postgis data types as Plain Dart Objects.

Since postgis is an extension, the datatype will not be available in databases where the extension has not been created/activated via CREATE EXTENSION postgis;

Due to that, there are some caveats to implementing this library here though as discussed by @isoos in #120 since the datatype's oid is dynamic, not static like other officially supported types.

A workaround has been suggested here #120 (comment)

This is the issue which needs reviewing as well as modifying the implementation in order to get the oid of the object as well as testing.

Kindly review. Looking foward to your feedback

isoos · 2020-06-09T14:26:05Z

lib/src/binary_codec.dart

+              utf8.encode(
+                value.toText(),
+              ),
+            ); //TODO: Convert to Uint8List. Need help here. Using toText() in raw sql inserts it well.


Please help me to understand this TODO: it seems that it is already implemented. Is there something missing?

Actually, this should have been added to the PostgresTextEncoder. I've added it in the last commit

b0c853a adds conversion of geometry objects from dart to string using toText() which returns the text in which postgis can understand

isoos · 2020-06-09T14:27:03Z

lib/src/types.dart

-  uuid
+  uuid,
+
+  /// Must be a [List] of [int] currently in Ewkb(Extended Well-Known Binary) format


Please link to the spec/documentation of this format.

https://postgis.net/docs/using_postgis_dbmanagement.html#OpenGISWKBWKT
I'll add it into the code and push the change

e23a3c6 I hope that helps

isoos · 2020-06-09T14:32:18Z

lib/src/binary_codec.dart

@@ -305,5 +326,7 @@ class PostgresBinaryDecoder extends Converter<Uint8List, dynamic> {
    1184: PostgreSQLDataType.timestampWithTimezone,
    2950: PostgreSQLDataType.uuid,
    3802: PostgreSQLDataType.json,
+    46315: PostgreSQLDataType.geometry, /// TODO: Oid Changes on different databases after running `CREATE EXTENSION postgis`


I think this should go into a separate PR. Do you mind if I create a stub and you can review it if it will work here?

I'd really appreciate the assistance on that. Sure that would be a good way to progress

I have an idea for implementing this with a kind of pluggable interface, which not only solves this obstacle, but also it could make sure we don't necessarily bind the postgres package to an API that is changing under us (package:dart_jts). Please be a bit patient while I do the necessary preparation work, I'll keep you posted on it.

I'd love to see how that will work. Thanks for your assistance

This rewrite / internal refactoring seems to be a much harder problem than I first anticipated. We need to define custom encoders and decoders, and pass them through every little place that may use them. Not sure when I will be ready with that... :(

Okay, I understand. I made some hacky solution where I send a query to read the datatypes when opening the connection,it's working well and all the other tests still work. If you'd love to check it out, and give some feedback...its on oid-connection-test branch.
The major change is that the typeMap is made dynamic and it has to be injected into the Query object.

I've used it on a variety of images including docker images and they seem to be working okay. The only caveat is that if someone run's CREATE extension custom-extension after connecting to the database, the typeMap will need to be updated to reflect this as the typeMap will not have the new types

Some tests are failing though.

here's a screenshot of failing tests and passing ones since Travis CI cannot run

https://imgur.com/Zj8qqTa

@isoos let me just close this into separate PR.s

isoos · 2020-06-09T21:55:26Z

lib/src/binary_codec.dart

@@ -305,5 +326,7 @@ class PostgresBinaryDecoder extends Converter<Uint8List, dynamic> {
    1184: PostgreSQLDataType.timestampWithTimezone,
    2950: PostgreSQLDataType.uuid,
    3802: PostgreSQLDataType.json,
+    46315: PostgreSQLDataType.geometry, /// TODO: Oid Changes on different databases after running `CREATE EXTENSION postgis`
+    46971: PostgreSQLDataType.geometry  /// `SELECT oid, typarray FROM pg_type WHERE typname in ('geometry','geography')`;


@bettdouglas: If you have two databases on the same server, you create the postgis extension on both, will this be one entry or two for 'geometry'?

So, postgis has two general types which is 'geography' and 'geometry'. When you create the extension on one database, the two types('geography' and 'geometry') will be registered to the pg_type table. Running the create extension postgis again replicates the same behavior but doesn't affect types registered in other databases.

In other words, it will have two new records per DB, and their OIDs will be different. It does complicate things, as we'll need a per-DB query (or another way to detect the type, which won't be cheap).

That would be kind of expensive,but i assume since the behaviour is different per database, and since every connection from dart has to specifies a specific database, the on connection query to get oids would just include one additional type 'geography' apart from 'geometry', or maybe I'm getting this wrong. or How were you planning to implement this?

bettdouglas · 2020-07-06T06:09:31Z

Closed due to #130 (comment) to separate the concerns

bettdouglas added 8 commits May 30, 2020 16:17

add dart_jts dependecy to parse EWKB format

c77478c

Updated all switch statements on the type

b5b12d8

added the various geometric types in Dart

191a713

Added geometry_tests

6a016d5

add LinearRing parse to interface

a9eb4c5

changed to ungrouped test

5260cd3

fix typo postgis and dartfmt

844c325

run dartfmt

9e2084b

bettdouglas changed the title ~~Add support for Postgis Geometry data types~~ [WIP] Add support for Postgis Geometry data types Jun 9, 2020

bettdouglas closed this Jun 9, 2020

bettdouglas reopened this Jun 9, 2020

bettdouglas added 3 commits June 9, 2020 09:12

Reduce boilerplate

876f039

added link to understand format of geometries

6887784

fix import

da9d9ed

bettdouglas closed this Jun 9, 2020

bettdouglas reopened this Jun 9, 2020

isoos reviewed Jun 9, 2020

View reviewed changes

bettdouglas added 4 commits June 9, 2020 20:55

tested a few

2f5a713

added Geometry to PostgresTextEncoder

b0c853a

added link to geometry specification

e23a3c6

added tests for PostgresTextEncoder for geometry

ea2fc20

isoos reviewed Jun 9, 2020

View reviewed changes

bettdouglas added 8 commits June 11, 2020 23:32

added read on connect query as well as test on other types

0e99449

crude fix to support postgis

6373561

added check for if not exists when creating extension

cd6731c

merge changes from upstream master

cf25154

merge changes from upstream master

3975520

merge changes from upstream master

c5e17a3

merge conflicts

ba15df0

ignore failing geometry collection equality test

36b9b9c

bettdouglas closed this Jul 6, 2020

bettdouglas mentioned this pull request Jul 12, 2020

[Enhancement] Add support for additonal types which are created via extensions e.g Geospatial objects. #139

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Add support for Postgis Geometry data types #130

[WIP] Add support for Postgis Geometry data types #130

bettdouglas commented Jun 8, 2020 •

edited

Loading

isoos Jun 9, 2020

bettdouglas Jun 9, 2020

bettdouglas Jun 9, 2020

isoos Jun 9, 2020

bettdouglas Jun 9, 2020 •

edited

Loading

bettdouglas Jun 9, 2020

isoos Jun 9, 2020

bettdouglas Jun 9, 2020 •

edited

Loading

isoos Jun 9, 2020

bettdouglas Jun 10, 2020

isoos Jul 2, 2020

bettdouglas Jul 3, 2020 •

edited

Loading

bettdouglas Jul 3, 2020

bettdouglas Jul 5, 2020 •

edited

Loading

isoos Jun 9, 2020

bettdouglas Jun 10, 2020

isoos Jun 10, 2020

bettdouglas Jun 10, 2020

bettdouglas commented Jul 6, 2020

[WIP] Add support for Postgis Geometry data types #130

[WIP] Add support for Postgis Geometry data types #130

Conversation

bettdouglas commented Jun 8, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bettdouglas Jun 9, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bettdouglas Jun 9, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bettdouglas Jul 3, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bettdouglas Jul 5, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bettdouglas commented Jul 6, 2020

bettdouglas commented Jun 8, 2020 •

edited

Loading

bettdouglas Jun 9, 2020 •

edited

Loading

bettdouglas Jun 9, 2020 •

edited

Loading

bettdouglas Jul 3, 2020 •

edited

Loading

bettdouglas Jul 5, 2020 •

edited

Loading