Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update data types for snowflake #52

Open
wants to merge 31 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
95c3965
Change gemspecs and gemfile
Shehbaz Sep 1, 2020
af47f3d
Upgrade to rails5.2.1
Shehbaz Sep 2, 2020
abdfd45
Upgrade to rails 6
Shehbaz Sep 29, 2020
b51a593
Backward compatibility
Shehbaz Sep 29, 2020
7a43f84
Update bind params
Shehbaz Sep 29, 2020
7c56bcc
Change params for Column class initializer
Shehbaz Oct 8, 2020
bf7a016
Support json type field
Shehbaz Oct 16, 2020
2696c7a
Support json and date datatype
Shehbaz Oct 29, 2020
fb7aa4a
Typo fix
Shehbaz Oct 29, 2020
6a94e07
Merge pull request #1 from patterninc/upgrade-to-rails6
smithworx Aug 4, 2021
c74949a
Minor updates to the README
smithworx Aug 4, 2021
d8f7c95
Updating for keyword arguments as per Ruby 3
satishaher Dec 8, 2022
2f3b4f6
Merge pull request #2 from patterninc/ruby_update_check
satishaher Dec 13, 2022
e1ff722
disbaling the call for disable_referential_integrity
AjitKonde-Pattern Feb 13, 2023
31e866e
Merge pull request #3 from patterninc/fix-odbc-error
jasonwells Feb 13, 2023
e9c43b9
Replaced fetch_all with fetch
AjitKonde-Pattern Feb 13, 2023
3976d58
Merge pull request #4 from patterninc/fix-odbc-error
jasonwells Feb 13, 2023
5ece8fe
Fixed boolean conversion to int bug
vishalzambre Mar 16, 2023
4ef32d7
Use native data type to map rails data types
vishalzambre Mar 16, 2023
869c5f2
Upgrade rails dependency
vishalzambre Mar 16, 2023
d9ef63a
Rubocop updated
vishalzambre Mar 17, 2023
c4c2c6d
Added additional datatypes
vishalzambre Mar 17, 2023
421b23c
Rubocop corrected
vishalzambre Mar 17, 2023
560c67e
Rubocop corrected
vishalzambre Mar 17, 2023
4606164
Undo changes to fetch_all and disable_referential_integrity
mattbrown-msb Mar 20, 2023
9178654
Datatypes corrected
vishalzambre Mar 21, 2023
52df6df
Readme
vishalzambre Mar 21, 2023
e2c278f
Merge pull request #5 from patterninc/boolean-fix
vishalzambre Mar 21, 2023
1306c5c
Updated mapping of number field to float.
mattbrown-msb Mar 21, 2023
7f75095
Data type changes.
mattbrown-msb Mar 21, 2023
8b44387
Removed condition.
mattbrown-msb Mar 22, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 13 additions & 20 deletions .rubocop.yml
Original file line number Diff line number Diff line change
@@ -1,35 +1,28 @@
require:
- rubocop-rake
- rubocop-minitest

inherit_from: .rubocop_todo.yml

AllCops:
NewCops: enable
DisplayCopNames: true
DisplayStyleGuide: true
TargetRubyVersion: 2.1
TargetRubyVersion: 2.7
Exclude:
- 'vendor/**/*'

Lint/AmbiguousBlockAssociation:
Enabled: false

Metrics/AbcSize:
Enabled: false

Metrics/ClassLength:
Enabled: false

Metrics/CyclomaticComplexity:
Minitest/MultipleAssertions:
Enabled: false

Metrics/MethodLength:
Enabled: false

Metrics/LineLength:
Enabled: false

Metrics/PerceivedComplexity:
Style/Documentation:
Enabled: false

Style/Documentation:
Metrics/ClassLength:
Enabled: false

Style/PercentLiteralDelimiters:
PreferredDelimiters:
default: '[]'
'%r': '{}'
Layout/LineLength:
Max: 170
36 changes: 36 additions & 0 deletions .rubocop_todo.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# This configuration was generated by
# `rubocop --auto-gen-config`
# on 2023-03-17 05:40:41 UTC using RuboCop version 1.48.1.
# The point is for the user to remove these configuration records
# one by one as the offenses are removed from the code base.
# Note that changes in the inspected code, or installation of new
# versions of RuboCop, may require this file to be generated again.

# Offense count: 2
# Configuration parameters: AllowedMethods, AllowedPatterns, CountRepeatedAttributes.
Metrics/AbcSize:
Max: 32

# Offense count: 1
# Configuration parameters: AllowedMethods, AllowedPatterns.
Metrics/CyclomaticComplexity:
Max: 9

# Offense count: 2
# Configuration parameters: Max, CountKeywordArgs.
Metrics/ParameterLists:
MaxOptionalParameters: 4

# Offense count: 1
# Configuration parameters: AllowedMethods, AllowedPatterns.
Metrics/PerceivedComplexity:
Max: 9

# Offense count: 5
# Configuration parameters: MinNameLength, AllowNamesEndingInNumbers, AllowedNames, ForbiddenNames.
# AllowedNames: as, at, by, cc, db, id, if, in, io, ip, of, on, os, pp, to
Naming/MethodParameterName:
Exclude:
- 'lib/active_record/connection_adapters/odbc_adapter.rb'
- 'lib/odbc_adapter/adapters/mysql_odbc_adapter.rb'
- 'lib/odbc_adapter/adapters/postgresql_odbc_adapter.rb'
5 changes: 2 additions & 3 deletions Gemfile
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
# frozen_string_literal: true

source 'https://rubygems.org'

gemspec

gem 'activerecord', '5.0.1'
gem 'pry', '~> 0.11.1'
92 changes: 88 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,6 @@
# ODBCAdapter

[![Build Status](https://travis-ci.org/localytics/odbc_adapter.svg?branch=master)](https://travis-ci.org/localytics/odbc_adapter)
[![Gem](https://img.shields.io/gem/v/odbc_adapter.svg)](https://rubygems.org/gems/odbc_adapter)

An ActiveRecord ODBC adapter. Master branch is working off of Rails 5.0.1. Previous work has been done to make it compatible with Rails 3.2 and 4.2; for those versions use the 3.2.x or 4.2.x gem releases.
An ActiveRecord ODBC adapter. Pattern has made some updates to get this working with Rails 6 as forked from repo was not actively being maintained to do so.

This adapter will work for basic queries for most DBMSs out of the box, without support for migrations. Full support is built-in for MySQL 5 and PostgreSQL 9 databases. You can register your own adapter to get more support for your DBMS using the `ODBCAdapter.register` function.

Expand Down Expand Up @@ -70,6 +67,93 @@ docker run -it --rm -v $(pwd):/workspace -v /tmp/mysql:/var/lib/mysql odbc-dev:l
docker/test.sh
```

## Datatype References


Reference https://docs.snowflake.com/en/sql-reference/intro-summary-data-types

https://github.com/rails/rails/blob/main/activerecord/lib/active_record/connection_adapters/abstract_adapter.rb#L877-L908

```
["number",
"decimal",
"numeric",
"int",
"integer",
"bigint",
"smallint",
"tinyint",
"byteint",
"float",
"float4",
"float8",
"double",
"real",
"varchar",
"char",
"character",
"string",
"text",
"binary",
"varbinary",
"boolean",
"date",
"datetime",
"time",
"timestamp",
"timestamp_ltz",
"timestamp_ntz",
"timestamp_tz",
"variant",
"object",
"array",
"geography",
"geometry"]
```


Possible mapping from chatgpt

| Snowflake Data Type | Ruby ActiveRecord Type | PostgreSQL Type |
| --------------------- | ---------------------- | ---------------- |
| number | :decimal | numeric
| decimal | :decimal | numeric
| numeric | :decimal | numeric
| int | :integer | integer
| integer | :integer | integer
| bigint | :bigint | bigint
| smallint | :integer | smallint
| tinyint | :integer | smallint
| byteint | :integer | smallint
| float | :float | double precision
| float4 | :float | real
| float8 | :float | double precision
| double | :float | double precision
| real | :float | real
| varchar | :string | character varying
| char | :string | character
| character | :string | character
| string | :string | character varying
| text | :text | text
| binary | :binary | bytea
| varbinary | :binary | bytea
| boolean | :boolean | boolean
| date | :date | date
| datetime | :datetime | timestamp without time zone
| time | :time | time without time zone
| timestamp | :timestamp | timestamp without time zone
| timestamp_ltz | :timestamp | timestamp without time zone
| timestamp_ntz | :timestamp | timestamp without time zone
| timestamp_tz | :timestamp | timestamp with time zone
| variant | :jsonb | jsonb
| object | :jsonb | jsonb
| array | :jsonb | jsonb
| geography | :st_point, :st_polygon,| geography
| | :st_multipolygon |
| geometry | :st_point, :st_polygon,| geometry
| | :st_multipolygon |


## Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/localytics/odbc_adapter.
Expand Down
2 changes: 2 additions & 0 deletions Rakefile
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
# frozen_string_literal: true

require 'bundler/gem_tasks'
require 'rake/testtask'
require 'rubocop/rake_task'
Expand Down
1 change: 1 addition & 0 deletions bin/console
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
#!/usr/bin/env ruby
# frozen_string_literal: true

require 'bundler/setup'
require 'odbc_adapter'
Expand Down
87 changes: 43 additions & 44 deletions lib/active_record/connection_adapters/odbc_adapter.rb
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
# frozen_string_literal: true

require 'active_record'
require 'arel/visitors/bind_visitor'
require 'odbc'
require 'odbc_utf8'

Expand Down Expand Up @@ -39,8 +40,8 @@ def odbc_connection(config)

# Connect using a predefined DSN.
def odbc_dsn_connection(config)
username = config[:username] ? config[:username].to_s : nil
password = config[:password] ? config[:password].to_s : nil
username = config[:username]&.to_s
password = config[:password]&.to_s
odbc_module = config[:encoding] == 'utf8' ? ODBC_UTF8 : ODBC
connection = odbc_module.connect(config[:dsn], username, password)

Expand All @@ -53,15 +54,16 @@ def odbc_dsn_connection(config)
# e.g. "DSN=virt5;UID=rails;PWD=rails"
# "DRIVER={OpenLink Virtuoso};HOST=carlmbp;UID=rails;PWD=rails"
def odbc_conn_str_connection(config)
attrs = config[:conn_str].split(';').map { |option| option.split('=', 2) }.to_h
attrs = config[:conn_str].split(';').to_h { |option| option.split('=', 2) }
odbc_module = attrs['ENCODING'] == 'utf8' ? ODBC_UTF8 : ODBC
driver = odbc_module::Driver.new
driver.name = 'odbc'
driver.attrs = attrs

connection = odbc_module::Database.new.drvconnect(driver)
# encoding_bug indicates that the driver is using non ASCII and has the issue referenced here https://github.com/larskanis/ruby-odbc/issues/2
[connection, config.merge(driver: driver, encoding: attrs['ENCODING'], encoding_bug: attrs['ENCODING'] == 'utf8')]
[connection,
config.merge(driver: driver, encoding: attrs['ENCODING'], encoding_bug: attrs['ENCODING'] == 'utf8')]
end
end
end
Expand All @@ -73,14 +75,14 @@ class ODBCAdapter < AbstractAdapter
include ::ODBCAdapter::Quoting
include ::ODBCAdapter::SchemaStatements

ADAPTER_NAME = 'ODBC'.freeze
BOOLEAN_TYPE = 'BOOLEAN'.freeze
ADAPTER_NAME = 'ODBC'
VARIANT_TYPE = 'VARIANT'

ERR_DUPLICATE_KEY_VALUE = 23_505
ERR_QUERY_TIMED_OUT = 57_014
ERR_QUERY_TIMED_OUT_MESSAGE = /Query has timed out/
ERR_CONNECTION_FAILED_REGEX = '^08[0S]0[12347]'.freeze
ERR_CONNECTION_FAILED_MESSAGE = /Client connection failed/
ERR_QUERY_TIMED_OUT_MESSAGE = /Query has timed out/.freeze
ERR_CONNECTION_FAILED_REGEX = '^08[0S]0[12347]'
ERR_CONNECTION_FAILED_MESSAGE = /Client connection failed/.freeze

# The object that stores the information that is fetched from the DBMS
# when a connection is first established.
Expand Down Expand Up @@ -137,49 +139,46 @@ def disconnect!
# Build a new column object from the given options. Effectively the same
# as super except that it also passes in the native type.
# rubocop:disable Metrics/ParameterLists
def new_column(name, default, sql_type_metadata, null, table_name, default_function = nil, collation = nil, native_type = nil)
::ODBCAdapter::Column.new(name, default, sql_type_metadata, null, table_name, default_function, collation, native_type)
def new_column(name, default, sql_type_metadata, null, table_name, native_type = nil)
::ODBCAdapter::Column.new(name, default, sql_type_metadata, null, table_name, native_type)
end
# rubocop:enable Metrics/ParameterLists

def clear_cache! # :nodoc:
reload_type_map
super
end

protected

# Build the type map for ActiveRecord
# Here, ODBC and ODBC_UTF8 constants are interchangeable
def initialize_type_map(map)
map.register_type 'boolean', Type::Boolean.new
map.register_type ODBC::SQL_CHAR, Type::String.new
map.register_type ODBC::SQL_LONGVARCHAR, Type::Text.new
map.register_type ODBC::SQL_TINYINT, Type::Integer.new(limit: 4)
map.register_type ODBC::SQL_SMALLINT, Type::Integer.new(limit: 8)
map.register_type ODBC::SQL_INTEGER, Type::Integer.new(limit: 16)
map.register_type ODBC::SQL_BIGINT, Type::BigInteger.new(limit: 32)
map.register_type ODBC::SQL_REAL, Type::Float.new(limit: 24)
map.register_type ODBC::SQL_FLOAT, Type::Float.new
map.register_type ODBC::SQL_DOUBLE, Type::Float.new(limit: 53)
map.register_type ODBC::SQL_DECIMAL, Type::Float.new
map.register_type ODBC::SQL_NUMERIC, Type::Integer.new
map.register_type ODBC::SQL_BINARY, Type::Binary.new
map.register_type ODBC::SQL_DATE, Type::Date.new
map.register_type ODBC::SQL_DATETIME, Type::DateTime.new
map.register_type ODBC::SQL_TIME, Type::Time.new
map.register_type ODBC::SQL_TIMESTAMP, Type::DateTime.new
map.register_type ODBC::SQL_GUID, Type::String.new

alias_type map, ODBC::SQL_BIT, 'boolean'
alias_type map, ODBC::SQL_VARCHAR, ODBC::SQL_CHAR
alias_type map, ODBC::SQL_WCHAR, ODBC::SQL_CHAR
alias_type map, ODBC::SQL_WVARCHAR, ODBC::SQL_CHAR
alias_type map, ODBC::SQL_WLONGVARCHAR, ODBC::SQL_LONGVARCHAR
alias_type map, ODBC::SQL_VARBINARY, ODBC::SQL_BINARY
alias_type map, ODBC::SQL_LONGVARBINARY, ODBC::SQL_BINARY
alias_type map, ODBC::SQL_TYPE_DATE, ODBC::SQL_DATE
alias_type map, ODBC::SQL_TYPE_TIME, ODBC::SQL_TIME
alias_type map, ODBC::SQL_TYPE_TIMESTAMP, ODBC::SQL_TIMESTAMP
def initialize_type_map(m = type_map)
super

m.register_type(/bigint/i, Type::BigInteger.new)
m.alias_type 'float4', 'float'
m.alias_type 'float8', 'float'
m.alias_type 'double', 'float'
m.alias_type 'number', 'decimal'
m.alias_type 'numeric', 'decimal'
m.alias_type 'real', 'float'
m.alias_type 'string', 'char'
m.alias_type 'bool', 'boolean'
m.alias_type 'varbinary', 'binary'
m.alias_type 'variant', 'json'
m.alias_type 'object', 'string'
m.alias_type 'array', 'string'
m.alias_type 'geography', 'char'
m.alias_type 'geometry', 'char'

# number() data types in Snowflake are interpreted as decimal and must be mapped back to a float
m.alias_type 'decimal', 'float'
end

# Translate an exception from the native DBMS to something usable by
# ActiveRecord.
def translate_exception(exception, message)
def translate_exception(exception, **message)
error_number = exception.message[/^\d+/].to_i

if error_number == ERR_DUPLICATE_KEY_VALUE
Expand All @@ -190,7 +189,7 @@ def translate_exception(exception, message)
begin
reconnect!
::ODBCAdapter::ConnectionFailedError.new(message, exception)
rescue => e
rescue StandardError => e
puts "unable to reconnect #{e}"
end
else
Expand Down
2 changes: 2 additions & 0 deletions lib/odbc_adapter.rb
Original file line number Diff line number Diff line change
@@ -1,2 +1,4 @@
# frozen_string_literal: true

# Requiring with this pattern to mirror ActiveRecord
require 'active_record/connection_adapters/odbc_adapter'
Loading