Tilda v2.5.1 is a large release due to the elapse time since v2.4. Due to the work, v2.5.0 was not even released publically.
In additional to many miscellaneous features and bug fixes, the focus of this release covered three areas:
- Major generalization of datetime and timezone handling
- Powerful new tools for data exchange between PG and BigQuery
- Masking enhancements on the heals of v2.4
New Features
DATETIME extensions
Dealing with date/times is always VERY complicated, and people claiming it's easy don't know what they are talking about. In particular, many developers trade-off short term gains with long-term pain when brushing away timezone concerns and datetime storage. In this modern age of more powerful traditional databases such as Postgres or cloud-scale databases such as BigQuery, the rule is always to use timestamps with time zones (https://wiki.postgresql.org/wiki/Don%27t_Do_This#Don.27t_use_timestamp_.28without_time_zone.29).
Tilda since the beginning took the approach of enforcing this view and ALL datetimes are stored as timestamp-with-time-zone (or equivalent) and an additional timezone field would be generated and stored so that timestamps could be retrieved and displayed with the expected timezone for an application. In this release, we have added more flexibility to how datetimes with timezones are handled in Tilda with far reaching impact. See this documentation for details. We think that these new capabilities give Tilda world-class timezone handling capabilities.
That being said, there is still work to be done, especially around migration. We would like in particular to enable moving between column and row-based TZ methodologies automatically. This is for a future release.
#780 tzMode: row/column -> Allow the storage of TZ columns to be specified.
#805 Adding comparators to QueryHelper for DateTime vs Date
#815 Add microseconds to timestamp exports (BREAKING CHANGE)
#817 Provide a flag to indicate whether a "DATETIME" column should be handled as a timestampz or just timestamp.
Data Exchange
We have been working with projects that handle more mixed environments between Postgres and BigQuery and needed a simpler high-performing way to move data back and forth. There are lots of tools out there and all, but having a simple mechanism as part of Tilda ready to use has proven to be a nice feature.
#814 Rewrite of Export utility to focus on PG<->BQ data transfers
#813 BigQuery To Postgres Streaming Utility
#789 Enable generic export from ANY table (not necessarily from a TILDA defined table).
Masking enhancements
In v2.1, we introduced masking capabilities to allow applications to dynamically implement rule-based masking of data. In this release, we implemented a number of improvements and fixed a few issues.
#776 Make masking work through views
#724 Allow masking to be turned on/off at the thread level.
#774 Add masking capabilities to JSON data type
#710 Masks for dates and datetimes are not working
#775 DATE and DATETIME Truncate capabilities to Year/Quarter/Month for masking
#799 Masks fail with view aggregate, transforming the data from a single value to an array.
Others
#821 Primary/Foreign key definitions for BigQuery (NOT ENFORCED)
#812 allowEmpty: currently, all "not-null" Strings are enforced to be not-empty either. With this feature, one can specify that a column is not-null but still allows empty string ''. By default, it's false.
#809 Allow empty strings for default values
#808 QueryHelper requires calling where() when creating a select clause. This makes dynamic clauses harder than needed. Need to make where() automated.
#782 Migration support for adding not-null columns
#801 Reverse support for BigQuery
#793 Allow STRING to CHAR migrations
#803 Renaming ClassStaticInit to ClassUtils
#810 Adding "salting" abilities to EncryptionUtil
#785 CheckDB enhancements
Build
Tilda is currently built using Eclipse-based build capabilities, which sucks. We are transitioning to a command-line approach using Gradle and getting rid in particular of dependencies we currently have around Eclipse plugins that are no longer supported (e.g., the Google Tools Plugin). This is an ongoing effort, but we are starting on this journey in this release. All library dependencies have been updated as of June 2024.
#820 Removing dependency on Google tools plugin and full update of all JAR dependencies.
#818 Move to 2_5 Tilda version
#802 Upgrade to Log4J 2.22
#794 Upgrade to Log4J 2.20
#787 JavaDocs update
#783 Update Postgres JDBC driver to postgresql-42.6.0.jar
Bug Fixes
#804 RS.absolute(Start) causes an exception when Start is invalid (i.e., in the past).
#788 Misleading error message when a foreign key is not automatically found
#800 Aggregates over a column with values generates invalid Java code
#798 Catalog and Acl updates during migrate are very long and causes massive query output to the logs which is not useful AND slows down things a lot.
#797 CODE_ONLY Views are being migrated causing errors.
#795 Column names in column expressions are not escaped properly.
#778 Calling newLine() right after the QueryHelper constructor causes syntax errors
#784 LookupWhereXXX methods can generate compile errors if a column is named "start" or "size"
#792 Migration for column renaming (and perhaps not-nulls as well) doesn't handle HIST patterns. Also check for clones.
#790 Queries over very large tables are not finishing and consuming all memory.
#786 CSHelper file download caching optimization
#779 check and map methods not generated properly for CHAR columns with valuesets
#807 Generated code for runSelect in Factory classes is not flexible enough to handle full select queries as well as where-clause-only queries.