Skip to content

Releases: deephaven/deephaven-core

v0.9.0

31 Jan 22:30
@devinrsmith devinrsmith
v0.9.0
08fea42
Compare
Choose a tag to compare
v0.9.0

What's Changed

New Contributors

Full Changelog: v0.8.0...v0.9.0

Contributors

niloc132, kosak, and 10 other contributors
Assets 2
Loading

v0.8.0

31 Dec 18:31
@devinrsmith devinrsmith
v0.8.0
4e828f3
Compare
Choose a tag to compare
v0.8.0

What's Changed

Full Changelog: v0.7.0...v0.8.0

Contributors

niloc132, jmao-denver, and 4 other contributors
Assets 2
Loading

v0.7.0

09 Dec 20:04
@devinrsmith devinrsmith
v0.7.0
2ce3ebe
Compare
Choose a tag to compare
v0.7.0

What's Changed

Full Changelog: v0.6.1...v0.7.0

Contributors

niloc132, nbauernfeind, and 7 other contributors
Assets 2
Loading

v0.6.1

04 Nov 22:21
@devinrsmith devinrsmith
v0.6.1
17170fd
Compare
Choose a tag to compare
v0.6.1

What's Changed

Full Changelog: v0.6.0...v0.6.1

Contributors

nbauernfeind, rcaudy, and devinrsmith
Assets 2
Loading

v0.6.0

01 Nov 17:30
@devinrsmith devinrsmith
v0.6.0
26d8ec8
Compare
Choose a tag to compare
v0.6.0

What's Changed

Full Changelog: v0.5.2...v0.6.0

Contributors

kosak, nbauernfeind, and 6 other contributors
Assets 2
Loading

v0.5.2

13 Oct 13:58
@devinrsmith devinrsmith
v0.5.2
e1ca9ca
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.5.2

What's Changed

Full Changelog: v0.5.1...v0.5.2

Contributors

niloc132, kosak, and 6 other contributors
Assets 2
Loading

v0.5.1

12 Oct 15:00
@devinrsmith devinrsmith
v0.5.1
ee16881
Compare
Choose a tag to compare
v0.5.1

What's Changed

Full Changelog: v0.5.0...v0.5.1

Contributors

nbauernfeind, jmao-denver, and 4 other contributors
Assets 2
Loading

[Release] 0.5.0

04 Oct 16:00
@devinrsmith devinrsmith
v0.5.0
aefe2d1
Compare
Choose a tag to compare
[Release] 0.5.0

Fixed bugs:

  • [Groovy] upperBin() (and lowerBin()) method signature doesn't match the javadocs #1341
  • Nightly Check Transient Failure: QueryTableWhereTest#testInterFilterInterruption #1317
  • Cannot add two series from the same table to a plot. #1292
  • diffDay, diffNanos, and diffYear have incorrect docs for the situation that gives negative values #1288
  • io.deephaven.db.v2.replay.Replayer tries to grab the LTM exclusive lock when it shouldn't #1283
  • Syntax error found in the overview example of pydeephaven #1279
  • StreamToTableAdapter.refresh error logging is substandard (because there isn't any) #1277
  • deephaven.ParquetTools.readTable() doesn't support Parquet date32 or date64 data types #1253
  • Python Issues with pre-defined constants #1247
  • The sort direction enum in the Python client isn't correctly assigned values #1233
  • Bug in ChunkColumnSources observed via streaming lastBy #1227
  • Copy Selection* menu options are copying the entire table not just the selection. #1225
  • The Python client releases tickets not generated by the client. #1219
  • No need to be verbose about cancelled barrage subscriptions #1205
  • "Running..." in IDE disappears before computation is complete #842
  • Port fix for DH-11620 (WouldMatch fillChunk bug fix) #1342 (rcaudy)
  • Move default nginx root outside of first location #1338 (nbauernfeind)
  • Make QueryTableWhereTest#testInterFilterInterruption more robust for CI #1318 (nbauernfeind)
  • Reduce noise on cancelled barrage subscriptions #1291 (nbauernfeind)
  • Fix plot not allowing more than one series per table instance #1289 (nbauernfeind)
  • Fix LTM lock usage in Replayer #1284 (rcaudy)
  • Log errors when a failure is encountered in StreamToTableAdapter.refresh #1280 (rcaudy)
  • Update web to v0.3.2 #1278 (mofojed)
  • DBLanguageParser allow unboxing and widening when finding available methods #1248 (nbauernfeind)
  • Fix bug in ChunkColumnSources #1243 (rcaudy)
  • Reduce verbosity of UNAUTHENTICATED calls from alternative GrpcUtil.rpcWrapper #1238 (nbauernfeind)
  • Fix case insensitive string filters #1229 (mofojed)
  • Fix zulu-open-jdk-debian base image dependency #1213 (devinrsmith)
  • TableService's gRPC lazyUpdate/update/select impls must acquire LTM shared lock #1211 (nbauernfeind)
  • BarrageStreamGenerator#Drainable should be empty after drainTo #1209 (nbauernfeind)
  • Send UI to Unknown Error if auth token cannot be refreshed after a few tries #1193 (nbauernfeind)
  • Fix console type for containers/ #1192 (devinrsmith)
  • Propagate query scope changes even if script throws a RuntimeError #1191 (nbauernfeind)
  • Add no-cache headers to responses from the web #1185 (mofojed)
  • Propagate Barrage propagation errors to the client table widget #1182 (nbauernfeind)
  • Properly ignore non-refreshing dynamic nodes; fix jsapi getObject to wait for Ticket #1173 (nbauernfeind)
  • ScriptSession#setVariable require subclasses to notify old vs new Value #1166 (nbauernfeind)

Closed issues:

  • Consolidate slf4j versions and make sure they all match #1332
  • Consider using standardized Java classes for date/time #1304
  • Make it easy to use PyCharm to remotely debug Python code inside the Deephaven grpc-api container #1302
  • Determine if an incremental-wrapping approach is possible or makes sense to make the server Python integration more Pythonic #1276
  • Make pydeeephaven.Session usable in Python 'with' statement. #1244
  • dummy issue #1197
  • deephaven-client-session should not depend on flight-core or arrow-x #1186
  • Code examples to demo how to use the major functions of the DH Python client. #1146
  • java-client DoPut #1124
  • Automate the code generation, building, and testing for the new Python client #1073
  • Kafka: Handle DBDateTime from JSON #1046
  • Kafka: KafkaTools: Make Avro/Json and Simple Raw Types Play Nicely Together as Key/Values #1025
  • Kafka: Support nested records in avro schemas by linearizing them. #985
  • Convert JSAPI examples from Groovy #904
  • Reduce log noise from autocomplete #796
  • Add convenience types for DynamicTableWriter #793
  • Establish and advertise minimum required docker version #769
  • CSV (and related table) uploads #696
  • EPIC: Java API supporting connection to remote worker + simple table ops #657
  • Consider open-telemetry #100

Merged pull requests:

Read more
Assets 2
Loading

[Release] 0.4.1

01 Sep 23:22
@devinrsmith devinrsmith
v0.4.1
c50a1bc
Compare
Choose a tag to compare
[Release] 0.4.1

New Feature

Kafka Ingestion Support

  • Schema service integration for Confluent/Apicurio.
  • Decode from Apache Avro, JSON, or primitives.
  • Start from beginning, end, current, or an arbitrary offset.
  • Stream tables to the Deephaven query engine for efficient event processing or aggregations.
  • Record full history in-memory to allow richer Deephaven queries.
  • Commingle all partitions, or segregate them into distinct sub-tables within Deephaven TableMaps.

Stream Table Optimizations

  • Developed in support of Kafka ingestion.
  • Allow for very efficient aggregations over the full data set, while only storing “new” rows in their entirety.
  • Can be transformed into full-history in-memory tables.
  • Suitable for any streaming ingestion use case.

Parquet Support Expansion

  • Use Spark/Dask style _metadata and _common_metadata files when present for faster partition discovery.
  • Support multiple row groups in a single .parquet file.
  • Major scalability enhancements for files with many dictionary pages and/or row groups.
  • Support dictionary size constraints in writing tools.
  • Single entry point for “do the right thing” reading.
  • This expands on existing support for Parquet, which allows for symmetrical writing/reading of Deephaven tables, in addition to ingestion support for single files, flat layouts, and hive-style partitioning with most standard compression options and data types.

Application Mode

  • Loads all .app files in deephaven.application.dir (CLI flag to directory).
  • Use ApplicationService to subscribe to field changes.
  • Script types of application must match deephaven.console.type.
  • Static (script-less, immutable) applications subclass StaticClassApplication.
  • Dynamic (script-less, non-immutable) applications subclass DynamicApplication.
  • Initialize REPL environment with a Script Application (Query Scope changes are available to REPL sessions).

Python Integration

  • New deephaven.Types import, defines constants for Deephaven column data types and adds support for table creation from constant data with an API resembling Panda’s DataFrame. This builds on the QST support for building tables also mentioned in the Java client section.

Python client

  • Session object connects to DH server, convenience methods for creating empty tables, time tables, importing table from Arrow data, merging tables, opening tables by names, and running scripts.
  • Table object represents a DH table, supports most table operations including filtering, selection, snapshotting, aggregation and joins.
  • Query object defines a set of table operations to be performed in a single batch on the server.
  • Available for download from PYPI.

Java client

  • Arrow Flight integration; doGet (doPut next release).
  • Query syntax tree (QST) support.
  • Serial mode.
  • Batch mode.
  • Console session support.
  • Command line utilities and examples.
  • Graphviz visualization for QST.
  • Jars published to Maven Central; https://search.maven.org/search?q=g:io.deephaven

C++ Client

  • Arrow flight integration (basically a passthrough to the C++ FlightClient provided by the Arrow libraries). Provides doGet, doPut, and others.
  • Async mode.
  • Unit tests and examples.

UI

  • CSV upload and download are now supported via the web UI.

Enhancement

Improved interactions between two windows communicating with the same worker.

  • The shared query-scope fields show up in the panels drop-down.
  • The UI recognizes initial change to an existing query-scope field as a new field.

Improved detection of scenarios where the backend is not working as expected.

  • If any of grpc-web, envoy, or the worker exit, the UI detects this and notifies the user.
  • If a Barrage subscription fails to snapshot, all subscribers are notified and closed.

Improvements to Arrow Flight integration

  • Migrated all Barrage metadata into FlightData metadata field.
  • doGet works from official Java / Python and C++ Arrow Flight clients.

Improved error messages related to tickets (missing and invalid) to gRPC clients

gRPC API

  • Added FetchTable to TableService; can be used in a batch.
  • Added ExportFromTicket to SessionService; does not validate type of export.
  • Added ExportNotification.RUNNING state to distinguish from QUEUED.
  • Reduced log volume related to UNAUTHENTICATED requests.

Javascript Client (jsapi)

  • Wired up selectDistinct (re-enables existing UI functionality for filter pop-ups).

Address bugs and inefficiencies in autocomplete.

Improved error handling in the JS API.

Bug fix

  • Fixed several issues regarding Liveness tracking in SessionState et. al.
  • Fix SessionState NPE caused by cancel / perform-export-work race.
  • Fix barrage subscription update race due to acquiring wrong lock.
  • Fix bugs in SortedRanges.SearchIterator for hasNext after exhaustion.

Internal

  • Cleanup many of the rawtypes related to ColumnSources.
  • Upgrade JavaParser from 2.0.0 to 3.23.0.
  • AutoComplete is now performed over a bi-directional stream.
  • Javascript Client
    • Prefers to use websockets in non-ssl for bi-directional streams.
    • Uses HTTP headers x-deephaven-stream-ticket, x-deephaven-stream-sequence, and x-deephaven-stream-halfclose to simulate client-streams (and bidirectional-streams) when using SSL.

System Administration

  • Flag io.deephaven.console.type renamed to deephaven.console.type.
  • New Flag deephaven.application.dir (see Application Mode in New Features).
  • Added no-cache headers to nginx_default.conf.
  • Added layouts directory adjacent to notebooks in nginx_default.conf.
  • Disable websocket idle timeout in envoy via stream_idle_timeout: 0s.

Commits

Raw Git release notes:
3349c42 Cut for 0.4.0
f9fb90b Remove old docker/ contents (#1170)
b7edf2c Deephaven C++ client (#1176)
5681bad Propagate Barrage propagation errors to the client table widget (#1182)
d11aa6c Give table update listeners short version of propagation error (#1179)
7b23050 MVP release, version set to 0.4.0, fixes #1171 (#1175)
d79bed1 Improve Ticket/Descriptor error messages for all gRPC usages (#1174)
b9bbef2 Added python tests for every Kafka spec type. Closes #1123. (#1168)
8361935 Properly ignore non-refreshing dynamic nodes; fix jsapi getObject to wait for Ticket (#1173)
2ff6962 Add execute script support for java client (#1156)
2b11f45 Add publishing for java-client jars and dependencies, fixes #1126 (#1149)
0753e2d Remove files that should have been never added. (#1169)
34cd17a ScriptSession#setVariable require subclasses to notify old vs new Value (#1166)
b181861 Add python documentation for Kafka integration. (#1160)
36b4c2d add compose files for standalone docker containers (#1058)
ff335f4 Use the correct location (Classpaths.groovy) to declare which java parser version (#1165)
e7ca952 Update web to v0.3.1 (#1164)
5cfd357 Improved validations for comboagg grpc request (#1014)
cf19dfa Initial Kafka python tests. (#1155)
657a716 fixed a compatibilty issue in the Python client due to the latest chanage to session proto, fixes #1158 (#1159)
ad6ff1b Update all .env files (#1161)
c2bd84f Updating web UI to v0.3.0 (#1154)
e0d85c4 Application mode for long running workers (#1082)
37e5f32 Adding Javadoc to KafkaTools.java. Closes #1122 (#1148)
6ef97c8 Deephaven Python API MVP (#1094)
404d5fd Upgrade javaparser-core to 3.23.0 (#1106)
a791fd3 Massive RawTypes Cleanup (#1145)
cd8745d Add safety check for .gitignore, fixes #1114 (#1147)
2e36920 Learn library logic, no dependencies (#1140)
51f19a5 Support bidirectional streaming on websockets (#1111)
a45154c Use DBDateTime insteat of qst...instant from python in deephaven.Types for the time being. (#1144)
602dfd9 Disambiguate DynamicTableWriter constructors taking an array from the point of view of jpy. (#1143)
590a746 Disable envoy websocket timeout (#1139)
7b2eacf Parquet: Lazy dictionary loading, shared page caches, and better file channel cache behavior (#1130)
d3d6aa9 QST to graphviz DOT format; and SVG, PNG, and others (#935)
34552e7 Fix CODEOWNERS references (#1137)
0add3f8 Run ./gradlew :Generators:generateAllPython (#1132)
473a40a DynamicTableWriter ctor to support using qst...Types (#1125)
f39d8f2 java-client session and flight (#953)
86319e2 Update code style (#1121)
a7a0454 Downgrade Alpine to avoid a DNS issue on older mac docker installs (#1034)
10fc2f6 Update CONTRIBUTING.md for styleguide (#1118)
9b851e5 Add DynamicTableWriter to deephaven module (#1117)
3d46a74 Fix tests after styling applied
8a5099f Apply style guide
038f8ee Remove style guide ratchet
abcfa45 Add style guide for most projects
14a997c Fix .gitignore rules (#1115)
c646eb2 Rename KafkaIngester module to Kafka. (#1110)
5e135dd Some Avro converter unit tests. (#1108)
8e64658 Adding Types.py for wrapping DH column types and table operations around them. (#1088)
11bda2e Fix a bug for the implementation of simple in the python side of KafkaTools. (#1109)
c932216 Add support for seek to end to Kafka API. (#1103)
440a166 rerun ./gradlew :DB:replicate (#1099)
2074ed2 Force jvm to load class in desired package (#1095)
7fe69b9 CrossJoin cardinality calculation may overflow integer (#1098)
87aa98e Properly use liveness API for dynamic non-refreshing nodes in SessionState exports (#1092)
8addc73 Revert "First iteration of Learn library (#974)" (#1096)
216ca7d First iteration of Learn library (#974)
0c22475 Native arrays, Db Arrays, and types. (#1076)
152b2b8 Add support for converting stream tables to append from python. (#1090)
41cd9a3 JS API Error handling (#1086)
686e5f8 TableMap support in KafkaTools (#1079)
c80f7c8...

Read more
Assets 2
Loading

[Release] 0.4.0

01 Sep 04:36
@devinrsmith devinrsmith
v0.4.0
3349c42
Compare
Choose a tag to compare
[Release] 0.4.0 
v0.4.0

[Release] 0.4.0
Assets 2
Loading