v0.33.1

• Ignore internal admin databases for GCP and Azure
  • This avoids collecting data from these internal databases, which produces
    unnecessary errors when using the all databases setting.
• Add log_line_prefix check to GCP self-test
• Schema stats handling: Avoid crash due to nil pointer dereference
• Add support for "%m [%p]: [%l-1] db=%d,user=%u " log_line_prefix

v0.33.0

• Add helper for log-based EXPLAIN access and use if available
  • This lets us avoid granting the pganalyze user any access to the data to follow
    the principle of least privilege
  • See https://github.com/pganalyze/collector#setting-up-log-explain-helper
• Avoid corrupted snapshots when OIDs get reused across databases
  • This would have shown as data not being visible in pganalyze,
    particularly for servers with many databases where tables were
    dropped and recreated often
• Locked relations: Ignore table statistics, handle other exclusive locks
  • Tables being rewritten would cause the relation statistics query to
    fail due to statement timeout (caused by lock being held)
  • Non-relation locks held in AccessExclusiveLock mode would cause all
    relation information to disappear, but only for everything thats not
    the top-level relation information. This is due to the behaviour of
    NOT IN when the list contains NULLs (never being true, even if an
    item doesn't match the list). The top-level relation information
    was using a LEFT JOIN that doesn't suffer from this problem. This likely
    caused problems reported as missing index information, or indices
    showing as being recently created even though they've exited for a
    while.
• Improvements to table partitioning reporting
• Enable additional settings to work correctly when used in Heroku/Docker
  • DB_NAME
  • DB_SSLROOTCERT_CONTENTS
  • DB_SSLCERT_CONTENTS
  • DB_SSLKEY_CONTENTS

v0.32.0

• Add ignore_schema_regexp / IGNORE_SCHEMA_REGEXP configuration option
  • This is like ignore_table_pattern, but pushed down into the actual
    stats-gathering queries to improve performance. This should work much
    better on very large schemas
  • We use a regular expression instead of the current glob-like matching
    since the former is natively supported in Postgres
  • We now warn on usage of the deprecated ignore_table_pattern field
• Add warning for too many tables being collected (and recommend ignore_schema_regexp)
• Allow keeping of unparsable query texts by setting filter_query_text: none
  • By default we replace everything with <unparsable query> (renamed
    from the previous <truncated query> for clarity), to avoid leaking
    sensitive data that may be contained in query texts that couldn't be
    parsed and that Postgres itself doesn't mask correctly (e.g. utility
    statements)
  • However in some situations it may be desirable to have the original
    query texts instead, e.g. when the collector parser is outdated
    (right now the parser is Postgres version 10, and some newer Postgres 12
    query syntax fails to parse)
  • To support this use case, a new "filter_query_text" / FILTER_QUERY_TEXT
    option is introduced which can be set to "none" to keep all query texts.
• EXPLAIN plans / Query samples: Support log line prefix without %d and %u
  • Whilst not recommended, in some scenarios changing the log_line_prefix
    is difficult, and we want to make it easy to get EXPLAIN data even in
    those scenarios
  • In case the log_line_prefix is missing the database (%d) and/or the user
    (%u), we simply use the user and database of the collector connection
• Log EXPLAIN: Run on all monitored databases, not just the primary database
• Add support for stored procedures (new with Postgres 11)
• Handle Postgres error checks using Go 1.13 error helpers
  • This is more correct going forward, and adds a required type check for
    the error type, since the database methods can also return net.OpError
  • Fixes "panic: interface conversion: error is *net.OpError, not *pq.Error"
• Collect information on table partitions
  • Relation parents as well as partition boundary (if any)
  • Partitionining strategy in use
  • List of partitioning fields and/or expression
• Log Insights: Track TLS protocol version as a log line detail
  • This allows verification of which TLS versions were used to connect to the
    database over time
• Log Insights: Track host as detail for connection received event
  • This allows more detailed analysis of which IPs/hostnames have connected
    to the database over time
• Example collector config: Use collect all databases option in the example
  • This improves the chance that this is set up correctly from the
    beginning, without requiring a backwards incompatible change in the
    collector

v0.31.0

• Add Log Insights support for Azure Database for PostgreSQL
• Log Insights: Avoid unnecessary "Timeout" error when there are other failures
• Log EXPLAIN: Don't run EXPLAIN logic when there are no query sample
• Improve non-fatal error messages to clarify the collector still works
• Log grant failure: Explain root cause better (plan doesn't support it / fair use limit reached)

v0.30.0

• Track local replication lag in bytes
• RDS: Handle end of log files correctly
• High-frequency query collection: Avoid race condition, run in parallel
  • This also resolves a memory leak in the collector that was causing
    increased memory usage over time for systems that have a lot of
    pg_stat_statements query texts (causing the full snapshot to take
    more than a minute, which triggered the race condition)

v0.29.0

• Package builds: Use Golang 1.14.3 patch release
  • This fixes golang/go#37436 which was causing
    "mlock of signal stack failed: 12" on Ubuntu systems
• Switch to simpler tail library to fix edge case bugs for self-managed systems
  • The hpcloud library has been unmaintained for a while, and whilst
    the new choice doesn't have much activity, in tests it has shown
    to work better, as well as having significantly less lines of code
  • This also should make "--test" work reliably for self-managed systems
    (before this returned "Timeout" most of the time)
• Index statistics: Don't run relation_size on exclusively locked indices
  • Previously the collector was effectively hanging when it encountered an
    index that has an ExclusiveLock held (e.g. due to a REINDEX)
• Add another custom log line prefix: "%m %r %u %a [%c] [%p] "
• RDS fixes
  • Fix handling of auto-detection of AWS regions outside of us-east-1
  • Remember log marker from previous runs, to avoid duplicate log lines
• Add support for Postgres 13
  • This adds support for running against Postgres 13, which otherwise breaks
    due to backwards-incompatible changes in pg_stat_statements
  • Note that there are many other new statistics views and metrics that
    will be added separately

v0.28.0

• Add "db_sslkey" and "db_sslcert" options to use SSL client certificates
• Add Ubuntu 20.04 packages
• Update to Go 1.14, latest libpq
• Ensure that we set system type correctly for Heroku full snapshots
• Detect cloud providers based on hostnames from DB_URL / db_url as well
  • Previously this was only detected for the DB_HOST / db_host setting, and that is unnecessarily restrictive
  • Note that this means your instance may show up under a new ID in pganalyze after upgrading to this version
• Log Explain
  • Ignore pg_start_backup queries
  • Support EXPLAIN for queries with parameters
• Log Insights improvements
  • Experimental: Google Cloud SQL log download
  • Remove unnecessary increment of log line byte end position
  • Make stream-based log processing more robust
• Add direct "http_proxy" & similar collector settings for Proxy config
  • This avoids problems in some environments where its not clear whether
    the environment variables are set. The environment variables HTTP_PROXY,
    http_proxy, HTTPS_PROXY, https_proxy, NO_PROXY and no_proxy continue to
    function as expected.
• Fix bug in handling of state mutex in activity snapshots
  • This may have been the cause of "unlock of unlocked mutex" errors
    when having multiple servers configured.

v0.27.0

• Activity snapshot: Track timestamp of previous activity snapshot
• Support setting custom AWS endpoints using environment variables
• Increase allowed characters for pid field to 7 (for 64-bit systems)
  • This supports environments where pid_max is set to 4194304 instead of 32768
  • Note that this means continuity with old backend/vacuum identities is lost
    i.e. data might show up incorrectly for existing processes after the upgrade

v0.26.0

• Add new wait events from Postgres 12
• Avoid unnecessary allocations in ReplaceSecrets function
• Rename "aws_endpoint_rds_signing_region" to "aws_endpoint_signing_region"
  • This more accurately reflects how the setting is used. Backwards
    compatibility is provided, but its recommended to migrate to the new
    config setting (when in use for custom AWS API endpoints)

v0.25.1

• Update rds-ca-2019-root.pem file to be correct certificate
  • This was identical to the 2015 certificate by accident, causing
    connection errors