Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix: Do not redact password for conn pool connection uri while connecting to DB #2203

Draft
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

priyanshi-yb
Copy link
Contributor

@priyanshi-yb priyanshi-yb commented Jan 17, 2025

Describe the changes in this pull request

Fixing the bug where import data always fails to connect to DB via the connection pool with a password authentication as the connection URLs are getting redacted while passing to the connection pool.
#2204
redaction still happens-

2025-01-18 12:46:14.483348 INFO yugabytedb.go:252 Initialized connection pool with settings: (tgtdb.ConnectionParams) {
 NumConnections: (int) 2,
 NumMaxConnections: (int) 4,
 ConnUriList: ([]string) (len=1 cap=1) {
  (string) (len=81) "postgresql://ybvoyager:[email protected]:5433/pg_datatypes_offline?sslmode=prefer"
 },
 SessionInitScript: ([]string) (len=3 cap=4) {
  (string) (len=29) "SET client_encoding TO 'UTF8'",
  (string) (len=39) "SET session_replication_role TO replica",
  (string) (len=53) "SET default_transaction_isolation = 'repeatable read'"
 }
}

Describe if there are any user-facing changes

Its a fix, no user-facing changes

How was this pull request tested?

automation tests are enough

Does your PR have changes that can cause upgrade issues?

Component Breaking changes?
MetaDB No
Name registry json No
Data File Descriptor Json No
Export Snapshot Status Json No
Import Data State No
Export Status Json No
Data .sql files of tables No
Export and import data queue No
Schema Dump No
AssessmentDB No
Sizing DB No
Migration Assessment Report Json No
Callhome Json No
YugabyteD Tables No
TargetDB Metadata Tables No

@priyanshi-yb priyanshi-yb force-pushed the priyanshi/fix-conn-pool-pass branch from c4b1b2b to 2f05de5 Compare January 17, 2025 10:36
@priyanshi-yb priyanshi-yb force-pushed the priyanshi/fix-conn-pool-pass branch from 2f05de5 to a83acb2 Compare January 17, 2025 10:49
@priyanshi-yb priyanshi-yb requested review from makalaaneesh and sanyamsinghal and removed request for makalaaneesh January 17, 2025 11:00
@@ -243,8 +243,12 @@ func (yb *TargetYugabyteDB) InitConnPool() error {
SessionInitScript: getYBSessionInitScript(yb.tconf),
}
yb.connPool = NewConnectionPool(params)
redactedParams := params
redactedParams.ConnUriList = utils.GetRedactedURLs(redactedParams.ConnUriList)
redactedParams := ConnectionParams{
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're logging the full struct below, so this will mean we'll have to come back to keep updating this copy here whenever we add a new field.

Let's go with redactedParams := *params ?

@@ -85,7 +85,7 @@ jobs:
docker run -d --name yugabytedb \
-p7000:7000 -p9000:9000 -p15433:15433 -p5433:5433 -p9042:9042 \
yugabytedb/yugabyte:${{ matrix.version }} \
bin/yugabyted start --background=false --ui=false
bin/yugabyted start --tserver_flags="ysql_hba_conf_csv={host all yugabyte all trust,host all all all md5}" --background=false --ui=false
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall we just have host all all all md5 ? Just want to avoid "trust" altogether 😅

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah I started with that only, but somehow yugabyted wasn't coming up with that conf as yugabyte is a default user maybe trust is required for that, I can take that up later after figuring out how to do that

@@ -243,8 +243,7 @@ func (yb *TargetYugabyteDB) InitConnPool() error {
SessionInitScript: getYBSessionInitScript(yb.tconf),
}
yb.connPool = NewConnectionPool(params)
redactedParams := params
redactedParams.ConnUriList = utils.GetRedactedURLs(redactedParams.ConnUriList)
redactedParams := *params
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should not do this.
https://go.dev/play/p/NjUBR-pdyd7
This is typical example of shallow copy vs deep copy and can potentially lead to a bug in future again.

I think as a standard practice also, we should always do Deep copy unless there is a usecase for shallow or significant perf implications of deepcopy.

  1. Either copy each field separately.
  2. Make use of standard deepcopy libraries available (to address @makalaaneesh's concern)
    https://github.com/mohae/deepcopy
    https://github.com/barkimedes/go-deepcopy (this seems to be better maintained)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, good point!
+1 to use deep copy by default. (unless there are pointers like mutexes/connections/etc. but then again if the struct has such fields, we wouldn't necessarily want to deep copy)

Copy link
Contributor Author

@priyanshi-yb priyanshi-yb Jan 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, missed it thanks for bringing it up @sanyamsinghal .
I made this change but missed one more thing to do redaction of urls back for logging.
I am thinking to not blindly deep copy that to just for logging or for the case where we can also add new fields which don't need to be copied, we can add any new field to logged params as well if required.
I can create the params with redaction for all current fields and log, basically revert the last commit
Sounds good?

Copy link
Collaborator

@sanyamsinghal sanyamsinghal Jan 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah you can copy each field manually also, just the overhead of maintaining the codepath for logging. But since the code is at one place so it will be fine.

btw do you see any issue with simply creating a deep copy and printing it? I believe this is a one time thing only? (i assume these deepcopy libraries will take care of nested pointer fields)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The issue with directly copying the params is that there will still be an overhead of redacting the URLs in ConnectionParams for logging, and this will still be present for any new fields we will add in the future to check if they need redaction.

redactedParams := &ConnectionParams{}
deepcopy.Copy(redactedParams, params)
redactedParams.ConnUriList = utils.GetRedactedURLs(redactedParams.ConnUriList)

So, I think it is better to have the overhead of adding the new field to the logged parameters if required, as it is a sensitive part where we have connection info with passwords.

This reverts commit b54c79a.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants