Skip to content

Commit

Permalink
revert (partial): write compressed configuration [RHELDST-25461]
Browse files Browse the repository at this point in the history
This commit partially reverts db4a344; specifically, it
disables the writing of compressed config and reverts back to writing
JSON config as before.

This is being reverted due to an issue noticed during testing. It seems
that the written items are ending up with *two* layers of base64
encoding, which is not intended.
The boto docs[1] use base64 strings as example arguments, giving the
impression the caller is expected to take care of base64 encoding, but
in fact botocore internally does the encoding; if the client also
encodes, we end up with two layers of encoding.

There is also a bug filed relating to this[2].

The code here still seems to "work" since the same mistake is made
on both the writing and reading end, but the goal is to make the
config smaller and having double-encoding works against that.

It should be easy enough to fix, but I'd like some time to confirm my
understanding of how it works, check whether exodus-lambda needs a fix
and also check whether localstack and AWS are behaving the same.
Hence I'll revert this for now and keep writing the old style of config.

[1] https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/dynamodb/client/put_item.html
[2] aws/aws-cli#1097
  • Loading branch information
rohanpm committed Aug 5, 2024
1 parent 4504e23 commit 005d272
Show file tree
Hide file tree
Showing 2 changed files with 3 additions and 17 deletions.
6 changes: 1 addition & 5 deletions exodus_gw/aws/dynamodb.py
Original file line number Diff line number Diff line change
Expand Up @@ -161,11 +161,7 @@ def create_config_request(self, config):
"Item": {
"from_date": {"S": self.from_date},
"config_id": {"S": "exodus-config"},
"config": {
"B": b64encode(
gzip.compress(json.dumps(config).encode())
).decode()
},
"config": {"S": json.dumps(config)},
}
}
},
Expand Down
14 changes: 2 additions & 12 deletions tests/worker/test_deploy.py
Original file line number Diff line number Diff line change
Expand Up @@ -54,11 +54,7 @@ def test_deploy_config(
"Item": {
"from_date": {"S": NOW_UTC},
"config_id": {"S": "exodus-config"},
"config": {
"B": b64encode(
gzip.compress(json.dumps(fake_config).encode())
).decode()
},
"config": {"S": json.dumps(fake_config)},
}
}
},
Expand Down Expand Up @@ -158,13 +154,7 @@ def test_deploy_config_with_flush(
"Item": {
"from_date": {"S": NOW_UTC},
"config_id": {"S": "exodus-config"},
"config": {
"B": b64encode(
gzip.compress(
json.dumps(updated_config).encode()
)
).decode()
},
"config": {"S": json.dumps(updated_config)},
}
}
},
Expand Down

0 comments on commit 005d272

Please sign in to comment.