Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

🐛 Source Chargebee: Ensure no pagination issues during concurrency #48510

Open
wants to merge 22 commits into
base: master
Choose a base branch
from
+345 −285
Open

🐛 Source Chargebee: Ensure no pagination issues during concurrency #48510

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
cfc0971
update cdk dependency
maxi297 Nov 15, 2024
8c32771
Update release information
maxi297 Nov 15, 2024
698de4d
fix tests
maxi297 Nov 15, 2024
c282dcf
format
maxi297 Nov 15, 2024
6e833fc
Merge branch 'master' into maxi297/update-chargebee-concurrency-release
maxi297 Nov 18, 2024
390b925
Disabling future state tests
maxi297 Nov 18, 2024
49af588
Merge branch 'master' into maxi297/update-chargebee-concurrency-release
maxi297 Nov 18, 2024
8b5f5c2
Merge branch 'master' into maxi297/update-chargebee-concurrency-release
maxi297 Nov 18, 2024
b5cd04d
Update future states
maxi297 Nov 19, 2024
9f73763
fix future tests changes
maxi297 Nov 19, 2024
1f86304
add assertion to debug
maxi297 Nov 19, 2024
674a78b
Revert "add assertion to debug"
maxi297 Nov 19, 2024
01057c1
add assertion to debug
maxi297 Nov 19, 2024
7d7e7b0
Print more
maxi297 Nov 20, 2024
2ac2704
Revert "Print more"
maxi297 Nov 20, 2024
927d216
Revert "add assertion to debug"
maxi297 Nov 20, 2024
38b50e2
Merge branch 'master' into maxi297/update-chargebee-concurrency-release
maxi297 Nov 20, 2024
95f6395
Add empty stream
maxi297 Nov 20, 2024
c46a387
remove expected records for contact
maxi297 Nov 20, 2024
42796d6
Fix primary key
maxi297 Nov 20, 2024
22b4738
Update empty test reason
maxi297 Nov 20, 2024
ea3d964
update schema
maxi297 Nov 20, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-chargebee/metadata.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ data:
connectorSubtype: api
connectorType: source
definitionId: 686473f1-76d9-4994-9cc7-9b13da46147c
dockerImageTag: 0.7.1
dockerImageTag: 0.7.2
dockerRepository: airbyte/source-chargebee
documentationUrl: https://docs.airbyte.com/integrations/sources/chargebee
githubIssueLabel: source-chargebee
Expand Down
471 changes: 249 additions & 222 deletions airbyte-integrations/connectors/source-chargebee/poetry.lock
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-chargebee/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ requires = [ "poetry-core>=1.0.0",]
build-backend = "poetry.core.masonry.api"

[tool.poetry]
version = "0.7.1"
version = "0.7.2"
name = "source-chargebee"
description = "Source implementation for Chargebee."
authors = [ "Airbyte <[email protected]>",]
Expand Down
4 changes: 2 additions & 2 deletions airbyte-integrations/connectors/source-chargebee/source_chargebee/manifest.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,8 +35,8 @@ definitions:
datetime: "{{ format_datetime(config['start_date'], '%s') }}"
datetime_format: "%s"
end_datetime:
datetime: "{{ now_utc().strftime('%s') }}"
datetime_format: "%s"
datetime: "{{ now_utc().strftime('%Y-%m-%dT%H:%M:%SZ') }}"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I got trolled by that. As this is not expose anywhere, I've decided to update the format to one that won't break with timezone differences.

datetime_format: "%Y-%m-%dT%H:%M:%SZ"
datetime_format: "%s"
cursor_granularity: PT1S
step: P1M
Expand Down
4 changes: 2 additions & 2 deletions airbyte-integrations/connectors/source-chargebee/unit_tests/integration/test_addon.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -195,7 +195,7 @@ def test_given_no_initial_state_when_read_then_return_state_based_on_most_recent
output = self._read(_config().with_start_date(self._start_date - timedelta(hours=8)), _NO_STATE)
most_recent_state = output.most_recent_state
assert most_recent_state.stream_descriptor == StreamDescriptor(name=_STREAM_NAME)
assert most_recent_state.stream_state == AirbyteStateBlob(updated_at=cursor_value)
assert most_recent_state.stream_state == AirbyteStateBlob(updated_at=str(cursor_value))
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The state has changed from a int to a string. This shouldn't be an issue because as shown here, the change is backward compatible


@HttpMocker()
def test_given_initial_state_use_state_for_query_params(self, http_mocker: HttpMocker) -> None:
Expand All @@ -210,4 +210,4 @@ def test_given_initial_state_use_state_for_query_params(self, http_mocker: HttpM
output = self._read(_config().with_start_date(self._start_date - timedelta(hours=8)), state)
most_recent_state = output.most_recent_state
assert most_recent_state.stream_descriptor == StreamDescriptor(name=_STREAM_NAME)
assert most_recent_state.stream_state == AirbyteStateBlob(updated_at=record_cursor_value)
assert most_recent_state.stream_state == AirbyteStateBlob(updated_at=str(record_cursor_value))
4 changes: 2 additions & 2 deletions airbyte-integrations/connectors/source-chargebee/unit_tests/integration/test_coupon.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -208,7 +208,7 @@ def test_given_no_initial_state_when_read_then_return_state_based_on_most_recent
output = self._read(_config().with_start_date(self._start_date - timedelta(hours=8)), _NO_STATE)
most_recent_state = output.most_recent_state
assert most_recent_state.stream_descriptor == StreamDescriptor(name=_STREAM_NAME)
assert most_recent_state.stream_state == AirbyteStateBlob(updated_at=cursor_value)
assert most_recent_state.stream_state == AirbyteStateBlob(updated_at=str(cursor_value))

@HttpMocker()
def test_given_initial_state_use_state_for_query_params(self, http_mocker: HttpMocker) -> None:
Expand All @@ -223,4 +223,4 @@ def test_given_initial_state_use_state_for_query_params(self, http_mocker: HttpM
output = self._read(_config().with_start_date(self._start_date - timedelta(hours=8)), state)
most_recent_state = output.most_recent_state
assert most_recent_state.stream_descriptor == StreamDescriptor(name=_STREAM_NAME)
assert most_recent_state.stream_state == AirbyteStateBlob(updated_at=record_cursor_value)
assert most_recent_state.stream_state == AirbyteStateBlob(updated_at=str(record_cursor_value))
4 changes: 2 additions & 2 deletions airbyte-integrations/connectors/source-chargebee/unit_tests/integration/test_customer.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -207,7 +207,7 @@ def test_given_no_initial_state_when_read_then_return_state_based_on_most_recent
output = self._read(_config().with_start_date(self._start_date - timedelta(hours=8)), _NO_STATE)
most_recent_state = output.most_recent_state
assert most_recent_state.stream_descriptor == StreamDescriptor(name=_STREAM_NAME)
assert most_recent_state.stream_state == AirbyteStateBlob(updated_at=cursor_value)
assert most_recent_state.stream_state == AirbyteStateBlob(updated_at=str(cursor_value))

@HttpMocker()
def test_given_initial_state_use_state_for_query_params(self, http_mocker: HttpMocker) -> None:
Expand All @@ -222,4 +222,4 @@ def test_given_initial_state_use_state_for_query_params(self, http_mocker: HttpM
output = self._read(_config().with_start_date(self._start_date - timedelta(hours=8)), state)
most_recent_state = output.most_recent_state
assert most_recent_state.stream_descriptor == StreamDescriptor(name=_STREAM_NAME)
assert most_recent_state.stream_state == AirbyteStateBlob(updated_at=record_cursor_value)
assert most_recent_state.stream_state == AirbyteStateBlob(updated_at=str(record_cursor_value))
4 changes: 2 additions & 2 deletions airbyte-integrations/connectors/source-chargebee/unit_tests/integration/test_event.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -193,7 +193,7 @@ def test_given_no_initial_state_when_read_then_return_state_based_on_most_recent
output = self._read(_config().with_start_date(self._start_date - timedelta(hours=8)), _NO_STATE)
most_recent_state = output.most_recent_state
assert most_recent_state.stream_descriptor == StreamDescriptor(name=_STREAM_NAME)
assert most_recent_state.stream_state == AirbyteStateBlob(occurred_at=cursor_value)
assert most_recent_state.stream_state == AirbyteStateBlob(occurred_at=str(cursor_value))

@HttpMocker()
def test_given_initial_state_use_state_for_query_params(self, http_mocker: HttpMocker) -> None:
Expand All @@ -208,4 +208,4 @@ def test_given_initial_state_use_state_for_query_params(self, http_mocker: HttpM
output = self._read(_config().with_start_date(self._start_date - timedelta(hours=8)), state)
most_recent_state = output.most_recent_state
assert most_recent_state.stream_descriptor == StreamDescriptor(name=_STREAM_NAME)
assert most_recent_state.stream_state == AirbyteStateBlob(occurred_at=record_cursor_value)
assert most_recent_state.stream_state == AirbyteStateBlob(occurred_at=str(record_cursor_value))
4 changes: 2 additions & 2 deletions airbyte-integrations/connectors/source-chargebee/unit_tests/integration/test_hosted_page.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -192,7 +192,7 @@ def test_given_no_initial_state_when_read_then_return_state_based_on_most_recent
output = self._read(_config().with_start_date(self._start_date - timedelta(hours=8)), _NO_STATE)
most_recent_state = output.most_recent_state
assert most_recent_state.stream_descriptor == StreamDescriptor(name=_STREAM_NAME)
assert most_recent_state.stream_state == AirbyteStateBlob(updated_at=cursor_value)
assert most_recent_state.stream_state == AirbyteStateBlob(updated_at=str(cursor_value))

@HttpMocker()
def test_given_initial_state_use_state_for_query_params(self, http_mocker: HttpMocker) -> None:
Expand All @@ -207,4 +207,4 @@ def test_given_initial_state_use_state_for_query_params(self, http_mocker: HttpM
output = self._read(_config().with_start_date(self._start_date - timedelta(hours=8)), state)
most_recent_state = output.most_recent_state
assert most_recent_state.stream_descriptor == StreamDescriptor(name=_STREAM_NAME)
assert most_recent_state.stream_state == AirbyteStateBlob(updated_at=record_cursor_value)
assert most_recent_state.stream_state == AirbyteStateBlob(updated_at=str(record_cursor_value))
4 changes: 2 additions & 2 deletions airbyte-integrations/connectors/source-chargebee/unit_tests/integration/test_plan.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -197,7 +197,7 @@ def test_given_no_initial_state_when_read_then_return_state_based_on_most_recent
output = self._read(_config().with_start_date(self._start_date - timedelta(hours=8)), _NO_STATE)
most_recent_state = output.most_recent_state
assert most_recent_state.stream_descriptor == StreamDescriptor(name=_STREAM_NAME)
assert most_recent_state.stream_state == AirbyteStateBlob(updated_at=cursor_value)
assert most_recent_state.stream_state == AirbyteStateBlob(updated_at=str(cursor_value))

@HttpMocker()
def test_given_initial_state_use_state_for_query_params(self, http_mocker: HttpMocker) -> None:
Expand All @@ -212,4 +212,4 @@ def test_given_initial_state_use_state_for_query_params(self, http_mocker: HttpM
output = self._read(_config().with_start_date(self._start_date - timedelta(hours=8)), state)
most_recent_state = output.most_recent_state
assert most_recent_state.stream_descriptor == StreamDescriptor(name=_STREAM_NAME)
assert most_recent_state.stream_state == AirbyteStateBlob(updated_at=record_cursor_value)
assert most_recent_state.stream_state == AirbyteStateBlob(updated_at=str(record_cursor_value))
4 changes: 2 additions & 2 deletions airbyte-integrations/connectors/source-chargebee/unit_tests/integration/test_subscription.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -208,7 +208,7 @@ def test_given_no_initial_state_when_read_then_return_state_based_on_most_recent
output = self._read(_config().with_start_date(self._start_date - timedelta(hours=8)), _NO_STATE)
most_recent_state = output.most_recent_state
assert most_recent_state.stream_descriptor == StreamDescriptor(name=_STREAM_NAME)
assert most_recent_state.stream_state == AirbyteStateBlob(updated_at=cursor_value)
assert most_recent_state.stream_state == AirbyteStateBlob(updated_at=str(cursor_value))

@HttpMocker()
def test_given_initial_state_use_state_for_query_params(self, http_mocker: HttpMocker) -> None:
Expand All @@ -223,4 +223,4 @@ def test_given_initial_state_use_state_for_query_params(self, http_mocker: HttpM
output = self._read(_config().with_start_date(self._start_date - timedelta(hours=8)), state)
most_recent_state = output.most_recent_state
assert most_recent_state.stream_descriptor == StreamDescriptor(name=_STREAM_NAME)
assert most_recent_state.stream_state == AirbyteStateBlob(updated_at=record_cursor_value)
assert most_recent_state.stream_state == AirbyteStateBlob(updated_at=str(record_cursor_value))
4 changes: 2 additions & 2 deletions ...egrations/connectors/source-chargebee/unit_tests/integration/test_virtual_bank_account.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -197,7 +197,7 @@ def test_given_no_initial_state_when_read_then_return_state_based_on_most_recent
output = self._read(_config().with_start_date(self._start_date - timedelta(hours=8)), _NO_STATE)
most_recent_state = output.most_recent_state
assert most_recent_state.stream_descriptor == StreamDescriptor(name=_STREAM_NAME)
assert most_recent_state.stream_state == AirbyteStateBlob(updated_at=cursor_value)
assert most_recent_state.stream_state == AirbyteStateBlob(updated_at=str(cursor_value))

@HttpMocker()
def test_given_initial_state_use_state_for_query_params(self, http_mocker: HttpMocker) -> None:
Expand All @@ -212,4 +212,4 @@ def test_given_initial_state_use_state_for_query_params(self, http_mocker: HttpM
output = self._read(_config().with_start_date(self._start_date - timedelta(hours=8)), state)
most_recent_state = output.most_recent_state
assert most_recent_state.stream_descriptor == StreamDescriptor(name=_STREAM_NAME)
assert most_recent_state.stream_state == AirbyteStateBlob(updated_at=record_cursor_value)
assert most_recent_state.stream_state == AirbyteStateBlob(updated_at=str(record_cursor_value))
7 changes: 4 additions & 3 deletions docs/integrations/sources/chargebee.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,9 +104,10 @@ The Chargebee connector should not run into [Chargebee API](https://apidocs.char

| Version | Date | Pull Request | Subject |
|:--------|:-----------|:---------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------|
| 0.7.1 | 2024-11-04 | [48133](https://github.com/airbytehq/airbyte/pull/48133) | Fix `error message pattern` to handle `Product 1.0` related errors |
| 0.7.0 | 2024-10-30 | [47978](https://github.com/airbytehq/airbyte/pull/47978) | Upgrade the CDK and startup files to sync incremental streams concurrently |
| 0.6.18 | 2024-10-31 | [47099](https://github.com/airbytehq/airbyte/pull/47099) | Update dependencies |
| 0.7.2 | 2024-11-18 | [48510](https://github.com/airbytehq/airbyte/pull/48510) | Ensure no pagination issues on concurrent syncs |
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I set up the date for Monday but if we think it is necessary, I can release today

| 0.7.1 | 2024-11-04 | [48133](https://github.com/airbytehq/airbyte/pull/48133) | Fix `error message pattern` to handle `Product 1.0` related errors |
| 0.7.0 | 2024-10-30 | [47978](https://github.com/airbytehq/airbyte/pull/47978) | Upgrade the CDK and startup files to sync incremental streams concurrently |
| 0.6.18 | 2024-10-31 | [47099](https://github.com/airbytehq/airbyte/pull/47099) | Update dependencies |
| 0.6.17 | 2024-10-28 | [46846](https://github.com/airbytehq/airbyte/pull/47387) | Update CDK dependencies to yield parent records more frequently |
| 0.6.16 | 2024-10-12 | [46846](https://github.com/airbytehq/airbyte/pull/46846) | Update dependencies |
| 0.6.15 | 2024-10-05 | [46478](https://github.com/airbytehq/airbyte/pull/46478) | Update dependencies |
Expand Down
Loading