[active-standby] Add oscillation logic when there is no heartbeat on both sides #221
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
yxieca
previously approved these changes
Nov 7, 2023
lolyu
reviewed
Nov 9, 2023
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
lolyu
reviewed
Nov 10, 2023
Merged
8 tasks
StormLiangMS
pushed a commit
to sonic-net/sonic-mgmt
that referenced
this pull request
Mar 5, 2024
…case (#11878) Description of PR Summary: Fixes # (issue) On latest master image, test_cacl_application_dualtor case could fail due to the following reason: Failed: Missing expected iptables rules: {'-A DHCP -m mark --mark 0x67005 -j DROP'} This is caused by the oscillation logic: sonic-net/sonic-linkmgrd#221 As there is no icmp_responder running, the mux will start flap, if it flaps betrween expected_dhcp_rules_for_standby fixture and the real iptables check, there could be some unexpected dhcp iptables which will cause case failure. What is the motivation for this PR? Fix Unexpected DHCP iptables rules for test_cacl_application_dualtor How did you do it? test_cacl_application_dualtor is used to verify dhcp iptables on dualtor, not oscillation. Before dualtor case, set mux mode to manual sudo config mux mode manual all in test teardown, run to set to auto back. sudo config mux mode auto all How did you verify/test it? Run cacl/test_cacl_application.py::test_cacl_application_dualtor on dualtor testbed against master image. Any platform specific information?
mssonicbld
pushed a commit
to mssonicbld/sonic-mgmt
that referenced
this pull request
May 6, 2024
…case (sonic-net#11878) Description of PR Summary: Fixes # (issue) On latest master image, test_cacl_application_dualtor case could fail due to the following reason: Failed: Missing expected iptables rules: {'-A DHCP -m mark --mark 0x67005 -j DROP'} This is caused by the oscillation logic: sonic-net/sonic-linkmgrd#221 As there is no icmp_responder running, the mux will start flap, if it flaps betrween expected_dhcp_rules_for_standby fixture and the real iptables check, there could be some unexpected dhcp iptables which will cause case failure. What is the motivation for this PR? Fix Unexpected DHCP iptables rules for test_cacl_application_dualtor How did you do it? test_cacl_application_dualtor is used to verify dhcp iptables on dualtor, not oscillation. Before dualtor case, set mux mode to manual sudo config mux mode manual all in test teardown, run to set to auto back. sudo config mux mode auto all How did you verify/test it? Run cacl/test_cacl_application.py::test_cacl_application_dualtor on dualtor testbed against master image. Any platform specific information?
8 tasks
mssonicbld
pushed a commit
to sonic-net/sonic-mgmt
that referenced
this pull request
May 6, 2024
…case (#11878) Description of PR Summary: Fixes # (issue) On latest master image, test_cacl_application_dualtor case could fail due to the following reason: Failed: Missing expected iptables rules: {'-A DHCP -m mark --mark 0x67005 -j DROP'} This is caused by the oscillation logic: sonic-net/sonic-linkmgrd#221 As there is no icmp_responder running, the mux will start flap, if it flaps betrween expected_dhcp_rules_for_standby fixture and the real iptables check, there could be some unexpected dhcp iptables which will cause case failure. What is the motivation for this PR? Fix Unexpected DHCP iptables rules for test_cacl_application_dualtor How did you do it? test_cacl_application_dualtor is used to verify dhcp iptables on dualtor, not oscillation. Before dualtor case, set mux mode to manual sudo config mux mode manual all in test teardown, run to set to auto back. sudo config mux mode auto all How did you verify/test it? Run cacl/test_cacl_application.py::test_cacl_application_dualtor on dualtor testbed against master image. Any platform specific information?
8 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of PR
Summary:
Fixes # (issue)
This PR is to cover an extreme edge case when none of the dualtors can receive ICMP heartbeat. The goal is to oscillate mux direction between two sides so if one side recovers, mux direction can have a chance to park on this side.
Note that toggles in an unhealthy scenario like this will cause longer disruption than in a healthy scenario.
sign-off: Jing Zhang [email protected]
Type of change
Approach
What is the motivation for this PR?
To avoid mux direction from being parked on one side when missing heartbeat.
Work item tracking
25367027
TODO: will submit a separate PR for option to disable oscillation or increase interval.
How did you do it?
How did you verify/test it?
Run the change on lab devices. Oscillation happened like expected (adjusted interval to 1min on lab device):
Any platform specific information?
Documentation