Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixes: 17292 - Detect infinite loop condition while iterating terminations in CablePath.from_origin #17293

Closed
wants to merge 11 commits into from

Conversation

bctiemann
Copy link
Contributor

@bctiemann bctiemann commented Aug 28, 2024

Fixes: #17292

Adds a check to Step 4 of CablePath.from_origin to make sure the path length is not exceeding CABLE_TRACE_MAX_LENGTH (a new setting).

Also adds an optional max_length parameter to from_origin for use as a testing hook, to determine whether the loop detection is working.

@bctiemann bctiemann self-assigned this Aug 28, 2024
@bctiemann bctiemann requested a review from DanSheps September 2, 2024 21:38
Copy link
Contributor

github-actions bot commented Oct 3, 2024

This PR has been automatically marked as stale because it has not had recent activity. It will be closed automatically if no further action is taken.

@github-actions github-actions bot added the pending closure Requires immediate attention to avoid being closed for inactivity label Oct 3, 2024
@DanSheps DanSheps removed the pending closure Requires immediate attention to avoid being closed for inactivity label Oct 3, 2024
Copy link
Contributor

github-actions bot commented Nov 3, 2024

This PR has been automatically marked as stale because it has not had recent activity. It will be closed automatically if no further action is taken.

@github-actions github-actions bot added the pending closure Requires immediate attention to avoid being closed for inactivity label Nov 3, 2024
@bctiemann bctiemann requested a review from arthanson November 4, 2024 19:11
Comment on lines 722 to 727
num_elements_added = len(path) - len(prev_path)
prev_elements_added = prev_path[-num_elements_added:]
elements_added = path[-num_elements_added:]
if elements_added == prev_elements_added:
logger.warning('Infinite loop detected while updating cable path trace')
break
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I am understanding this correctly, this iwll only detect a loop in the last element, not if it is 3 elements back

Eg:

FP1 <> RP1 <> RP2 <> FP2 <> FP3 <> RP3 <> RP4 <> FP4 <> FP1

Could youy confirm this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please re-review; I moved the check to earlier in the process and made it more straightforward. It only checks cables in the body of the trace, not the beginning and end terminations, but I doubt there's risk there.

@DanSheps DanSheps self-requested a review November 18, 2024 22:24
@@ -2343,3 +2344,41 @@ def test_401_exclude_midspan_devices(self):
is_active=True
)
self.assertEqual(CablePath.objects.count(), 0)

def test_detect_infinite_loop(self):
Copy link
Member

@DanSheps DanSheps Nov 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this test suppose to fail or succeed? As it stands I don't see a loop in the topology unless I am missing something.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If there's an infinite loop then the test will just keep executing. I'm not sure of an elegant way to test the infinite looping condition, but as long as it executes without the whole test process hanging then the test passes.

I've tried a few different approaches for making it assert if the execution takes too long or loops too many times, mostly involving timeouts and/or threading, but I don't know if it's worth going to that extreme. What would you suggest?

In any case if you comment out lines 600-602 in cables.py, the test will hang indefinitely.

Copy link
Member

@DanSheps DanSheps Nov 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO, for testing purposes we may want to add a parameter that sets a maximum path length (might not be bad anyways to set it to an unreasonable number) and still create a loop.

IDK if it is worth it though TBH. It is just this test doesn't really do anything as far as testing for a "loop" though., which was more my point. Might not be worth including a test even for this.

@bctiemann bctiemann removed the pending closure Requires immediate attention to avoid being closed for inactivity label Nov 20, 2024
Copy link
Member

@jeremystretch jeremystretch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems like this condition can be avoided by checking for the presence of each encountered termination in the path so far: If we see the same termination twice, we're in a loop. Could this be implemented under Cable.trace()?

@bctiemann
Copy link
Contributor Author

Short answer -- no, I don't think so. The from_origin method that has the infinite loop possibility in it is called from a variety of places, including create_cablepath and retrace_cable_paths, and defending against it at that level would have to be done in all those locations. Similarly Cable.trace is not called in the post-save signal flow that triggers the infinite loop condition I found; it goes straight to from_origin where it falls into the loop. I think the only reasonable place to handle this is in from_origin itself.

@bctiemann
Copy link
Contributor Author

I'm going to close this PR, as it's sucking up a lot of time that is better spent elsewhere. I believe the solution here is the correct and least invasive way to deal with this issue, so if anyone picks up this issue in the future this is a potential valid place to start from.

@bctiemann bctiemann closed this Dec 16, 2024
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jan 16, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Detect infinite loop in cable paths
3 participants