Skip to content
This repository has been archived by the owner on Nov 19, 2020. It is now read-only.

Start monitoring TCP Connection counts. Enforce TCP Connection count limits by killing. #11

Open
justenwalker opened this issue Oct 19, 2018 · 0 comments
Labels
enhancement New feature or request help wanted Extra attention is needed

Comments

@justenwalker
Copy link
Contributor

There are a limited number of ephemeral ports on the machine. In Windows 2016 this is around 16k. A process which opens up many connections at once, or has a connection leak is likely to use up all available ephemeral ports - causing other services to fail to bind to their assigned nomad ports, and other outbound connection failures.

There already exists functionality in the win32 package to query the TCP Tables for connections by PID. We should use this to:

  1. Expose a damon_tcp_connection_count metric so that monitoring and alerting can be done in Prometheus
  2. Add an optional DAMON_TCP_CONNECTION_LIMIT configuration to set an upper bound on the number of TCP connections a process can make. If this option is set, we should terminate the child process when it exceeds this count.

There isn't a lot we can do to prevent an application from opening another connection; so the safest option would be to terminate it to prevent it from growing unbounded.

This also mean we'd need to enumerate the entire process tree, since there could be more than one process under the child that is asking for connections. Terminating the parent process should be enough to kill the child processes as well given that, once Damon exits, the job object will terminate all processes still in the JobObject since the last handle will have closed.

@justenwalker justenwalker added enhancement New feature or request help wanted Extra attention is needed labels Oct 19, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement New feature or request help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

1 participant