Using 802.3ad bonded interfaces #151

marthin-vandyk · 2023-02-09T10:25:18Z

marthin-vandyk
Feb 9, 2023

Hi all,

Am trying to use bonded interfaces on the Linux host, since 'Multithreaded IO is not supported for LAG member interfaces' as per the docs.

Not getting the network interfaces to resolve however.

Can ping the network interface, as per its config in the json, form the BNG DUT.
Have done a test VLAN subinterface and can ping the BNG DUT from the bngblaster host.

However when running the test, it does not progress. If I run the exact same scenario without bonding, it works fine. Have checked
whether the links correspond between the bngblaster host and BNG DUT many times.

Any thoughts perhaps?

many thanks

Here is the bngblaster and json I am using.

root@dell630:~# bngblaster -v
Version: DEV
Compiler: GNU (11.3.0)
GIT:
  REF: main
  SHA: 42975adeeac7944ddecb7cba441925531088d321
IO Modes: packet_mmap_raw (default), packet_mmap, raw, dpdk

{
    "interfaces": {
	"io-slots": 16384,
	"network": {
            "interface": "bond1",
            "address": "100.127.6.2/24",
            "gateway": "100.127.6.1",
            "address-ipv6": "2001:db8::2/64",
            "gateway-ipv6": "2001:db8::1",
            "vlan": 99
        },
        "access": {
            "interface": "bond2",
            "vlan-mode": "N:1",
	    "outer-vlan": 100,
            "type": "pppoe",
            "stream-group-id": 1
        }
    },
    "streams": [
	{
	    "name": "general4",
	    "stream-group-id": 1,
	    "type": "ipv4",
	    "direction": "downstream",
	    "bps": 1536000,
	    "length": 1440,
	    "network-interface": "bond1:99"
        }
    ],
    "pppoe": {
        "reconnect": true
    },
    "ppp": {
        "mru": 1492,
        "authentication": {
            "username": "test@bng",
            "password": "password",
            "timeout": 3,
            "retry": 5,
            "authentication-protocol": "PAP"
        },
        "lcp": {
            "conf-request-timeout": 1,
            "conf-request-retry": 10,
            "keepalive-interval": 30,
            "keepalive-retry": 3
        },
        "ipcp": {
            "enable": true,
            "request-ip": true,
            "request-dns1": true,
            "request-dns2": true,
            "conf-request-timeout": 1,
            "conf-request-retry": 10
        },
        "ip6cp": {
            "enable": false,
            "conf-request-timeout": 1,
            "conf-request-retry": 10
        }
    },
    "dhcpv6": {
        "enable": false,
        "retry": 2
    }
}

Answered by marthin-vandyk

Feb 9, 2023

Found it. With the various changes on the BNG DUT did not remember to update the attached routing interfaces.

Have it working with 802.3ad interfaces on the Linux host, with bngblaster calling them direct. Have not gotten it working yet with bgnblaster defining the LAG though ... for another day :)

View full answer

marthin-vandyk · 2023-02-09T16:31:42Z

marthin-vandyk
Feb 9, 2023
Author

Tried using LAG config inside bngblaster, with 802.3ad on the BNG DUT. No go yet ..

{
    "interfaces": {
	    "io-slots": 16384,
        "lag": [
            {
                "interface": "lag1",
                "lacp": true,
                "lacp-timeout-short": true
            },
            {
                "interface": "lag2",
                "lacp": true,
                "lacp-timeout-short": true
            }

        ],
        "links": [
            {
                "interface": "enp129s0f0",
                "lag-interface": "lag1"
            },
            {
                "interface": "enp129s0f1",
                "lag-interface": "lag1"
            },
            {
                "interface": "enp129s0f2",
                "lag-interface": "lag2"
            },
            {
                "interface": "enp129s0f3",
                "lag-interface": "lag2"
            }

        ],
	"network": {
            "interface": "lag1",
            "address": "100.127.6.2/24",
            "gateway": "100.127.6.1",
            "address-ipv6": "2001:db8::2/64",
            "gateway-ipv6": "2001:db8::1",
            "vlan": 99
        },
        "access": {
            "interface": "lag2",
            "vlan-mode": "N:1",
	    "outer-vlan": 100,
            "type": "pppoe",
            "stream-group-id": 1
        }
    },
    "streams": [
	{
	    "name": "general4",
	    "stream-group-id": 1,
	    "type": "ipv4",
	    "direction": "downstream",
	    "bps": 1536000,
	    "length": 1440,
	    "network-interface": "lag1:99"
        }
    ],
    "pppoe": {
        "reconnect": true
    },
    "ppp": {
        "mru": 1492,
        "authentication": {
            "username": "test@bng",
            "password": "password",
            "timeout": 3,
            "retry": 5,
            "authentication-protocol": "PAP"
        },
        "lcp": {
            "conf-request-timeout": 1,
            "conf-request-retry": 10,
            "keepalive-interval": 30,
            "keepalive-retry": 3
        },
        "ipcp": {
            "enable": true,
            "request-ip": true,
            "request-dns1": true,
            "request-dns2": true,
            "conf-request-timeout": 1,
            "conf-request-retry": 10
        },
        "ip6cp": {
            "enable": false,
            "conf-request-timeout": 1,
            "conf-request-retry": 10
        }
    },
    "dhcpv6": {
        "enable": false,
        "retry": 2
    }
}

0 replies

marthin-vandyk · 2023-02-09T17:36:47Z

marthin-vandyk
Feb 9, 2023
Author

Found it. With the various changes on the BNG DUT did not remember to update the attached routing interfaces.

Have it working with 802.3ad interfaces on the Linux host, with bngblaster calling them direct. Have not gotten it working yet with bgnblaster defining the LAG though ... for another day :)

0 replies

GIC-de · 2023-02-09T20:52:55Z

GIC-de
Feb 9, 2023
Maintainer

Thanks for your feedback. Actually we use the LAG feature internally quite extensive and should work fine. You can also start the BNG Blaster with the logging flag -l lag or use the cli/control command lag-info to get more details.

Please be also aware that there must be no bridge between BNG Blaster interfaces and device under test (DUT) because any bridges (including linux bridges) will drop LACP packets, but veth interfaces are fine! We use BNG Blaster LAG either directly or via veth interfaces.

0 replies

marthin-vandyk · 2023-05-29T11:11:37Z

marthin-vandyk
May 29, 2023
Author

hi Christian,

A follow-up on this one.

ran the scenario first time without LACP and nothing resolved.

root@bng-poc1-bngblaster:~# sudo bngblaster-cli run.sock lag-info
{
    "status": "ok",
    "code": 200,
    "lag-info": [
        {
            "id": 1,
            "interface": "lag1",
            "state": "Down",
            "state-transitions": 0,
            "members": [
                {
                    "interface": "enp130s0f4",
                    "state": "Up",
                    "state-transitions": 0,
                    "packets-rx": 0,
                    "packets-tx": 0,
                    "lacp-state": "Disabled"
                },
                {
                    "interface": "enp130s0f4d1",
                    "state": "Up",
                    "state-transitions": 0,
                    "packets-rx": 0,
                    "packets-tx": 0,
                    "lacp-state": "Disabled"
                }
            ]
        }
    ]
}

Second time the scenario ran with LACP, and it came up ok.

root@bng-poc1-bngblaster:~# sudo bngblaster-cli run.sock lag-info
{
    "status": "ok",
    "code": 200,
    "lag-info": [
        {
            "id": 1,
            "interface": "lag1",
            "state": "Up",
            "state-transitions": 1,
            "members": [
                {
                    "interface": "enp130s0f4",
                    "state": "Up",
                    "state-transitions": 1,
                    "packets-rx": 3485,
                    "packets-tx": 3566,
                    "lacp-state": "Current",
                    "lacp": {
                        "bpdu-rx": 6,
                        "bpdu-tx": 5,
                        "bpdu-dropped": 0,
                        "actor-system-id": "02:ff:ff:ff:ff:00",
                        "actor-system-priority": 32768,
                        "actor-key": 1,
                        "actor-port-priority": 32768,
                        "actor-port-id": 1,
                        "actor-state": 61,
                        "partner-system-id": "52:54:00:06:eb:7f",
                        "partner-system-priority": 32768,
                        "partner-key": 1,
                        "partner-port-priority": 32768,
                        "partner-port-id": 1,
                        "partner-state": 61
                    }
                },
                {
                    "interface": "enp130s0f4d1",
                    "state": "Up",
                    "state-transitions": 1,
                    "packets-rx": 29829,
                    "packets-tx": 29869,
                    "lacp-state": "Current",
                    "lacp": {
                        "bpdu-rx": 6,
                        "bpdu-tx": 4,
                        "bpdu-dropped": 0,
                        "actor-system-id": "02:ff:ff:ff:ff:00",
                        "actor-system-priority": 32768,
                        "actor-key": 1,
                        "actor-port-priority": 32768,
                        "actor-port-id": 2,
                        "actor-state": 61,
                        "partner-system-id": "52:54:00:06:eb:7f",
                        "partner-system-priority": 32768,
                        "partner-key": 1,
                        "partner-port-priority": 32768,
                        "partner-port-id": 2,
                        "partner-state": 61
                    }
                }
            ]
        }
    ]
}

Here is the json I am using for reference:

{
    "interfaces": {
	            "lag": [
            {
                "interface": "lag1",
                "lacp": true,
                "lacp-timeout-short": false
            }
        ],
        "links": [
            {
                "interface": "enp130s0f4",
                "lag-interface": "lag1"
            },
            {
                "interface": "enp130s0f4d1",
                "lag-interface": "lag1"
            }
        ],
	"network": {
            "interface": "lag1",
            "address": "100.127.1.2/24",
            "gateway": "100.127.1.1",
            "address-ipv6": "2001:db8:1::2/64",
            "gateway-ipv6": "2001:db8:1::1",
            "vlan": 100
        },
        "access": {
            "interface": "lag1",
            "vlan-mode": "N:1",
	    "outer-vlan": 702,
            "type": "pppoe",
            "stream-group-id": 1
        }
    },
    "streams": [
	{
	    "name": "general4",
	    "stream-group-id": 1,
	    "type": "ipv4",
	    "direction": "downstream",
	    "bps": 3072000,
	    "length": 1400,
	    "network-interface": "lag1:100"
        }
    ],
    "sessions": {
        "start-rate": 30,
        "stop-rate": 30
    },
    "pppoe": {
        "reconnect": true
    },
    "ppp": {
        "mru": 1492,
        "authentication": {
            "username": "[email protected]",
            "password": "testing",
            "timeout": 3,
            "retry": 5,
	    "protocol": "PAP"
        },
        "lcp": {
            "conf-request-timeout": 1,
            "conf-request-retry": 10,
            "keepalive-interval": 30,
            "keepalive-retry": 3
        },
        "ipcp": {
            "enable": true,
            "request-ip": true,
            "request-dns1": true,
            "request-dns2": true,
            "conf-request-timeout": 1,
            "conf-request-retry": 10
        },
        "ip6cp": {
            "enable": true,
            "conf-request-timeout": 1,
            "conf-request-retry": 10
        }
    },
    "dhcpv6": {
        "enable": true,
        "retry": 2
    }
}

Here is the log output from '-l lag' logging:

May 29 13:07:30.384655 LAG (lag1) New lag-interface created
May 29 13:07:30.427646 LAG (lag1) Interface enp130s0f4 added
May 29 13:07:30.471410 LAG (lag1) Interface enp130s0f4d1 added
May 29 13:07:30.471668 Opened control socket run.sock
May 29 13:07:31.471610 Resolve network interfaces
May 29 13:07:32.471795 Resolve network interfaces
May 29 13:07:33.471932 Resolve network interfaces
May 29 13:07:34.472069 Resolve network interfaces
May 29 13:07:35.472242 Resolve network interfaces
May 29 13:07:36.472430 Resolve network interfaces
May 29 13:07:37.472567 Resolve network interfaces
May 29 13:07:38.472761 Resolve network interfaces
May 29 13:07:39.472947 Resolve network interfaces
May 29 13:07:40.473084 Resolve network interfaces
May 29 13:07:41.473264 Resolve network interfaces
May 29 13:07:42.473401 Resolve network interfaces
May 29 13:07:43.473535 Resolve network interfaces
May 29 13:07:44.473715 Resolve network interfaces
May 29 13:07:45.473873 Resolve network interfaces
May 29 13:07:45.490358 LAG (lag1) Member interface enp130s0f4 state changed from Down to Up
May 29 13:07:45.490439 LAG (lag1) Interface state changed from Down to Up
May 29 13:07:46.474010 Resolve network interfaces
May 29 13:07:47.474192 Resolve network interfaces
May 29 13:07:47.474270 All network interfaces resolved
May 29 13:07:47.770978 ALL SESSIONS ESTABLISHED
May 29 13:07:48.073001 ALL STREAM TRAFFIC FLOWS VERIFIED
May 29 13:08:00.490682 LAG (lag1) Member interface enp130s0f4d1 state changed from Down to Up
May 29 13:10:41.948155 Teardown request
May 29 13:10:43.502115 ALL SESSIONS TERMINATED

0 replies

marthin-vandyk · 2023-05-29T18:42:28Z

marthin-vandyk
May 29, 2023
Author

Hi,

Some interesting points.

We opted to go for the Linux bonded interface, since using LAG inside bngblaster meant we could not do multi-threaded and lost performance.
Unfortunately DPDK is not an option since our NICs do not support it, otherwise we would go for the bngblaster LAG and DPDK.
When we run Linux bonded interfaces we could not get the xmit_hash_policy to have both 10G in the bond be used for transmit, no matter which we used (https://www.kernel.org/doc/Documentation/networking/bonding.txt).

Is this something perhaps anyone has seen, ie. where using Linux bonded interfaces will only hash to a single member and not consider the other, irrespective of the xmit_hash_policy?

How does bngblaster LAG decide to load-balance traffic on egress?

thanks!

2 replies

GIC-de May 30, 2023
Maintainer

If BNG Blaster is used for LAG (BNG Blaster handle LACP, ...), TX threads are not allowed, RX threads are still okay. The streams itself are distributed round robin over all active LAG member interfaces.

GIC-de May 30, 2023
Maintainer

Depending on the NIC model, it is very common that PPPoE traffic can't be hashed. I have similar issues with Intel X710 in RX path, where all traffic is received by single queue, even with DPDK. For DPDK I have some plans to fix this for at least some NIC models.

marthin-vandyk · 2023-05-30T19:23:31Z

marthin-vandyk
May 30, 2023
Author

many thanks for the info Christian. we are progressing ok with our use cases with the current setup and nuances.

blessings and chat later

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Using 802.3ad bonded interfaces #151

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 6 comments 2 replies

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Using 802.3ad bonded interfaces #151

marthin-vandyk Feb 9, 2023

Replies: 6 comments · 2 replies

marthin-vandyk Feb 9, 2023 Author

marthin-vandyk Feb 9, 2023 Author

GIC-de Feb 9, 2023 Maintainer

marthin-vandyk May 29, 2023 Author

marthin-vandyk May 29, 2023 Author

GIC-de May 30, 2023 Maintainer

GIC-de May 30, 2023 Maintainer

marthin-vandyk May 30, 2023 Author

marthin-vandyk
Feb 9, 2023

Replies: 6 comments 2 replies

marthin-vandyk
Feb 9, 2023
Author

marthin-vandyk
Feb 9, 2023
Author

GIC-de
Feb 9, 2023
Maintainer

marthin-vandyk
May 29, 2023
Author

marthin-vandyk
May 29, 2023
Author

GIC-de May 30, 2023
Maintainer

GIC-de May 30, 2023
Maintainer

marthin-vandyk
May 30, 2023
Author