cEOS MPLS working example?

[bgolab]

Hi,
I wonder if anyone could share any MPLS working example for the cEOS.

I am facing some issues (LFIB is not populated even though the LDP is successfully neighbored)with the LDP I would like to verify working configuration for the CEOS to exclude system issues related to WSL2/ubuntu, etc.

Thanks,
Bogdan

[hellt]

Hi
did we chat on discord with you about it? If not, I suggest you have a look at arista channel.
TLDR it seems ceos doens't support MPLS dataplane, you may have better luck asking this on arista forum

[bgolab]

Probably it wasn't me as I do not like discord (it is too messy for me).

Yes, it looks like the data plain is broken.

Still looking for ideal container-based platform;) At least working one...

[hellt]

ah, okay, because the timing was very similar hah

you can check the (growing) MPLS support that we have in SR Linux. Ivan and Jeroen wrote about it recently https://github.com/ipspace/netsim-examples/tree/master/routing/sr-mpls-bgp-srlinux

[bgolab]

oh, yes. I am in touch with Ivan.

Thank you

[bgolab]

Just to update: the LFIB was not updated because of WSL2 - I moved to the VirtualBox - ubuntu VM and everything is fine.

[hellt]

can you also share if you are able to verify that the dataplane is working? By using mpls ping for example?

[bgolab]

Yes, sure.

ceos1#sh mpls lfib route
[...]
L 116384 [1], 20.0.0.1/32
via M, 8.0.0.2, pop
payload autoDecide, ttlMode uniform, apply egress-acl
interface Ethernet2

ceos1#ping mpls ldp ip 20.0.0.1/32
LSP ping to LDP route 20.0.0.1/32
timeout is 5000ms, interval is 1000ms
Via 8.0.0.2, Ethernet2, label stack: [3]
Reply from 8.0.0.2: seq=1, time=3ms, success: egress ok
Via 8.0.0.2, Ethernet2, label stack: [3]
Reply from 8.0.0.2: seq=2, time=13ms, success: egress ok

Let me know if this is what you wanted to verify.

[bgolab]

Captured on the 2nd router
bash-4.2# tcpdump -i eth2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth2, link-type EN10MB (Ethernet), snapshot length 262144 bytes
20:42:04.338726 aa:c1:ab:f2:43:5d (oui Unknown) > 01:00:5e:00:00:02 (oui Unknown), ethertype IPv4 (0x0800), length 84: 8.0.0.1.ldp > 224.0.0.2.ldp: LDP, Label-Space-ID: 10.0.0.1:0, pdu-length: 38
20:42:04.434068 aa:c1:ab:6b:71:b3 (oui Unknown) > 01:00:5e:00:00:02 (oui Unknown), ethertype IPv4 (0x0800), length 84: ceos2.ldp > 224.0.0.2.ldp: LDP, Label-Space-ID: ceos2:0, pdu-length: 38
20:42:04.558393 aa:c1:ab:f2:43:5d (oui Unknown) > aa:c1:ab:6b:71:b3 (oui Unknown), ethertype IPv4 (0x0800), length 94: 8.0.0.1.35001 > localhost.localdomain.lsp-ping: LSP-PINGv1, MPLS Echo Request, seq 260, length: 48
20:42:04.559449 aa:c1:ab:6b:71:b3 (oui Unknown) > aa:c1:ab:f2:43:5d (oui Unknown), ethertype IPv4 (0x0800), length 74: ceos2.lsp-ping > 8.0.0.1.35001: LSP-PINGv1, MPLS Echo Reply, seq 260, length: 32
20:42:05.558598 aa:c1:ab:f2:43:5d (oui Unknown) > aa:c1:ab:6b:71:b3 (oui Unknown), ethertype IPv4 (0x0800), length 94: 8.0.0.1.35001 > localhost.localdomain.lsp-ping: LSP-PINGv1, MPLS Echo Request, seq 261, length: 48
20:42:05.559301 aa:c1:ab:6b:71:b3 (oui Unknown) > aa:c1:ab:f2:43:5d (oui Unknown), ethertype IPv4 (0x0800), length 74: ceos2.lsp-ping > 8.0.0.1.35001: LSP-PINGv1, MPLS Echo Reply, seq 261, length: 32
20:42:06.659272 aa:c1:ab:f2:43:5d (oui Unknown) > aa:c1:ab:6b:71:b3 (oui Unknown), ethertype IPv4 (0x0800), length 94: 8.0.0.1.35001 > localhost.localdomain.lsp-ping: LSP-PINGv1, MPLS Echo Request, seq 262, length: 48
20:42:06.660087 aa:c1:ab:6b:71:b3 (oui Unknown) > aa:c1:ab:f2:43:5d (oui Unknown), ethertype IPv4 (0x0800), length 74: ceos2.lsp-ping > 8.0.0.1.35001: LSP-PINGv1, MPLS Echo Reply, seq 262, length: 32

[bgolab]

router 1
interface Ethernet1
no switchport
ip address 7.0.0.200/24
ip ospf area 0.0.0.0
interface Ethernet2
no switchport
ip address 8.0.0.1/24
ip ospf area 0.0.0.0
interface Loopback0
ip address 10.0.0.1/32
ip ospf area 0.0.0.0

ip routing
router ospf 1

mpls ip
mpls ldp
router-id interface Loopback0
no shutdown

router2
interface Ethernet1
no switchport
ip address 9.0.0.1/24
ip ospf area 0.0.0.0
interface Ethernet2
no switchport
ip address 8.0.0.2/24
ip ospf area 0.0.0.0
interface Loopback0
ip address 20.0.0.1/32
ip ospf area 0.0.0.0

ip routing
router ospf 1

mpls ip
mpls ldp
router-id interface Loopback0
no shutdown

[hmntsharma]

I am sorry to be the bearer of bad news but that is not the full picture of the topology.

hostname R1
!
spanning-tree mode mstp
!
management api http-commands
no shutdown
!
management api gnmi
transport grpc default
!
management api netconf
transport ssh default
!
interface Ethernet1
no switchport
ip address 8.0.0.1/24
ip ospf area 0.0.0.0
!
interface Loopback0
ip address 10.0.0.1/32
ip ospf area 0.0.0.0
!
interface Management0
ip address 172.20.20.4/24
ipv6 address 2001:172:20:20::4/64
!
ip routing
!
mpls ip
!
mpls ldp
router-id interface Loopback0
no shutdown
!
router ospf 1
max-lsa 12000
!
end

hostname R2
!
spanning-tree mode mstp
!
management api http-commands
no shutdown
!
management api gnmi
transport grpc default
!
management api netconf
transport ssh default
!
interface Ethernet1
no switchport
ip address 8.0.0.2/24
ip ospf area 0.0.0.0
!
interface Ethernet2
no switchport
ip address 9.0.0.1/24
ip ospf area 0.0.0.0
!
interface Loopback0
ip address 20.0.0.1/32
ip ospf area 0.0.0.0
!
interface Management0
ip address 172.20.20.3/24
ipv6 address 2001:172:20:20::3/64
!
ip routing
!
mpls ip
!
mpls ldp
router-id interface Loopback0
no shutdown
!
router ospf 1
max-lsa 12000
!
end

hostname R3
!
spanning-tree mode mstp
!
management api http-commands
no shutdown
!
management api gnmi
transport grpc default
!
management api netconf
transport ssh default
!
interface Ethernet2
no switchport
ip address 9.0.0.2/24
ip ospf area 0.0.0.0
!
interface Loopback0
ip address 30.0.0.1/32
ip ospf area 0.0.0.0
!
interface Management0
ip address 172.20.20.2/24
ipv6 address 2001:172:20:20::2/64
!
ip routing
!
mpls ip
!
mpls ldp
router-id interface Loopback0
no shutdown
!
router ospf 1
max-lsa 12000
!
end

Let's see some results:

So just like yours, the mpls ping between the directly connected routers seems to be working okay.

R1#ping mpls ldp ip 20.0.0.1/32 repeat 1
LSP ping to LDP route 20.0.0.1/32
timeout is 5000ms, interval is 1000ms
Via 8.0.0.2, Ethernet1, label stack: [3]
Reply from 8.0.0.2: seq=1, time=0ms, success: egress ok

--- LDP target fec 20.0.0.1/32 : lspping statistics ---
Via 8.0.0.2, Ethernet1, label stack: [3]
1 packets transmitted, 1 received, 0% packet loss, time 152ms
1 received from 8.0.0.2, rtt min/max/avg 0/0/0 ms

R1#

But ping to the indirect hop is not working.

R1#ping mpls ldp ip 30.0.0.1/32 repeat 1
LSP ping to LDP route 30.0.0.1/32
timeout is 5000ms, interval is 1000ms
Via 8.0.0.2, Ethernet1, label stack: [116385]
Request timeout

--- LDP target fec 30.0.0.1/32 : lspping statistics ---
Via 8.0.0.2, Ethernet1, label stack: [116385]
1 packets transmitted, 0 received, 100% packet loss, time 5158ms

R1#

The LFIB is okay

R1#sh mpls lfib route detail | sec payload
L 116384 [1], 20.0.0.1/32
via M, 8.0.0.2, pop
payload autoDecide, ttlMode uniform, apply egress-acl
L 116385 [1], 30.0.0.1/32
via M, 8.0.0.2, swap 116385
payload autoDecide, ttlMode uniform, apply egress-acl
R1#

R2#sh mpls lfib route detail | sec payload
L 116384 [1], 10.0.0.1/32
via M, 8.0.0.1, pop
payload autoDecide, ttlMode uniform, apply egress-acl
L 116385 [1], 30.0.0.1/32
via M, 9.0.0.2, pop
payload autoDecide, ttlMode uniform, apply egress-acl
R2#

R3#sh mpls lfib route detail | sec payload
L 116384 [1], 20.0.0.1/32
via M, 9.0.0.1, pop
payload autoDecide, ttlMode uniform, apply egress-acl
L 116385 [1], 10.0.0.1/32
via M, 9.0.0.1, swap 116384
payload autoDecide, ttlMode uniform, apply egress-acl
R3#

At first I thought it is still okay, if vrf traffic can get across, even if it is between directly connected routers.

So I went ahead and created a vrf with a loopback1 interface as below on all three routers, but it did not work.

What surprises me is that remote routes wouldn't even get installed in the vrf table.

R1
vrf instance CEOS
ip routing vrf CEOS
interface Loopback1
vrf CEOS
ip address 192.168.1.1/32
!
router bgp 65001
neighbor 20.0.0.1 remote-as 65001
neighbor 20.0.0.1 update-source Loopback0
neighbor 30.0.0.1 remote-as 65001
neighbor 30.0.0.1 update-source Loopback0
!
address-family vpn-ipv4
neighbor 20.0.0.1 activate
neighbor 30.0.0.1 activate
neighbor 20.0.0.1 encapsulation mpls next-hop-self source-interface Loopback0
neighbor 30.0.0.1 encapsulation mpls next-hop-self source-interface Loopback0
!
vrf CEOS
rd 65001:100
route-target import vpn-ipv4 65001:100
route-target export vpn-ipv4 65001:100
redistribute connected
!

R2
vrf instance CEOS
ip routing vrf CEOS
interface Loopback1
vrf CEOS
ip address 192.168.1.2/32
!
router bgp 65001
neighbor 10.0.0.1 remote-as 65001
neighbor 10.0.0.1 update-source Loopback0
neighbor 30.0.0.1 remote-as 65001
neighbor 30.0.0.1 update-source Loopback0
!
address-family vpn-ipv4
neighbor 10.0.0.1 activate
neighbor 30.0.0.1 activate
neighbor 10.0.0.1 encapsulation mpls next-hop-self source-interface Loopback0
neighbor 30.0.0.1 encapsulation mpls next-hop-self source-interface Loopback0
!
vrf CEOS
rd 65001:100
route-target import vpn-ipv4 65001:100
route-target export vpn-ipv4 65001:100
redistribute connected
!

R3
vrf instance CEOS
ip routing vrf CEOS
interface Loopback1
vrf CEOS
ip address 192.168.1.3/32
!
router bgp 65001
neighbor 10.0.0.1 remote-as 65001
neighbor 10.0.0.1 update-source Loopback0
neighbor 20.0.0.1 remote-as 65001
neighbor 20.0.0.1 update-source Loopback0
!
address-family vpn-ipv4
neighbor 10.0.0.1 activate
neighbor 20.0.0.1 activate
neighbor 10.0.0.1 encapsulation mpls next-hop-self source-interface Loopback0
neighbor 20.0.0.1 encapsulation mpls next-hop-self source-interface Loopback0
!
vrf CEOS
rd 65001:100
route-target import vpn-ipv4 65001:100
route-target export vpn-ipv4 65001:100
redistribute connected
!

Output

R1#sh mpls lfib route
B3 100000 [0]
via I, ipv4, vrf CEOS
L 116384 [1], 20.0.0.1/32
via M, 8.0.0.2, pop
payload autoDecide, ttlMode uniform, apply egress-acl
interface Ethernet1
L 116385 [1], 30.0.0.1/32
via M, 8.0.0.2, swap 116385
payload autoDecide, ttlMode uniform, apply egress-acl
interface Ethernet1
R1#

R1#sh bgp vpn-ipv4 summary
BGP summary information for VRF default
Router identifier 10.0.0.1, local AS number 65001
Neighbor Status Codes: m - Under maintenance
Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc
20.0.0.1 4 65001 31 33 0 0 00:17:27 Estab 1 1
30.0.0.1 4 65001 32 31 0 0 00:17:02 Estab 1 1
R1#
R1#sh bgp vpn-ipv4
BGP routing table information for VRF default
Router identifier 10.0.0.1, local AS number 65001
Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP
S - Stale, c - Contributing to ECMP, b - backup, L - labeled-unicast
% - Pending BGP convergence
Origin codes: i - IGP, e - EGP, ? - incomplete
AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop

      Network                Next Hop              Metric  LocPref Weight  Path

• RD: 65001:100 IPv4 prefix 192.168.1.1/32
  

                            -                     -       -       0       i
  

• RD: 65001:100 IPv4 prefix 192.168.1.2/32
  

                            20.0.0.1              -       100     0       i
  

• RD: 65001:100 IPv4 prefix 192.168.1.3/32
  

                            30.0.0.1              -       100     0       i
  

R1#
R1#sh bgp vpn-ipv4 rd 65001:100 detail
BGP routing table information for VRF default
Router identifier 10.0.0.1, local AS number 65001
BGP routing table entry for IPv4 prefix 192.168.1.1/32, Route Distinguisher: 65001:100
Paths: 1 available
Local
- from - (0.0.0.0)
Origin IGP, metric -, localpref -, weight 0, valid, local, best, redistributed (Connected)
Extended Community: Route-Target-AS:65001:100
Local MPLS label (VRF label): 100000
BGP routing table entry for IPv4 prefix 192.168.1.2/32, Route Distinguisher: 65001:100
Paths: 1 available
Local
20.0.0.1 from 20.0.0.1 (20.0.0.1)
Origin IGP, metric -, localpref 100, weight 0, valid, internal, best
Remote MPLS label: 100000
BGP routing table entry for IPv4 prefix 192.168.1.3/32, Route Distinguisher: 65001:100
Paths: 1 available
Local
30.0.0.1 from 30.0.0.1 (30.0.0.1)
Origin IGP, metric -, localpref 100, weight 0, valid, internal, best
Remote MPLS label: 100000
R1#
R1#
R1#sh ip route vrf CEOS

VRF: CEOS
Codes: C - connected, S - static, K - kernel,
O - OSPF, IA - OSPF inter area, E1 - OSPF external type 1,
E2 - OSPF external type 2, N1 - OSPF NSSA external type 1,
N2 - OSPF NSSA external type2, B - Other BGP Routes,
B I - iBGP, B E - eBGP, R - RIP, I L1 - IS-IS level 1,
I L2 - IS-IS level 2, O3 - OSPFv3, A B - BGP Aggregate,
A O - OSPF Summary, NG - Nexthop Group Static Route,
V - VXLAN Control Service, M - Martian,
DH - DHCP client installed default route,
DP - Dynamic Policy Route, L - VRF Leaked,
G - gRIBI, RC - Route Cache Route

Gateway of last resort is not set

C 192.168.1.1/32 is directly connected, Loopback1

R1#

R2#sh bgp vpn-ipv4 rd 65001:100 detail
BGP routing table information for VRF default
Router identifier 20.0.0.1, local AS number 65001
BGP routing table entry for IPv4 prefix 192.168.1.1/32, Route Distinguisher: 65001:100
Paths: 1 available
Local
10.0.0.1 from 10.0.0.1 (10.0.0.1)
Origin IGP, metric -, localpref 100, weight 0, valid, internal, best
Remote MPLS label: 100000
BGP routing table entry for IPv4 prefix 192.168.1.2/32, Route Distinguisher: 65001:100
Paths: 1 available
Local
- from - (0.0.0.0)
Origin IGP, metric -, localpref -, weight 0, valid, local, best, redistributed (Connected)
Extended Community: Route-Target-AS:65001:100
Local MPLS label (VRF label): 100000
BGP routing table entry for IPv4 prefix 192.168.1.3/32, Route Distinguisher: 65001:100
Paths: 1 available
Local
30.0.0.1 from 30.0.0.1 (30.0.0.1)
Origin IGP, metric -, localpref 100, weight 0, valid, internal, best
Remote MPLS label: 100000
R2#

I think I am missing something as to why the vrf routes are not installed but I will get back to it after some time.
Also, there are some platforms, where the vpn route is not installed in the vrf table if the transport label is missing, perhaps it is checking the linux mpls fib and does not see any label, hence does not install the route.

Or, as mentioned initially, MPLS is not supported in cEOS.

Kindly, let me know if you want more outputs.

Cheers!

[bgolab]

Thank you for your notes.

Yes, I was going to try the MPLS VPN. I will try it.

[bgolab]

Yes, I see the same. The mpls ping is broken. The MPLS Echo Request reaches the R2 router but it is not forwarded to R3 (verified with tcpdump)

[hmntsharma]

Now, check this out...

FRRouting/frr#4872

Someone had similar issue and there is a reply which claims to have made it work but I couldn't replicate it, yet.
Appears to be a limitation of docker networking.

See if you make any progress, I will also try to get back to it at some point this week, with fresh virtual server,.

Forgot to mention, there is this post as well.

https://github.com/ipspace/netsim-examples/tree/master/routing/sr-mpls-bgp-srlinux

Let us know, if you can replicate at least this one.

[bgolab]

Thank you for the hint.

When you try to enforce mpls processing in eth interfaces through the sysctl you get:
sysctl: cannot stat /proc/sys/net/mpls/conf/eth1/input: No such file or directory
sysctl: cannot stat /proc/sys/net/mpls/conf/eth2/input: No such file or directory

Someone already brought this up here:
FRRouting/frr#1710

Maybe we need to use modified (custom version) of the linux kernel...

Will look into this this week.

[hmntsharma]

I read somewhere that mpls is supported in kernel 4.3 onwards and I was using 4.15.

I was testing this last night and I enabled mpls on the ethX interfaces of cEOS routers, by accessing their shell instead of CLI. increased label limit too. Did the same on the linux host as well, for the vETH interfaces, which are actually used by the containers for routing.

Nothing worked.

The mpls labels are not getting programmed, only the routing part does.

[hmntsharma]

Might need to do that manually, to at least test the function.
https://liuhangbin.netlify.app/post/mpls-on-linux/

But then there is no stopping to it, then we'd need to program the vpn labels for the vrf as well, eventually moving to either srlinux or cumulus.

https://legacy.netdevconf.info/1.1/proceedings/slides/ahern-vrf-tutorial.pdf

[bgolab]

My kernel version is 5.13. Yes, I was thinking about mowing to SR Linux or Cumulus.

The beauty of the Arista cEOS is CLI as I come from the Cisco world;)

[bgolab]

Agree. I think we should do the test with pure docker bridges (named) to interconnect docker containers.
Not sure if this is possible directly from the containerlab - I used manually created bridge with the container lab only (NOT created through the docker).

I guess we need to do it manually using docker only - without the containerlab.

[hmntsharma]

I had the same thought and just finished testing that, still no luck

sudo docker network create --internal R1_R2
sudo docker network create --internal R2_R3

sudo docker run -d --privileged --name R1 --mount type=bind,source=/home/lab/clab/frrlab/daemons,target=/etc/frr/daemons frrouting/frr:latest
sudo docker run -d --privileged --name R2 --mount type=bind,source=/home/lab/clab/frrlab/daemons,target=/etc/frr/daemons frrouting/frr:latest
sudo docker run -d --privileged --name R3 --mount type=bind,source=/home/lab/clab/frrlab/daemons,target=/etc/frr/daemons frrouting/frr:latest

sudo docker network connect R1_R2 R1
sudo docker network connect R1_R2 R2
sudo docker network connect R2_R3 R2
sudo docker network connect R2_R3 R3

Then set the sysctl mpls interfaces to 1 on the host, as well as the FRR bash for each router, increased the label limit too to 100k

lab@ubuntu1804:~/clab/frrlab$ sudo docker exec -it R1 bash
bash-5.1# sysctl -a | grep mpls
net.mpls.conf.eth0.input = 1
net.mpls.conf.eth1.input = 1
net.mpls.conf.lo.input = 1
net.mpls.default_ttl = 255
net.mpls.ip_ttl_propagate = 1
net.mpls.platform_labels = 100000
bash-5.1#

lab@ubuntu1804:~/clab/frrlab$ sudo docker exec -it R2 bash
bash-5.1# sysctl -a | grep mpls
net.mpls.conf.eth0.input = 1
net.mpls.conf.eth1.input = 1
net.mpls.conf.eth2.input = 1
net.mpls.conf.lo.input = 1
net.mpls.default_ttl = 255
net.mpls.ip_ttl_propagate = 1
net.mpls.platform_labels = 100000
bash-5.1#

lab@ubuntu1804:~/clab/frrlab$ sudo docker exec -it R3 bash
bash-5.1# sysctl -a | grep mpls
net.mpls.conf.eth0.input = 1
net.mpls.conf.eth1.input = 1
net.mpls.conf.lo.input = 1
net.mpls.default_ttl = 255
net.mpls.ip_ttl_propagate = 1
net.mpls.platform_labels = 100000
bash-5.1#

lab@ubuntu1804:~/clab/frrlab$ sudo sysctl -a | grep mpls
net.mpls.conf.br-339d8d5d0b7d.input = 1
net.mpls.conf.br-ad19fdf6fdd8.input = 1
net.mpls.conf.docker0.input = 1
net.mpls.conf.ens3.input = 0
net.mpls.conf.lo.input = 0
net.mpls.conf.veth02ab428.input = 1
net.mpls.conf.veth37d52af.input = 1
net.mpls.conf.veth693c8be.input = 1
net.mpls.conf.veth6b1ff4e.input = 1
net.mpls.conf.vethe3b6a47.input = 1
net.mpls.conf.vethee1e0dc.input = 1
net.mpls.conf.vethfcf851a.input = 1
net.mpls.default_ttl = 255
net.mpls.ip_ttl_propagate = 1
net.mpls.platform_labels = 1048575
lab@ubuntu1804:~/clab/frrlab$

All the other outputs are same from FRR vtysh and bash shell, without LDP the routing works, with LDP it doesn't

R1# sh ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

K>* 0.0.0.0/0 [0/0] via 172.17.0.1, eth0, 00:26:23
O   1.1.1.1/32 [110/0] is directly connected, lo, weight 1, 00:23:40
C>* 1.1.1.1/32 is directly connected, lo, 00:23:47
O>* 2.2.2.2/32 [110/10] via 172.18.0.3, eth1, label implicit-null, weight 1, 00:22:10
O>* 3.3.3.3/32 [110/20] via 172.18.0.3, eth1, label 17, weight 1, 00:21:21
C>* 172.17.0.0/16 is directly connected, eth0, 00:26:23
O   172.18.0.0/16 [110/10] is directly connected, eth1, weight 1, 00:23:40
C>* 172.18.0.0/16 is directly connected, eth1, 00:24:53
O>* 172.19.0.0/16 [110/20] via 172.18.0.3, eth1, label implicit-null, weight 1, 00:22:10
R1#

R1# sh mpls table
 Inbound Label  Type  Nexthop     Outbound Label
 -------------------------------------------------
 16             LDP   172.18.0.3  implicit-null
 17             LDP   172.18.0.3  17
 18             LDP   172.18.0.3  implicit-null

R1# ping 3.3.3.3
PING 3.3.3.3 (3.3.3.3): 56 data bytes
^C
--- 3.3.3.3 ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss
R1#

All the other config remains the same, except the interface IP addresses were assigned by docker from 172.x.x.x

Not sure what did I miss or do wrong this time.

[bgolab]

I also tried to disable the RP filter:
systcl -w net.ipv4.conf.all.rp_filter=0
sysctl -w net.ipv4.conf.lo.rp_filter=0

following the link regarding SR routing:
https://github.com/FRRouting/frr/blob/master/doc/developer/ospf-sr.rst#configuration

still no success.

[bgolab]

Now phase II - advanced troubleshooting. Started talking to docker guys;)

[bgolab]

Talked to a guy who managed to setup the Arista EOS with MPLS on VM and he said that MPLS modules are not used in this case.
It looks like they use different mechanism to forward the MPLS packets. It would explain why playing with the MPLS settings does NOT help.

[hmntsharma]

I think they are using Arista vEOS not cEOS, VMs dont have any issue as such, I haven't tried Arista myself but never had any trouble with Cisco XRv9k and Nokia vSROS. Could you please check and confirm if my understanding is right?

Meanwhile, I will try SRLinux with containerlab, it has been suggested on discord that there is no issue with mpls forwarding on this platform.

And there is only one way to find out!

[bgolab]

Yes, for sure they use vEOS. Versus our cEOS.

[bravindranath-bsn]

For anyone who comes across this. It's a cEOS issue that makes it fail. Not a kernel/docker bridge issue here.
On a vEOS or a physical switch (EOS) the L3 interfaces use the system MAC address where as with cEOS the L3 interfaces uses the mac address provided by the docker. This causes some issues when the MPLS packets are received on cEOS which needs to be routed.

On cEOS to make this work we just need to configure the OSPF neighborship over SVIs. Here is a working configurations:

ceos1 --- ceos2 --- ceos3

ceos1:

service routing protocols model multi-agent
!
hostname ceos1
!
spanning-tree mode mstp
!
vlan 10
!
management api http-commands
   no shutdown
!
management api gnmi
   transport grpc default
!
management api netconf
   transport ssh default
!
interface Ethernet1
   switchport access vlan 10
!
interface Loopback0
   ip address 10.0.0.1/32
   ip ospf area 0.0.0.0
!
interface Management0
   ip address 172.20.20.2/24
   ipv6 address 2001:172:20:20::2/64
!
interface Vlan10
   ip address 8.0.0.1/24
   ip ospf area 0.0.0.0
!
ip routing
!
mpls ip
!
mpls ldp
   router-id interface Loopback0
   no shutdown
!
router ospf 1
   max-lsa 12000
!
end

ceos2:

service routing protocols model multi-agent
!
hostname ceos2
!
spanning-tree mode mstp
!
vlan 10,20
!
management api http-commands
   no shutdown
!
management api gnmi
   transport grpc default
!
management api netconf
   transport ssh default
!
interface Ethernet1
   switchport access vlan 10
!
interface Ethernet2
   switchport access vlan 20
!
interface Loopback0
   ip address 20.0.0.1/32
   ip ospf area 0.0.0.0
!
interface Management0
   ip address 172.20.20.4/24
   ipv6 address 2001:172:20:20::4/64
!
interface Vlan10
   ip address 8.0.0.2/24
   ip ospf area 0.0.0.0
!
interface Vlan20
   ip address 9.0.0.1/24
   ip ospf area 0.0.0.0
!
ip virtual-router mac-address aa:c1:ab:c3:c0:a3
!
ip routing
!
mpls ip
!
mpls ldp
   router-id interface Loopback0
   no shutdown
!
router ospf 1
   max-lsa 12000
!
end

ceos3:

service routing protocols model multi-agent
!
hostname ceos3
!
spanning-tree mode mstp
!
vlan 20
!
management api http-commands
   no shutdown
!
management api gnmi
   transport grpc default
!
management api netconf
   transport ssh default
!
interface Ethernet1
   switchport access vlan 20
!
interface Loopback0
   ip address 30.0.0.1/32
   ip ospf area 0.0.0.0
!
interface Management0
   ip address 172.20.20.3/24
   ipv6 address 2001:172:20:20::3/64
!
interface Vlan20
   ip address 9.0.0.2/24
   ip ospf area 0.0.0.0
!
ip routing
!
mpls ip
!
mpls ldp
   router-id interface Loopback0
   no shutdown
!
router ospf 1
   max-lsa 12000
!
end

Output:

ceos1(config-if-Et1)#ping mpls ldp ip 20.0.0.1/32 repeat 1
LSP ping to LDP route 20.0.0.1/32
   timeout is 5000ms, interval is 1000ms
Via 8.0.0.2, Vlan10, label stack: [3]
   Reply from 8.0.0.2: seq=1, time=8ms, success: egress ok

--- LDP target fec 20.0.0.1/32 : lspping statistics ---
Via 8.0.0.2, Vlan10, label stack: [3]
   1 packets transmitted, 1 received, 0% packet loss, time 153ms
   1 received from 8.0.0.2, rtt min/max/avg 8/8/8 ms

ceos1(config-if-Et1)#ping mpls ldp ip 30.0.0.1/32 repeat 1
LSP ping to LDP route 30.0.0.1/32
   timeout is 5000ms, interval is 1000ms
Via 8.0.0.2, Vlan10, label stack: [116385]
   Reply from 9.0.0.2: seq=1, time=18ms, success: egress ok

--- LDP target fec 30.0.0.1/32 : lspping statistics ---
Via 8.0.0.2, Vlan10, label stack: [116385]
   1 packets transmitted, 1 received, 0% packet loss, time 152ms
   1 received from 9.0.0.2, rtt min/max/avg 18/18/18 ms

ceos1(config-if-Et1)#

[hmntsharma]

Hi Bharath,
Thank you very much.

At least it negates the response I had found on discord that cEOS does not support mpls.

@bgolab I believe you can now mark the response from @bravindranath as answer.

[hmntsharma]

Also, I know I am being pedantic here but is there any way to get to the bottom of the issue of mac addresses assigned by docker and its impact on cEOS, so that it can be alleviated in future releases?

[bgolab]

@w1nt3rfell Yes, sure we can mark this as the answer.

I want to thank the Arista TAC team here.
I opened a case at the Arista TAC. They reproduced the problem within 24h, and delivered the workaround within next 24h!
I have no valid contract at Arista!

[sulrich]

just to follow up on this a bit ...

ceos-lab does support MPLS forwarding. however, (for the moment) it's only
operational on interfaces that ceos-lab actually processes. this means that it
requires your IP and MPLS configuration must take place on either an SVI
(interface VLAN) or a port-channel. a single member port-channel is
sufficient to invoke this operation.

[hmntsharma]

Good idea, I will try this on port-channel as well.

[hmntsharma]

@bravindranath I did not need to apply "ip virtual-router mac-address aa:c1:ab:c3:c0:a3" on R2 and it still worked.

[bgolab]

Agree. Tested without "ip virtual-router mac-address". But I gues this line is added sometimes automatically but I have not investigated further...

[bravindranath-bsn]

Sorry for the confusion there. The virtual address was not part of the configuration. It was for a different test (I was trying if I can add the physical interface IP as virtual IP to make this work). It’s not added automatically here. We can definitely remove it

[bgolab]

@w1nt3rfell
I just tried to setup L3 VPN and noticed that vlan interface subnets have no LFIB entries and as a consequence of this the BGP next-hope is not available (and the BGP delivered vpn-ipv4 entries are of course not installed).
I noticed that you use "neighbor 20.0.0.1 encapsulation mpls next-hop-self source-interface Loopback0" in your configuration file to change the next-hop to loopback0 IP which is seen in the LFIB table. And thanks to this config knob there is no problem with next-hop reachability.

I am sorry fo r the ignorant question: did you encounter the same issue and you decided to add this config to fix it
OR
there are other reasons why the subnets on the links between R1-R2 and R2-R3 (currently subnets used by the interface vlans) are not visible as LFIB entries i.e. they have no label assigned.

Thank you for any hint.

[hmntsharma]

@bgolab
I used this user manual as reference for all the commands, where I found the encapsulation mpls bit, so I had it right from the beginning itself in my config.
https://www.arista.com/en/um-eos/eos-overview

Glad to know that L3VPN now works!

Cheers!

[bgolab]

@w1nt3rfell
Unfortunately I cannot make it work.
Now the BGP routes are valid. The matching VRF route is also installed.
But the 'ping vrf' does not work (also tested with the 'source' set to the IP known by the vrf on the opposite side).

[bgolab]

The problem is that MPLS ICMP request is seen on the R2 P router on both ingress and egress router BUT not seen on the PE router.
The LFIB seems to be OK.

I lost a couple of hours today.

It looks like GNS3 + Cisco gear is the way to go.
Cisco has better manuals and many examples.

[hmntsharma]

Interesting, ping mpls works but ping vrf doesn't.
I will check myself during the weekend, it has been a long day today.

[hmntsharma]

and for L&D, yes, Cisco online knowledge base is very good, as for the labs, GNS3/EVE-NG both are good but resource hungry, depends on what you want to simulate.

[bgolab]

I spent too much tome on the L3 VPN today. I need sometime to try it again. Maybe I am doing something really stupid.
The Arista config knobs are not very clear for me.

Anyway, if you have a chance to look into I'll be very grateful.

You know I come from the Cisco world (worked as CCIE around 2000+) but for 10+ years I have not touched Cisco and would like to refresh some topics... Arista looked as a good replacement for Cisco, particularly lightweight cEOS version....

[hmntsharma]

Strikes a chord, I understand the frustration, so tried it.
ping vrf is still not working but at least the routes are installed. @bravindranath any idea?

send-community is required, just as in legacy Cisco IOS

router bgp 65001
   neighbor <nbr-ip> send-community standard extended

After that, the routes are installed

R1#sh ip route vrf CEOS | b Gate
Gateway of last resort is not set

 C        192.168.1.1/32 is directly connected, Loopback1
 B I      192.168.1.2/32 [200/0] via 2.2.2.2/32, LDP tunnel index 2, label 116384
                                    via 10.1.2.2, Vlan10, label imp-null(3)
 B I      192.168.1.3/32 [200/0] via 3.3.3.3/32, LDP tunnel index 1, label 116384
                                    via 10.1.2.2, Vlan10, label 100000

R1#

I will continue to work on it.

[bgolab]

Exactly the same results.
Yes, send community is required. Plus important address-family vpn-ipv4 active for specific neighbor.

[hmntsharma]

Ping vrf ICMP Echo request from R1 is received on R3, but it is not generating any Echo Reply

Output on R3

bash-4.2# tcpdump -i eth2 | grep "192.168"
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth2, link-type EN10MB (Ethernet), snapshot length 262144 bytes
23:05:54.685548 00:1c:73:2b:08:67 (oui Arista Networks) > 00:1c:73:f9:30:32 (oui Arista Networks), ethertype MPLS unicast (0x8847), length 118: MPLS (label 116384, tc 0, [S], ttl 63) 192.168.1.1 > 192.168.1.3: ICMP echo request, id 12945, seq 1, length 80
23:05:54.691855 00:1c:73:2b:08:67 (oui Arista Networks) > 00:1c:73:f9:30:32 (oui Arista Networks), ethertype MPLS unicast (0x8847), length 118: MPLS (label 116384, tc 0, [S], ttl 63) 192.168.1.1 > 192.168.1.3: ICMP echo request, id 12945, seq 2, length 80
23:05:54.703343 00:1c:73:2b:08:67 (oui Arista Networks) > 00:1c:73:f9:30:32 (oui Arista Networks), ethertype MPLS unicast (0x8847), length 118: MPLS (label 116384, tc 0, [S], ttl 63) 192.168.1.1 > 192.168.1.3: ICMP echo request, id 12945, seq 3, length 80
23:05:54.711465 00:1c:73:2b:08:67 (oui Arista Networks) > 00:1c:73:f9:30:32 (oui Arista Networks), ethertype MPLS unicast (0x8847), length 118: MPLS (label 116384, tc 0, [S], ttl 63) 192.168.1.1 > 192.168.1.3: ICMP echo request, id 12945, seq 4, length 80
23:05:54.721572 00:1c:73:2b:08:67 (oui Arista Networks) > 00:1c:73:f9:30:32 (oui Arista Networks), ethertype MPLS unicast (0x8847), length 118: MPLS (label 116384, tc 0, [S], ttl 63) 192.168.1.1 > 192.168.1.3: ICMP echo request, id 12945, seq 5, length 80

If ping mpls can work for a loopback then should not be an issue for ping vrf either

[hmntsharma]

Figured it out.

Updated the topology to PC1 -- [ R1 --- R2 --- R3 ] -- PC3
Connected the PC1 and PC3 as CPEs, with default route to R1 and R3, respectively

It appears that transit traffic is working, PC1 to PC3 ping, but not router destined traffic, we can live with that, I hope

R1#sh run
! Command: show running-config
! device: R1 (cEOSLab, EOS-4.27.2F-26069621.4272F (engineering build))
!
no aaa root
!
username admin privilege 15 role network-admin secret sha512 $6$8y7rIZM9MpWPRz8B$QahjXoNBHHr37oCY1hSXDvH.Brkhtu9Rt.2Ahdbzq.KzG72yd5djyx5VEF2f7FDJpdSL0.xAjI6MfZMIiyTv6/
!
transceiver qsfp default-mode 4x10G
!
service routing protocols model multi-agent
!
hostname R1
!
spanning-tree mode mstp
!
vlan 10,100
!
vrf instance CEOS
!
management api http-commands
   no shutdown
!
management api gnmi
   transport grpc default
!
management api netconf
   transport ssh default
!
interface Ethernet1
   switchport access vlan 10
!
interface Ethernet3
   switchport access vlan 100
!
interface Loopback0
   ip address 1.1.1.1/32
   isis enable ISIS
   isis circuit-type level-2
   isis passive
   isis network point-to-point
!
interface Loopback1
   vrf CEOS
   ip address 192.168.1.1/32
!
interface Management0
   ip address 172.20.20.3/24
   ipv6 address 2001:172:20:20::3/64
!
interface Vlan10
   ip address 10.1.2.1/24
   isis enable ISIS
   isis circuit-type level-2
   isis metric 100
   isis network point-to-point
!
interface Vlan100
   vrf CEOS
   ip address 192.168.100.2/30
!
ip routing
ip routing vrf CEOS
!
mpls ip
!
mpls ldp
   router-id interface Loopback0
   no shutdown
!
router bgp 65001
   neighbor 2.2.2.2 remote-as 65001
   neighbor 2.2.2.2 update-source Loopback0
   neighbor 2.2.2.2 send-community standard extended
   neighbor 3.3.3.3 remote-as 65001
   neighbor 3.3.3.3 update-source Loopback0
   neighbor 3.3.3.3 send-community standard extended
   !
   address-family vpn-ipv4
      neighbor 2.2.2.2 activate
      neighbor 3.3.3.3 activate
      neighbor 2.2.2.2 encapsulation mpls next-hop-self source-interface Loopback0
      neighbor 3.3.3.3 encapsulation mpls next-hop-self source-interface Loopback0
   !
   vrf CEOS
      rd 65001:100
      route-target import vpn-ipv4 65001:100
      route-target export vpn-ipv4 65001:100
      redistribute connected
!
router isis ISIS
   net 49.0001.0000.0000.0001.00
   router-id ipv4 1.1.1.1
   is-type level-2
   mpls ldp sync default
   !
   address-family ipv4 unicast
!
end
R1#

R2#sh run
! Command: show running-config
! device: R2 (cEOSLab, EOS-4.27.2F-26069621.4272F (engineering build))
!
no aaa root
!
username admin privilege 15 role network-admin secret sha512 $6$3LDMvPkEriAF3oys$hKFqK/iIn7vr3FpKSeocYRtMi16tJFhKp73sh.9t5k9.Z/8CDWtuMni/el3qyFMVg8eucg6c3q6KYB4VNl0NV/
!
transceiver qsfp default-mode 4x10G
!
service routing protocols model multi-agent
!
hostname R2
!
spanning-tree mode mstp
!
vlan 10,20
!
vrf instance CEOS
!
management api http-commands
   no shutdown
!
management api gnmi
   transport grpc default
!
management api netconf
   transport ssh default
!
interface Ethernet1
   switchport access vlan 10
!
interface Ethernet2
   switchport access vlan 20
!
interface Loopback0
   ip address 2.2.2.2/32
   isis enable ISIS
   isis passive
!
interface Loopback1
   vrf CEOS
   ip address 192.168.1.2/32
!
interface Management0
   ip address 172.20.20.5/24
   ipv6 address 2001:172:20:20::5/64
!
interface Vlan10
   ip address 10.1.2.2/24
   isis enable ISIS
   isis circuit-type level-2
   isis metric 100
   isis network point-to-point
!
interface Vlan20
   ip address 10.2.3.2/24
   isis enable ISIS
   isis circuit-type level-2
   isis metric 100
   isis network point-to-point
!
ip routing
ip routing vrf CEOS
!
mpls ip
!
mpls ldp
   router-id interface Loopback0
   no shutdown
!
router bgp 65001
   neighbor 1.1.1.1 remote-as 65001
   neighbor 1.1.1.1 update-source Loopback0
   neighbor 1.1.1.1 send-community standard extended
   neighbor 3.3.3.3 remote-as 65001
   neighbor 3.3.3.3 update-source Loopback0
   neighbor 3.3.3.3 send-community standard extended
   !
   address-family vpn-ipv4
      neighbor 1.1.1.1 activate
      neighbor 3.3.3.3 activate
      neighbor 1.1.1.1 encapsulation mpls next-hop-self source-interface Loopback0
      neighbor 3.3.3.3 encapsulation mpls next-hop-self source-interface Loopback0
   !
   vrf CEOS
      rd 65001:100
      route-target import vpn-ipv4 65001:100
      route-target export vpn-ipv4 65001:100
      redistribute connected
!
router isis ISIS
   net 49.0001.0000.0000.0002.00
   router-id ipv4 2.2.2.2
   is-type level-2
   mpls ldp sync default
   !
   address-family ipv4 unicast
!
end
R2#

R3#sh run
! Command: show running-config
! device: R3 (cEOSLab, EOS-4.27.2F-26069621.4272F (engineering build))
!
no aaa root
!
username admin privilege 15 role network-admin secret sha512 $6$gnxeafg.vKzdQY.X$49jW5sNToBMZV9hTwwUixDCw1icUQnGmWh2cGSh1MbftpWATRdvk1b1WvCqkVRO7/57IXPo73BrtCQhoTu1JB1
!
transceiver qsfp default-mode 4x10G
!
service routing protocols model multi-agent
!
hostname R3
!
spanning-tree mode mstp
!
vlan 20,100
!
vrf instance CEOS
!
management api http-commands
   no shutdown
!
management api gnmi
   transport grpc default
!
management api netconf
   transport ssh default
!
interface Ethernet2
   switchport access vlan 20
!
interface Ethernet3
   switchport access vlan 100
!
interface Loopback0
   ip address 3.3.3.3/32
   isis enable ISIS
   isis passive
!
interface Loopback1
   vrf CEOS
   ip address 192.168.1.3/32
!
interface Management0
   ip address 172.20.20.2/24
   ipv6 address 2001:172:20:20::2/64
!
interface Vlan20
   ip address 10.2.3.3/24
   isis enable ISIS
   isis circuit-type level-2
   isis metric 100
   isis network point-to-point
!
interface Vlan100
   vrf CEOS
   ip address 192.168.200.2/30
!
ip routing
ip routing vrf CEOS
!
mpls ip
!
mpls ldp
   router-id interface Loopback0
   no shutdown
!
router bgp 65001
   neighbor 1.1.1.1 remote-as 65001
   neighbor 1.1.1.1 update-source Loopback0
   neighbor 1.1.1.1 send-community standard extended
   neighbor 2.2.2.2 remote-as 65001
   neighbor 2.2.2.2 update-source Loopback0
   neighbor 2.2.2.2 send-community standard extended
   !
   address-family vpn-ipv4
      neighbor 1.1.1.1 activate
      neighbor 2.2.2.2 activate
      neighbor 1.1.1.1 encapsulation mpls next-hop-self source-interface Loopback0
      neighbor 2.2.2.2 encapsulation mpls next-hop-self source-interface Loopback0
   !
   vrf CEOS
      rd 65001:100
      route-target import vpn-ipv4 65001:100
      route-target export vpn-ipv4 65001:100
      redistribute connected
!
router isis ISIS
   net 49.0001.0000.0000.0003.00
   router-id ipv4 3.3.3.3
   is-type level-2
   mpls ldp sync default
   !
   address-family ipv4 unicast
!
end
R3#

PC1#sh run
! Command: show running-config
! device: PC1 (cEOSLab, EOS-4.27.2F-26069621.4272F (engineering build))
!
no aaa root
!
username admin privilege 15 role network-admin secret sha512 $6$gkOIchuJTrgQVA6g$AVsLLmApohMJpki3g5mHBrg0NYyMAUCILEI.0YBubZUU6pQ9vlbZ8Qtm7opWRB5gR0hZhnNWpDyMvp9ApTeHm.
!
transceiver qsfp default-mode 4x10G
!
service routing protocols model multi-agent
!
hostname PC1
!
spanning-tree mode mstp
!
vlan 100
!
management api http-commands
   no shutdown
!
management api gnmi
   transport grpc default
!
management api netconf
   transport ssh default
!
interface Ethernet3
   switchport access vlan 100
!
interface Management0
   ip address 172.20.20.6/24
   ipv6 address 2001:172:20:20::6/64
!
interface Vlan100
   ip address 192.168.100.1/30
!
ip routing
!
ip route 0.0.0.0/0 Vlan100 192.168.100.2
!
end
PC1#

PC3#sh run
! Command: show running-config
! device: PC3 (cEOSLab, EOS-4.27.2F-26069621.4272F (engineering build))
!
no aaa root
!
username admin privilege 15 role network-admin secret sha512 $6$c7n313m8rNcWPhJ2$3ZO38uSJ5wg0eSU25xULNVJBpz/alcZszdgl/LBqFuAXUkY6huMfzOMEMMaDn.oRGifENBfKUofv.rwPxtypo.
!
transceiver qsfp default-mode 4x10G
!
service routing protocols model multi-agent
!
hostname PC3
!
spanning-tree mode mstp
!
vlan 100
!
management api http-commands
   no shutdown
!
management api gnmi
   transport grpc default
!
management api netconf
   transport ssh default
!
interface Ethernet3
   switchport access vlan 100
!
interface Management0
   ip address 172.20.20.4/24
   ipv6 address 2001:172:20:20::4/64
!
interface Vlan100
   ip address 192.168.200.1/30
!
ip routing
!
ip route 0.0.0.0/0 Vlan100 192.168.200.2
!
end
PC3#

[hmntsharma]

@bgolab
Ping Test and tcpdump

PC1# ping 192.168.200.1
PING 192.168.200.1 (192.168.200.1) 72(100) bytes of data.
80 bytes from 192.168.200.1: icmp_seq=1 ttl=61 time=52.6 ms
80 bytes from 192.168.200.1: icmp_seq=2 ttl=61 time=45.7 ms
80 bytes from 192.168.200.1: icmp_seq=3 ttl=61 time=42.3 ms
80 bytes from 192.168.200.1: icmp_seq=4 ttl=61 time=35.9 ms
80 bytes from 192.168.200.1: icmp_seq=5 ttl=61 time=28.4 ms

--- 192.168.200.1 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 40ms
rtt min/avg/max/mdev = 28.449/41.052/52.658/8.287 ms, pipe 5, ipg/ewma 10.139/46.249 ms
PC1#

lab@ubuntu1804:~$ sudo docker exec -it PC3 bash
bash-4.2# tcpdump -i eth3 icmp
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth3, link-type EN10MB (Ethernet), snapshot length 262144 bytes
23:49:00.140248 00:1c:73:f9:30:32 (oui Arista Networks) > 00:1c:73:16:d3:35 (oui Arista Networks), ethertype IPv4 (0x0800), length 114: 192.168.100.1 > PC3: ICMP echo request, id 2, seq 1, length 80
23:49:00.141732 00:1c:73:f9:30:32 (oui Arista Networks) > 00:1c:73:16:d3:35 (oui Arista Networks), ethertype IPv4 (0x0800), length 114: 192.168.100.1 > PC3: ICMP echo request, id 2, seq 2, length 80
23:49:00.144381 00:1c:73:f9:30:32 (oui Arista Networks) > 00:1c:73:16:d3:35 (oui Arista Networks), ethertype IPv4 (0x0800), length 114: 192.168.100.1 > PC3: ICMP echo request, id 2, seq 3, length 80
23:49:00.149798 00:1c:73:16:d3:35 (oui Arista Networks) > 00:1c:73:f9:30:32 (oui Arista Networks), ethertype IPv4 (0x0800), length 114: PC3 > 192.168.100.1: ICMP echo reply, id 2, seq 1, length 80
23:49:00.150570 00:1c:73:16:d3:35 (oui Arista Networks) > 00:1c:73:f9:30:32 (oui Arista Networks), ethertype IPv4 (0x0800), length 114: PC3 > 192.168.100.1: ICMP echo reply, id 2, seq 2, length 80
23:49:00.153080 00:1c:73:16:d3:35 (oui Arista Networks) > 00:1c:73:f9:30:32 (oui Arista Networks), ethertype IPv4 (0x0800), length 114: PC3 > 192.168.100.1: ICMP echo reply, id 2, seq 3, length 80
23:49:00.160363 00:1c:73:f9:30:32 (oui Arista Networks) > 00:1c:73:16:d3:35 (oui Arista Networks), ethertype IPv4 (0x0800), length 114: 192.168.100.1 > PC3: ICMP echo request, id 2, seq 4, length 80
23:49:00.162415 00:1c:73:16:d3:35 (oui Arista Networks) > 00:1c:73:f9:30:32 (oui Arista Networks), ethertype IPv4 (0x0800), length 114: PC3 > 192.168.100.1: ICMP echo reply, id 2, seq 4, length 80
23:49:00.168172 00:1c:73:f9:30:32 (oui Arista Networks) > 00:1c:73:16:d3:35 (oui Arista Networks), ethertype IPv4 (0x0800), length 114: 192.168.100.1 > PC3: ICMP echo request, id 2, seq 5, length 80
23:49:00.170284 00:1c:73:16:d3:35 (oui Arista Networks) > 00:1c:73:f9:30:32 (oui Arista Networks), ethertype IPv4 (0x0800), length 114: PC3 > 192.168.100.1: ICMP echo reply, id 2, seq 5, length 80

---

PC3#ping 192.168.100.1
PING 192.168.100.1 (192.168.100.1) 72(100) bytes of data.
80 bytes from 192.168.100.1: icmp_seq=1 ttl=61 time=50.3 ms
80 bytes from 192.168.100.1: icmp_seq=2 ttl=61 time=39.6 ms
80 bytes from 192.168.100.1: icmp_seq=3 ttl=61 time=32.3 ms
80 bytes from 192.168.100.1: icmp_seq=4 ttl=61 time=24.8 ms
80 bytes from 192.168.100.1: icmp_seq=5 ttl=61 time=30.9 ms

--- 192.168.100.1 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 42ms
rtt min/avg/max/mdev = 24.898/35.641/50.395/8.751 ms, pipe 5, ipg/ewma 10.627/42.543 ms
PC3#

lab@ubuntu1804:~$ sudo docker exec -it PC1 bash
bash-4.2#  tcpdump -i eth3 icmp
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth3, link-type EN10MB (Ethernet), snapshot length 262144 bytes
23:50:28.455777 00:1c:73:68:e4:87 (oui Arista Networks) > 00:1c:73:7a:80:5c (oui Arista Networks), ethertype IPv4 (0x0800), length 114: 192.168.200.1 > PC1: ICMP echo request, id 3, seq 1, length 80
23:50:28.465123 00:1c:73:68:e4:87 (oui Arista Networks) > 00:1c:73:7a:80:5c (oui Arista Networks), ethertype IPv4 (0x0800), length 114: 192.168.200.1 > PC1: ICMP echo request, id 3, seq 2, length 80
23:50:28.468886 00:1c:73:7a:80:5c (oui Arista Networks) > 00:1c:73:68:e4:87 (oui Arista Networks), ethertype IPv4 (0x0800), length 114: PC1 > 192.168.200.1: ICMP echo reply, id 3, seq 1, length 80
23:50:28.470471 00:1c:73:7a:80:5c (oui Arista Networks) > 00:1c:73:68:e4:87 (oui Arista Networks), ethertype IPv4 (0x0800), length 114: PC1 > 192.168.200.1: ICMP echo reply, id 3, seq 2, length 80
23:50:28.472798 00:1c:73:68:e4:87 (oui Arista Networks) > 00:1c:73:7a:80:5c (oui Arista Networks), ethertype IPv4 (0x0800), length 114: 192.168.200.1 > PC1: ICMP echo request, id 3, seq 3, length 80
23:50:28.475556 00:1c:73:7a:80:5c (oui Arista Networks) > 00:1c:73:68:e4:87 (oui Arista Networks), ethertype IPv4 (0x0800), length 114: PC1 > 192.168.200.1: ICMP echo reply, id 3, seq 3, length 80
23:50:28.483251 00:1c:73:68:e4:87 (oui Arista Networks) > 00:1c:73:7a:80:5c (oui Arista Networks), ethertype IPv4 (0x0800), length 114: 192.168.200.1 > PC1: ICMP echo request, id 3, seq 4, length 80
23:50:28.485341 00:1c:73:7a:80:5c (oui Arista Networks) > 00:1c:73:68:e4:87 (oui Arista Networks), ethertype IPv4 (0x0800), length 114: PC1 > 192.168.200.1: ICMP echo reply, id 3, seq 4, length 80
23:50:28.494818 00:1c:73:68:e4:87 (oui Arista Networks) > 00:1c:73:7a:80:5c (oui Arista Networks), ethertype IPv4 (0x0800), length 114: 192.168.200.1 > PC1: ICMP echo request, id 3, seq 5, length 80
23:50:28.496809 00:1c:73:7a:80:5c (oui Arista Networks) > 00:1c:73:68:e4:87 (oui Arista Networks), ethertype IPv4 (0x0800), length 114: PC1 > 192.168.200.1: ICMP echo reply, id 3, seq 5, length 80

[hmntsharma]

Sure. I like the idea.

[bgolab]

Just tried your example (and ping externally connected PC instead of 'ping vrf') and it works!.
It looks like the vlan interfaces still matters here.
I tried to connect the PCs directly to the Ethernet interfaces with IP address assigned and failed miserably - the ICMP Request packets reached the R3 (assumed the ping was send from the PC1 connected to R1) BUT was not forwarded to PC3 .

[hmntsharma]

I had a feeling about it which is why I went to the vlan interface directly even for the vrf, but thanks for the confirmation.

[bravindranath-bsn]

Hi guys, sorry for the delayed response. I tried this as well and like you said, it works fine if you have an SVI. This is because the vrf is not truly "Active" as long as there is no "SVI" OR physical interface that is part of that vrf (vrf CEOS in this case). So its necessary to have an SVI or a physical interface (Ethernet or port-channel) within the same VRF for this to work.

I tried this using Port-Channel configuration (no SVIs for point to point interfaces), and that works too (we still need SVIs for the end to end communications).

@bgolab , you said I tried to connect the PCs directly to the Ethernet interfaces with IP address assigned and failed miserably --> If you wrap this ethernet interface in a port-channel interface, you should be able to make this work without SVIs on the PCs.

P.S. It might just be me, but I find it easier to configure all the ethernet interfaces with the corresponding port-channel numbers, and then use Po1,2 etc as if they are the physical (Ethernet) interfaces interconnecting the cEOS instances.

[bgolab]

@bravindranath Yes, good point.

Thank you for your insights - it's good to know where the limitations come from (actually not a problem if we consider real life scenarios).

Personally I also prefer this kind of 'layer of indirection' (both SVI and port-channel) as I can easier to manage the physical resources e.g. the IP is assignments , etc

[hmntsharma]

I have documented this as a proper repository here at w1nt3rfell/clab-ceosmpls
Similarly, I tested Juniper cRPD as well, here at w1nt3rfell/clab-crpdmpls

[bgolab]

My 3 cents about the testing environment.

I use Ubuntu installed on VirtualBox VM and in this case everything is fine (no MPLS kernel modules are required, no sysctl MPLS settings then). It looks like the Arista uses different way of handling non-IP packets i.e. MPLS packets.

In the case of WSL2/Ubuntu the MPLS does not work here (the IP connectivity is fine - of course the 'iptables -P INPUT ACCEPT' is required to fix docker incompatibilities).
The LDP neighboring state is operational but the LFIB is empty. I have not looked into deeply.

[bgolab]

I tested cRPD a couple of weeks ago. I suggested the vendor to handle IP setup better way (of course it easy to automate this) but it would be more practical to have single place to configure the device (both IP settings and other Junos related stuff).

[hmntsharma]

I think I had read it somewhere else too about mpls forwarding issue in WSL, but yes, Arista does it in a different way when implemented in native linux or vm.
wrt crpd, it'd be very helpful indeed, if there is only one interface to configure everything.

cRPS consumes less resources (~300MB RAM) as compared to cEOS (~1GB RAM) and since configuring IP was the only requirement from bash for version21, I liked it better. :-)