Nexus 9000v
L2 to Routed Port Conversion and OSPF Setup
Multicast Deployment and NVE Multicast Update
BGP L2VPN EVPN and NVE Multicast
BGP L2VPN and NVE Ingress Replication
Adding Additional VLAN, VNI and SVI
LACP Port Channel between IOS and Nexus
Inter VNI Routing with Symmetric IRB
Multi Tenancy Adding Second Customer
vPC and VXLAN with BGP L2VPN EVPN or AnyCast VTEP
Service Leaf Setup and Single ASAv Firewall Deployment
New Service Leaf Setup and Tshooting and ASA HA Part
New Service Leaf Setup and Tshooting and ASA HA Part2
MPLS L3 VPN Setup for External WAN Access to the
External Connectivity to Private WAN via eBGP Peerings
VXLAN EVPN Multi Site Overview and Config Walkthrough
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Preparation
Cisco Data Center Nexus 9000v VXLAN 001 - Series Intro and What We'll Cover!
From <https://www.youtube.com/watch?v=x4mMxYYb2JQ&ab_channel=RobRiker%27sTechChannel>
VXLAN Multipod Design for Intra-Data Center and Geographically Dispersed Data Center Sites
Nexus 9000v is a demo version of the Nexus Operating System:
Software
BIOS: version
NXOS: version 9.2(2)
show cli history unformatted | last 20
Run ASAv on EVE
ASAV report error"IO memory blocks requested from bigphys 32bit: 87680" and can't use console
got this working ok .. needed to set console type to vnc and apply the following as detailed in other posts:
copy disk0:/coredumpinfo/coredump.cfg disk0:/use_ttyS0
wr
copy startup-config disk0:/startup-config
conf t
boot config disk0:/startup-config
hostname ASAv951
wr
copy running-config disk0:/startup-config
reload
Stop ASAv then,
Then right click ASA and configure console type back to Telnet and startup output will eventually appear on putty console.
From <https://www.gns3.com/community/featured/asav-not-booting-correctly>
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Switch bootstrap
skip
dir
switch# conf t
Enter configuration commands, one per line. End with CNTL/Z.
switch(config)# boot nxos bootflash:nxos.9.2.2.bin
Performing image verification and compatibility check, please wait....
switch(config)# hostname XXX
N1(config)# cli alias name wr copy run start
N1(config)# wr
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete.
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Legacy L2 Network
N9K1
vlan 10
name VLAN10
vrf context management
interface Ethernet1/1
switchport mode trunk
interface Ethernet1/2
switchport mode trunk
interface Ethernet1/9
switchport access vlan 10
spanning-tree port type edge
N9K2
vlan 10
name VLAN10
vrf context management
interface Ethernet1/1
switchport mode trunk
interface Ethernet1/2
switchport mode trunk
interface Ethernet1/7
switchport access vlan 10
spanning-tree port type edge
interface Ethernet1/8
switchport access vlan 10
spanning-tree port type edge
N9K5 and N9K6
vlan 10
name VLAN10
vrf context management
interface Ethernet1/1
switchport mode trunk
interface Ethernet1/2
switchport mode trunk
SPINE 1 & 2
vlan 10
name VLAN10
vrf context management
interface Ethernet1/1
switchport mode trunk
interface Ethernet1/2
switchport mode trunk
interface Ethernet1/3
switchport mode trunk
interface Ethernet1/4
switchport mode trunk
SPINE1# show system internal l2fwder mac
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link,
(T) - True, (F) - False, C - ControlPlane MAC
VLAN MAC Address Type age Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
* 10 0050.7966.680d dynamic 00:04:20 F F Eth1/1
* 10 0050.7966.680e dynamic 00:04:19 F F Eth1/2
* 10 0050.7966.680f dynamic 00:04:17 F F Eth1/2
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
L2 to Routed Port Conversion and OSPF Setup
SPINE1(config-if)# sh ip int bri
IP Interface Status for VRF "default"(1)
Interface IP Address Interface Status
Lo0 10.0.0.2 protocol-up/link-up/admin-up
Eth1/1 10.1.2.2 protocol-up/link-up/admin-up
Eth1/2 10.2.3.2 protocol-up/link-up/admin-up
Eth1/3 10.2.5.2 protocol-up/link-up/admin-up
Eth1/4 10.2.6.2 protocol-up/link-up/admin-up
SPINE2# show ip int bri
IP Interface Status for VRF "default"(1)
Interface IP Address Interface Status
Lo0 10.0.0.4 protocol-up/link-up/admin-up
Eth1/1 10.1.4.4 protocol-up/link-up/admin-up
Eth1/2 10.3.4.4 protocol-up/link-up/admin-up
Eth1/3 10.4.5.4 protocol-up/link-up/admin-up
Eth1/4 10.4.6.4 protocol-up/link-up/admin-up
N9K1(config-if)# show ip int bri
IP Interface Status for VRF "default"(1)
Interface IP Address Interface Status
Lo0 10.0.0.1 protocol-up/link-up/admin-up
Eth1/1 10.1.2.1 protocol-up/link-up/admin-up
Eth1/2 10.1.4.1 protocol-up/link-up/admin-up
N9K3(config-if)# show ip int bri
IP Interface Status for VRF "default"(1)
Interface IP Address Interface Status
Lo0 10.0.0.3 protocol-up/link-up/admin-up
Eth1/1 10.2.3.3 protocol-up/link-up/admin-up
Eth1/2 10.3.4.3 protocol-up/link-up/admin-up
N9k5# show ip int bri
IP Interface Status for VRF "default"(1)
Interface IP Address Interface Status
Lo0 10.0.0.5 protocol-up/link-up/admin-up
Eth1/1 10.2.5.5 protocol-up/link-up/admin-up
Eth1/2 10.4.5.5 protocol-up/link-up/admin-up
N9K6(config-if)# show ip int bri
IP Interface Status for VRF "default"(1)
Interface IP Address Interface Status
Lo0 10.0.0.6 protocol-up/link-up/admin-up
Eth1/1 10.2.6.6 protocol-up/link-up/admin-up
Eth1/2 10.4.6.6 protocol-up/link-up/admin-up
feature ospf
show cli history unformatted | last 20
N9K1-6
router ospf 1
int e1/1-2
ip router ospf 1 area 0
ip ospf network point-to-point
int lo 0
ip router ospf 1 area 0
SPINE 1 & 2
router ospf 1
int e1/1-4
ip router ospf 1 area 0
ip ospf network point-to-point
int lo 0
ip router ospf 1 area 0
SPINE1# show ip ospf nei
OSPF Process ID 1 VRF default
Total number of neighbors: 4
Neighbor ID Pri State Up Time Address Interface
10.0.0.1 1 FULL/ - 00:00:09 10.1.2.1 Eth1/1
10.0.0.3 1 FULL/ - 00:00:14 10.2.3.3 Eth1/2
10.0.0.5 1 FULL/ - 00:00:07 10.2.5.5 Eth1/3
10.0.0.6 1 FULL/ - 00:00:15 10.2.6.6 Eth1/4
SPINE2# show ip ospf nei
OSPF Process ID 1 VRF default
Total number of neighbors: 4
Neighbor ID Pri State Up Time Address Interface
10.0.0.1 1 FULL/ - 00:01:20 10.1.4.1 Eth1/1
10.0.0.3 1 FULL/ - 00:01:25 10.3.4.3 Eth1/2
10.0.0.5 1 FULL/ - 00:01:19 10.4.5.5 Eth1/3
10.0.0.6 1 FULL/ - 00:01:27 10.4.6.6 Eth1/4
SPINE1# show ip ospf database
OSPF Router with ID (10.0.0.2) (Process ID 1 VRF default)
Router Link States (Area 0.0.0.0)
Link ID ADV Router Age Seq# Checksum Link Count
10.0.0.1 10.0.0.1 255 0x8000000a 0xb93d 5
10.0.0.2 10.0.0.2 252 0x8000000f 0x1197 9
10.0.0.3 10.0.0.3 262 0x80000009 0x5b8a 5
10.0.0.4 10.0.0.4 274 0x8000000f 0xa0e9 9
10.0.0.5 10.0.0.5 253 0x80000009 0xfed4 5
10.0.0.6 10.0.0.6 261 0x80000009 0xd1f8 5
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
VXLAN Ingress Replication
VLAN 10 = VN-Seg 10000, 10010
VLANs are switch specific, whereas VNI is fabric wide
N9K1
feature nv overlay
feature vn-segment-vlan-based
vlan 10
name VLAN10
vn-segment 10010
interface nve1
no shutdown
source-interface loopback0
member vni 10010
ingress-replication protocol static
peer-ip 10.0.0.3
N9K1# sho nve peers
Interface Peer-IP State LearnType Uptime Router-Mac
--------- --------------- ----- --------- -------- -----------------
nve1 10.0.0.3 Up DP 00:01:10 n/a
N9K3
feature nv overlay
feature vn-segment-vlan-based
vlan 10
name VLAN10
vn-segment 10010
interface nve1
no shutdown
source-interface loopback0
member vni 10010
ingress-replication protocol static
peer-ip 10.0.0.1
N9K3# show nve peers
Interface Peer-IP State LearnType Uptime Router-Mac
--------- --------------- ----- --------- -------- -----------------
nve1 10.0.0.1 Up DP 00:00:22 n/a
PC3
VPCS> ping 10.1.10.14
84 bytes from 10.1.10.14 icmp_seq=1 ttl=64 time=23.879 ms
84 bytes from 10.1.10.14 icmp_seq=2 ttl=64 time=30.895 ms
84 bytes from 10.1.10.14 icmp_seq=3 ttl=64 time=22.754 ms
84 bytes from 10.1.10.14 icmp_seq=4 ttl=64 time=20.035 ms
84 bytes from 10.1.10.14 icmp_seq=5 ttl=64 time=17.743 ms
N9K1# show system internal l2fwder mac
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link,
(T) - True, (F) - False, C - ControlPlane MAC
VLAN MAC Address Type age Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
* 10 0050.7966.680d dynamic 00:00:37 F F Eth1/9
* 10 0050.7966.680e dynamic 00:00:37 F F nve-peer1 10.0.0.3
N9K3# show system internal l2fwder mac
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link,
(T) - True, (F) - False, C - ControlPlane MAC
VLAN MAC Address Type age Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
* 10 0050.7966.680d dynamic 00:00:57 F F nve-peer1 10.0.0.1
* 10 0050.7966.680e dynamic 00:00:57 F F Eth1/7
Interface: nve1, State: Up, encapsulation: VXLAN
VPC Capability: VPC-VIP-Only [not-notified]
Local Router MAC: 5000.0004.0007
Host Learning Mode: Data-Plane
Source-Interface: loopback0 (primary: 10.0.0.3, secondary: 0.0.0.0)
N9K3# show nve vxlan-params
VxLAN Dest. UDP Port: 4789
N9K3# show nve vni ingress-replication
Interface VNI Replication List Source Up Time
--------- -------- ----------------- ------- -------
nve1 10010 10.0.0.1 CLI 00:13:06
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Multicast Deployment and NVE Multicast Update
All switches
feature pim
ip pim rp-address 10.0.0.2 group-list 224.0.0.0/4
Leafs
int e1/1-2
ip pim sparse-mode
int lo 0
ip pim sparse-mode
Spines
int e1/1-4
ip pim sparse-mode
int lo 0
ip pim sparse-mode
N9K6# show ip pim rp
PIM RP Status Information for VRF "default"
BSR disabled
Auto-RP disabled
BSR RP Candidate policy: None
BSR RP policy: None
Auto-RP Announce policy: None
Auto-RP Discovery policy: None
RP: 10.0.0.2, (0),
uptime: 00:01:11 priority: 255,
RP-source: (local),
group ranges:
224.0.0.0/4
N9K1 & N9K3
N9K1(config-if-nve)interface nve1
N9K1(config-if-nve) shutdown
N9K1(config-if-nve)# member vni 10010
N9K1(config-if-nve-vni)# no ingress-replication protocol static
N9K1(config-if-nve-vni)# mcast-group 224.1.1.1
N9K1(config-if-nve-vni)# no shut
N9K1# show run int nve1
interface nve1
no shutdown
source-interface loopback0
member vni 10010
mcast-group 224.1.1.1
N9K1# show nve vni
Codes: CP - Control Plane DP - Data Plane
UC - Unconfigured SA - Suppress ARP
SU - Suppress Unknown Unicast
Xconn - Crossconnect
MS-IR - Multisite Ingress Replication
Interface VNI Multicast-group State Mode Type [BD/VRF] Flags
--------- -------- ----------------- ----- ---- ------------------ -----
nve1 10010 224.1.1.1 Up DP L2 [10]
PC13> ping 10.1.10.14
84 bytes from 10.1.10.14 icmp_seq=1 ttl=64 time=57.945 ms
84 bytes from 10.1.10.14 icmp_seq=2 ttl=64 time=27.958 ms
84 bytes from 10.1.10.14 icmp_seq=3 ttl=64 time=31.916 ms
84 bytes from 10.1.10.14 icmp_seq=4 ttl=64 time=19.012 ms
84 bytes from 10.1.10.14 icmp_seq=5 ttl=64 time=22.312 ms
N9K1# show nve peers
Interface Peer-IP State LearnType Uptime Router-Mac
--------- --------------- ----- --------- -------- -----------------
nve1 10.0.0.3 Up DP 00:00:27 n/a
N9K3# show ip mroute
IP Multicast Routing Table for VRF "default"
(*, 224.1.1.1/32), uptime: 00:07:12, nve ip pim
Incoming interface: Ethernet1/1, RPF nbr: 10.2.3.2
Outgoing interface list: (count: 1)
nve1, uptime: 00:07:12, nve
(10.0.0.1/32, 224.1.1.1/32), uptime: 00:01:28, ip pim mrib
Incoming interface: Ethernet1/1, RPF nbr: 10.2.3.2
Outgoing interface list: (count: 1)
nve1, uptime: 00:01:28, mrib
(10.0.0.3/32, 224.1.1.1/32), uptime: 00:07:12, nve mrib ip pim
Incoming interface: loopback0, RPF nbr: 10.0.0.3
Outgoing interface list: (count: 1)
Ethernet1/1, uptime: 00:06:43, pim
(*, 232.0.0.0/8), uptime: 00:52:47, pim ip
Incoming interface: Null, RPF nbr: 0.0.0.0
Outgoing interface list: (count: 0)
N9K3# show system internal l2fwder mac
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link,
(T) - True, (F) - False, C - ControlPlane MAC
VLAN MAC Address Type age Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
* 10 0050.7966.680d dynamic 00:02:34 F F nve-peer1 10.0.0.1
* 10 0050.7966.680e dynamic 00:02:36 F F Eth1/7
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
BGP L2VPN EVPN and NVE Multicast
Spines
feature bgp
nv overlay evpn
router bgp 65001
address-family l2vpn evpn
neighbor 10.0.0.1
remote-as 65001
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
route-reflector-client
neighbor 10.0.0.3
remote-as 65001
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
route-reflector-client
neighbor 10.0.0.5
remote-as 65001
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
route-reflector-client
neighbor 10.0.0.6
remote-as 65001
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
route-reflector-client
Leafs
feature bgp
nv overlay evpn
feature fabric forwarding
fabric forward anycast-gateway-mac 0001.0001.0001
feature interface-vlan
router bgp 65001
address-family l2vpn evpn
neighbor 10.0.0.2
remote-as 65001
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
neighbor 10.0.0.4
remote-as 65001
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
evpn
vni 10010 l2
rd auto
route-target both auto
interface Vlan10
no shutdown
ip address 10.1.10.254/24
fabric forwarding mode anycast-gateway
N9K1(config)# int nve 1
N9K1(config-if-nve)# shutdown
N9K1(config-if-nve)# host-reachability protocol bgp
N9K1(config-if-nve)# no shutdown
interface nve1
no shutdown
host-reachability protocol bgp
source-interface loopback0
member vni 10010
mcast-group 224.1.1.1
PC13> ping 10.1.10.14
84 bytes from 10.1.10.14 icmp_seq=1 ttl=64 time=43.241 ms
84 bytes from 10.1.10.14 icmp_seq=2 ttl=64 time=30.577 ms
84 bytes from 10.1.10.14 icmp_seq=3 ttl=64 time=35.636 ms
CP = control plane learned
N9K1# show nve peers
Interface Peer-IP State LearnType Uptime Router-Mac
--------- --------------- ----- --------- -------- -----------------
nve1 10.0.0.3 Up CP 00:00:04 n/a
N9K3# show nve peers
Interface Peer-IP State LearnType Uptime Router-Mac
--------- --------------- ----- --------- -------- -----------------
nve1 10.0.0.1 Up CP 00:00:40 n/a
N9K1# show nve vni control-plane
Codes: CP - Control Plane DP - Data Plane
UC - Unconfigured SA - Suppress ARP
SU - Suppress Unknown Unicast
Xconn - Crossconnect
MS-IR - Multisite Ingress Replication
Interface VNI Multicast-group State Mode Type [BD/VRF] Flags
--------- -------- ----------------- ----- ---- ------------------ -----
nve1 10010 224.1.1.1 Up CP L2 [10]
N9K1# show system internal l2fwder mac
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link,
(T) - True, (F) - False, C - ControlPlane MAC
VLAN MAC Address Type age Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
* 10 0050.7966.680d dynamic 00:02:42 F F Eth1/9
* 10 0050.7966.680e static - F F nve-peer1 10.0.0.3
N9K3# show system internal l2fwder mac
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link,
(T) - True, (F) - False, C - ControlPlane MAC
VLAN MAC Address Type age Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
* 10 0050.7966.680d static - F F nve-peer1 10.0.0.1
* 10 0050.7966.680e dynamic 00:03:03 F F Eth1/7
1 1 -00:01:00:01:00:01 - 1
N9K1# show bgp l2vpn evpn
BGP routing table information for VRF default, address family L2VPN EVPN
BGP table version is 12, Local Router ID is 10.0.0.1
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-i
njected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup, 2 - b
est2
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 10.0.0.1:32777 (L2VNI 10010)
*>l[2]:[0]:[0]:[48]:[0050.7966.680d]:[0]:[0.0.0.0]/216
10.0.0.1 100 32768 i
*>i[2]:[0]:[0]:[48]:[0050.7966.680e]:[0]:[0.0.0.0]/216
10.0.0.3 100 0 i
*>l[2]:[0]:[0]:[48]:[0050.7966.680d]:[32]:[10.1.10.13]/248
10.0.0.1 100 32768 i
*>i[2]:[0]:[0]:[48]:[0050.7966.680e]:[32]:[10.1.10.14]/248
10.0.0.3 100 0 i
Route Distinguisher: 10.0.0.3:32777
* i[2]:[0]:[0]:[48]:[0050.7966.680e]:[0]:[0.0.0.0]/216
10.0.0.3 100 0 i
*>i 10.0.0.3 100 0 i
* i[2]:[0]:[0]:[48]:[0050.7966.680e]:[32]:[10.1.10.14]/248
10.0.0.3 100 0 i
*>i 10.0.0.3 100 0 i
N9K1# show bgp l2vpn evpn su
BGP summary information for VRF default, address family L2VPN EVPN
BGP router identifier 10.0.0.1, local AS number 65001
BGP table version is 12, L2VPN EVPN config peers 2, capable peers 2
6 network entries and 8 paths using 1320 bytes of memory
BGP attribute entries [3/492], BGP AS path entries [0/0]
BGP community entries [0/0], BGP clusterlist entries [2/8]
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.0.0.2 4 65001 93 80 12 0 0 01:08:06 2
10.0.0.4 4 65001 91 79 12 0 0 01:06:36 2
Debug bgp updates
N9K1# show nve pee detail
Details of nve Peers:
----------------------------------------
Peer-Ip: 10.0.0.3
NVE Interface : nve1
Peer State : Up
Peer Uptime : 00:10:23
Router-Mac : n/a
Peer First VNI : 10010
Time since Create : 00:10:23
Configured VNIs : 10010
Provision State : peer-add-complete
Learnt CP VNIs : 10010
vni assignment mode : SYMMETRIC
Peer Location : N/A
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
BGP L2VPN and NVE Ingress Replication
No Multicast deployed, propagation using BGP for Ingress replication as Point to Point peer
Spines
interface nve1
no shutdown
host-reachability protocol bgp
source-interface loopback0
member vni 10010
suppress-arp
ingress-replication protocol bgp
N9K1# show bgp l2vpn evpn
BGP routing table information for VRF default, address family L2VPN EVPN
BGP table version is 2269, Local Router ID is 10.0.0.1
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-i
njected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup, 2 - b
est2
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 10.0.0.1:32777 (L2VNI 10010)
*>i[2]:[0]:[0]:[48]:[0050.7966.680e]:[0]:[0.0.0.0]/216
10.0.0.3 100 0 i
*>l[2]:[0]:[0]:[48]:[0050.7966.680d]:[32]:[10.1.10.13]/248
10.0.0.1 100 32768 i
*>i[2]:[0]:[0]:[48]:[0050.7966.680e]:[32]:[10.1.10.14]/248
10.0.0.3 100 0 i
*>l[3]:[0]:[32]:[10.0.0.1]/88
10.0.0.1 100 32768 i
*>i[3]:[0]:[32]:[10.0.0.3]/88
10.0.0.3 100 0 I
## [3] route type 3
Route Distinguisher: 10.0.0.3:32777
* i[2]:[0]:[0]:[48]:[0050.7966.680e]:[0]:[0.0.0.0]/216
10.0.0.3 100 0 i
*>i 10.0.0.3 100 0 i
* i[2]:[0]:[0]:[48]:[0050.7966.680e]:[32]:[10.1.10.14]/248
10.0.0.3 100 0 i
*>i 10.0.0.3 100 0 i
* i[3]:[0]:[32]:[10.0.0.3]/88
10.0.0.3 100 0 i
*>i 10.0.0.3 100 0 i
N9K3# show nve peers
Interface Peer-IP State LearnType Uptime Router-Mac
--------- --------------- ----- --------- -------- -----------------
nve1 10.0.0.1 Up CP 00:02:44 n/a
N9K3# show nve peers de
Details of nve Peers:
----------------------------------------
Peer-Ip: 10.0.0.1
NVE Interface : nve1
Peer State : Up
Peer Uptime : 00:02:52
Router-Mac : n/a
Peer First VNI : 10010
Time since Create : 00:02:52
Configured VNIs : 10010
Provision State : peer-add-complete
Learnt CP VNIs : 10010
vni assignment mode : SYMMETRIC
Peer Location : N/A
N9K3# show nve vni
Codes: CP - Control Plane DP - Data Plane
UC - Unconfigured SA - Suppress ARP
SU - Suppress Unknown Unicast
Xconn - Crossconnect
MS-IR - Multisite Ingress Replication
Interface VNI Multicast-group State Mode Type [BD/VRF] Flags
--------- -------- ----------------- ----- ---- ------------------ -----
nve1 10010 UnicastBGP Up CP L2 [10]
N9K1# show system internal l2fwder mac
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link,
(T) - True, (F) - False, C - ControlPlane MAC
VLAN MAC Address Type age Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
* 10 0050.7966.680d dynamic 00:03:35 F F Eth1/9
* 10 0050.7966.680e static - F F nve-peer1 10.0.0.3
G 10 5000.0003.0007 static - F F sup-eth1(R)
1 1 -00:01:00:01:00:01 - 1
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Adding Additional VLAN, VNI and SVI
Leafs revert to use multicast for IR
interface nve1
no shutdown
host-reachability protocol bgp
source-interface loopback0
member vni 10010
mcast-group 224.1.1.1
Leafs
vlan 20
name VLAN20
vn-segment 10020
interface Vlan20
no shutdown
ip address 10.1.20.254/24
fabric forwarding mode anycast-gateway
interface nve1
member vni 10020
mcast-group 224.1.1.1
N9K3
interface Ethernet1/8
switchport access vlan 20
spanning-tree port type edge
N9K3# show system internal l2fwder mac
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link,
(T) - True, (F) - False, C - ControlPlane MAC
VLAN MAC Address Type age Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
G 20 5000.0004.0007 static - F F sup-eth1(R)
G 10 5000.0004.0007 static - F F sup-eth1(R)
* 10 0050.7966.680d static - F F nve-peer1 10.0.0.1
* 10 0050.7966.680e dynamic 00:00:30 F F Eth1/7
* 20 0050.7966.680f dynamic 00:02:46 F F Eth1/8
1 1 -00:01:00:01:00:01 - 1
N9K1# show system internal l2fwder mac
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link,
(T) - True, (F) - False, C - ControlPlane MAC
VLAN MAC Address Type age Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
* 10 0050.7966.680d dynamic 00:00:41 F F Eth1/9
G 20 5000.0003.0007 static - F F sup-eth1(R)
* 10 0050.7966.680e static - F F nve-peer1 10.0.0.3
G 10 5000.0003.0007 static - F F sup-eth1(R)
* 20 0050.7966.680f static - F F nve-peer1 10.0.0.3
1 1 -00:01:00:01:00:01 - 1
N9K1# show bgp l2vpn evpn
BGP routing table information for VRF default, address family L2VPN EVPN
BGP table version is 2379, Local Router ID is 10.0.0.1
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-i
njected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup, 2 - b
est2
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 10.0.0.1:32777 (L2VNI 10010)
*>l[2]:[0]:[0]:[48]:[0050.7966.680d]:[0]:[0.0.0.0]/216
10.0.0.1 100 32768 i
*>i[2]:[0]:[0]:[48]:[0050.7966.680e]:[0]:[0.0.0.0]/216
10.0.0.3 100 0 i
*>l[2]:[0]:[0]:[48]:[0050.7966.680d]:[32]:[10.1.10.13]/248
10.0.0.1 100 32768 i
*>i[2]:[0]:[0]:[48]:[0050.7966.680e]:[32]:[10.1.10.14]/248
10.0.0.3 100 0 i
Route Distinguisher: 10.0.0.1:32787 (L2VNI 10020)
*>i[2]:[0]:[0]:[48]:[0050.7966.680f]:[0]:[0.0.0.0]/216
10.0.0.3 100 0 i
*>l[2]:[0]:[0]:[48]:[0001.0001.0001]:[32]:[10.1.20.254]/248
10.0.0.1 100 32768 i
*>i[2]:[0]:[0]:[48]:[0050.7966.680f]:[32]:[10.1.20.20]/248
10.0.0.3 100 0 i
Route Distinguisher: 10.0.0.3:32777
* i[2]:[0]:[0]:[48]:[0050.7966.680e]:[0]:[0.0.0.0]/216
10.0.0.3 100 0 i
*>i 10.0.0.3 100 0 i
*>i[2]:[0]:[0]:[48]:[0050.7966.680e]:[32]:[10.1.10.14]/248
10.0.0.3 100 0 i
* i 10.0.0.3 100 0 i
Route Distinguisher: 10.0.0.3:32787
* i[2]:[0]:[0]:[48]:[0050.7966.680f]:[0]:[0.0.0.0]/216
10.0.0.3 100 0 i
*>i 10.0.0.3 100 0 i
* i[2]:[0]:[0]:[48]:[0050.7966.680f]:[32]:[10.1.20.20]/248
10.0.0.3 100 0 i
*>i 10.0.0.3 100 0 i
N9K1# show nve vni
Codes: CP - Control Plane DP - Data Plane
UC - Unconfigured SA - Suppress ARP
SU - Suppress Unknown Unicast
Xconn - Crossconnect
MS-IR - Multisite Ingress Replication
Interface VNI Multicast-group State Mode Type [BD/VRF] Flags
--------- -------- ----------------- ----- ---- ------------------ -----
nve1 10010 224.1.1.1 Up CP L2 [10]
nve1 10020 224.1.1.1 Up CP L2 [20]
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
LACP Port Channel between IOS and Nexus
SW9
interface range GigabitEthernet0/0-1
shutdown
switchport trunk encapsulation dot1q
switchport mode trunk
channel-protocol lacp
channel-group 1 mode active
no shutdown
SW9#show etherchannel summary
Number of channel-groups in use: 1
Number of aggregators: 1
Group Port-channel Protocol Ports
------+-------------+-----------+-----------------------------------------------
1 Po1(SU) LACP Gi0/0(P) Gi0/1(P)
SW9(config)#vlan 10
SW9(config-vlan)#name VLAN10
SW9(config-vlan)#exit
SW9(config)#vlan 20
SW9(config-vlan)#name VLAN20
N9K1
N9K1(config)# int e1/7-8
N9K1(config-if-range)# shutdown
N9K1(config-if-range)# switchport mode trunk
N9K1(config-if-range)# channel-group 1 mode active
command failed: port not compatible [port mode]
** You can use force option to override the port's parameters
** (e.g. "channel-group X force")
** Use "show port-channel compatibility-parameters" to get more information on failure
N9K1(config-if-range)# exit
N9K1(config)# no int port-channel 1
N9K1(config)# int e1/7-8
N9K1(config-if-range)# switchport mode trunk
N9K1(config-if-range)# channel-group 1 mode active
N9K1(config-if-range)# show run int port-chan 1
show running-config interface port-channel1
interface port-channel1
switchport mode trunk
N9K1(config)# int e1/7-8
N9K1(config-if-range)# no shut
N9K1# show port-channel summary
Flags: D - Down P - Up in port-channel (members)
I - Individual H - Hot-standby (LACP only)
s - Suspended r - Module-removed
b - BFD Session Wait
S - Switched R - Routed
U - Up (port-channel)
p - Up in delay-lacp mode (member)
M - Not in use. Min-links not met
--------------------------------------------------------------------------------
Group Port- Type Protocol Member Ports
Channel
--------------------------------------------------------------------------------
1 Po1(SU) Eth LACP Eth1/7(P) Eth1/8(P)
N9K1# show vlan bri
VLAN Name Status Ports
---- -------------------------------- --------- -------------------------------
…
10 VLAN10 active Po1, Eth1/7, Eth1/8, Eth1/9
20 VLAN20 active Po1, Eth1/7, Eth1/8
PC21> ping 10.1.20.20
84 bytes from 10.1.20.20 icmp_seq=1 ttl=64 time=32.017 ms
84 bytes from 10.1.20.20 icmp_seq=2 ttl=64 time=24.190 ms
84 bytes from 10.1.20.20 icmp_seq=3 ttl=64 time=23.672 ms
SW9#show mac address-table dynamic vlan 20
Mac Address Table
-------------------------------------------
Vlan Mac Address Type Ports
---- ----------- -------- -----
20 0050.7966.680f DYNAMIC Gi0/0
20 0050.7966.6810 DYNAMIC Gi1/0
N9K3# show system internal l2fwder mac
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link,
(T) - True, (F) - False, C - ControlPlane MAC
VLAN MAC Address Type age Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
G 20 5000.0004.0007 static - F F sup-eth1(R)
G 10 5000.0004.0007 static - F F sup-eth1(R)
* 10 0050.7966.680d static - F F nve-peer1 10.0.0.1
* 10 0050.7966.680e dynamic 00:04:08 F F Eth1/7
* 20 0050.7966.6810 static - F F nve-peer1 10.0.0.1
N9K3# show bgp l2vpn evpn
…
Route Distinguisher: 10.0.0.3:32787 (L2VNI 10020)
*>l[2]:[0]:[0]:[48]:[0050.7966.680f]:[0]:[0.0.0.0]/216
10.0.0.3 100 32768 i
*>i[2]:[0]:[0]:[48]:[0050.7966.6810]:[0]:[0.0.0.0]/216
10.0.0.1 100 0 i
*>l[2]:[0]:[0]:[48]:[0050.7966.680f]:[32]:[10.1.20.20]/248
10.0.0.3 100 32768 i
*>i[2]:[0]:[0]:[48]:[0050.7966.6810]:[32]:[10.1.20.21]/248
10.0.0.1 100 0 I
SW9#show mac address-table dynamic
Mac Address Table
-------------------------------------------
Vlan Mac Address Type Ports
---- ----------- -------- -----
10 0001.0001.0001 DYNAMIC Gi0/0
20 0001.0001.0001 DYNAMIC Gi0/1
20 0050.7966.680f DYNAMIC Gi0/1
20 0050.7966.6810 DYNAMIC Gi1/0
20 5000.0003.0007 DYNAMIC Gi0/0
Total Mac Addresses for this criterion: 5
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Inter VNI Routing with Symmetric IRB
VXLAN Network with MP-BGP EVPN Control Plane Design Guid
Asymmetric IRB
Symmetric IRB
Leafs
N9K1(config)# vlan 100
N9K1(config-vlan)# vn-segment 100100
N9K1(config-vlan)# name VLAN100
N9K1(config)# vrf context CUST1
N9K1(config-vrf)# vni 100100
N9K1(config-vrf)# rd auto
N9K1(config-vrf)# address-family ipv4 unicast
N9K1(config-vrf-af-ipv4)# route-target both auto
N9K1(config-vrf-af-ipv4)# route-target both auto evpn
N9K1(config)# int vlan 10
N9K1(config-if)# vrf member CUST1
N9K1(config-if)# ip address 10.1.10.254/24
N9K1(config-if)# fabric forwarding mode anycast-gateway
N9K1(config-if)# int vlan 20
N9K1(config-if)# vrf member CUST1
N9K1(config-if)# ip address 10.1.20.254/24
N9K1(config-if)# fabric forwarding mode anycast-gateway
N9K1(config)# int nve 1
N9K1(config-if-nve)# shutdown
N9K1(config-if-nve)# member vni 100100 associate-vrf
N9K1(config-if-nve-vni)# no shut
N9K1(config)# int vlan 100
N9K1(config-if)# vrf member CUST1
Warning: Deleted all L3 config on interface Vlan100
N9K1(config-if)# ip forward
N9K1(config-if)# no shut
N9K1(config)# router bgp 65001
N9K1(config-router)# vrf CUST1
N9K1(config-router-vrf)# address-family ipv4 unicast
N9K1(config-router-vrf-af)# network 10.1.10.0/24
N9K1(config-router-vrf-af)# network 10.1.20.0/24
N9K1# show ip int bri vrf CUST1
IP Interface Status for VRF "CUST1"(3)
Interface IP Address Interface Status
Vlan10 10.1.10.254 protocol-up/link-up/admin-up
Vlan20 10.1.20.254 protocol-up/link-up/admin-up
Vlan100 forward-enabled protocol-up/link-up/admin-up
PC21> ping 10.1.10.14
10.1.10.14 icmp_seq=1 timeout
10.1.10.14 icmp_seq=2 timeout
84 bytes from 10.1.10.14 icmp_seq=3 ttl=62 time=40.824 ms
84 bytes from 10.1.10.14 icmp_seq=4 ttl=62 time=25.085 ms
84 bytes from 10.1.10.14 icmp_seq=5 ttl=62 time=32.089 ms
N9K3# show bgp l2 evpn
BGP routing table information for VRF default, address family L2VPN EVPN
BGP table version is 7682, Local Router ID is 10.0.0.3
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-i
njected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup, 2 - b
est2
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 10.0.0.1:3
*>i[5]:[0]:[0]:[24]:[10.1.10.0]/224
10.0.0.1 100 0 i
* i 10.0.0.1 100 0 i
*>i[5]:[0]:[0]:[24]:[10.1.20.0]/224
10.0.0.1 100 0 i
* i 10.0.0.1 100 0 i
Route Distinguisher: 10.0.0.1:32777
* i[2]:[0]:[0]:[48]:[0050.7966.680d]:[0]:[0.0.0.0]/216
10.0.0.1 100 0 i
*>i 10.0.0.1 100 0 i
* i[2]:[0]:[0]:[48]:[0050.7966.680d]:[32]:[10.1.10.13]/272
10.0.0.1 100 0 i
*>i 10.0.0.1 100 0 i
Route Distinguisher: 10.0.0.1:32787
*>i[2]:[0]:[0]:[48]:[0050.7966.6810]:[0]:[0.0.0.0]/216
10.0.0.1 100 0 i
* i 10.0.0.1 100 0 i
*>i[2]:[0]:[0]:[48]:[0050.7966.6810]:[32]:[10.1.20.21]/272
10.0.0.1 100 0 i
* i 10.0.0.1 100 0 i
Route Distinguisher: 10.0.0.3:32777 (L2VNI 10010)
*>i[2]:[0]:[0]:[48]:[0050.7966.680d]:[0]:[0.0.0.0]/216
10.0.0.1 100 0 i
*>l[2]:[0]:[0]:[48]:[0050.7966.680e]:[0]:[0.0.0.0]/216
10.0.0.3 100 32768 i
*>i[2]:[0]:[0]:[48]:[0050.7966.680d]:[32]:[10.1.10.13]/272
10.0.0.1 100 0 i
*>l[2]:[0]:[0]:[48]:[0050.7966.680e]:[32]:[10.1.10.14]/272
10.0.0.3 100 32768 i
Route Distinguisher: 10.0.0.3:32787 (L2VNI 10020)
*>l[2]:[0]:[0]:[48]:[0050.7966.680f]:[0]:[0.0.0.0]/216
10.0.0.3 100 32768 i
*>i[2]:[0]:[0]:[48]:[0050.7966.6810]:[0]:[0.0.0.0]/216
10.0.0.1 100 0 i
*>l[2]:[0]:[0]:[48]:[0050.7966.680f]:[32]:[10.1.20.20]/272
10.0.0.3 100 32768 i
*>i[2]:[0]:[0]:[48]:[0050.7966.6810]:[32]:[10.1.20.21]/272
10.0.0.1 100 0 i
Route Distinguisher: 10.0.0.3:3 (L3VNI 100100)
*>l[5]:[0]:[0]:[24]:[10.1.10.0]/224
10.0.0.3 100 32768 i
*>l[5]:[0]:[0]:[24]:[10.1.20.0]/224
10.0.0.3 100 32768 i
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Multi Tenancy Adding Second Customer
Leafs
vlan 200
name VLAN200
vn-segment 200200
vlan 210
name VLAN210
vn-segment 20010
vlan 220
name VLAN220
vn-segment 20020
vrf context CUST2
vni 200200
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
interface Vlan210
no shutdown
vrf member CUST2
ip address 10.2.10.254/24
fabric forwarding mode anycast-gateway
interface Vlan220
no shutdown
vrf member CUST2
ip address 10.2.20.254/24
fabric forwarding mode anycast-gateway
interface Vlan200
no shutdown
vrf member CUST2
ip forward
interface nve1
no shutdown
host-reachability protocol bgp
source-interface loopback0
member vni 10010
mcast-group 224.1.1.1
member vni 10020
mcast-group 224.1.1.1
member vni 20010
mcast-group 224.1.1.2
member vni 20020
mcast-group 224.1.1.2
member vni 100100 associate-vrf
member vni 200200 associate-vrf
evpn
vni 10010 l2
rd auto
route-target import auto
route-target export auto
vni 20010 l2
rd auto
route-target import auto
route-target export auto
vni 20020 l2
rd auto
route-target import auto
route-target export auto
router bgp 65001
vrf CUST2
address-family ipv4 unicast
network 10.2.10.0/24
network 10.2.20.0/24
SW9
SW9(config)#vlan 210
SW9(config-vlan)#name VLAN210
SW9(config-vlan)#VLAN 220
SW9(config-vlan)#NAME VLAN220
SW9(config-vlan)interface GigabitEthernet0/2
switchport access vlan 210
switchport mode access
spanning-tree portfast edge
SW9(config-vlan)interface GigabitEthernet0/3
switchport access vlan 220
switchport mode access
spanning-tree portfast edge
Both N9K1 and N9K3 can see mac address from vlan 210/220
N9K1# show system internal l2fwder mac
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link,
(T) - True, (F) - False, C - ControlPlane MAC
VLAN MAC Address Type age Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
* 210 0050.7966.6811 dynamic 0.651186 F F Po1
* 10 0050.7966.680d dynamic 00:04:14 F F Eth1/9
* 220 0050.7966.6812 dynamic 00:09:24 F F Po1
* 1 5000.0009.0001 dynamic 00:00:36 F F Eth1/3
G 20 5000.0003.0007 static - F F sup-eth1(R)
* 10 0050.7966.680e static - F F nve-peer1 10.0.0.3
G 10 5000.0003.0007 static - F F sup-eth1(R)
G 210 5000.0003.0007 static - F F sup-eth1(R)
G 220 5000.0003.0007 static - F F sup-eth1(R)
G 200 5000.0003.0007 static - F F sup-eth1(R)
* 20 0050.7966.680f static - F F nve-peer1 10.0.0.3
1 1 -00:01:00:01:00:01 - 1
N9K3# show system internal l2fwder mac
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link,
(T) - True, (F) - False, C - ControlPlane MAC
VLAN MAC Address Type age Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
G 100 5000.0004.0007 static - F F sup-eth1(R)
G 20 5000.0004.0007 static - F F sup-eth1(R)
G 10 5000.0004.0007 static - F F sup-eth1(R)
* 210 0050.7966.6811 static - F F nve-peer1 10.0.0.1
G 210 5000.0004.0007 static - F F sup-eth1(R)
G 220 5000.0004.0007 static - F F sup-eth1(R)
G 200 5000.0004.0007 static - F F sup-eth1(R)
* 10 0050.7966.680d static - F F nve-peer1 10.0.0.1
* 220 0050.7966.6812 static - F F nve-peer1 10.0.0.1
* 1 5000.0009.0001 dynamic 00:01:11 F F Eth1/6
* 10 0050.7966.680e dynamic 00:04:48 F F Eth1/7
* 20 0050.7966.680f dynamic 00:03:05 F F Eth1/8
1 1 -00:01:00:01:00:01 - 1
N9K1# show bgp l2vpn evpn
Route Distinguisher: 10.0.0.1:32977 (L2VNI 20010)
*>l[2]:[0]:[0]:[48]:[0050.7966.6811]:[0]:[0.0.0.0]/216
10.0.0.1 100 32768 i
*>l[2]:[0]:[0]:[48]:[0050.7966.6811]:[32]:[10.2.10.15]/272
10.0.0.1 100 32768 i
Route Distinguisher: 10.0.0.1:32987 (L2VNI 20020)
*>l[2]:[0]:[0]:[48]:[0050.7966.6812]:[0]:[0.0.0.0]/216
10.0.0.1 100 32768 i
*>l[2]:[0]:[0]:[48]:[0050.7966.6812]:[32]:[10.2.20.17]/272
10.0.0.1 100 32768 i
…
Route Distinguisher: 10.0.0.1:4 (L3VNI 200200)
*>l[5]:[0]:[0]:[24]:[10.2.10.0]/224
10.0.0.1 100 32768 i
*>l[5]:[0]:[0]:[24]:[10.2.20.0]/224
10.0.0.1 100 32768 I
…
Different vrf PC can't ping (multi tenant)
PC21> ping 10.2.10.15
10.2.10.15 icmp_seq=1 timeout
10.2.10.15 icmp_seq=2 timeout
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
vPC and VXLAN with BGP L2VPN EVPN or AnyCast VTEP
N9K1 and N9K3
Both
feature vpc
feature lacp
vrf context VPC
N9K1# show run int e1/3
interface Ethernet1/3
no switchport
vrf member VPC
ip address 10.1.3.1/24
no shutdown
N9K3# show run int e1/3
interface Ethernet1/3
no switchport
vrf member VPC
ip address 10.1.3.3/24
no shutdown
interface Ethernet1/4-5
switchport mode trunk
channel-group 13 mode active
N9K1
Vpc domain 13
peer-keepalive destination 10.1.3.3 source 10.1.3.1 vrf VPC
N9K3
Vpc domain 13
peer-keepalive destination 10.1.3.1 source 10.1.3.3 vrf VPC
role priority 1
interface port-channel13
switchport mode trunk
vpc peer-link
Both
interface Ethernet1/6
switchport mode trunk
channel-group 10 mode active
interface port-channel10
switchport mode trunk
vpc 10
Both
interface loopback0
ip address 10.0.0.13/32 secondary
int nve 1
shutdown
no shutdown
SW8
interface range GigabitEthernet0/0-1
switchport trunk encapsulation dot1q
switchport mode trunk
media-type rj45
negotiation auto
channel-group 1 mode active
SW8#show run int g0/2
Building configuration...
Current configuration : 128 bytes
!
interface GigabitEthernet0/2
switchport access vlan 210
media-type rj45
negotiation auto
spanning-tree portfast edge
end
SW8#show run int g0/3
Building configuration...
Current configuration : 128 bytes
!
interface GigabitEthernet0/3
switchport access vlan 220
media-type rj45
negotiation auto
spanning-tree portfast edge
end
SW8#show vlan bri
VLAN Name Status Ports
---- -------------------------------- --------- -------------------------------
1 default active Gi1/0, Gi1/1, Gi1/2, Gi1/3
210 VLAN210 active Gi0/2
220 VLAN220 active Gi0/3
N9K3(config-if-nve)# show vpc
Legend:
(*) - local vPC is down, forwarding via vPC peer-link
vPC domain id : 13
Peer status : peer adjacency formed ok
vPC keep-alive status : peer is alive
Configuration consistency status : success
Per-vlan consistency status : success
Type-2 consistency status : success
vPC role : primary
Number of vPCs configured : 1
Peer Gateway : Disabled
Dual-active excluded VLANs : -
Graceful Consistency Check : Enabled
Auto-recovery status : Disabled
Delay-restore status : Timer is off.(timeout = 30s)
Delay-restore SVI status : Timer is off.(timeout = 10s)
Operational Layer3 Peer-router : Disabled
vPC Peer-link status
---------------------------------------------------------------------
id Port Status Active vlans
-- ---- ------ -------------------------------------------------
1 Po13 up 1,10,20,100,200,210,220
vPC status
----------------------------------------------------------------------------
Id Port Status Consistency Reason Active vlans
-- ------------ ------ ----------- ------ ---------------
10 Po10 up success success 1,10,20,100,200,210,220
PC17> ping 10.2.10.15
84 bytes from 10.2.10.15 icmp_seq=1 ttl=64 time=15.406 ms
84 bytes from 10.2.10.15 icmp_seq=2 ttl=64 time=14.237 ms
84 bytes from 10.2.10.15 icmp_seq=3 ttl=64 time=17.544 ms
^C
PC17> ping 10.2.20.19
84 bytes from 10.2.20.19 icmp_seq=1 ttl=63 time=75.465 ms
84 bytes from 10.2.20.19 icmp_seq=2 ttl=63 time=26.018 ms
84 bytes from 10.2.20.19 icmp_seq=3 ttl=63 time=17.534 ms
84 bytes from 10.2.20.19 icmp_seq=4 ttl=63 time=15.587 ms
^C
PC17> ping 10.2.20.16
84 bytes from 10.2.20.16 icmp_seq=1 ttl=63 time=36.854 ms
84 bytes from 10.2.20.16 icmp_seq=2 ttl=63 time=96.014 ms
84 bytes from 10.2.20.16 icmp_seq=3 ttl=63 time=26.097 ms
N9K1# show bgp l2vpn evpn vni-id 20010
BGP routing table information for VRF default, address family L2VPN EVPN
BGP table version is 8728, Local Router ID is 10.0.0.1
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-i
njected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup, 2 - b
est2
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 10.0.0.1:32977 (L2VNI 20010)
…
*>l[2]:[0]:[0]:[48]:[0050.7966.6811]:[32]:[10.2.10.15]/272
10.0.0.13 100 32768 i
*>l[2]:[0]:[0]:[48]:[0050.7966.6813]:[32]:[10.2.10.17]/272
10.0.0.13 100 32768 I
N9K1# show bgp l2vpn evpn vni-id 20020
BGP routing table information for VRF default, address family L2VPN EVPN
BGP table version is 8728, Local Router ID is 10.0.0.1
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-i
njected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup, 2 - b
est2
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 10.0.0.1:32987 (L2VNI 20020)
…
*>l[2]:[0]:[0]:[48]:[0050.7966.6812]:[32]:[10.2.20.16]/272
10.0.0.13 100 32768 i
*>l[2]:[0]:[0]:[48]:[0050.7966.6814]:[32]:[10.2.20.19]/272
10.0.0.13 100 32768 i
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Service Leaf Setup and Single ASAv Firewall Deployment
N9K5
nv overlay evpn
feature ospf
feature bgp
feature pim
feature fabric forwarding
feature interface-vlan
feature vn-segment-vlan-based
feature nv overlay
fabric forwarding anycast-gateway-mac 0001.0001.0001
vlan 10
name VLAN10
vn-segment 10010
vlan 100
name VLAN100
vn-segment 100100
ip prefix-list DEFAULT seq 5 permit 0.0.0.0/0
route-map STATIC_TO_BGP permit 10
match ip address prefix-list DEFAULT
vrf context CUST1
vni 100100
ip route 0.0.0.0/0 10.1.10.1
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
vrf context management
interface Vlan10
no shutdown
vrf member CUST1
ip address 10.1.10.254/24
interface Vlan100
no shutdown
vrf member CUST1
ip forward
interface nve1
no shutdown
host-reachability protocol bgp
source-interface loopback0
member vni 10010
mcast-group 224.1.1.1
member vni 100100 associate-vrf
interface Ethernet1/3
switchport mode trunk
router bgp 65001
address-family l2vpn evpn
neighbor 10.0.0.2
remote-as 65001
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
neighbor 10.0.0.4
remote-as 65001
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
vrf CUST1
address-family ipv4 unicast
network 10.1.10.0/24
redistribute static route-map STATIC_TO_BGP
default-information originate
evpn
vni 10010 l2
rd auto
route-target import auto
route-target export auto
SW6
interface GigabitEthernet0/0
switchport trunk encapsulation dot1q
switchport mode trunk
interface GigabitEthernet0/2
switchport trunk encapsulation dot1q
switchport mode trunk
Vlan 10
Name VLAN10
ASAv11
interface GigabitEthernet0/0.10
vlan 10
nameif inside-vlan10
security-level 100
ip address 10.1.10.1 255.255.255.0
!
interface GigabitEthernet0/2
nameif outside
security-level 0
ip address 101.0.0.11 255.255.255.0
!
object network VLAN10
subnet 10.1.10.0 255.255.255.0
!
object network VLAN10
nat (inside-vlan10,outside) dynamic interface
route outside 0.0.0.0 0.0.0.0 101.0.0.1 1
!
policy-map global_policy
class inspection_default
inspect icmp
INT
interface GigabitEthernet0/1
ip address 101.0.0.1 255.255.255.0
N9K1# show ip route vrf CUST1
IP Route Table for VRF "CUST1"
…
0.0.0.0/0, ubest/mbest: 1/0
*via 10.0.0.5%default, [200/0], 00:01:38, bgp-65001, internal, tag 65001 (ev
pn) segid: 100100 tunnelid: 0xa000005 encap: VXLAN
…
PC13> ping 101.0.0.1
84 bytes from 101.0.0.1 icmp_seq=1 ttl=255 time=47.921 ms
84 bytes from 101.0.0.1 icmp_seq=2 ttl=255 time=44.485 ms
84 bytes from 101.0.0.1 icmp_seq=3 ttl=255 time=44.479 ms
84 bytes from 101.0.0.1 icmp_seq=4 ttl=255 time=27.854 ms
84 bytes from 101.0.0.1 icmp_seq=5 ttl=255 time=43.083 ms
ASA11# show conn long
2 in use, 4 most used
…
UDP outside: 101.0.0.1/4241 (101.0.0.1/4241) inside-vlan10: 10.1.10.13/4240 (100
UDP outside: 101.0.0.1/46242 (101.0.0.1/46242) inside-vlan10: 10.1.10.13/46241 0
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
New Service Leaf Setup and Tshooting and ASA HA Part 1
N9K6
nv overlay evpn
feature ospf
feature bgp
feature pim
feature fabric forwarding
feature interface-vlan
feature vn-segment-vlan-based
feature nv overlay
fabric forwarding anycast-gateway-mac 0001.0001.0001
ip pim rp-address 10.0.0.2 group-list 224.0.0.0/4
ip pim ssm range 232.0.0.0/8
vlan 1,10,100
vlan 10
name VLAN10
vn-segment 10010
vlan 100
name VLAN100
vn-segment 100100
ip prefix-list DEFAULT seq 5 permit 0.0.0.0/0
route-map STATIC_TO_BGP permit 10
match ip address prefix-list DEFAULT
vrf context CUST1
vni 100100
ip route 0.0.0.0/0 10.1.10.1
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
interface Vlan10
no shutdown
vrf member CUST1
ip address 10.1.10.254/24
interface Vlan100
no shutdown
vrf member CUST1
ip forward
interface nve1
no shutdown
host-reachability protocol bgp
source-interface loopback0
member vni 10010
mcast-group 224.1.1.1
member vni 100100 associate-vrf
router bgp 65001
address-family l2vpn evpn
neighbor 10.0.0.2
remote-as 65001
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
neighbor 10.0.0.4
remote-as 65001
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
vrf CUST1
address-family ipv4 unicast
network 10.1.10.0/24
redistribute static route-map STATIC_TO_BGP
default-information originate
evpn
vni 10010 l2
rd auto
route-target import auto
route-target export auto
SPINE1# show bgp l2vpn evpn su
BGP summary information for VRF default, address family L2VPN EVPN
BGP router identifier 10.0.0.2, local AS number 65001
BGP table version is 8447, L2VPN EVPN config peers 4, capable peers 4
26 network entries and 26 paths using 5720 bytes of memory
BGP attribute entries [16/2624], BGP AS path entries [0/0]
BGP community entries [0/0], BGP clusterlist entries [0/0]
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.0.0.1 4 65001 9062 7194 8447 0 0 17:49:22 10
10.0.0.3 4 65001 9134 7320 8447 0 0 17:49:15 11
10.0.0.5 4 65001 1742 2363 8447 0 0 00:00:19 3
10.0.0.6 4 65001 11 7 8447 0 0 00:00:32 2
N9K5 and N9K6
feature lacp
feature vpc
vrf context VPC
interface Ethernet1/4
no switchport
vrf member VPC
ip address 10.5.6.5/24
no shutdown
N9K5
vpc domain 56
peer-keepalive destination 10.5.6.6 source 10.5.6.5 vrf VPC
N9K6
vpc domain 56
peer-keepalive destination 10.5.6.5 source 10.5.6.6 vrf VPC
interface Ethernet1/5-6
switchport mode trunk
channel-group 56 mode active
no shut
interface port-channel56
vpc peer-link
interface loopback0
ip address 10.0.0.56/32 secondary
interface Ethernet1/3
switchport mode trunk
channel-group 11 mode active
interface port-channel11
switchport mode trunk
vpc 11
SW6
SW6(config)#default int range g0/0-1
interface range GigabitEthernet0/0-1
switchport trunk encapsulation dot1q
switchport mode trunk
no negotiation auto
channel-protocol lacp
channel-group 11 mode active
N9k5# show vpc
Legend:
(*) - local vPC is down, forwarding via vPC peer-link
vPC domain id : 56
Peer status : peer adjacency formed ok
vPC keep-alive status : peer is alive
Configuration consistency status : success
Per-vlan consistency status : success
Type-2 consistency status : success
vPC role : primary
Number of vPCs configured : 1
Peer Gateway : Disabled
Dual-active excluded VLANs : -
Graceful Consistency Check : Enabled
Auto-recovery status : Disabled
Delay-restore status : Timer is off.(timeout = 30s)
Delay-restore SVI status : Timer is off.(timeout = 10s)
Operational Layer3 Peer-router : Disabled
vPC Peer-link status
---------------------------------------------------------------------
id Port Status Active vlans
-- ---- ------ -------------------------------------------------
1 Po56 up 1,10,100
vPC status
----------------------------------------------------------------------------
Id Port Status Consistency Reason Active vlans
-- ------------ ------ ----------- ------ ---------------
11 Po11 up success success 1,10,100
N9k5(config-if-nve)# show ip int bri
IP Interface Status for VRF "default"(1)
Interface IP Address Interface Status
Lo0 10.0.0.5 protocol-up/link-up/admin-up
SW6#show etherchannel summary
…
Number of channel-groups in use: 1
Number of aggregators: 1
Group Port-channel Protocol Ports
------+-------------+-----------+-----------------------------------------------
11 Po11(SU) LACP Gi0/0(P) Gi0/1(P)
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
New Service Leaf Setup and Tshooting and ASA HA Part 2
SW6
interface GigabitEthernet0/2
switchport trunk encapsulation dot1q
switchport mode trunk
!
interface GigabitEthernet0/3
switchport trunk encapsulation dot1q
switchport mode trunk
ASA11 and ASA12
ASA11(config)# failover lan interface FAILOVER g0/1
ASA11(config)# int g0/1
ASA11(config-if)# no shut
ASA11(config)# failover link SFFO g0/3
ASA11(config)# int g0/3
ASA11(config-if)# no shut
failover interface ip FAILOVER 10.11.12.11 255.255.255.0 standby 10.11.12.12
failover interface ip SFFO 172.16.12.11 255.255.255.0 standby 172.16.12.12
failover
ASA11 only
failover lan unit primary
ASA11# show failover
Failover On
Failover unit Primary
Failover LAN Interface: FAILOVER GigabitEthernet0/1 (up)
Reconnect timeout 0:00:00
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 1 of 61 maximum
MAC Address Move Notification Interval not set
Version: Ours 9.8(4), Mate 9.8(4)
Serial Number: Ours 9ASK2X8XJB7, Mate 9ALD2T3DPGG
Last Failover at: 18:51:11 UTC Apr 12 2024
This host: Primary - Active
Active time: 341 (sec)
slot 0: empty
Interface inside-vlan10 (10.1.10.1): Normal (Not-Monitored)
Interface outside (101.0.0.11): Normal (Waiting)
Other host: Secondary - Standby Ready
Active time: 0 (sec)
Interface inside-vlan10 (0.0.0.0): Normal (Not-Monitored)
Interface outside (0.0.0.0): Normal (Waiting)
Stateful Failover Logical Update Statistics
Link : SFFO GigabitEthernet0/3 (up)
…
PC14> ping 101.0.0.1 -c 100
84 bytes from 101.0.0.1 icmp_seq=1 ttl=255 time=128.999 ms
84 bytes from 101.0.0.1 icmp_seq=2 ttl=255 time=47.045 ms
84 bytes from 101.0.0.1 icmp_seq=3 ttl=255 time=45.176 ms
84 bytes from 101.0.0.1 icmp_seq=4 ttl=255 time=47.905 ms
84 bytes from 101.0.0.1 icmp_seq=5 ttl=255 time=34.555 ms
PC13> ping 101.0.0.1 -c 100
84 bytes from 101.0.0.1 icmp_seq=1 ttl=255 time=232.132 ms
101.0.0.1 icmp_seq=2 timeout
84 bytes from 101.0.0.1 icmp_seq=3 ttl=255 time=38.146 ms
84 bytes from 101.0.0.1 icmp_seq=4 ttl=255 time=40.523 ms
84 bytes from 101.0.0.1 icmp_seq=5 ttl=255 time=34.523 ms
ASA11# show conn long
…
ICMP outside: 101.0.0.1/0 (101.0.0.1/0) inside-vlan10: 10.1.10.14/30597 (101.0.0.11/30597), , flags , idle 0s, uptime 0s, timeout 2s, bytes 112, xlate id 0x0
ASA11# show failover | in General
General 120 0 100 0
Go to console ASAv12
ASA11# failover active
Switching to Active
ASA11(config)# prompt hostname priority state
ASA11/sec/act(config)#
PC14
84 bytes from 101.0.0.1 icmp_seq=77 ttl=255 time=51.106 ms
84 bytes from 101.0.0.1 icmp_seq=78 ttl=255 time=62.141 ms
84 bytes from 101.0.0.1 icmp_seq=79 ttl=255 time=34.147 ms
84 bytes from 101.0.0.1 icmp_seq=80 ttl=255 time=35.246 ms
84 bytes from 101.0.0.1 icmp_seq=81 ttl=255 time=37.494 ms
84 bytes from 101.0.0.1 icmp_seq=82 ttl=255 time=104.571 ms
84 bytes from 101.0.0.1 icmp_seq=83 ttl=255 time=35.195 ms
84 bytes from 101.0.0.1 icmp_seq=84 ttl=255 time=55.959 ms
84 bytes from 101.0.0.1 icmp_seq=85 ttl=255 time=30.951 ms
84 bytes from 101.0.0.1 icmp_seq=86 ttl=255 time=64.638 ms
84 bytes from 101.0.0.1 icmp_seq=87 ttl=255 time=51.829 ms
84 bytes from 101.0.0.1 icmp_seq=88 ttl=255 time=36.156 ms
84 bytes from 101.0.0.1 icmp_seq=89 ttl=255 time=51.113 ms
ASAv12 console
ASA11/sec/act(config)# show conn long
…
ICMP outside: 101.0.0.1/0 (101.0.0.1/0) inside-vlan10: 10.1.10.14/41094 (101.0.0.11/41094), , flags , id0
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
MPLS L3 VPN Setup for External WAN Access to the Fabric
PE1
vrf definition C1
rd 1:65100
route-target export 1:65100
route-target import 1:65100
!
address-family ipv4
exit-address-family
!
vrf definition C2
rd 2:65100
route-target export 2:65100
route-target import 2:65100
!
address-family ipv4
exit-address-family
!
interface Loopback0
ip address 1.1.1.1 255.255.255.255
ip ospf 1 area 0
!
interface GigabitEthernet0/0
ip address 10.1.1.1 255.255.255.0
ip ospf 1 area 0
!
router ospf 1
mpls ldp autoconfig
!
router bgp 65100
bgp log-neighbor-changes
no bgp default ipv4-unicast
neighbor 1.1.1.2 remote-as 65100
neighbor 1.1.1.2 update-source Loopback0
neighbor 1.1.1.3 remote-as 65100
neighbor 1.1.1.3 update-source Loopback0
!
address-family ipv4
exit-address-family
!
address-family vpnv4
neighbor 1.1.1.2 activate
neighbor 1.1.1.2 send-community extended
neighbor 1.1.1.3 activate
neighbor 1.1.1.3 send-community extended
exit-address-family
PE2
!
vrf definition C1
rd 1:65100
route-target export 1:65100
route-target import 1:65100
!
address-family ipv4
exit-address-family
!
vrf definition C2
rd 2:65100
route-target export 2:65100
route-target import 2:65100
!
address-family ipv4
exit-address-family
!
!
interface Loopback0
ip address 1.1.1.2 255.255.255.255
ip ospf 1 area 0
!
interface GigabitEthernet0/0
ip address 10.1.2.2 255.255.255.0
ip ospf 1 area 0
!
router ospf 1
mpls ldp autoconfig
!
router bgp 65100
bgp log-neighbor-changes
no bgp default ipv4-unicast
neighbor 1.1.1.1 remote-as 65100
neighbor 1.1.1.1 update-source Loopback0
neighbor 1.1.1.3 remote-as 65100
neighbor 1.1.1.3 update-source Loopback0
!
address-family ipv4
exit-address-family
!
address-family vpnv4
neighbor 1.1.1.1 activate
neighbor 1.1.1.1 send-community extended
neighbor 1.1.1.3 activate
neighbor 1.1.1.3 send-community extended
exit-address-family
!
PE3
vrf definition C1
rd 1:65100
route-target export 1:65100
route-target import 1:65100
!
address-family ipv4
exit-address-family
!
vrf definition C2
rd 2:65100
route-target export 2:65100
route-target import 2:65100
!
address-family ipv4
exit-address-family
!
!
!
interface Loopback0
ip address 1.1.1.3 255.255.255.255
ip ospf 1 area 0
!
interface GigabitEthernet0/0
ip address 10.1.3.3 255.255.255.0
ip ospf 1 area 0
!
!
router ospf 1
mpls ldp autoconfig
!
router bgp 65100
bgp log-neighbor-changes
no bgp default ipv4-unicast
neighbor 1.1.1.1 remote-as 65100
neighbor 1.1.1.1 update-source Loopback0
neighbor 1.1.1.2 remote-as 65100
neighbor 1.1.1.2 update-source Loopback0
!
address-family ipv4
exit-address-family
!
address-family vpnv4
neighbor 1.1.1.1 activate
neighbor 1.1.1.1 send-community extended
neighbor 1.1.1.2 activate
neighbor 1.1.1.2 send-community extended
exit-address-family
P1
interface Loopback0
ip address 1.1.1.11 255.255.255.255
ip ospf 1 area 0
!
interface GigabitEthernet0/0
ip address 10.1.1.11 255.255.255.0
ip ospf 1 area 0
!
interface GigabitEthernet0/1
ip address 10.1.2.11 255.255.255.0
ip ospf 1 area 0
!
interface GigabitEthernet0/2
ip address 10.1.3.11 255.255.255.0
ip ospf 1 area 0
!
!
router ospf 1
mpls ldp autoconfig
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
External Connectivity to Private WAN via eBGP Peerings
N9K3
interface Ethernet1/9
no switchport
no shutdown
interface Ethernet1/9.11
encapsulation dot1q 11
vrf member CUST1
ip address 10.101.11.3/24
no shutdown
!
interface Ethernet1/9.12
encapsulation dot1q 12
vrf member CUST2
ip address 10.101.12.3/24
no shutdown
!
N9K3(config)# route-map DIRECT-CUST1
N9K3(config-route-map)# match interface ethernet1/9.11
N9K3(config-route-map)# exit
N9K3(config)# route-map DIRECT-CUST2
N9K3(config-route-map)# match interface ethernet 1/9.12
router bgp 65001
…
vrf CUST1
address-family ipv4 unicast
network 10.1.10.0/24
network 10.1.20.0/24
redistribute direct route-map DIRECT-CUST1
neighbor 10.101.11.1
remote-as 65100
address-family ipv4 unicast
vrf CUST2
address-family ipv4 unicast
network 10.2.10.0/24
network 10.2.20.0/24
redistribute direct route-map DIRECT-CUST2
neighbor 10.101.12.1
remote-as 65100
address-family ipv4 unicast
PE1
interface GigabitEthernet0/1
no ip address
interface GigabitEthernet0/1.11
encapsulation dot1Q 11
vrf forwarding C1
ip address 10.101.11.1 255.255.255.0
!
interface GigabitEthernet0/1.12
encapsulation dot1Q 12
vrf forwarding C2
ip address 10.101.12.1 255.255.255.0
!
router bgp 65100
address-family ipv4 unicast
exit
address-family ipv4 vrf C1
neighbor 10.101.11.3 remote-as 65001
neighbor 10.101.11.3 activate
exit-address-family
!
address-family ipv4 vrf C2
neighbor 10.101.12.3 remote-as 65001
neighbor 10.101.12.3 activate
exit-address-family
PE1#show bgp vrf C1 vpnv4 unicast su
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.101.11.3 4 65001 7 5 4 0 0 00:01:12
PE1#show bgp vrf C2 vpnv4 unicast su
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.101.12.3 4 65001 5 4 6 0 0 00:00:17
PE2
interface GigabitEthernet0/1
No shut
!
interface GigabitEthernet0/1.11
encapsulation dot1Q 11
vrf forwarding C1
ip address 10.101.21.2 255.255.255.0
router bgp 65100
address-family ipv4 vrf C1
neighbor 10.101.21.5 remote-as 65001
neighbor 10.101.21.5 activate
exit-address-family
N9K5
interface Ethernet1/7
no switchport
no shutdown
!
interface Ethernet1/7.11
encapsulation dot1q 11
vrf member CUST1
ip address 10.101.21.5/24
no shutdown
!
router bgp 65001
vrf CUST1
address-family ipv4 unicast
neighbor 10.101.21.2
remote-as 65100
address-family ipv4 unicast
PE2#show bgp vrf C1 vpnv4 unicast su
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.101.21.5 4 65001 9 8 20 0 0 00:00:13 6
PE3
interface GigabitEthernet0/1
no shut
!
interface GigabitEthernet0/1.11
encapsulation dot1Q 11
vrf forwarding C1
ip address 10.101.31.3 255.255.255.0
!
interface GigabitEthernet0/1.12
encapsulation dot1Q 12
vrf forwarding C2
ip address 10.101.32.3 255.255.255.0
Router bgp 65100
address-family ipv4 vrf C1
neighbor 10.101.31.11 remote-as 1
neighbor 10.101.31.11 activate
exit-address-family
PC1
vrf definition C1
rd 1:65100
route-target export 1:65100
route-target import 1:65100
!
address-family ipv4
exit-address-family
vrf definition C2
rd 2:65100
route-target export 2:65100
route-target import 2:65100
!
address-family ipv4
exit-address-family
interface GigabitEthernet0/0
no shut
!
interface GigabitEthernet0/0.11
encapsulation dot1Q 11
vrf forwarding C1
ip address 10.101.31.11 255.255.255.0
!
interface GigabitEthernet0/0.12
encapsulation dot1Q 12
vrf forwarding C2
ip address 10.101.32.11 255.255.255.0
!
router bgp 1
bgp router-id 1.1.1.1
bgp log-neighbor-changes
no bgp default ipv4-unicast
!
address-family ipv4 vrf C1
network 10.101.31.0 mask 255.255.255.0
neighbor 10.101.31.3 remote-as 65100
neighbor 10.101.31.3 activate
exit-address-family
!
address-family ipv4 vrf C2
network 10.101.32.0 mask 255.255.255.0
neighbor 10.101.32.3 remote-as 65100
neighbor 10.101.32.3 activate
exit-address-family
N9k5# show ip route vrf CUST1
10.101.31.0/24, ubest/mbest: 1/0
*via 10.101.21.2, [20/0], 00:00:07, bgp-65001, external, tag 65100
N9k5# ping 10.101.31.11 vrf CUST1
PING 10.101.31.11 (10.101.31.11): 56 data bytes
64 bytes from 10.101.31.11: icmp_seq=1 ttl=250 time=24.351 ms
64 bytes from 10.101.31.11: icmp_seq=3 ttl=250 time=13.252 ms
PC1#ping vrf C1 10.1.10.254
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.1.10.254, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 6/8/15 ms
PC1#traceroute vrf C1 10.1.10.254
Type escape sequence to abort.
Tracing the route to 10.1.10.254
VRF info: (vrf in name/id, vrf out name/id)
1 10.101.31.3 9 msec 4 msec 4 msec
2 10.1.3.11 [MPLS: Labels 16/22 Exp 0] 21 msec 7 msec 7 msec
3 10.101.11.1 [MPLS: Label 22 Exp 0] 7 msec 6 msec 5 msec
4 10.1.10.254 10 msec 7 msec 7 msec
PC1#ping vrf C1 10.1.10.13
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.1.10.13, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 31/43/68 ms
PC1#ping vrf C1 10.1.10.14
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.1.10.14, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 25/59/89 m
PC13> ping 10.101.31.11
84 bytes from 10.101.31.11 icmp_seq=1 ttl=250 time=85.292 ms
84 bytes from 10.101.31.11 icmp_seq=2 ttl=250 time=40.349 ms
84 bytes from 10.101.31.11 icmp_seq=3 ttl=250 time=25.321 ms
84 bytes from 10.101.31.11 icmp_seq=4 ttl=250 time=39.991 ms
84 bytes from 10.101.31.11 icmp_seq=5 ttl=250 time=33.821 ms
PC1#traceroute vrf C1 10.1.20.20
Type escape sequence to abort.
Tracing the route to 10.1.20.20
VRF info: (vrf in name/id, vrf out name/id)
1 10.101.31.3 9 msec 3 msec 2 msec
2 10.1.3.11 [MPLS: Labels 17/26 Exp 0] 13 msec 8 msec 7 msec
3 10.101.21.2 [MPLS: Label 26 Exp 0] 7 msec 27 msec 7 msec
4 10.101.21.5 9 msec 6 msec 6 msec
5 10.101.11.3 20 msec 14 msec 13 msec
6 10.1.20.20 58 msec 15 msec 15 msec
PC20> trace 10.101.31.11
trace to 10.101.31.11, 8 hops max, press Ctrl+C to stop
1 10.1.20.254 8.260 ms 7.879 ms 8.146 ms
2 10.101.11.1 23.795 ms 14.365 ms 15.237 ms
3 10.1.1.11 33.457 ms 34.786 ms 14.364 ms
4 10.1.1.11 11.621 ms 10.599 ms 4.480 ms
5 10.101.31.3 18.826 ms 4.167 ms 4.679 ms
6 10.101.31.3 4.489 ms 6.511 ms
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
VXLAN EVPN Multi Site Overview and Config Walkthrough
Configure and Verify EVPN/VxLAN in Multisite Environment
Leaf3
nv overlay evpn
feature ospf
feature bgp
feature pim
feature fabric forwarding
feature interface-vlan
feature vn-segment-vlan-based
feature nv overlay
fabric forwarding anycast-gateway-mac 0001.0001.0001
ip pim rp-address 10.0.1.23 group-list 224.0.0.0/4
ip pim ssm range 232.0.0.0/8
vlan 1,10,100
vlan 10
name VLAN10
vn-segment 10010
vlan 100
name VLAN100
vn-segment 100100
vrf context CUST1
vni 100100
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
vrf context management
interface Vlan1
interface Vlan10
no shutdown
vrf member CUST1
ip address 10.1.10.254/24
fabric forwarding mode anycast-gateway
interface Vlan100
no shutdown
vrf member CUST1
ip forward
interface nve1
no shutdown
host-reachability protocol bgp
source-interface loopback0
member vni 10010
mcast-group 224.1.1.1
member vni 100100 associate-vrf
interface Ethernet1/1
no switchport
ip address 10.3.3.3/24
ip ospf network point-to-point
ip router ospf 1 area 0.0.0.0
ip pim sparse-mode
no shutdown
interface Ethernet1/2
switchport access vlan 10
spanning-tree port type edge
interface loopback0
ip address 10.0.1.33/32
ip router ospf 1 area 0.0.0.0
ip pim sparse-mode
cli alias name wr copy run start
line console
line vty
boot nxos bootflash:/nxos.9.2.2.bin
router ospf 1
router bgp 65002
address-family l2vpn evpn
neighbor 10.0.1.23
remote-as 65002
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
vrf CUST1
address-family ipv4 unicast
network 10.1.10.0/24
evpn
vni 10010 l2
rd auto
route-target import auto
route-target export auto
Leaf4
nv overlay evpn
feature ospf
feature bgp
feature pim
feature fabric forwarding
feature interface-vlan
feature vn-segment-vlan-based
feature nv overlay
fabric forwarding anycast-gateway-mac 0001.0001.0001
ip pim rp-address 10.0.1.23 group-list 224.0.0.0/4
ip pim ssm range 232.0.0.0/8
vlan 1,10,100
vlan 10
name VLAN10
vn-segment 10010
vlan 100
name VLAN100
vn-segment 100100
vrf context CUST1
vni 100100
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
vrf context management
interface Vlan1
interface Vlan10
no shutdown
vrf member CUST1
ip address 10.1.10.254/24
fabric forwarding mode anycast-gateway
interface Vlan100
no shutdown
vrf member CUST1
ip forward
interface nve1
no shutdown
host-reachability protocol bgp
source-interface loopback0
member vni 10010
mcast-group 224.1.1.1
member vni 100100 associate-vrf
interface Ethernet1/1
no switchport
ip address 10.3.4.4/24
ip ospf network point-to-point
ip router ospf 1 area 0.0.0.0
ip pim sparse-mode
no shutdown
interface Ethernet1/2
switchport access vlan 10
spanning-tree port type edge
interface loopback0
ip address 10.0.1.34/32
ip router ospf 1 area 0.0.0.0
ip pim sparse-mode
cli alias name wr copy run start
line console
line vty
boot nxos bootflash:/nxos.9.2.2.bin
router ospf 1
router bgp 65002
address-family l2vpn evpn
neighbor 10.0.1.23
remote-as 65002
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
vrf CUST1
address-family ipv4 unicast
network 10.1.10.0/24
evpn
vni 10010 l2
rd auto
route-target import auto
route-target export auto
Spine3
nv overlay evpn
feature ospf
feature bgp
feature pim
feature fabric forwarding
feature interface-vlan
feature vn-segment-vlan-based
feature nv overlay
fabric forwarding anycast-gateway-mac 0001.0001.0001
ip pim rp-address 10.0.1.23 group-list 224.0.0.0/4
ip pim ssm range 232.0.0.0/8
vlan 1,10
vlan 10
name VLAN10
vrf context management
interface Vlan1
interface Ethernet1/1
no switchport
ip address 10.2.3.3/24
ip ospf network point-to-point
ip router ospf 1 area 0.0.0.0
ip pim sparse-mode
no shutdown
interface Ethernet1/2
no switchport
ip address 10.3.3.23/24
ip ospf network point-to-point
ip router ospf 1 area 0.0.0.0
ip pim sparse-mode
no shutdown
interface Ethernet1/3
no switchport
ip address 10.3.4.23/24
ip ospf network point-to-point
ip router ospf 1 area 0.0.0.0
ip pim sparse-mode
no shutdown
interface loopback0
ip address 10.0.1.23/32
ip router ospf 1 area 0.0.0.0
ip pim sparse-mode
cli alias name wr copy run start
line console
line vty
boot nxos bootflash:/nxos.9.2.2.bin
router ospf 1
router bgp 65002
address-family l2vpn evpn
neighbor 10.0.1.13
remote-as 65002
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
route-reflector-client
neighbor 10.0.1.33
remote-as 65002
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
route-reflector-client
neighbor 10.0.1.34
remote-as 65002
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
route-reflector-client
BGW3
nv overlay evpn
feature ospf
feature bgp
feature pim
feature fabric forwarding
feature interface-vlan
feature vn-segment-vlan-based
feature nv overlay
fabric forwarding anycast-gateway-mac 0001.0001.0001
ip pim rp-address 10.0.1.23 group-list 224.0.0.0/4
ip pim ssm range 232.0.0.0/8
vlan 1,10,100
vlan 10
name VLAN10
vn-segment 10010
vlan 100
name VLAN100
vn-segment 100100
route-map RM_MULTISITE_ADDRESSING permit 10
match interface loopback100 Ethernet1/2 loopback0
vrf context CUST1
vni 100100
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
vrf context management
interface Vlan1
interface Vlan100
no shutdown
vrf member CUST1
ip forward
interface nve1
no shutdown
host-reachability protocol bgp
source-interface loopback0
multisite border-gateway interface loopback100
member vni 10010
multisite ingress-replication
ingress-replication protocol bgp
member vni 100100 associate-vrf
interface Ethernet1/1
no switchport
ip address 10.2.3.13/24
ip ospf network point-to-point
ip router ospf 1 area 0.0.0.0
ip pim sparse-mode
evpn multisite fabric-tracking
no shutdown
interface Ethernet1/2
no switchport
mtu 9216
ip address 103.0.0.3/24
evpn multisite dci-tracking
no shutdown
interface loopback0
ip address 10.0.1.13/32
ip router ospf 1 area 0.0.0.0
ip pim sparse-mode
interface loopback100
ip address 172.16.100.3/32
ip router ospf 1 area 0.0.0.0
cli alias name wr copy run start
line console
line vty
boot nxos bootflash:/nxos.9.2.2.bin
router ospf 1
router bgp 65002
address-family ipv4 unicast
redistribute direct route-map RM_MULTISITE_ADDRESSING
address-family l2vpn evpn
neighbor 10.0.1.23
remote-as 65002
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
neighbor 103.0.0.1
remote-as 1
address-family ipv4 unicast
neighbor 172.16.100.1
remote-as 65001
update-source loopback100
ebgp-multihop 5
peer-type fabric-external
address-family l2vpn evpn
send-community
send-community extended
rewrite-evpn-rt-asn
evpn
vni 10010 l2
rd auto
route-target import auto
route-target export auto
DC1
interface GigabitEthernet0/0
mtu 9216
ip address 101.0.0.1 255.255.255.0
duplex auto
speed auto
media-type rj45
!
!
interface GigabitEthernet0/2
mtu 9216
ip address 103.0.0.1 255.255.255.0
duplex auto
speed auto
media-type rj45
!
!
router bgp 1
bgp log-neighbor-changes
neighbor 101.0.0.11 remote-as 65001
neighbor 103.0.0.3 remote-as 65002
!
BGW1
nv overlay evpn
feature ospf
feature bgp
feature pim
feature fabric forwarding
feature interface-vlan
feature vn-segment-vlan-based
feature nv overlay
fabric forwarding anycast-gateway-mac 0001.0001.0001
ip pim rp-address 10.0.1.23 group-list 224.0.0.0/4
ip pim ssm range 232.0.0.0/8
vlan 1,10,100
vlan 10
name VLAN10
vn-segment 10010
vlan 100
name VLAN100
vn-segment 100100
route-map RM_MULTISITE_ADDRESSING permit 10
match interface loopback100 Ethernet1/2 loopback0
vrf context CUST1
vni 100100
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
vrf context management
interface Vlan1
interface Vlan100
no shutdown
vrf member CUST1
ip forward
interface nve1
no shutdown
host-reachability protocol bgp
source-interface loopback0
multisite border-gateway interface loopback100
member vni 10010
multisite ingress-replication
ingress-replication protocol bgp
member vni 100100 associate-vrf
interface Ethernet1/1
no switchport
ip address 10.1.1.1/24
ip ospf network point-to-point
ip router ospf 1 area 0.0.0.0
ip pim sparse-mode
evpn multisite fabric-tracking
no shutdown
interface Ethernet1/2
no switchport
mtu 9216
ip address 101.0.0.11/24
evpn multisite dci-tracking
no shutdown
interface loopback0
ip address 10.0.0.11/32
ip router ospf 1 area 0.0.0.0
ip pim sparse-mode
interface loopback100
ip address 172.16.100.1/32
ip router ospf 1 area 0.0.0.0
cli alias name wr copy run start
line console
line vty
boot nxos bootflash:/nxos.9.2.2.bin
router ospf 1
router bgp 65001
address-family ipv4 unicast
redistribute direct route-map RM_MULTISITE_ADDRESSING
address-family l2vpn evpn
neighbor 10.0.0.2
remote-as 65001
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
neighbor 101.0.0.1
remote-as 1
address-family ipv4 unicast
neighbor 172.16.100.3
remote-as 65002
update-source loopback100
ebgp-multihop 5
peer-type fabric-external
address-family l2vpn evpn
send-community
send-community extended
rewrite-evpn-rt-asn
evpn
vni 10010 l2
rd auto
route-target import auto
route-target export auto
Spine1
nv overlay evpn
feature ospf
feature bgp
feature pim
feature fabric forwarding
feature interface-vlan
feature vn-segment-vlan-based
feature nv overlay
fabric forwarding anycast-gateway-mac 0001.0001.0001
ip pim rp-address 10.0.0.2 group-list 224.0.0.0/4
ip pim ssm range 232.0.0.0/8
vlan 1,10
vlan 10
name VLAN10
vrf context management
interface Vlan1
interface Ethernet1/1
no switchport
ip address 10.1.2.2/24
ip ospf network point-to-point
ip router ospf 1 area 0.0.0.0
ip pim sparse-mode
no shutdown
interface Ethernet1/2
no switchport
ip address 10.2.3.2/24
ip ospf network point-to-point
ip router ospf 1 area 0.0.0.0
ip pim sparse-mode
no shutdown
interface Ethernet1/3
no switchport
ip address 10.2.5.2/24
ip ospf network point-to-point
ip router ospf 1 area 0.0.0.0
ip pim sparse-mode
no shutdown
interface Ethernet1/4
no switchport
ip address 10.2.6.2/24
ip ospf network point-to-point
ip router ospf 1 area 0.0.0.0
ip pim sparse-mode
no shutdown
interface Ethernet1/5
no switchport
ip address 10.1.1.2/24
ip ospf network point-to-point
ip router ospf 1 area 0.0.0.0
ip pim sparse-mode
no shutdown
interface loopback0
ip address 10.0.0.2/32
ip router ospf 1 area 0.0.0.0
ip pim sparse-mode
cli alias name wr copy run start
line console
line vty
boot nxos bootflash:/nxos.9.2.2.bin
router ospf 1
router bgp 65001
address-family l2vpn evpn
neighbor 10.0.0.1
remote-as 65001
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
route-reflector-client
neighbor 10.0.0.3
remote-as 65001
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
route-reflector-client
neighbor 10.0.0.5
remote-as 65001
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
route-reflector-client
neighbor 10.0.0.6
remote-as 65001
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
route-reflector-client
neighbor 10.0.0.11
remote-as 65001
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
route-reflector-client
