Packet Forwarding Between Leaf Switches
A) Source leaf knows the destination—destination is on the same leaf.
If the destination MAC address is known to an ingress leaf and it is on the same leaf, the packet is forwarded to the local port, like traditional Layer 2 or Layer 3 forwarding.
B) Source leaf knows the destination—destination is on a different leaf.
If the destination MAC address is known to an ingress leaf and it is not on the local leaf, the packet is forwarded directly to the destination leaf.
C) Source leaf does not know the destination—decides to use Spine-Proxy mode.
(1) The packet is a unicast Layer 3 routed packet, and the destination IP (unknown to the ingress leaf) belongs to one of the BD subnets on the ingress leaf.
(2) The packet is a unicast Layer 2 switched packet, and the BD Layer 2 Unknown Unicast mode is set to hardware-proxy.
(3) The packet is a broadcast ARP request, the BD ARP Flooding is disabled, and the ARP target IP (unknown to the ingress leaf) belongs to one of the BD subnets on the ingress leaf.
D) Source leaf does not know the destination—decides to use Flood mode.
(1) The packet is a unicast Layer 2 switched packet, and the BD Layer 2 Unknown Unicast mode is set to flood.
(2) The packet is a broadcast ARP request, and the BD ARP Flooding is enabled.
Endpoint : It consists of one MAC address and zero or more IP addresses.
In a traditional network, three tables are used to maintain the network addresses of external devices:
A MAC address table for Layer 2 Forwarding
A Routing Information Base (RIB) for Layer 3 forwarding
An Address Resolution Protocol (ARP) table for the association of IP addresses and MAC addresses
If leaf needs to send packet from Host A and Host B
Both A & B are connected to same leaf
Same vlan and subnet… locally route packet
Different vlan or subnet … use anycast Gateway to send packet A to B
Both A & B connected to remote leaf
Same Bridge Domain → Encapsulate packet with Vxlan of BD VNID
Different Bridge Domain → Encapsulate packet with Vxlan of VRF VNID
Vxlan has vrf or BD VNID
Cisco ACI replaced the MAC address table and ARP table with a single table called the endpoint table.
Types of VLANs
Access Encap VLAN ID
It is used to communicate with external devices. This is a VLAN ID on the ACI and on the external device—a VLAN on the wire.
Access Encap VLAN ID is internally mapped to platform independent VLAN ID
Platform-Independent (PI) VLAN ID
It is used internally on a particular leaf i.e. not shared with other leafs. PI VLAN ID can map the internal ID to an access encap VLAN or VXLAN ID, or bridge domain switch virtual interface (SVI) ID. You can use the PI VLAN ID to get information about the EPG.
Note : Even though you can have the same access encap VLAN IDs on another leaf, PI VLAN IDs may be different.
VXLAN ID
It is to make forwarding across the entire fabric, you need to have an identifier that is shared among all the nodes. This identifier is VNID.
Mainly this ID is either used as a bridge domain VNID for bridging the traffic or as a VRF VNID for routing the traffic.
If you want to see endpoints learned in specific EPG then simply go to the path
Tenant -> Application Profile → EGP
Check number of endpoints present in particular VRF in leaf
Leaf 1 # Show vrf
# show endpoint vrf Tenant:VRF
Types of Endpoints
Physical local endpoint (PL)
This endpoint is learned and physically attached to the local leaf.
2) Virtual local endpoint (VL)
This endpoint is still local to the leaf, but it is behind a virtual switch (Cisco Application Virtual Switch [AVS] or Cisco ACI Virtual Edge [AVE]).
Leaf# show endpoint ip 192.168.66.2
3) Remote endpoint (XR)
It represents an endpoint on another leaf. Remote endpoint does not have the L flag.
Leaf# show endpoint mac 0000.5555.2222
Leaf# show endpoint IP 192.168.0.53
4) On-peer endpoint (O)
This endpoint is on another leaf but that leaf is a vPC peer of your local leaf switch and the endpoint is connected to an orphan port on that vPC peer. This endpoint type has a specific O flag that is shown in the show endpoint command output.
COOP DATABASE
To get VRF & BD Vnid
Leaf # show system internal epm endpoint ip 10.0.1.1
Spine# show coop internal info repo ep key “Bridge Domain VNID” “EP Mac” | egrep ‘vnid|mac|id|Real’
Spine# show coop internal info repo ep key 15826915 0000.5555.1111 | egrep ‘vnid|mac|id|Real’
Endpoint Database on GUI ---> Inventory --> Spine switch --> Protocols --> COOP --> Endpoint Database.
NIC Teaming to ACI Fabric
1) Active/Standby NIC Teaming
With active/standby NIC Teaming, one interface is active and one or more is in a standby state.
Two different implementations of the failover process :
A) The MAC address of the active interface stays identical after a failover.
The bridge domain configuration does not require any change if the newly active interface starts sending traffic immediately after the failover as the endpoint information (MAC, IP and its combination) didn’t change.
The endpoint to TEP mapping in the COOP database on spine switches is also updated accordingly just like a normal endpoint move.
B) When a failover happens, the newly active interface uses its own MAC address to send traffic.
The IP-to-MAC mapping must be updated on all the servers in the same Layer 2 domain.
Therefore, with this type of implementation, the server sends a GARP/RARP request after a failover.
Here, the bridge domain must be configured for ARP flooding for the GARP request to reach the servers in the bridge domain.
Ip data plane learning must be set to off (by default enable)
2) Active/Active NIC Teaming
Servers configured with NIC Teaming active/active, such as Transmit Load Balancing, send the same source IP from multiple NIC cards with different MAC addresses, causing IP flapping between those two MAC addresses on ACI endpoint learning due to the IP data plane learning.
With these types of servers, the best connectivity option is to change the teaming to one of the following options:
Active/standby NIC Teaming
Port Channel for active/active scenario
If the teaming configuration cannot be changed, you could disable the IP data plane learning in the VRF configuration.
With the IP data plane learning disabled, the Endpoint table is updated by ARP i.e. MAC-IP mapping.
Endpoint learning optimization options.
Through Bridge Domain → Limit IP Learning to Subnet
If the limit IP learning to subnet option is enabled, the local endpoint IP learning will be limited to only IP addresses for subnets configured on the bridge domain.
Note : This option does not limit remote endpoint IP learning & is enabled by default.
If a bridge domain is configured with a subnet address of 192.168.1.254/24, the fabric does not learn a local endpoint IP address, such as 192.168.2.1/24, that is outside this range. This behaviour prevents unnecessary IP learning.
Note: This feature prevents local IP learning of hosts whose subnet aren’t configured in Bridge Domain, yet the local leaf still learns the MAC address, and the remote leaf still learns both the IP and MAC addresses. It’s important to keep in mind that although the local leaf does not learn the IP address, it does not drop the packet .
2) At VRF level but for all Tenants —> Through Global enforce subnet check
Enforce subnet check feature ensures that Cisco ACI learns endpoints whose IP addresses belong to the bridge domain subnet.
This feature also ensures that leaf switches learn remote entries whose IP addresses belong to the VRF that they are associated with, preventing the learning of IP addresses that are not configured as subnets on the bridge domains of the VRF.
You can enable the enforce subnet check under System Settings > Fabric Wide Settings.
It limits learning of both the MAC address and IP address when IP learning is triggered.
This behavior prevents IP spoofing scenarios, in which an endpoint sends a packet with an unexpected source IP address that does not belong to any of the bridge domains on the VRF instance, such as an IP address that exists behind the Layer 3 Outside connection.
Note: The entries in the COOP database are not cleared.
At VRF Level → IP Data-Plane Learning (on Specific VRF only)
The IP Data-plane Learning option is located at Tenant > Networking > VRFs.
Select any VRF
This option is enabled by default.
When the IP Data-plane learning option under VRF is disabled, endpoint learning behavior on an ACI leaf changes as follows:
Local MACs and remote MACs are learned via the data plane (no change with this option).
Local IPs are not learned via the data plane.
Local IPs are learned from ARP/GARP/Neighbor Discovery (ND) via the control plane.
Remote IPs are not learned from unicast packets via the data plane.
Remote IPs are learned from multicast packets via the data plane.
When the IP Data-plane Learning option is disabled, existing remote IP endpoints are flushed immediately.
Existing local IP endpoints are not flushed either, but they will age out eventually unless control plane packets such as ARP keep them alive.
The IP Data-plane Learning option, under the VRF, must be disabled if
1) If there is a possibility that the ACI fabric receives traffic with the same-source IP address from different locations, which causes endpoint IP and MAC binding updates that occur due to data plane traffic.
For example, mainframe virtual IP address (VIPA) connectivity and dynamic load-balancing mode for NIC Teaming on Microsoft Windows behave in this manner.
2) Another use case is when multiple devices share the same IP such as virtual IP (VIP).
When F5 failover happens, it could result in the ACI fabric learning the VIP from multiple places via the data plane. This issue can be avoided by disabling IP Data-plane Learning.
Endpoint Loop Protection
The endpoint loop protection is a feature that is configured at the global level.
The feature is effective on all bridge domains.
The endpoint loop protection feature can be enabled by choosing
System > System Settings > Endpoint Controls > Ep Loop Protection (nearby Rougue EP Control)
Endpoint loop protection acts if the Cisco ACI fabric detects an endpoint mac address flapping between two interfaces more than a specified number of times during a given time interval.
By default it’s disabled.
You can specify:
Loop Detection Interval: Specifies the time to detect a loop. The default setting is 60 seconds.
Loop Detection Multiplication Factor: Sets the number of times a single endpoint moves between ports within the loop detection interval. The default is 4.
Action: When too many moves are detected within a loop detection interval you can choose whether Cisco ACI should disable one of the ports to which the endpoint is connected (Port Disable option) or disable learning within the bridge domain (BD Learn Disable option).
If a single endpoint is moving too often (which most likely is not a loop but maybe an incorrect NIC-teaming configuration) whereas multiple endpoints are moving too often (as happens with loops).
Rogue Endpoint Control
Rogue endpoint control is a feature that can help in case there are MAC or IP addresses that are moving too often between ports.
With rogue endpoint control, only the misbehaving endpoint (MAC/IP) is quarantined, which means that Cisco ACI keeps its TEP and port that is fixed for a certain amount of time when learning is disabled for this endpoint.
The feature also raises a fault to allow easy identification of the problematic endpoint.
Rogue endpoint detection interval: Specifies the time to detect rogue endpoints. The default is 60 seconds.
Rogue endpoint detection multiplicator factor: If the endpoint moves more times than this number, within the endpoint detection interval, the endpoint is declared rogue. The default is 4.
Hold interval: Defines how long after the endpoint is declared rogue, the endpoint is kept static, so learning is prevented and the traffic to and from the rogue endpoint is dropped. After this interval, the endpoint is deleted. The default is 1800.
Note: Rogue endpoint control does not stop a Layer 2 loop, but it provides mitigation of the impact of a loop on the COOP control plane by quarantining the endpoints.
Inventory → POD → switch → Right Click → clear Rougue endpoint
Check if Rougue Endpoint Entry is present or not.
Leaf # Show system internal epm endpoint all
Rogue endpoint control also helps if there are incorrect configurations on servers (such as incorrect NIC Teaming configuration), which may cause endpoint flapping.
In such a case, Cisco ACI, instead of disabling the server ports, as endpoint loop detection may do, stops the learning for the endpoint that is moving too often, and it provides a fault with the IP of the endpoint that is moving too often so that the administrator can verify its configuration.
It is recommended to enable rogue endpoint control (the feature is disabled by default).
Clearing The ENDPOINT
APIC1 # clear endpoints leaf 101 tenant T1 vrf V1
To verify if endpoint got deleted or not ?
APIC# fabric 101 show endpoint
Clearing the endpoint on chip level
leaf1# vsh -c "clear system internal epm endpoint key vrf T1:V1 ip 1.1.1.1"
Once the endpoints have been cleared from the leaves, the endpoints will then be cleared via COOP on the spines.
Apic# fabric 201 show coop internal info ip-db | grep -A10 10.1.1.1 <-- Here node 201 is spine switch
Spine # show coop internal info repo ap --> all endpoint in fabric
Comments