In-Band and Out-of-Band Management

In-Band and Out-of-Band Management

• Two Management Options

  • Out-of-Band (OOB): Uses a dedicated physical management port on the back of the device.

  • In-Band (INB): Uses a special EPG/BD/VRF inside the management tenant. You can configure various parameters for it.

• Default OOB EPG

  • The management tenant already has an OOB EPG by default.

  • You cannot modify this default OOB EPG except to configure its Provided OOB Contracts.

• Interfaces on the APIC

  • The OOB interface appears as oobmgmt in the output of the ifconfig command.

  • The In-Band interface appears as bond.x, where x is the VLAN ID used for the in-band EPG.

apic1# ifconfig oobmgmt

oobmgmt: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

inet 192.168.4.20 netmask 255.255.255.0 broadcast 192.168.4.255

inet6 fe80::7269:5aff:feca:2986 prefixlen 64 scopeid 0x20<link>

ether 70:69:5a:ca:29:86 txqueuelen 1000 (Ethernet)

RX packets 495815 bytes 852703636 (813.2 MiB)

RX errors 0 dropped 0 overruns 0 frame 0

TX packets 432927 bytes 110333594 (105.2 MiB)

TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

apic1# ifconfig bond0.300

bond0.300: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1496

inet 10.30.30.254 netmask 255.255.255.0 broadcast 10.30.30.255

inet6 fe80::25d:73ff:fec1:8d9e prefixlen 64 scopeid 0x20<link>

ether 00:5d:73:c1:8d:9e txqueuelen 1000 (Ethernet)

RX packets 545 bytes 25298 (24.7 KiB)

RX errors 0 dropped 0 overruns 0 frame 0

TX packets 6996 bytes 535314 (522.7 KiB)

TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

• Interfaces on Leaf Switches

  • The OOB interface appears as eth0 in the ifconfig command.

  • The In-Band interface is a dedicated SVI (Switch Virtual Interface).

  • You can see the In-Band interface by using:

    • ifconfig

    • show ip interface vrf mgmt:<vrf> (where <vrf> is the name of the In-Band VRF).

leaf101# show interface mgmt 0

mgmt0 is up

admin state is up,

Hardware: GigabitEthernet, address: 00fc.baa8.2760 (bia 00fc.baa8.2760)

Internet Address is 192.168.4.23/24

MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation ARPA, medium is broadcast

Port mode is routed

full-duplex, 1000 Mb/s

Beacon is turned off

Auto-Negotiation is turned on

Input flow-control is off, output flow-control is off

Auto-mdix is turned off

EtherType is 0x0000

30 seconds input rate 3664 bits/sec, 4 packets/sec

30 seconds output rate 4192 bits/sec, 4 packets/sec

Rx

14114 input packets 8580 unicast packets 5058 multicast packets

476 broadcast packets 2494768 bytes

Tx

9701 output packets 9686 unicast packets 8 multicast packets

7 broadcast packets 1648081 bytes

• In-Band on Leaf Switches (Secondary IP)

  • The command show ip interface vrf mgmt:<vrf> will display the in-band management BD (Bridge Domain) subnet IP as a secondary IP address.

  • This is the expected behavior.

leaf# show ip interface vrf mgmt:inb

IP Interface Status for VRF "mgmt:inb-vrf"

vlan16, Interface status: protocol-up/link-up/admin-up, iod: 4, mode: pervasive

IP address: 10.30.30.1, IP subnet: 10.30.30.0/24

secondary IP address: 10.30.30.3, IP subnet: 10.30.30.0/24

IP broadcast address: 255.255.255.255

IP primary address route-preference: 0, tag: 0

• Interfaces on Spine Switches

  • The In-Band management IP is added as a dedicated loopback interface in the mgmt:<vrf> VRF.

  • This is different from how leaf switches handle In-Band management.

spine201# show ip interface vrf mgmt:inb

IP Interface Status for VRF "mgmt:inb"

lo10, Interface status: protocol-up/link-up/admin-up, iod: 98, mode: pervasive

IP address: 10.30.30.12, IP subnet: 10.30.30.12/32

IP broadcast address: 255.255.255.255

IP primary address route-preference: 0, tag: 0

• APIC Connectivity Preference

  • Under System Settings, you can select In-Band or Out-of-Band as the preferred connectivity method for the APICs.

  • Only traffic sent from the APIC uses the selected management preference.

  • The APIC can still receive traffic on both In-Band and Out-of-Band interfaces if they are configured.

APIC Routing Table (Simplified)

• When OOB (Out-of-Band) is Selected:

  • The metric value for oobmgmt (OOB interface) is 16.

  • The metric value for bond0.300 (In-Band interface) is 32 (higher than 16).

  • Since lower metric values are preferred, the OOB interface (oobmgmt) is used for outgoing management traffic.

apic1# bash

admin@apic1:~> route -n

Kernel IP routing table

Destination Gateway Genmask Flags Metric Ref Use Iface

0.0.0.0 192.168.4.1 0.0.0.0 UG 16 0 0 oobmgmt

0.0.0.0 10.30.30.1 0.0.0.0 UG 32 0 0 bond0.300

• When In-Band is Selected:

  • The metric value for bond0.300 (In-Band interface) is 8.

  • The metric value for oobmgmt (OOB interface) is 16 (higher than 8).

  • Since lower metric values are preferred, the In-Band interface (bond0.300) is used for outgoing management traffic.

admin@apic1:~> route -n

Kernel IP routing table

Destination Gateway Genmask Flags Metric Ref Use Iface

0.0.0.0 10.30.30.1 0.0.0.0 UG 8 0 0 bond0.300

0.0.0.0 192.168.4.1 0.0.0.0 UG 16 0 0 oobmgmt

Note : The leaf and spine node management preferences are not affected by this setting.

Verification of Out-of-band Configuration

To verify out-of-band configuration, check four folders within a specific tenant named 'mgmt':

• Node Management Addresses.

Node Management Addresses can either be assigned statically or from a pool.

• Node Management EPGs.

The out-of-band EPG should be present under the Node Management EPGs folder.

• Out-of-band Contracts (located under Contracts).

The contracts that determine which management services the out-of-band EPG provides are unique contracts set up in the out-of-band contracts folder.

• External Network Instance Profiles.

Then, ensure that the External Management Network Instance Profile has been created and that the appropriate out-of-band contract is set up as the 'Consumed Out-Of-Band Contract'.

Network Connectivity (through OOB)

To verify network connectivity via the OOB, use ping to test the path

of the packet through the out-of-band network.

apic# ping 192.168.4.1

PING 192.168.4.1 (192.168.4.1) 56(84) bytes of data.

64 bytes from 192.168.4.1: icmp_seq=1 ttl=255 time=0.409 ms

64 bytes from 192.168.4.1: icmp_seq=2 ttl=255 time=0.393 ms

64 bytes from 192.168.4.1: icmp_seq=3 ttl=255 time=0.354 ms

apic# bash

admin@apic1:~> traceroute 10.55.0.16

traceroute to 10.55.0.16 (10.55.0.16), 30 hops max, 60 byte packets

1 192.168.4.1 (192.168.4.1) 0.368 ms 0.355 ms 0.396 ms

2 *

3 *

4 10.0.255.221 (10.0.255.221) 6.419 ms 10.0.255.225 (10.0.255.225) 6.447 ms *

5 *

6 *

7 10.55.0.16 (10.55.0.16) 8.652 ms 8.676 ms 8.694 ms

Mgmt Tenant

Make sure the management tenant includes a BD with a subnet for assigning in-band node management addresses to the fabric nodes for in-band connectivity, and verify that the L3Out is associated with the in-band management BD.

Like out-of-band, fabric node in-band management IP addresses can be assigned either statically or dynamically.

Note :

The spine nodes won't reply to ping on the in-band because they utilize loopback

interfaces for connectivity, which do not respond to ARP.

-------------------------------------------------------------------------------------------------------------------------------

ACI OOB Management Architecture

In Cisco ACI, Out-of-Band management traffic is controlled using the mgmt tenant.

Object	Purpose
External Management Network Instance Profile	Defines external management source networks
External Management EPG	Represents external management hosts
Node Management EPG	Represents APIC / leaf / spine management interfaces
Contract	Defines allowed management protocols

Tenants

→ mgmt

→ External Management Network Instance Profiles

→ MGMT-EXT-INSTANCES

→ Subnets [These are usually management subnet]

These subnets define Which external networks are allowed to access the ACI OOB management interfaces.

Traffic Flow for OOB Access

External Management Host

│

│ (Source subnet defined in MGMT-EXT-INSTANCES)

▼

External Management EPG

│

│ Contract

▼

Node Management EPG

│

▼

ACI Nodes (APIC / Leaf / Spine)

Note : If no subnet is configured, ACI cannot match the source IP to the External Management EPG.

So the traffic classification looks like this:

External Host

│

▼

ACI OOB Interface

│

▼

Source not matched to External Mgmt EPG

Since the External Management EPG is not matched, the contract between:

External Mgmt EPG ↔ Node Mgmt EPG will not apply.

Result: External hosts blocked

Will OOB Subnet Access May Still Work ?

If a host exists inside the same OOB management subnet, the traffic may succeed because:

Host → same L2 OOB network → node

In that case, traffic may bypass the External Mgmt EPG classification.