L3-Out : Normal vs Floating

Step-by-Step Configuration Guide

Normal L3Out

A Normal L3Out binds Layer 3 connectivity to specific leaf nodes and specific physical interfaces. Routing adjacency exists only on those defined nodes.

Prerequisites

• Create a VLAN Pool (Static or Dynamic depending on use case)

• Create an L3 Domain

• Associate VLAN Pool with the L3 Domain

• Create a Physical Domain

• Associate Physical Domain to the AEP

• Bind AEP to leaf interfaces connected to the external device (e.g., F5)

Step-by-Step (GUI)

Step 1 — Create L3Out

Navigate:Tenants → → Networking → L3Outs → Create L3Out

Identity

• Name: e.g., Normal_L3Out

• VRF: Select appropriate VRF

• L3 Domain: Select created L3 domain

• Routing Protocol: BGP / OSPF / Static

Step 2 — Create Logical Node Profile

Inside the L3Out:

• Logical Node Profiles → Create

• Add specific leaf nodes (e.g., 101, 102)

• Configure:

  • Router ID

  • Loopback (if required for BGP)

These nodes become your Border Leafs.

Step 3 — Create Logical Interface Profile (Critical Step)

Inside L3Out:

• Logical Interface Profiles → Create

• Select Interface Type: SVI (or Routed Interface)

Now explicitly bind the path:

• Path: Select specific interface (e.g., eth1/10) or vPC

• Encapsulation: e.g., vlan-205

• IP Address: e.g., 192.168.1.1/24

• Configure routing protocol neighbor parameters

This explicit path binding is what makes it a "Normal" L3Out.

Control-plane adjacency will exist only on these defined leaf nodes.

Step 4 — Configure Routing Protocol

Under the protocol tab:

• For BGP:

  • Peer IP

  • Remote ASN

  • BFD (optional)

• For OSPF:

  • Area

  • Interface policy

Step 5 — Create External EPG

Inside L3Out:

• External EPG → Create

• Add external subnets (e.g., 0.0.0.0/0)

• Attach Contracts (Provide/Consume)

Behavioural Summary (Normal L3Out)

• Control plane adjacency only on defined border leaves

• Traffic must reach those leaves

• No dynamic instantiation on other leaves

• Ideal for fixed physical router connections

Floating L3Out

A Floating L3Out allows Layer 3 connectivity to be defined at the domain level instead of per-interface path binding.

Routing adjacency is formed only on Anchor Nodes, but the L3Out can dynamically instantiate across the fabric.

This is commonly used for:

• Virtual routers (F5 VE, CSR, Fw)

• VM mobility

• Service insertion designs

Prerequisites

• VLAN Pool must use Static Allocation

• L3 Domain must reference that VLAN Pool

• Physical Domain must reference same VLAN Pool

• AEP must include required domain

• Leaf switches must support Floating L3Out (Gen2+ platforms such as EX/FX/FX2/GX — verify per release)

Important Architectural Clarification

Floating L3Out does NOT eliminate binding.

Instead of:

Binding to specific leaf interface paths

You bind:

At the Domain level, allowing dynamic SVI instantiation on leaves where endpoints exist.

Step-by-Step

Step 1 — Create L3Out

Navigate:Tenants → Networking → L3Outs → Create L3Out

Identity

• Name: Floating_L3Out

• VRF: Select VRF

• L3 Domain: Select L3 Domain

• Routing Protocol: BGP / OSPF

Step 2 — Configure Floating SVI

Under Nodes and Interfaces:

• Interface Type: Select Floating SVI

• Domain Type: Physical

• Domain: Select Physical Domain

• Encapsulation: vlan-205

• Floating Address: e.g., 192.168.1.250/24

The Floating Address represents the SVI IP used by non-anchor leaves for consistent routing behavior.

Step 3 — Define Anchor Nodes

Add specific leaf nodes that will establish routing adjacency:

Example:

• Node 101

  • Router ID

  • Primary IP: 192.168.1.251/24

• Node 102

  • Router ID

  • Primary IP: 192.168.1.252/24

These nodes:

• Form BGP/OSPF adjacency

• Advertise routes

• Act as control-plane endpoints

Step 4 — Configure Routing Protocol

• For BGP:

  • Peer IP (F5 Self-IP)

  • Remote ASN

• For OSPF:

  • Area configuration

Only Anchor nodes establish adjacency.

Step 5 — Create External EPG

Same as Normal L3Out:

• Define external subnets

• Attach contracts

Behavioural Summary (Floating L3Out)

Control Plane:

• Adjacency only on Anchor nodes

Data Plane:

• Traffic from any leaf can be forwarded internally to anchor nodes

• Non-anchor nodes do not establish routing adjacency

Key difference:

Normal L3Out → Control & Data localised to border leavesFloating L3Out → Control on anchors, Data flexible across fabric

Converting Normal L3Out to Floating L3Out

Technically possible, but requires structural modification.

For production environments, it is recommended to:

Delete and recreate L3Out instead of modifying in-place

Reason:

• Changing interface type (SVI → Floating SVI)

• Can cause routing adjacency flap

• May recreate internal BD objects

• Can temporarily withdraw routes

If Conversion Is Required (Lab / Controlled Environment)

Step 1 — Validate VLAN Pool

Ensure:

• VLAN Pool uses Static allocation

• L3 Domain and Physical Domain are aligned

Step 2 — Remove Explicit Path Bindings

Navigate:L3Out → Logical Node Profile → Logical Interface Profile

Under SVI/Sub-interface:

• Remove explicit interface paths (e.g., eth1/10 bindings)

Step 3 — Create Floating SVI

Under same Logical Interface Profile:

• Select Floating SVI

• Enter:

  • VLAN

  • Floating Address

  • Domain association

• Add Anchor nodes

• Reconfigure Primary IP per node

Step 4 — Validate Routing

Ensure:

• BGP/OSPF adjacency forms on Anchor nodes

• Routes are advertised correctly

• Traffic forwarding behaves as expected

Comparison

Feature	Normal L3Out	Floating L3Out
Interface Binding	Per-leaf physical path	Domain-based
Routing Adjacency	On defined leafs only	On Anchor nodes only
Data Plane Flexibility	Limited	Fabric-wide
Best Use Case	Fixed border routers	Virtual / mobile routers
Floating IP Required	No	Yes

Floating L3Out Version Support

• Floating L3Out Introduction:

  • Started in ACI Version 4.2(1): The floating L3Out feature became available starting with Cisco ACI version 4.2(1) for VMware vSphere Distributed Switch (VDS) in Virtual Machine Manager (VMM) domains.

    
    

  • Expanded in ACI Version 5.0(1): From version 5.0(1), the support for floating L3Out was extended to physical domains. This allows using floating L3Out with virtual routers without needing VMM domain integration, or with physical routers without complex interface path configurations.

    
    

• Configuration Simplification:

  
  

  • Previous Setup Requirements: Before version 4.2(1), when connecting to an external virtual router, you had to configure the L3Out logical interface path from each border leaf switch to the hypervisor's uplink where the virtual device was running. If the virtual machines moved to different hosts within a hypervisor cluster, you had to manually configure the L3Out on all potential leaf switches where the virtual machine could run.

    
    

  • Problem Example: For instance, if a hypervisor cluster has 12 leaf switches, you had to create L3Out configurations for every leaf switch to ensure connectivity as virtual machines moved across the cluster.

    
    

• Floating L3Out Benefits:

  
  

  • Simplified Configuration: With floating L3Out, the process is much easier. After setting it up, you no longer need to manually configure the L3Out logical interfaces on all the leaf switches in the cluster.

    
    

  • Automatic Handling: This feature automatically manages the connectivity, even as virtual machines move between different leaf switches, removing the need for complex and time-consuming manual configurations.

    
    

Example  to use the Floating Layer 3 Outside Network Connection (Floating L3Out) feature in Cisco ACI.

The example includes a Virtual Machine Manager (VMM) domain and a Virtual Port Channel (vPC) setup. However, Floating L3Out also works with physical domains, and using vPC is optional.

Key Components

1. Virtual Routers:

   • Can be a regular router, a virtual firewall, or any virtual device.

   • Acts as the next hop for static routes in the Cisco ACI fabric.

   • Establishes a routing connection with the ACI fabric.

     
     

2. Anchor Leaf Nodes:

   • Example Setup: Two leaf switches named Leaf1 and Leaf2.

   • Role:

     • Connect directly with external routers.

     • Establish Layer 3 (L3) routing connections with these external routers.

   • Scalability:

     • Up to 6 anchor leaf nodes are supported per L3Out in Cisco ACI release 6.0(1).

   • IP Addresses:

     • Use primary and floating IP addresses.

     • Can also have secondary and floating secondary IP addresses if needed.

       
       

3. Non-Anchor Leaf Nodes:

   • Example Setup: Two leaf switches named Leaf3 and Leaf4.

   • Role:

     • Do not connect directly with external routers.

     • Act as "pass-through" nodes for traffic between external routers and anchor leaf nodes.

   • Scalability:

     • Up to 32 non-anchor leaf nodes are supported per L3Out in Cisco ACI release 6.0(1).

   • IP Addresses:

     • Use a common floating IP address.

     • Can have floating secondary IP addresses if needed.

     • In VMM domains with VMware vDS:

       • Floating IP is used only when the virtual router is connected to the leaf node.

     • In physical domains:

       • Floating IP is used if the leaf port uses an Attachment Endpoint Profile (AEP) associated with the Floating L3Out.

How Floating L3Out Works

1. Creating L3Out Bridge Domain:

   • When you set up an L3Out, an L3Out bridge domain is created on the anchor leaf nodes.

   • This bridge domain is often called the “L3Out’s SVI subnet.”

     
     

2. Floating L3Out with VMM Domain:

   • If a virtual router moves to a host connected to a non-anchor leaf switch:

     • Cisco APIC automatically deploys the L3Out bridge domain on the non-anchor leaf switch.

     • Installs the floating IP address (and floating secondary IPs if needed) on the non-anchor leaf switch.

   • If an external Endpoint Group (EPG) under the L3Out has a contract with another EPG:

     • Routes to the EPG and policy rules for the contract are also installed on the non-anchor leaf switch.

       
       

3. Maintaining Routing Adjacencies:

   • Even if the virtual router moves, it maintains routing connections with the SVI interfaces on the anchor leaf nodes.

   • This is possible because ACI extends connectivity for the L3Out bridge domain across both anchor and non-anchor leaf nodes.

Let's Understand

Behavior:

• Only Anchor nodes should be physically connected to the F5 (or external router).

• Non-anchor nodes typically do NOT have physical L3 adjacency to the F5.

• Even if a non-anchor leaf has the VLAN present, it does NOT form routing adjacency.

Floating L3Out design assumption:

• Routing adjacency exists only on anchor nodes.

• Traffic from non-anchor nodes is forwarded internally to anchor nodes.

Who are Non-Anchor nodes ?

Every other leaf in the fabric:

• That belongs to the same VRF

• That has the relevant domain association

• That is not selected as Anchor

You do not:

• Mark them as non-anchor

• Configure them as non-anchor

• Apply any special policy

They are simply “not anchor”

Let’s say:

• Node 101 & 102 = Anchor

• Node 103, 104, 105 = Not defined

Node	Role
101	Anchor
102	Anchor
103	Non-anchor (automatic)
104	Non-anchor (automatic)
105	Non-anchor (automatic)

No configuration needed for 103/104/105.

If a VM connected to Node 104 sends traffic toward external:

• Node 104 forwards to Anchor node via fabric

• Anchor node routes externally

How Does APIC Know Which Leafs Get the Floating SVI?

Floating L3Out works through:

• Domain association

• Static VLAN allocation

• Endpoint presence

APIC dynamically instantiates the SVI on leafs where needed.

But:

• Only anchor nodes run routing protocol

• Non-anchor nodes do not run routing protocol

• Floating IP exists logically across nodes for consistent forwarding

Floating L3Out design

You want a controlled topology:

• Leaf1 & Leaf2 → Anchor nodes

• Leaf3 & Leaf4 → Non-anchor nodes

• Leaf5 & Leaf6 → Completely excluded

First: Important Architectural Reality

In Floating L3Out:

• You explicitly configure Anchor nodes

• All other leaves in the same VRF + Domain scope become eligible non-anchor

• Leaves are “excluded” only if:

  • They do NOT belong to the same domain/AEP

  • OR they do NOT participate in the VRF

  • OR the Floating VLAN is not allowed to instantiate there

So we must design carefully.

Desired Fabric Topology

Leaf	Role
Leaf1	Anchor
Leaf2	Anchor
Leaf3	Non-anchor
Leaf4	Non-anchor
Leaf5	Fully excluded
Leaf6	Fully excluded

Design Strategy

We control behavior using:

1️⃣ VRF association

2️⃣ Domain association

3️⃣ VLAN pool scope

4️⃣ AEP bindings

Floating L3Out only instantiates where:

• The leaf participates in the VRF

• The domain/AEP relationship exists

• The VLAN can be allocated

So exclusion = remove one of these.

Step-by-Step Deployment

STEP 1 — Create Dedicated L3 Domain & VLAN Pool

This isolates your Floating L3Out.

Create VLAN Pool

• Static allocation only

• Example: VLAN 205

Create L3 Domain

• Associate VLAN Pool

Create Physical Domain

• Associate same VLAN Pool

Do NOT reuse a domain used by Leaf5/6 workloads if you want them excluded.

STEP 2 — Control AEP Scope

Create a new AEP:

Example:AEP-Floating-L3Out

Associate:

• Physical Domain (created above)

Now bind this AEP ONLY to:

• Leaf1 uplink ports

• Leaf2 uplink ports

• Any ports on Leaf3/4 where endpoints needing L3Out exist

Do NOT bind this AEP to Leaf5 or Leaf6 ports.

STEP 3 — VRF Placement Control

Ensure:

• Bridge Domains and EPGs using this L3Out are only deployed on Leaf1–4.

Endpoints connected to Leaf5/6 should not belong to:

• The same tenant/VRF

  OR

• Should not use EPGs associated with this L3Out

If Leaf5/6 do not host endpoints in that VRF → Floating SVI will not instantiate.

STEP 4 — Create Floating L3Out

Now configure the Floating L3Out.

Navigate:Tenants → Networking → L3Out → Create

Identity

• VRF = your controlled VRF

• L3 Domain = your dedicated L3 domain

• Protocol = BGP/OSPF

STEP 5 — Configure Floating SVI

Under Nodes & Interfaces:

• Interface Type → Floating SVI

• Domain Type → Physical

• Domain → your new Physical Domain

• Encap → vlan-205

• Floating Address → e.g., 192.168.1.250/24

STEP 6 — Define Anchor Nodes

Add:

• Leaf1 → Primary IP 192.168.1.251/24

• Leaf2 → Primary IP 192.168.1.252/24

These will form routing adjacency.

Leaf3/4 are not configured here → automatically non-anchor.

Leaf5/6 also not configured — but exclusion depends on domain/VRF scope.

STEP 7 — Protocol Configuration

Configure BGP/OSPF peer:

• Peer IP = F5 self-IP

• Remote ASN

Only Leaf1/2 will form adjacency.

What Happens Now?

Leaf1 & Leaf2

• Have Floating SVI

• Run routing protocol

• Peer with F5

Leaf3 & Leaf4

• Floating SVI may instantiate

• Do NOT run routing protocol

• Forward traffic internally to Anchor nodes

Leaf5 & Leaf6

If:

• They have no endpoints in this VRF

• Their ports do not use the AEP with this domain

Then:

• No Floating SVI instantiated

• No participation in this L3Out

• Fully excluded

Expected:

Leaf	BGP Neighbor?	SVI Present?
1	Yes	Yes
2	Yes	Yes
3	No	Possibly (if endpoints present)
4	No	Possibly
5	No	No
6	No	No

Note : If you reuse an existing Physical Domain used across all leaves:

Then VLAN 205 may instantiate everywhere → Leaf5/6 will not be excluded.

Domain scoping is critical.