Syslog & Backup in ACI

Step 1: Let’s use one of the leafs with ip 10.197.205.38 as remote server (Practically it would be server in your network)

Step 2: We are going to use directory /var/log/Test

Step 3: Now go to Admin Import/Export Export Policies Remote location

Create New Location Leaf1

Step 4 : Now go to Admin Import/Export Export Policies Configuration

Step 5 : Go to Leaf and verify.

leaf1# pwd

/var/log/Test

leaf1# ls

ce2_Leaf1-Export-Policy-2023-11-13T09-00-27.tar.gz

Sending Backup to Snapshot Folder on APIC controller.

Now go to directory “/data2/snapshots” on APIC

apic1# pwd

/data2/snapshots

apic1# ls

ce2_Backup-Snapshot-2023-11-13T09-12-08.tar.gz

Now go to AdminConfig Rollbacks Take snapshot with location as “APIC”

apic1# pwd

/data2/snapshots

apic1# ls -l

total 216

-rw-r--r-- 1 ifc admin 109929 Nov 13 09:12 ce2_Backup-Snapshot-2023-11-13T09-12-08.tar.gz

-rw-r--r-- 1 ifc admin 109771 Nov 13 09:15 ce2_defaultOneTime-2023-11-13T09-15-34.tar.gz

Syslog

Local file in APIC containing Syslog message is /var/log/external/messages

Syslog Configuration

Admin --> External Data Collectors --> Monitoring Destinations ---> Syslog --> Create Syslog Monitoring Destination Group

Syslog Message Policies

Fabric --> Fabric Policies --> Monitoring --> Common Policy --> Syslog Message Policies ----> we can delete or modify severity levels of policies.

-----------------------------------------------------------------------------------------------------------------------

Cisco ACI Syslog Architecture

In Cisco ACI, syslog messages can originate from multiple components within the fabric. The architecture is designed so that individual fabric nodes generate and send syslog messages directly to the configured external servers.

Key Components

1. APIC (Application Policy Infrastructure Controller)APIC manages the configuration of syslog destinations and monitoring policies.

2. Fabric Nodes (Leaf and Spine Switches)Each node generates system logs and sends them directly to the external syslog server.

3. Monitoring PoliciesMonitoring policies determine which events are exported and from which objects.

4. Syslog Destination GroupsDestination groups define the external syslog servers that will receive the messages.

Simplified Flow

ACI Node (Leaf/Spine)

        |

Monitoring Policy

        |

Syslog Source

        |

Destination Group

        |

External Syslog Server

Important point:

Syslog messages are generated by the node itself and transmitted directly to the syslog server. APIC does not proxy or relay these messages.

How Syslog Transport Works in ACI

Cisco ACI supports two common transport protocols:

UDP Syslog

• Default syslog protocol

• Uses port 514

• Connectionless

• Lower overhead

TCP Syslog

• Reliable transport

• Used in many enterprise logging environments

• Uses configurable ports (for example TCP 9003)

Important Behaviour of TCP Syslog

When TCP is used, the process works as follows:

1. The ACI node establishes a TCP session with the syslog server

2. Syslog messages are transmitted over this persistent session

3. The session remains open until:

   • The server closes it

   • The node restarts the syslog process

   • A network interruption occurs

A new TCP connection is not created for each syslog message. Instead, messages are streamed over the existing session.

Configuring Syslog in Cisco ACI

Syslog configuration in ACI consists of three main steps.

Step 1 – Configure a Syslog Destination

Navigate to:

Admin → External Data Collectors → Syslog

Create a new remote destination.

Example configuration:

Name: Syslog_Server_1

Host: 10.10.10.50

Transport: TCP

Port: 9003

Severity: warnings

Management EPG: default (Out-of-Band)

Key fields explained:

TransportDefines whether syslog uses TCP or UDP.

PortMust match the listening port on the syslog server.

Management EPGDefines whether syslog traffic is sent using:

• Out-of-band management network

• In-band management network

Most deployments use Out-of-Band (OOB).

Step 2 – Create a Destination Group

Destination groups allow administrators to group multiple syslog servers together.

Example:

Destination Group: SYSLOG_GROUP

Members:

  - Syslog_Server_1

  - Syslog_Server_2

Best practice:

In some environments, it is preferable to create one destination per group to ensure deterministic delivery and simplify troubleshooting.

Step 3 – Configure Monitoring Policy

Monitoring policies define which events should be exported.

Example monitoring policy settings include:

• Fault severity

• Audit events

• System events

• Endpoint events

These policies can be applied to:

• Fabric

• Nodes

• Tenants

• EPGs

• Contracts

The monitoring policy ultimately determines which syslog messages are generated and exported.

Verifying Syslog Configuration

After configuration, it is important to verify that the syslog configuration has been correctly applied to fabric nodes.

One useful command is:

moquery -c syslogDestState

This command shows the operational state of syslog destinations on nodes.

Example fields to check:

host

port

protocol

operSt

If the destination is active, the operational state should be up.

Generating a Test Syslog Message

Cisco ACI provides a very useful command for generating test syslog events.

logit severity critical dest-grp SYSLOG_GROUP "Test message"

This command creates a synthetic syslog message that is immediately sent to the configured destination.

It can also be targeted to a specific node:

logit severity critical dest-grp SYSLOG_GROUP "Test message" node 101

This is extremely helpful when validating node-specific logging behaviour.

Packet-Level Verification

If logs are not reaching the syslog server, packet captures can be used to verify whether traffic is leaving the node.

Engineers can SSH directly to a leaf switch and run:

tcpdump -i any host <syslog-server-ip>

Example output:

IP 10.10.1.25 > 10.10.10.50.syslog: SYSLOG local4.critical

This confirms that the node is actively transmitting syslog messages.

If packets are visible on the node but not received by the server, the issue may be related to:

• Firewall policies

• Network routing

• Syslog server configuration

Verifying Management Network Connectivity

When syslog is configured to use the Out-of-Band network, the management VRF must have proper connectivity.

Engineers can verify this using:

show ip route vrf management

Typical output:

0.0.0.0/0 via 10.10.1.1 mgmt0

This confirms that the node has a default route toward the management network gateway.

Common Troubleshooting Scenarios

Scenario 1 – Syslog Server Not Receiving Logs

Possible causes include:

• Firewall blocking syslog port

• Incorrect syslog server port

• Server not listening on the configured port

• Monitoring policy not generating events

Packet capture on the node is the fastest way to isolate the issue.

Scenario 2 – Only Some Nodes Send Logs

If some nodes send logs but others do not, check:

• Monitoring policy scope

• Destination configuration

• Node-level connectivity

• Management VRF routing

Scenario 3 – TCP Syslog Session Not Re-Established

In rare cases, if a syslog server closes the TCP session unexpectedly, the node may not immediately re-establish the connection.

A common workaround is to reapply the syslog destination configuration, which restarts the syslog process and creates a new TCP session.