ACI Multi-Site and Active-Standby Firewall Clusters with Service Graph Redirect
| Technology in Nexus Dashboard Orchestrator | Equivalent on APIC |
|---|---|
| define a Service Service Device Cluster | L4-7 Device, SGT |
EPG-to-EPG PBR contract in ACI multi-site with independent HA-firewalls: I must define the BD subnet also under the consumer EPG and set the no Gateway SVI flag.
Contract Enforcement vs PBR Contract Enforcement vs PBR Policy Enforcement Contract enforcement == zoning rules are deployed to a leaf X. So leaf X takes action on the contract-matched traffic based on the deployed zoning rules. Leaf X is determined according to the Contract Policy Enforcement Locations table in the document ACI Contract Guide White Paper, page=20.
PBR policy enforcement == traffic redirection to service nodes is performed effectively by a leaf Y. Leaf Y is determined according to the PBR Policy Enforcement Locations table in ACI Multi-Site in the document ACI Multi-Site Service Node Insertion White Paper 2025, page=18.
PBR contract enforcement == Contract enforcement + PBR Policy enforcement
A PBR contract can be enforced on a leaf X. But the PBR policy might be enforced on a leaf Y, i.e. leaf Y is the one which redirects traffic to service nodes. –> This explains why cisco defined distinct tables for contract enforcement locations ( table 3 ‘Where Contract policy is applied aka Contract Policy Enforcement Location’ of the document ACI Contract Guide White Paper, page=20) and for PBR policy enforcement locations in ACI Multi-Site (document ACI Multi-Site Service Node Insertion White Paper, page=18, PBR Policy Enforcement Locations in ACI Multi-Site)
PBR Policy Enforcement Locations#
These tables describe the location where PBR policy is enforced for the different types of ACI traffic.
See the document ACI Multi-Site Service Node Insertion White Paper, page=18.
Considerations#
Using a contract with PBR in a ACI Multi-Site context requires the Service Nodes not to be connected with L3Out to ACI. However, I can use the same physical interfaces of the service devices for both L3Out (by using a VLAN X with SVI as the layer-3 interface type instead of routed port or routed sub-interface) and SGR connectivity (by using VLAN Y for the VLAN interface on the transit network between ACI and the service device, and the transit network is configured on the Service BD).
The same Service BD must be used in all sites (L2 stretched) where Service Nodes are integrated. Cisco recommends to disable BUM flooding on the Service BD.
By default a leaf switch redirects traffic only to local PBR nodes. Thus it is critical that the local service nodes are high available.
With vzAny PBR Contracts, cisco requires L4-7 Device in one-arm design. Cisco recommends configuring IP SLA Monitoring policy. IP SLA Monitoring Policy is required for:
- vzAny to vzAny PBR contracts
- L3Out-to-L3Out PBR contracts
- EPG to L3Out PBR contracts #verifyThis
- vzAny to EPG contracts #verifyThis
The configuration is performed in NDO and I must associate the policy with the ACI Sites.
I shall remember the directionality of a PBR Policy.
Recommended Design for Active-Standby HA-Firewalls with SGR#
Cisco recommends this topology when I want to implement Active-Standby HA-firewalls: independent Active-Standby HA-firewalls on the ACI Sites. See the document ACI Multi-Site Architecture White Paper, page=130.
For the rest of the note I am referring to the independent active-standby HA-firewalls design.
Traffic Types, Challenges and Considerations#
North/South Traffic between appEPG and L3extEPG, PBR Policy applied in both directions#
Intra-VRF#
See the document ACI Multi-Site Service Node Insertion White Paper 2025, page=19.
Assumptions#
I have SGR with the PBR policy being applied to both Provider and Consumer sides of the contract.
Source and destination pcTags are known (pcTag of the L3extEPG has known values depending on traffic direction, and the pcTag of the app EPG is known to its attached compute leaf) -> the non-border leaf enforces the PBR contract.
from-WAN-ingress traffic#
The traffic traverses a border leaf without PBR policy enforcement, because only non-border compute leaves should do that according to the PBR Policy Enforcement Locations table in ACI Multi-Site, in the document ACI Multi-Site Service Node Insertion White Paper 2025, page=18.
The non-border compute leaf receives the traffic and enforces the PBR policy; Traffic arrives at a local Service Node which applies its security policy configuration and sends back traffic to the ACI service leaf. The service leaf forwards traffic to the destination compute leaf (where the destination endpoint is attached). #QA what if the EPG were instantiated on the border leaf itself?
to-WAN-egress traffic#
The compute leaf enforces the PBR policy; It redirects traffic to the local active firewall, which applies its configure security policy and sends the traffic back to the service leaves. ACI applies layer-3 routing semantics and forwards the traffic to the local border leaf, which routes traffic to the external network destination. #QA what if the EPG were instantiated on the border leaf itself?
Observations#
I could have asymmetrical selection of the L3Out (L3Out of Site 1 for one traffic flow and L3Out of Site 2 for another traffic flow) for from-WAN-ingress and to-WAN-egress traffic. This is normal but it is not problematic, because the compute leaf will always enforce the PBR policy toward the site-local Service Node for this type of traffic (N/S between appEPG and L3extEPG).
To minimize L3Out selection asymmetry, I must implement Host Routing. But this might not impact the symmetry of the selection of the SGR active firewall. #QA how do I mitigate the risk of asymmetric PBR node selection without Host Routing?
Cisco recommends using stretched L3extEPG, i.e. associate the same L3extEPG with each site-specific L3Out.
#QA no preference which one is the provider and which one the consumer?
Configuration#
Create IP SLA Monitoring policy #QA How to reference it? Create a Service Device Template and associate it with all Sites. Create a Service Device Cluster.
Go the the template that contains the desired (EPG, EPG) contract. Create Service Chaining and associate the contract with the Service Device Cluster.
Apply the template ( #QA which template, the Service Device template or the EPG template?) => in APIC, a Service Graph Template is created and deployed, Logical Device Context(s) is created and associated with the deployed SGT, PBR policy(es) is created and associated with the deployed SGT.
Inter-VRF#
The L3extEPG must be the contract provider.
East/West Traffic between two appEPGs, PBR Policy applied in both directions#
Intra-VRF#
See the document ACI Multi-Site Service Node Insertion White Paper, page=23.
According to the table PBR Policy Enforcement Locations in ACI Multi-Site in the document ACI Multi-Site Service Node Insertion White Paper 2025.pdf#page=18, the PBR policy is always enforced at the Provider leaf, even if the destination EPG’s pcTag is learned by the source leaf; This is performed by ACI dynamically, by setting a flag on the zoning rules that tells a non-Provider leaf not to apply the PBR policy. The flag is called ==redirect override==. This ensures the PBR policy is always applied at the Provider leaf for east/west (appEPG, appEPG) traffic, thus selecting always the same service node, thus achieving symmetry in the service node selection in ACI Multi-Site for east/west EPG to EPG traffic.
Requirements#
- If I want to deploy a PBR Contract for an East/West specific (EPG, EPG) pair - which in ACI Multi-Site is not the same design as having a vzAny PBR Contract - then I must configure the IP subnet under the Consumer EPG in a way that covers all the endpoints that belong to that EPG. Normally, the IP subnet of the BD’s IP address covers all endpoints under an attached EPG. But in some designs, $2^+$ EPGs might share the same IP subnet: So if I configure the IP subnet under the Consumer EPG while another EPG has the same subnet, ACI is confused because there might be endpoints in the other EPG which fall under the same IP subnet. In this case, under the Consumer EPG, I configure only the /32 IP addresses of its endpoints.
This requirement is meant only to help the leaf resolve the pcTag, not participate to routing. Routing of the IP subnet is performed at the bridge domain level.
The IP subnet that’s configured under the Consumer EPG must have the no SVI Default Gateway flag on.
- When I have a border leaf with an intersite L3Out ( #verifyThis a L3Out that is configured on more than one ACI Site and whose external EPG is stretched) and I want to implement a East/West PBR Contract, then Cisco does not support having the Consumer EPG on that border leaf. #QA is this requirement also valid for vzAny PBR contracts?
Inter-VRF#
#TBC
East/West Traffic for (vzAny, vzAny) PBR Contracts, PBR Policy applied in both directions, intra-VRF#
When a (vzAny,vzAny) PBR contract is implemented east-west in ACI Multi-Site and both the source and destination leaves are not in the same Site, traffic redirection is applied on both of them (See [[Attachments/lib/ACI Multi-Site Service Node Insertion White Paper 2025.pdf#page=18&rect=35,141,575,510|PBR Policy Enforcement Locations in ACI Multi-Site, second table]]).
Due to the nature of vzAny, ACI leaves can not determine which EPG is the Provider and which EPG is the Consumer. –> They simply can not find out which leaf the Provider leaf is -> They can not implement the ‘redirect override’ flag like in the east/west (appEPG, appEPG) traffic type.
Starting with version 6.0(4c), ACI introduced an algorithm that is based on special flags (SP and DP) activated on the iVXLAN header, that decide whether leaf switches redirect traffic to site-local service nodes or not. #verifyThis The decision whether to redirect traffic to the site-local firewalls depends on whether the pcTag of the destination EPG is known or not.
Requirements#
- minimum ACI 6.0(4c) and NDO 4.2.x.
- IP SLA Monitoring policy.
Ingress leaf knows pcTag of source EPG and pcTag of dst EPG#
. The flags are SP and DP. The possible binary combinations are:
- SP = 0 and DP = 1 in a received iVXLAN packet makes the leaf “believe” that the source leaf did not apply the PBR policy -> the leaf applies the PBR policy. This is the combination which ensures that traffic redirection takes place on every Site that has an active-standby cluster, when source pcTag and destination pcTag are known.
- SP = 1 and DP = 1 on a received iVXLAN packet makes the leaf “believe” that the source leaf did already apply the PBR policy -> It does not apply the PBR policy.
Example: Source leaf switch in Site 1 applies the PBR policy by redirecting ingress traffic to the site-local firewalls. The site-local firewalls apply the security configuration and send traffic back to ACI. The service leaf (the leaf attached to the firewalls) sets SP = 0 and DP = 1 on the iVXLAN header of the packet toward the destination leaf ( #QA what is the value of PA bit?). Traffic is processed according to layer-3 semantics and reaches the destination leaf in Site 2 (so this is intersite traffic). The destination leaf in Site 2 believes that the originating leaf did not enforce PBR policy (because of the combination SP = 0, DP = 1). So, it applies the PBR policy and redirects traffic to the firewalls in Site 2. After the firewalls apply the configured security policies, they send the traffic back to ACI. The service leaf in Site 2 sets SP = 0 and DP = 1 on the iVXLAN header of the packet toward the destination leaf. The destination leaf receives the iVXLAN packet. But since this is intra-Site traffic, the SP and DP flags are ignored.
Ingress leaf knows pcTag of source EPG, but not of dst EPG#
Example: In Site 1, an ingress leaf can not enforce the PBR contract (because I know that enforcing a contract requires being able to resolve the source and destination pcTags) -> it sets PA = 0 in the iVXLAN header of the packet toward the destination leaf in Site 2 ( #QA the destination VTEP being learned from the endpoint table?). The destination leaf in Site 2 reads PA =0 and concludes that no contract enforcement took place on the source leaf. So the destination leaf enforces the contract and applies the PBR policy: It redirects traffic to Site 1 firewalls. Site 1 firewalls apply their configured security policies then send traffic back to ACI. The attached service leaf in Site 1 activates SP = 0 and DP = 1 on the iVXLAN header of the packet toward the destination leaf in Site 2. The destination leaf in Site 2 receives the traffic again, reads the special flags, believes that PBR policy has not been enforced in Site 1. The destination leaf in Site 2 applies the PBR policy by redirecting the traffic to the local firewalls in Site 2. The local firewalls in Site 2 apply their configured security policies and send the traffic back to ACI. The attached service leaves apply the special flags SP = 0 and DP = 1 to the iVXLAN header of the packet toward the destination leaf. The destination leaf receives the traffic with the special flags. But since this time it is intra-Site traffic, it ignores the special flags and forwards the traffic to the attached destination endpoint.
Eliminating Hairpinning between Sites#
Although the algorithm with the special flags eliminated the risk for traffic asymmetry, traffic hairpins between Site 1 and Site 2. This can be reduced by another algorithm called ==Conversational learning==: When the destination leaf in Site 2 received a iVXLAN packet from the source leaf in Site 1 containing PA = 0, the destination leaf applies the PBR policy as per the algorithm of the special flags, plus it generates a control packet containing the IP address and pcTag of the destination endpoint and sends it to the source leaf in Site 1. This helps the source leaf in the future apply the PBR policy by redirecting traffic to the local PBR destination (as part of the special flags algorithm) -> reduces traffic hairpinning between SItes.
North/South Traffic for (vzAny, vzAny) PBR Contracts, PBR Policy applied in both directions#
The risk of asymmetric N/S traffic is mitigated by Implementing Host Routing for stretched BDs. The reason is explained on [[Attachments/lib/ACI Multi-Site Service Node Insertion White Paper 2025.pdf#page=33|ACI Multi-Site Service Node Insertion White Paper 2025, page 33]]
Requirements#
- minimum ACI 6.0(4c) and NDO 4.2.x.
- IP SLA Monitoring policy.
East/West Traffic for (appEPG, vzAny) PBR Contracts, PBR Policy applied in both directions#
Configuration#
takes place on Nexus Dashboard. there is a template-specific and site-specific configs Config in pre-version 4: See the document ACI Multi-Site Service Node Insertion White Paper, page=75. Config in version 4 and later: (get screenshots from pcloud folder.) Configure IP SLA Monitoring policy for the tenant. Associate it with all sites where service nodes are deployed. Configure a Service Device.
Use cases#
Intra-VRF north/south traffic intra-VRF east/west traffic inter-VRF east/west traffic vzAny-involved traffic.