Overview:
Two pod VVD design with an underlay built on Cisco ACI and using OSPF adjacencies with a single area with BGP providing adjacencies for the overlay provided by NSX.
Problem description: Inconsistent routing of traffic on the overlay.
The root of the Problem:
Edge-1 was publishing routes to ACI fabric to allowing underlay to route traffic to overlay networks. ACI was then publishing those routes to Edge-2 with higher priority than what the DLR was publishing the routes to Edge-2 as this causing traffic on Edge-2 could not forward to the DLR connected to Edge-2.
Resolution: Was to change the design from uplinks on multiple edges to uplink on single EDGE with HA enabled.
Reconfiguration:
Edge Configuration: Uplink to vDS and Internal connection to Global Transport Network
OSPF routing on EDGE: Uplink1 will connect to ToR for OSPF traffic.
BGP routing on EDGE: Configure Neighbors.
Route Distribution on EDGE: Needs two distributions from BGP to OSPF and accept all BGP.
DLR configuration: Uplink to EDGE-01 with three local virtual switches.
OSPF Configuration on DLR: Disable
BGP Configuration on the DLR: BGP connection to the Edge.
Route redistribution: Only BGP connected needed.
Final configuration architecture:
vPSO 
Seems like having two ESG’s on each cluster with ECMP enabled might provide additional throughput, rather than the active/standby solution implemented herein. Also, the diagram is showing only one switch/subnet inside the DLR. Is that a diagramming space consideration and there are multiple subnets in the overlay? Can you explain the use of BGP on the overlay when OSPF was the IGP for the underlay. Why not just set a separate OSPF area for the overlay and aggregate/filter routes as needed?
Tim – Agree with your assessment and in fact, the VMware Validated Design (VVD) calls for two ESG’s using ECMP for that reason. The reason the change from the first picture with ICMP to HA design was due to the OSPF layout of the underlay. There was a single OSPF area that covered both the overlay and other physical assets in production changes to the underlay OSPF design was not an option.
BGP was chosen for a couple reasons, first was that VVD guidance calls for BGP on the overlay even if OSPF is used in the underlay. We could have gone back to OSPF, but troubleshooting steps in order were changing the dynamic overlay routing to BGP. Then we switch to HA configuration so that single OSPF area would not advertise down routes from the first ESG to second with higher weight than the DLR.
Hi Joe, you said one of the reasons is the songle OSPF area, but I had exactly the same issue though I’ve furtherly adhered to vmware recommendations (https://docs.vmware.com/en/VMware-Validated-Design/4.1/vmware-validated-design-41-sddc-ospf-bgp-routing.pdf) stating:
– Configure NSSA area types
– Do not use NSX ESGs as OSPF ABRs.
– Do not include NSX ESGs in Area 0
– Validated Design deployment and upgrade guidance is not relevant to hybrid OSPF-BGP routing
So my routing topology was:
OSPF-area0—-OSPF-area1-NSSA—–iBGP—
The problem is exaclty the same: on ESG devices OSPF learnt routes (admin distance 110) takes precedence over iBGP learnt prefixes (admin distance 200) this causing one of the two ESG blackhauling traffic.
Maybe this is one of the reasons OSPF+BGP why implementation is NOTcompletely validated…