VMware NSX and BGP Design and Configuration

VMware NSX and BGP Design and Configuration

Introduction

In this blog post I would like to share how BGP is configured on VMware NSX to automatically update routing information with the physical network.

Design

Recently I was involved in a project where we used BGP to peer the NSX environment with the physical network. The design we did was challenging because of scale. The NSX environment we deployed meant for a multi-tenant Cloud platform scaled for several hundreds of tenants. We deployed a multi-site NSX environment.  Each tenant has its own Universal Distributed Logical Router (UDLR) spanned across both datacenters, and an Edge Services Gateway (called the Tenant Logical Router or Tenant ESG) behind a Perimeter Logical Router (PLR or Perimeter ESG), which holds a maximum of 49 tenants or less depending the performance or bandwidth requirements.

In this post a reference design is used for a fictional tenant to proof the configuration and design.

Below this design is displayed. Note the the UDLR and Logical Switches are available on both sites. A tenant can deployed on both sites.

NSX multi-site design with BGP

 

Configuration

Logical Switches

First the necessary Logical Switches are created. The screen shot below shows the created Logical Switches on the primary site. Note the Universal Logical Switches which were created to make the network available on the secondary site.

NSX Logical Switches

Logical Switches

 

Universal Distributed Logical Router

An Universal Distributed Logical Router (UDLR) is deployed on the primary site and connected the associated Logical Switches. Local Egress is enabled.

UDLR Configuration

UDLR Configuration on Primary Site

 

The interfaces for this UDLR are configured as shown below. Interface vNIC# 2 is the Uplink to the Tenant ESG. Note that interface #3 is the uplink for the secondary site to the Tenant ESG on that site.

ULDR Interface Configuration

ULDR Interface Configuration

 

Tenant Logical Router

Each Tenant is configured with a Edge Services Gateway (ESG) as the Tenant Logical Router which is the gateway to the tenant internal networks and services.

The ESG is deployed with the following configuration.

Tenant ESG Configuration

Tenant ESG Configuration

 

The interfaces are configured as shown below. The Tenant ESG inteface vNIC# 0 is configured as an uplink to the Perimeter ESG, which is configured later in this blog post. For scalability a /28 subnet is used to connect several tenant Tenant ESG’s to a Perimeter ESG. Interface vNIC# 2 is the link to the Tenant UDLR. Interface vNIC# 1 is reserved for future use (ECMP). Note that on interface vNIC#3 we’ve configured a Tenant BGP Test Public IP address (hidden in this screenshot) for a Transparant Mode Load Balancer.

Tenant ESG Interface Configuration

Tenant ESG Interface Configuration

 

Perimeter Logical Router

The Perimeter Logical Router is the ESG which has an uplink to the physical network configured. This ESG also has BGP peering configured with the physical network. In this design the Perimeter ESG is used to scale up to 35 tenants per ESG. The Perimeter ESG is configured with HA and deployed as X-Large to have enough resources available to route and process all the traffic for the tenants.

Perimeter ESG Configuration

Perimeter ESG Configuration

 

The interfaces are configured as shown below. Interface vNIC0 is configured as the uplink to the physical network. Because we are limited to 10 vNIC interfaces a Trunk is created on interface vNIC2 to scale up to 200 sub interfaces.

Perimeter ESG Interface Configuration

Perimeter ESG Interface Configuration

 

Each sub interface is configured as the transit link to each seperate Tenant ESG. In the example below interface vNIC10 is the transit link to the tenant ESG which was created.

NSX Edge Interface Configuration

NSX Edge Interface Configuration

 

Routing and BGP

Now BGP is configured on the UDLR, Tenant ESG and finally the Perimeter ESG.

UDLR

Edit the Global Routing Configuration on the UDLR on the Routing Page.

Routing Global Configuration

Routing Global Configuration

 

Edit the Dynamic Routing Configuration and select the Router ID.

Edit Dynamic Routing Configuration

Dynamic Routing Configuration

 

Continue on the BGP Configuration page.

BGP Configuration

BGP Configuration

 

Click on the Edit button to configure BGP. In this example the Local Autonomous System (AS) is 65029.

Edit the BGP Configuration

Edit the BGP Configuration

 

Now the BGP neighbours are configured. In this configuration the Tenant ESG is the neighbour. The neighbour IP Address is 10.10.10.1 (Internal interface of the Tenant ESG). The forwarding address is the configured IP address of the UDLR and on the UDLR a Protocol Address is configured. The remote AS number is 65029 in this example.

BGP Neighbour Configuration on the UDLR

BGP Neighbour Configuration on the UDLR

 

Continue on the Route Redistribution page and enable BGP.

Add a Route Redistribution criteria in the Route Redistribution table. Select BGP as the learner protocol and make sure it is connected.

Add BGP Route Redistribution Criteria

Add BGP Route Redistribution Criteria

 

Tenant ESG

Continue with the Global Routing Configuration on the Tenant ESG.

Tenant ESG Routing Configuration

 

Edit the Dynamic Routing Configuration and select the Router ID.

Edit Dynamic Routing Configuration

Edit Dynamic Routing Configuration

 

Continue on the BGP Configuration page.

BGP Configuration

BGP Configuration

 

Click on the Edit button to configure BGP. In this example the Local Autonomous System (AS) is 65029.

Edit the BGP Configuration

Edit the BGP Configuration

 

Now the BGP neighbours are configured. In this configuration the Perimeter ESG and the UDLR are the neighbouring routers. The neighbour IP Address is for the the Perimeter ESG is 172.16.20.1 (Internal interface of the Perimeter ESG) and the IP address of the neighbour UDLR is 10.10.10.3 (Protocol Address the UDLR).

Add UDLR Neighbour

UDLR Neighbour Configuration

UDLR Neighbour Configuration

 

Add Perimeter ESG Neighbour

Tenant ESG Neighbour Configuration

Tenant ESG Neighbour Configuration

 

Continue on the Route Redistribution page and enable BGP.

Add a Route Redistribution criteria in the Route Redistribution table. Select BGP as the learner protocol and make sure it is connected.

Add BGP Route Redistribution Criteria

Add BGP Route Redistribution Criteria

 

Perimeter ESG

Continue with the Global Routing Configuration on the Perimeter ESG.

Perimeter ESG Global Configuration

Perimeter ESG Global Configuration

 

Edit the Dynamic Routing Configuration and select the Router ID.

Edit Dynamic Routing Configuration

Edit Dynamic Routing Configuration

 

Continue on the BGP Configuration page.

BGP Configuration

BGP Configuration

 

Click on the Edit button to configure BGP. In this example the Local Autonomous System (AS) is 65029.

Edit the BGP Configuration

Edit the BGP Configuration

 

Next the BGP neighbours are configured. In this configuration the Physical Router and the Tenant ESG are the neighbouring routers. The neighbour IP Address is for the Physical Router is a Public IP address ending with .1 and the IP address of the neighbour Tenant ESG is 172.16.20.2 (Uplink interface on the Tenant ESG)

Add Physical Router Neighbour. Note that the Physical Router has a different AS configured.

Edit Physical BGP Neighbour

Edit Physical BGP Neighbour

 

Add Perimeter ESG Neighbour

Edit Tenant ESG as a Neighbour

Edit Tenant ESG as a Neighbour

 

Continue on the Route Redistribution page and enable BGP.

Add a Route Redistribution criteria in the Route Redistribution table. Select BGP as the learner protocol and make sure it is connected.

Add BGP Route Redistribution Criteria

Add BGP Route Redistribution Criteria

 

Routing and BGP configuration is complete.

Test the BGP Configuration

Open the console or a SSH session to the UDLR, Tenant ESG or Perimeter ESG to test the BGP configuration

Use the show ip route command to see if there are routes derived from BGP.

show ip route
Show IP Route TESG

Show IP Route on Tenant ESG

 

Use the show ip bgp neighbours command to read detailed BGP information of the BGP neighbours

show ip bgp neighbours

Show IP BGP Neighbours

Test the routing with a ping to a VM in one of the Tenant networks behind the UDLR.

Ping Test from the Tenant ESG to a VM behind a UDLR

Ping Test from the Tenant ESG to a VM behind a UDLR

 

Facebooktwittergoogle_plusredditpinterestlinkedinmail

There is 1 comment. Add yours.

  1. Jim Streit

    Great information. Thank you!

Leave a Reply

Your email address will not be published. Required fields are marked *