VMware NSX Multi-tenancy; True Tenant Isolation?

March 6, 2023

What is VMware NSX multi-tenancy? Historically multi-tenancy in VMware NSX was a Tier-0 gateway, otherwise known as the provider router, with one or many child Tier-1 gateways. Each Tier-1 gateway is created with a specific use-case; it could be for different customers, different environments, etc, the end goal is to provide tenancy segmentation.

Multi-tenancy was first introduced in VMware NSX 4.0.1.1, however, was driven via API. This article will focus on multi-tenancy in VMware NSX 4.1 which now allows users to have logical tenancy separation in the UI as well as API. This is fundamentally different to manually segmenting each tenant using logical gateways. Administrators now have the ability to provide true multi-tenancy, this article will cover all aspects of VMware NSX multi-tenancy; what it is, what it looks like, how to configure it, how to verify the configuration, and how to monitor it.

Prior to VMware NSX 4.0.1.1, multi-tenancy could be achieved with some trickery. This was achieved with using principal identities, but it wasn’t clean or easy, and not very widely adopted.

Topology and Design

This section will cover the logical topology and design of the VMware NSX multi-tenancy solution, it will not provide design examples for environments.

Image from VMware website, modified by lab2prod

Each deployment of VMware NSX has a single default org, which cannot be changed or modified. Objects created in the default space can be shared with projects / tenants, which you will see in this article.

The default org is where infrastructural fabric objects such as Tier-0 gateways, VRFs, Edges, Edge clusters, Transport Zones, hosts, etc reside. Within the default view, you will also be able to create users and their role mappings for each project. A tenant administrator, and subsequently a tenant will not have access to the system tab in VMware NSX Manager, as a result will not have access to create the objects mentioned earlier.

This section is a precursor to the remainder of this article, and is to provide a holistic view of VMware NSX multi-tenancy, objects, projects and how they all interact.

Requirements and Supported Services

Requirements

VMware NSX 4.1 is required to configure multi-tenancy (projects) via the user interface, otherwise VMware NSX 4.0.1.1 will suffice when creating tenancies via the API.

If you have created projects in the earlier version and plan on upgrading, the projects will appear in NSX 4.1.

Supported Services

Tier-1 gateways, static routing on Tier-1 gateways, Overlay Segments, Segment Profiles, NAT, DNS Forwarder, IPAM, DHCP, Gateway QoS, Distributed Firewall (DFW), and Gateway Firewall (GFW).

Note: there doesn’t seem to be a mechanism to stop two tenancies creating a segment and assigning the same subnet. Which is likely by design (similar to AWS etc). I put together another article to deep-dive into the routing aspects of multi-tenancy.

Configuring VMware NSX Multi-tenancy

Configuring multi-tenancy not overly difficult, at a high level the steps are listed below;

Create a Tier-0 gateway, VRFs(if required), deploy Edges, create an Edge cluster, essentially a standard deployment. This step assumes that none of this exists yet.
Create/ define project users
Create a project, assign users
Assign quotas if required
Create required networking constructs
Create / define required tags / groups / profiles / services
Share any resources that are required from the default tenancy to projects
Verify tenant isolation configuration
Verify tenant specific alarms, logs, and operations

This is not a strict order, some of the steps may be done at different times depending on your deployment. This section will cover the steps listed above excluding step 1, as I already have a Tier-0 gateway configured.

Create Project Users

Log into NSX, navigate to System, User Management, we will be focussing on the areas highlighted in the image below. In this write up I will be using locally created users, not users synced via an identity provider. Click on Local Users.

You will be presented with the below screen, click add, local user.

Create the user/name you require and click save.

You will now need to activate the user, to do so, find the user in the local users list, click the three dots and select activate.

You will need to enter a user password, click save.

Next click on User Role Assignment and find the user you created in the list. Click on the three dots, select edit.

Click on the 1 under roles.

You will now be able to assign roles to the user. Click on add role.

Add the role you want for the user, I am using enterprise admin, but you should really select a role such as Project Admin, or another role with lower permissions.

You will need to select the Project Scope (the project to apply the user to).

Click apply, once you are done. You can also remove any roles that you don’t want there.

Note: This will only work if you already have a project created. If you are following along, you will not have a project yet. You can also set the role under the user role assignment in the project user settings, you will see this in the next section.

Create a Project

Log into VMware NSX, ensure it is version 4.1, if it is you should see the drop-down shown in the image below. Click on the little down arrow.

After you have clicked on the arrow, you will see the projects menu. Click on manage.

This screen is where you create projects, you will need to create a project on a per-tenant basis. In other words, if you require 10 tenants all requiring separation, create 10 projects. Click on Add Project.

Populate the details relevant to the tenant you are creating. I will discuss the use and purpose of the Short Log Identifier later in this article. Click save once done.

You will be taken back to the projects screen and should be able to see the project listed. If you created project users earlier, you can assign the users by editing the project and clicking set under users.

You will be able to see the User assignment screen in the image below. Click on add role assignment (project admin in this case), select the source (Local User, LDAP, VIDM User) I will be using Local User. Type in the name of the user created earlier, select the role/scope (project admin in this case), click save.

Configuring Quotas

Quota’s allow you to create a limits for objects in each tenancy, this becomes increasingly important as the number of tenants scale out. The NSX platform has overarching limits for objects across every tenancy, these limits can be seen here. It is crucial that these limits are adhered to, exceeding these tested limits may lead to system instability, as I have seen in various customers.

There are 3 ‘pillars’ that quotas can be created for, they are Networking, Security, and Inventory. The objects that quota’s can be configured for currently are in the table below.

Networking	Security	Inventory
Tier-1 Gateway	Distributed Firewall Security Policy	Service
Segment	Distributed Firewall Rules	Service Entry
Segment Port	Gateway Firewall Policy	Group
NAT Rule	Gateway Firewall Rules	Context Profile
DNS Service		L7 Access Profile
DNS Zone
IP Address Pool
IP Address Pool Subnet
IP Address Block
IP Address Allocation
ND Profile
DAD Profile
Gateway QoS Profile
DHCP Relay
DHCP Server

As you can see, the table does not cover all objects that can be created or exist within VMware NSX, I assume as new versions release, this list will continue to expand. It’s relatively easy to create a quota, as can be seen in the image below, add the object and enter the limit, then click apply.

Configure Networking

I will be completing all the steps in this section using the local user created earlier, called tenant1-admin. Log back into NSX using the local user account, you should notice some options are greyed out, essentially everything outside of the tenant. Based off these results, it looks as though the Project Admin role allows the user to see other objects, but not modify or create.

You will also notice that it logs into the default namespace, which is not where you want to create the tenant specific objects anyway. Click on the drop down and select the tenancy for this user. The moment you do that, your options are reduced.

As mentioned earlier in this article, the supported services/features for VMware NSX multi-tenancy are reduced. This section will focus on creating Tier-1’s, Segments, and how they look in the tenancy, and in the default scope, however, there is no difference in the process of creating these objects within a tenant.

Creating a Tier-1 Gateway

Within the tenancy, navigate to Networking, then Tier-1 Gateways. Click on Add Tier-1 Gateway.

Populate all the details as you normally would, notice, the Linked Tier-0 Gateway provides a choice of the gateways that were selected when the project was created. If other Tier-0 gateways are required for the tenant, you will need to log in as the administrator account or an account with the relevant permissions and assign the gateway to the tenant.

In this example, I will not be using any stateful services, so I have selected the Distributed Only HA Mode, if stateful services are required; select either Active-Standby or Active-Active. If north-south connectivity is required, ensure you select a Tier-0 gateway. Click Save once done.

Creating a Segment

Navigate to Networking, Segments, Click on Add Segment. You will see the below screen.

The name is required, if you select none for the connected gateway, then you will not be required to add a subnet. The moment you select a linked gateway, the subnet field becomes mandatory. The choices of the gateway are the Tier-0 gateway that was linked to the tenant and the Tier-1 gateway that was created earlier.

Notice that the Tier-0 gateway is not tagged with the tenant name, only the one created in the tenancy is. In this scenario, I will be attached the gateway to the Tier-1 gateway, not the Tier-0. Click save once you are done.

Creating and Defining Network and Policy Objects

You may be asking yourself, how does NSX know which VM’s and/or objects belong to each tenancy? If you think about it, NSX syncs with vCenter and gets its inventory that way, so how does it differentiate between the default org or the tenant?

It’s a good question, and if you look in the default org of NSX you will see all the VMs in vCenters inventory.

At this point, if you move into the tenant view for VM’s you will see none listed.

Similarly, any services, groups, profiles, and tags that were created in the default org will not be visible to the tenant. This can be remediated with resource sharing, which we will cover later in this article.

At this point, you are able to define these objects within the tenancy if you do not want to share resources or if they are going to be tenant specific. But you can’t create a virtual machine, so how do you get VM’s so show up? Simple, attach the required VMs in vCenter to the tenancy segment.

You will see the VM appear within the tenancy under Virtual Machines now.

The object within the tenancy looks as it would normally.

It is no longer visible in the default org.

Tags and objects

This section will focus on tags and objects, however, DFW and GFW policy behaves in the exact same fashion.

Similar to the Virtual Machines, you may also be asking what objects created in a tenancy can be seen in the default org. Let’s test with tags, navigate to Inventory, Tags, click Add Tag (in the tenant space). Add in the tag details then click save.

You will see a similar output as the below image.

Now, I will navigate back to the default org and check to see if the tag is visible.

As you can see, the tag is not visible, it is safe to say objects defined within the tenancy are not visible ‘upstream’.

There may be scenarios where administrators are required to share certain resources with tenancies, this can be achieved using Resource Sharing.

The objects in the image below are currently supported to be shared with tenants.

As mentioned earlier, VLAN backed segments are not supported with tenancies, only overlay segments. What happens when a tenant requires access to a VLAN / VLAN segment for let’s say a service interface?

The administrator can share the VLAN backed segment with the tenant, for them to use as a service interface or whatever their use case may be.

The image below shows a VLAN backed segment in the default org called ‘nsx-vlan-63‘, which needs to be shared to tenant1.

Navigate to Inventory, Resource Sharing, click on Add Resource Share.

Enter the share name, then click on set.

In the next section, click on segments, select the segment you want to share, in this case ‘nsx-vlan-63’, and click apply.

Next, click set under Shared With, select the project to share the resource with. Click Apply.

Click Save.

To verify, log out as administrator, then log back in as the tenant administrator, navigate to Networking, Segments. By default, you will not see shared resources, ensure you select the checkbox below ‘Show Shared Objects’.

You can now see the shared segment, notice it has the default org tagged next to the name.

Verify Tenant Isolation

To ensure tenants are not able to ‘cross-communicate’ or see each others resources, I have created a second tenant, called tenant2, along with its own administrator account, effectively followed all the steps outlined above.

One thing to note, while creating the tenant2-t1, the tenant1 Tier-1 gateway was not visible.

From a configuration perspective, it seems as though the tenants are isolated. Tenant2-admin can still click on all projects which is a read-only view of NSX and lists all objects underneath the default org. You will also notice, there are no configuration options, it is essentially read only.

The tenant administrator also has visibility across the network topology view.

Whilst under all projects, the tenant administrator can also see all functions under Plan & Troubleshoot, however, cannot run tools such as Traffic Analysis under all projects, regardless of being able to list VMs etc.

Tenant Alarms, Operations, Logging

Alarms

In order to test tenant specific alarms, I have created a quota for tenant 2 with a maximum limit of 1 segment.

Checking the project quota now shows that it has been exceeded.

In theory, this should raise an alert/alarm for tenant 2, so I will now log in as tenant 2 and check the status. No alarms were triggered, let’s try creating another segment to see if it triggers an alarm.

The good news is, it doesn’t allow you to exceed the quota, the bad news is, there is no alarm triggered from hitting the quota.

It looks like the quota limit alarm is raised under the default / all projects pane, as it was created in the default org, which makes sense.

The alarm definitions seem mostly infrastructural, so getting them to trigger will take some testing. Based off my looking around, it seems as though the alarms would not be tenant specific ie, edge down, TEP down etc, and would therefore appear in default, as well as tenancy alarms.

Logging

Earlier when creating the projects, there was a field called ‘Short log identifier‘. This is useful for log tracing and debugging.

I use vRealize Log Insight (vRLI) in this environment for logging, below is an image from log insight using the search term ‘tenant1’.

Filtering by the log identifier shows all log events for the project. This will be handy for troubleshooting purposes when you are required to track down events for a project specifically.

Summary

VMware NSX 4.1 has come leaps and bounds for multi-tenancy, it effectively allows administrators to isolate their logical constructs from one another. There are some features missing, there definitely isn’t feature parity between the default org and a tenant. I found it quite easy to navigate and configure, it will be much simpler to do so in Greenfields, however, that does not mean it is not possible for brownfields.

Administrators would have to consider outages, downtime, and order of operations for moving from the default space to a project. In saying all of that, I think it is a step in the right direction and look forward to seeing how it develops in the future.

Further details can be found on the VMware website.