Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Permalink
live
Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?
Go to file
 
 
Cannot retrieve contributors at this time
title titleSuffix description author ms.author ms.reviewer ms.date ms.service ms.subservice ms.topic ms.custom
Connectivity architecture
Azure SQL Managed Instance
Learn about Azure SQL Managed Instance communication and connectivity architecture and how the components direct traffic for a managed instance.
srdan-bozovic-msft
srbozovi
mathoma, bonova
11/16/2022
sql-managed-instance
service-overview
conceptual
fasttrack-edit

Connectivity architecture for Azure SQL Managed Instance

[!INCLUDEappliesto-sqlmi]

This article describes the connectivity architecture in Azure SQL Managed Instance and how components direct communication traffic for a managed instance.

Overview

In SQL Managed Instance, an instance is placed inside the Azure virtual network and inside the subnet that's dedicated to managed instances. The deployment provides:

  • A secure virtual network-local (VNet-local) IP address.
  • The ability to connect an on-premises network to SQL Managed Instance.
  • The ability to connect SQL Managed Instance to a linked server or to another on-premises data store.
  • The ability to connect SQL Managed Instance to Azure resources.

[!NOTE] The November 2022 feature wave introduced changes to the default connectivity structure of SQL Managed Instance. This article provides information about the current architecture and the architecture of instances that haven't yet enrolled in the feature wave. For more information, see November 2022 feature wave.

November 2022 feature wave

The November 2022 feature wave introduced the following changes to the SQL Managed Instance connectivity architecture:

  • Removed the management endpoint.
  • Simplified mandatory network security group rules (removed one mandatory rule).
  • Revised mandatory network security group rules so that they no longer include outbound to AzureCloud on port 443.
  • Simplified the route table (reduced mandatory routes from 13 to 5).

Communication overview

Current architecture

The following diagram shows entities that connect to SQL Managed Instance. It also shows the resources that need to communicate with a managed instance. The communication process at the bottom of the diagram represents customer applications and tools that connect to SQL Managed Instance as data sources.

:::image type="content" source="media/connectivity-architecture-overview/1-connectivity-architecture-diagram-entities.png" border="false" alt-text="Diagram that shows entities in the connectivity architecture for Azure SQL Managed Instance after November 2022.":::

SQL Managed Instance is a single-tenant, platform as a service offering that operates in two planes: a data plane and a control plane.

The data plane is deployed inside the customer's subnet for compatibility, connectivity, and network isolation. The data plane typically is accessed through its VNet-local endpoint. The data plane depends on Azure services like Azure Storage, Azure Active Directory (Azure AD) for authentication, and telemetry collection services. You'll see traffic that originates in subnets that contain SQL Managed Instance going to those services.

The control plane carries the deployment, management, and core service maintenance functions via automated agents. These agents have exclusive access to the compute resources that operate the service. You can't use ssh or Remote Desktop Protocol to access those hosts. All control plane communications are encrypted and signed by using certificates. To check the trustworthiness of communicating parties, SQL Managed Instance constantly verifies these certificates by using certificate revocation lists.

Architecture before November 2022

The following diagram shows entities that connect to SQL Managed Instance. It also shows the resources that need to communicate with a managed instance. The communication process at the bottom of the diagram represents customer applications and tools that connect to SQL Managed Instance as data sources.

:::image type="content" source="media/connectivity-architecture-overview/01-connectivity-architecture-entities.png" border="false" alt-text="Diagram that shows entities in the connectivity architecture for SQL Managed Instance before November 2022.":::

SQL Managed Instance is a single-tenant, platform as a service (PaaS) offering. Its compute and networking elements are deployed inside the customer's subnet. SQL Managed Instance typically is accessed via its VNet-local endpoint. SQL Managed Instance depends on Azure services like Azure Storage, Azure Active Directory (Azure AD), Azure Key Vault, Azure Event Hubs, and telemetry collection services. Traffic that originates in subnets that contain SQL Managed Instance might go to those services.

Deployment, management, and core service maintenance operations are carried out via automated agents. These agents have exclusive access to the compute resources that operate the service. You can't use ssh or Remote Desktop Protocol to access those hosts. All internal communications are encrypted and signed by using certificates. To check the trustworthiness of communicating parties, SQL Managed Instance constantly verifies these certificates by using certificate revocation lists.


High-level connectivity architecture

Current architecture

At a high level, SQL Managed Instance is a set of service components that are hosted on a dedicated set of isolated virtual machines that are joined to a virtual cluster. Some service components are deployed inside the customer's virtual network subnet. Other services operate in a secure network environment that Microsoft manages.

A virtual cluster can host multiple managed instances. The cluster automatically expands or contracts as needed to accommodate new and removed instances.

Customer applications can connect to SQL Managed Instance and can query and update databases inside the virtual network, peered virtual network, or network connected by VPN or Azure ExpressRoute.

:::image type="content" source="media/connectivity-architecture-overview/2-connectivity-architecture-diagram-sql-managed-instance.png" border="false" alt-text="Diagram that shows the high-level connectivity architecture for Azure SQL Managed Instance after November 2022.":::

To facilitate connectivity with customer applications, SQL Managed Instance offers two types of endpoints: a VNet-local endpoint and a public endpoint.

VNet-local endpoint

The VNet-local endpoint is the default way to connect to SQL Managed Instance. The VNet-local endpoint is a domain name that has the form <mi_name>.<dns_zone>.database.windows.net and resolves to an IP address from the subnet's address pool. The endpoint is local to the virtual network. You can use a VNet-local endpoint to connect a managed instance in all standard connectivity scenarios.

Public endpoint

The public endpoint is an optional domain name that has the form <mi_name>.public.<dns_zone>.database.windows.net and resolves to a public IP address that's reachable from the internet. This endpoint allows only TDS traffic to reach SQL Managed Instance and can't be used for integration scenarios like failover groups, SQL Managed Instance link, and similar technologies.

Architecture before November 2022

At a high level, SQL Managed Instance is a set of service components that are hosted on a dedicated set of isolated virtual machines that are joined to a virtual cluster. Some service components are deployed inside the customer's virtual network subnet. Other services operate in a secure network environment that Microsoft manages.

A virtual cluster can host multiple managed instances. The cluster automatically expands or contracts as needed to accommodate new and removed instances.

Customer applications can connect to SQL Managed Instance and can query and update databases inside the virtual network, peered virtual network, or network connected by VPN or Azure ExpressRoute.

:::image type="content" source="media/connectivity-architecture-overview/02-connectivity-architecture-sql-managed-instance.png" border="false" alt-text="Diagram that shows the high-level connectivity architecture for Azure SQL Managed Instance before November 2022.":::

To facilitate connectivity with customer applications, SQL Managed Instance offers two types of endpoints: a VNet-local endpoint and a public endpoint.

VNet-local endpoint

The VNet-local endpoint is the default way to connect to SQL Managed Instance. The VNet-local endpoint is a domain name that has the form <managed-instance-name>.<dns-zone>.database.windows.net and resolves to an IP address from the subnet's address pool. The endpoint is local to the virtual network. You can use a VNet-local endpoint to connect a managed instance in all standard connectivity scenarios.

Public endpoint

The public endpoint is an optional domain name that has the form <mi_name>.public.<dns_zone>.database.windows.net and resolves to a public IP address that's reachable from the internet. This endpoint allows only TDS traffic to reach SQL Managed Instance and can't be used for integration scenarios like failover groups, SQL Managed Instance link, and similar technologies.

Management endpoint

To facilitate communication between the control plane and components that are deployed inside the customer's subnet, managed instances that haven't enrolled in the November 2022 feature wave use a management endpoint. Elements of the virtual network's infrastructure can harm management traffic by making the instance fail and become unavailable. Management and deployment services connect to the SQL Managed Instance management endpoint that maps to an external load balancer. Traffic is routed to the nodes only if it's received on a predefined set of ports that only the management components of SQL Managed Instance use. A built-in firewall on the nodes is set up to allow traffic only from Microsoft IP address ranges. Certificates mutually authenticate all communication between management components and the management plane.

[!NOTE] Traffic that goes to Azure services that are inside the SQL Managed Instance region is optimized. Because the traffic is optimized, traffic isn't sent by Network Address Translation (NAT) to the public IP address for the management endpoint. If you need to use IP-based firewall rules, most commonly for storage, the service must be in a different region than SQL Managed Instance.

The Azure SQL Managed Instance mandatory inbound security rules require management ports 9000, 9003, 1438, 1440, and 1452 to be open from any source on the network security group (NSG) that protects SQL Managed Instance. Although these ports are open at the network security group level, they're protected at the network level by the built-in firewall.

The management endpoint is protected by a built-in firewall on the network level. On the application level, it's protected by mutual certificate verification. When connections start inside SQL Managed Instance, like in backups and audit logs, traffic appears to start from the management endpoint's public IP address.


Virtual cluster connectivity architecture

Current architecture

Let's take a closer look at the connectivity architecture of SQL Managed Instance. The following diagram shows the conceptual layout of the virtual cluster:

:::image type="content" source="media/connectivity-architecture-overview/3-connectivity-architecture-diagram-virtual-cluster.png" border="false" alt-text="Diagram that shows the virtual cluster connectivity architecture for Azure SQL Managed Instance after November 2022.":::

Clients connect to SQL Managed Instance by using a host name that has the form <mi_name>.<dns_zone>.database.windows.net. The host name resolves to a private IP address, although it's registered in a public Domain Name System (DNS) zone and is publicly resolvable. The value for zone-id is automatically generated when you create the cluster. If a newly created cluster hosts a secondary managed instance, it shares its zone ID with the primary cluster. For more information, see Use auto failover groups to enable transparent and coordinated failover of multiple databases.

This private IP address belongs to the internal load balancer for SQL Managed Instance. The load balancer directs traffic to the SQL Managed Instance gateway. Because multiple managed instances can run inside the same cluster, the gateway uses the SQL Managed Instance host name to redirect traffic to the correct SQL engine service.

Architecture before November 2022

Let's take a closer look at the connectivity architecture of SQL Managed Instance. The following diagram shows the conceptual layout of the virtual cluster:

:::image type="content" source="media/connectivity-architecture-overview/03-connectivity-architecture-virtual-cluster.png" border="false" alt-text="Diagram that shows the virtual cluster connectivity architecture for Azure SQL Managed Instance before November 2022.":::

Clients connect to SQL Managed Instance by using a host name that has the form <mi_name>.<dns_zone>.database.windows.net. The host name resolves to a private IP address, although it's registered in a public Domain Name System (DNS) zone and is publicly resolvable. The value for zone-id is automatically generated when you create the cluster. If a newly created cluster hosts a secondary managed instance, it shares its zone ID with the primary cluster. For more information, see Use auto failover groups to enable transparent and coordinated failover of multiple databases.

The private IP address belongs to the internal load balancer for SQL Managed Instance. The load balancer directs traffic to the SQL Managed Instance gateway. Because multiple managed instances can run inside the same cluster, the gateway uses the SQL Managed Instance host name to redirect traffic to the correct SQL engine service.


Service-aided subnet configuration

To improve service security, manageability, and availability, SQL Managed Instance applies a network intent policy on some elements of the Azure virtual network infrastructure. The policy configures the subnet, the associated network security group, and the route table to ensure that the minimum requirements for SQL Managed Instance are met. This platform mechanism transparently communicates networking requirements to users. The policy's main goal is to prevent network misconfiguration and to ensure normal SQL Managed Instance operations and service-level agreement commitment. When you delete a managed instance, the network intent policy is also removed.

A service-aided subnet configuration builds on top of the virtual network subnet delegation feature to provide automatic network configuration management and to enable service endpoints.

You can use service endpoints to configure virtual network firewall rules on storage accounts that keep backups and audit logs. Even with service endpoints enabled, customers are encouraged to use Azure Private Link to access their storage accounts. Private Link provides more isolation than service endpoints.

[!IMPORTANT] Due to control plane configuration specificities, a service-aided subnet configuration doesn't enable service endpoints in national clouds.

Network requirements

The subnet in which SQL Managed Instance is deployed must have the following characteristics:

  • Dedicated subnet: The subnet SQL Managed Instance uses can be delegated only to the SQL Managed Instance service. The subnet can't be a gateway subnet, and you can deploy only SQL Managed Instance resources in the subnet.
  • Subnet delegation: The SQL Managed Instance subnet must be delegated to the Microsoft.Sql/managedInstances resource provider.
  • Network security group: A network security group must be associated with the SQL Managed Instance subnet. You can use a network security group to control access to the SQL Managed Instance data endpoint by filtering traffic on port 1433 and ports 11000-11999 when SQL Managed Instance is configured for redirect connections. The service automatically provisions rules and keeps them current as required to allow uninterrupted flow of management traffic.
  • Route table: A route table must be associated with the SQL Managed Instance subnet. You can add entries to the route table to route traffic that has on-premises private IP address ranges as a destination through the virtual network gateway or virtual network appliance. The service automatically provisions entries and keeps them current as required to allow uninterrupted flow of management traffic.
  • Sufficient IP addresses: The SQL Managed Instance subnet must have at least 32 IP addresses. For more information, see Determine the size of the subnet for SQL Managed Instance. You can deploy managed instances in the existing network after you configure it to satisfy the networking requirements for SQL Managed Instance. Otherwise, create a new network and subnet.
  • Allowed by Azure policies: If you use Azure Policy to prevent resource creation or modification in a scope that includes a SQL Managed Instance subnet or virtual network, your policies must not prevent SQL Managed Instance from managing its internal resources. The following resources need to be excluded from policy deny effects for normal operation:
    • Resources of type Microsoft.Network/serviceEndpointPolicies, when resource name begins with \_e41f87a2\_
    • All resources of type Microsoft.Network/networkIntentPolicies
    • All resources of type Microsoft.Network/virtualNetworks/subnets/contextualServiceEndpointPolicies
  • Locks on virtual network: Locks on the dedicated subnet's virtual network, its parent resource group, or subscription, might occasionally interfere with SQL Managed Instance management and maintenance operations. Take special care when you use resource locks.
  • Replication traffic: Replication traffic for auto-failover groups between two managed instances should be direct and not routed through a hub network.
  • Custom DNS server: If the virtual network is configured to use a custom DNS server, the DNS server must be able to resolve public DNS records. Using features like Azure AD authentication might require resolving more fully qualified domain names (FQDNs). For more information, see Resolving private DNS names in Azure SQL Managed Instance.

Mandatory security rules with service-aided subnet configuration

Current architecture

To ensure inbound management traffic flow, the rules described in the following table are required. The rules are enforced by the network intent policy and don't need to be deployed by the customer. For more information about the connectivity architecture and management traffic, see High-level connectivity architecture.

Name Port Protocol Source Destination Action
healthprobe-in Any Any AzureLoadBalancer subnet Allow
internal-in Any Any subnet subnet Allow

To ensure outbound management traffic flow, the rules described in the following table are required. For more information about the connectivity architecture and management traffic, see High-level connectivity architecture.

Name Port Protocol Source Destination Action
AAD-out 443 TCP subnet AzureActiveDirectory Allow
OneDsCollector-out 443 TCP subnet OneDsCollector Allow
internal-out Any Any subnet subnet Allow
StorageP-out 443 Any subnet Storage.primaryRegion Allow
StorageS-out 443 Any subnet Storage.secondaryRegion Allow

Architecture before November 2022

To ensure inbound management traffic flow, the rules described in the following table are required:

Name Port Protocol Source Destination Action
management 9000, 9003, 1438, 1440, 1452 TCP SqlManagement MI SUBNET Allow
9000, 9003 TCP CorpnetSaw MI SUBNET Allow
9000, 9003 TCP CorpnetPublic MI SUBNET Allow
mi_subnet Any Any MI SUBNET MI SUBNET Allow
health_probe Any Any AzureLoadBalancer MI SUBNET Allow

To ensure outbound management traffic flow, the rules described in the following table are required:

Name Port Protocol Source Destination Action
management 443, 12000 TCP MI SUBNET AzureCloud Allow
mi_subnet Any Any MI SUBNET MI SUBNET Allow

Mandatory routes with service-aided subnet configuration

Current architecture

The routes that are described in the following table are necessary to ensure that management traffic is routed directly to a destination. The routes are enforced by the network intent policy and don't need to be deployed by the customer. For more information about connectivity architecture and management traffic, see High-level connectivity architecture.

Name Address prefix Next hop
AzureActiveDirectory AzureActiveDirectory Internet*
OneDsCollector OneDsCollector Internet*
Storage.primaryRegion Storage.primaryRegion Internet*
Storage.secondaryRegion Storage.secondaryRegion Internet*
subnet-to-vnetlocal subnet Virtual network

[!NOTE] * The Internet value in the Next hop column instructs the gateway to route the traffic outside the virtual network. However, if the destination address is for an Azure service, Azure routes the traffic directly to the service over the Azure network instead of outside the Azure cloud. Traffic between Azure services doesn't traverse the internet, regardless of which Azure region the virtual network exists in or which Azure region an instance of the Azure service is deployed to. For more information, see Azure virtual network traffic routing.

You also can add entries to the route table to route traffic that has on-premises private IP ranges as a destination through the virtual network gateway or virtual network appliance.

Architecture before November 2022

The routes that are described in the following table are necessary to ensure that management traffic is routed directly to a destination:

Name Address prefix Next hop 2
subnet-to-vnetlocal MI SUBNET 1 Virtual network
mi-azurecloud-REGION-internet AzureCloud.REGION Internet
mi-azurecloud-REGION_PAIR-internet AzureCloud.REGION_PAIR Internet
mi-azuremonitor-internet AzureMonitor Internet
mi-corpnetpublic-internet CorpNetPublic Internet
mi-corpnetsaw-internet CorpNetSaw Internet
mi-eventhub-REGION-internet EventHub.REGION Internet
mi-eventhub-REGION_PAIR-internet EventHub.REGION_PAIR Internet
mi-sqlmanagement-internet SqlManagement Internet
mi-storage-internet Storage Internet
mi-storage-REGION-internet Storage.REGION Internet
mi-storage-REGION_PAIR-internet Storage.REGION_PAIR Internet
mi-azureactivedirectory-internet AzureActiveDirectory Internet

Networking constraints

TLS 1.2 is enforced on outbound connections: Beginning in January 2020, Microsoft enforces TLS 1.2 for intra-service traffic in all Azure services. For SQL Managed Instance, this resulted in TLS 1.2 being enforced on outbound connections that are used for replication and on linked server connections to SQL Server. If you use a version of SQL Server that's earlier than 2016 with SQL Managed Instance, make sure that you apply TLS 1.2-specific updates.

The following virtual network features are currently not supported with SQL Managed Instance:

  • Microsoft peering: Enabling Microsoft peering on ExpressRoute circuits that are peered directly or transitively with a virtual network in which SQL Managed Instance resides affects traffic flow between SQL Managed Instance components inside the virtual network and services it depends on. Availability issues result. SQL Managed Instance deployments to a virtual network that already has Microsoft peering enabled are expected to fail.
  • Global virtual network peering: Virtual network peering connectivity across Azure regions doesn't work for instances of SQL Managed Instance that are placed in subnets that were created before September 9, 2020.
  • AzurePlatformDNS: Using the AzurePlatformDNS service tag to block platform DNS resolution would render SQL Managed Instance unavailable. Although SQL Managed Instance supports customer-defined DNS for DNS resolution inside the engine, there's a dependency on platform DNS for platform operations.
  • NAT gateway: Using Azure Virtual Network NAT to control outbound connectivity with a specific public IP address renders SQL Managed Instance unavailable. The SQL Managed Instance service is currently limited to use the basic load balancer, which doesn't provide coexistence of inbound and outbound flows with Azure Virtual Network NAT.
  • IPv6 for Azure Virtual Network: Deploying SQL Managed Instance to dual stack IPv4/IPv6 virtual networks is expected to fail. Associating a network security group or a route table with user-defined routes (UDRs) that contains IPv6 address prefixes to a SQL Managed Instance subnet renders SQL Managed Instance unavailable. Also, adding IPv6 address prefixes to a network security group or UDR that's already associated with a managed instance subnet renders SQL Managed Instance unavailable. SQL Managed Instance deployments to a subnet with a network security group and UDR that already have IPv6 prefixes are expected to fail.
  • Azure DNS private zones with a name reserved for Microsoft services: The following domain names are reserved names: windows.net, database.windows.net, core.windows.net, blob.core.windows.net, table.core.windows.net, management.core.windows.net, monitoring.core.windows.net, queue.core.windows.net, graph.windows.net, login.microsoftonline.com, login.windows.net, servicebus.windows.net, and vault.azure.net. Deploying SQL Managed Instance to a virtual network that has an associated Azure DNS private zone that uses a name reserved for Microsoft services fails. Associating an Azure DNS private zone that uses a reserved name with a virtual network that contains a managed instance renders SQL Managed Instance unavailable. For information about Private Link configuration, see Azure Private Endpoint DNS configuration.

Next steps