Events
Power BI DataViz World Championships
Feb 14, 4 PM - Mar 31, 4 PM
With 4 chances to enter, you could win a conference package and make it to the LIVE Grand Finale in Las Vegas
Learn moreThis browser is no longer supported.
Upgrade to Microsoft Edge to take advantage of the latest features, security updates, and technical support.
Note
This article forms part of the Power BI implementation planning series of articles. This series focuses primarily on the Power BI experience within Microsoft Fabric. For an introduction to the series, see Power BI implementation planning.
This article helps you to plan and implement on-premises data gateways and virtual network (VNet) data gateways for Microsoft Fabric. It's primarily targeted at:
To access source data for Power BI semantic models, dataflows, and other Fabric data items, you might need a data gateway. A data gateway securely transfers data between private networks or on-premises data sources and cloud services, including Fabric.
Note
This article provides an overview of gateways. It focuses on key considerations and actions for planning and implementing gateways to support your Fabric content.
For more information about how gateways work, see:
Gateways are often integral to successful Power BI and Fabric implementations. A COE or central data and BI team typically plans and centrally manages gateways, although some organizations use decentralized teams to manage gateways. To mitigate the possibility of future disruption and governance risks, it's important to carefully plan how and when you'll use gateways.
You typically plan gateway implementation during two distinct stages.
Planning a gateway implementation begins by making key decisions, starting with whether you need a gateway or not.
Tip
First, create an inventory of your data sources. You should evaluate the following key decisions for each data source. Be sure to document your results and store them in a central location that's easily accessible, like a communication hub or your centralized portal.
In general, you need a gateway for your data source when:
In these situations, you need a gateway to:
The following sections describe when you need a gateway.
You need a gateway to connect to on-premises data sources from the Fabric portal. The gateway serves as a bridge, evaluating query expressions on the gateway machine and securely transferring on-premises data to the cloud.
This scenario is relevant when connecting to:
You need a gateway to connect to data sources that are located in a private network, such as an Azure Virtual Network (Azure VNet). A virtual network, or VNet, is a logically isolated segment of a network that insulates traffic from the public internet. A VNet provides enhanced network security.
This scenario is relevant when the data source:
Note
It's a common misconception that you don't need a gateway for cloud data sources. When a cloud data source resides within a private organizational network, a gateway is required.
Sometimes you might need a gateway to host supporting items that are required to connect to your data source. This software might include custom data connectors, drivers, or libraries that you install on the gateway machine. The Power BI service doesn't have access to this software, so it's unable to refresh data sources that use it without relying on the gateway—even when you're connecting to a cloud data source.
This scenario is relevant when you connect to a data source with connectors such as a:
Important
When content creators use a client tool such as Power BI Desktop and their solutions rely on drivers, connectors, or providers, you should install the same components on the gateway machine as are installed on the content creators machines. Missing or mismatched components between creator machines and data gateways is a common reason for data refresh failures of published content. For more information, see User tools and devices.
You need a gateway to use certain Power Query connectors or functions, such as the Web connector or Web.BrowserContents function. These connectors and functions require a gateway for many reasons, including security isolation.
Tip
Consider the following alternatives when connecting to web page data sources.
This scenario is relevant when you connect to a data source by using connectors and drivers such as:
When you've identified that you need a gateway, you should next determine which type of gateway to install. There are three types of gateway.
The gateway type you choose will depend on your requirements and the data sources. The following sections describe each of the three gateway types.
An on-premises data gateway (standard mode) allows multiple users to connect to data sources that through a single shared gateway. Typically, you centrally install and manage standard mode gateways on an always-on VM. With a standard mode gateway, you can connect to data from multiple services, such as Fabric, Power BI, and other Power Platform services.
The following diagram depicts a high-level overview of a standard mode gateway.
Important
This diagram doesn't depict the architecture of an on-premises data gateway.
The diagram depicts the following concepts.
Item | Description |
---|---|
The on-premises data gateway (standard mode) securely transfers data from on-premises data source to cloud services. | |
A standard mode data gateway is required for cloud data sources in specific scenarios (described in the previous section). | |
The standard mode data gateway is installed on an always-on VM. Administrators centrally manage the VM and data gateway. Gateway administrators install software required for data source connections, if necessary. | |
Multiple users can connect to the data gateway data sources. | |
Users can use the data gateway for items published to Fabric workspaces, like semantic models, dataflows, pipelines, or paginated reports. | |
Users can use the data gateway for other Power Platform cloud services, like Power Platform dataflows. |
A standard mode gateway is required in the following specific situations.
An on-premises gateway (personal mode), commonly known as a personal gateway, allows a user to connect to on-premises data sources that reside on the same machine. A user typically installs and manages a personal gateway from their own machine. With a personal gateway, users can't connect to data from other Power Platform services. They also can't share the gateway or connections with other users.
A personal gateway is intended for limited, personal use by a single individual. Typically, content creators install and use these gateways to conduct personal BI. These gateways are limited to personal BI because they can't be shared. Additionally, a personal gateway requires that the user has machine rights and policy approval to download and install the personal gateway software.
Tip
Don't use a personal gateway with team, departmental, or enterprise BI solutions.
For most scenarios where you connect to on-premises data, you should use a gateway in standard mode (described in the previous section). That's because you can share a standard mode gateway with multiple users, it supports DirectQuery queries and live connections, and it has more options to centralize gateway governance and management.
Caution
Because a personal gateway is typically installed on a user machine, it is more difficult to manage and govern. If you do need to use a personal gateway, consider moving it to a centrally-managed VM that uses a service account. This approach ensures gateway availability isn't dependent on a user machine (which might be turned off) and improves gateway governance and management.
The following diagram depicts a high-level overview of a personal gateway.
Important
This diagram doesn't depict the architecture of an on-premises data gateway.
The diagram depicts the following concepts.
Item | Description |
---|---|
A personal gateway is typically installed on a user machine. | |
The personal gateway securely transfers data from local data sources on the user machine to cloud services. | |
The personal gateway is typically managed by the user who installed it. | |
A single user uses the personal gateway for limited, personal use. A personal mode gateway can't be shared. | |
A personal gateway can only be used for items published to a Power BI workspace, like semantic models or Power BI dataflows. |
To reiterate, a personal gateway is intended for limited, personal use by a single individual. However, there are two specific scenarios that require you to use a personal gateway.
Tip
Whenever possible, avoid using a personal gateway. Instead, consider the following alternatives.
A VNet gateway allows multiple users to connect to data sources secured with private networks, including data sources that use private endpoints. With a VNet gateway, you can connect to data with multiple services, and you can share the gateway or connections with multiple users.
A VNet gateway is a Microsoft managed service. If your organization uses private networks, you need a VNet gateway.
Important
If you're considering using the VNet gateway service, discuss it with your IT teams that handle networking and security. These teams can help ensure that everything is set up, such as private endpoints (if applicable) and gateway communication.
A VNet gateway is only supported for Power BI Fabric or Premium capacities. The VNet gateway is billed as an additive premium infrastructure charge for that capacity.
Important
At times this article refers to Power BI Premium or its capacity subscriptions (P SKUs). Be aware that Microsoft is currently consolidating purchase options and retiring the Power BI Premium per capacity SKUs. New and existing customers should consider purchasing Fabric capacity subscriptions (F SKUs) instead.
For more information, see Important update coming to Power BI Premium licensing and Power BI Premium FAQ.
The following diagram depicts a high-level overview of a VNet gateway.
Important
This diagram doesn't depict the architecture of a VNet data gateway.
The diagram depicts the following concepts.
Item | Description |
---|---|
You use a virtual network (VNet) gateway is to connect to data sources on a private network, like those in an Azure VNet. | |
The VNet data gateway is a Microsoft managed service. You centrally manage the VNet data gateway from the Azure portal and the Power Platform admin portal. | |
Multiple users can use a VNet data gateway. | |
Users can use a VNet data gateway for items published to a Fabric workspace, like semantic models. | |
Users can use a VNet data gateway for other Power Platform services, like Power Platform dataflows. |
Warning
VNet gateways have some limitations, and they don't support all data sources or scenarios. Validate that your data sources and scenarios are supported and consult the frequently asked questions before you proceed with VNet gateway implementation and solution planning.
When you've determined that you need a gateway, and which type of gateways, you should next determine how many gateways you need.
Depending on your needs, you might require multiple gateways. Consider the following factors when deciding how many gateways to install and use.
It's important that gateways have high availability to avoid disruption caused by refresh or query delays. One way to ensure gateway availability is to install multiple gateways in a high-availability gateway cluster. A gateway cluster is a collection of gateways that you install on different VMs, and that are logically associated together as a single, functional unit (the cluster). Each gateway machine is sometimes called a node.
Here are the benefits of using a gateway cluster.
Important
We strongly recommend that you use gateway clusters for business-critical workloads.
For more information and guidance on setting up a gateway cluster, see Plan, scale, and maintain a business-critical gateway solution..
Content creators typically use separate environments to develop and manage business-critical solutions, like development, test, and production. Depending on the number of environments you use and how you use them, you might want to have separate gateway clusters for each environment.
Separating gateway clusters into different environments can:
Important
We recommend that you have separate gateway clusters for production workloads. If you have one gateway cluster across all environments, that can represent additional risk. To minimize cost and management effort, it's common to allocate fewer resources (such as memory and CPU) to a development gateway cluster.
To ensure good performance of data refreshes, it's important that you consider the location of your data sources, gateways, and where your users are located. To reduce latency, you should install gateways as close to your data sources as possible. For this reason, you might need to install multiple gateway clusters to support different regions or tenants.
Caution
Ensure that your gateway installation complies with any data residency requirements for your organization.
Important
To minimize latency, we recommend that you install gateways on machines that are in the same region as your data sources. Additionally, for VNet gateways, the gateways and data sources should be on the same subnet.
Checklist - When planning a gateway implementation, key decisions and actions include:
At this point, you know which types of gateway you need and how many. Next, you need to plan for the gateway installation. Typically, gateways are installed on VMs that you dedicate to this purpose (often called gateway machines). Each machine in the gateway cluster should always be on to ensure continuous support of user activity and data refresh operations.
Note
Because a VNet gateway is a managed service, you don't download and install it. Instead, you provision and set up a VNet gateway in your Azure portal, and then bind it to a Fabric or Power BI Premium capacity. For more information, see Create virtual network data gateways.
Before installing the gateway, identify who will install and own the gateway.
Typically, the gateway owner is a technical person who installs, owns, and manages the gateway. Gateway owners are responsible for various activities.
Important
Ensure that the gateway owner is aware of, and agrees with, these responsibilities. If the gateway owner isn't prepared to manage the gateway, it can quickly become a dependency that blocks content owners and creators. Additionally, identify whether the gateway owner understands how to install and manage a gateway and, if not, how you'll train them to do this.
Tip
Some organizations successfully allow gateway ownership within business units and departments, whereas others reserve gateway ownership for a centralized team (such as IT). One way to handle it is to form a partnership where IT manages the gateway cluster nodes, and the business unit manages the data source connections.
Because gateway ownership is an important responsibility, you should clearly define who can install gateways in your organization.
To reduce management overhead and mitigate governance risk, it's important to limit the number of active gateways in your organization. To this end, we recommend that you limit the number of users who can install gateways.
Warning
Gateway owners have full control over the gateways that they manage. This means that malicious gateway owners could potentially intercept information as it flows through an on-premises data gateway. For this reason, it's essential that you restrict the ability to install gateways to trusted people.
For standard mode gateways, you manage gateway installers either from the Fabric portal or the Power Platform admin center. You also manage who can create VNet data gateways by using the gateway installer setting.
You can also manage gateway installers programmatically by using the PowerShell cmdlets for on-premises gateway management. For personal gateways and standard mode gateways, you can use these cmdlets to set the gateway tenant policy. Setting the gateway tenant policy by using PowerShell is the only way to manage who can install personal gateways in your tenant.
Important
We recommend that you closely regulate who can install personal gateways, limiting their installation and use for valid, approved business cases.
When you've identified who will install and own the gateway, you should prepare for the gateway installation. You should:
The following sections describe these key considerations for planning a gateway installation.
Typically, you install a gateway on an always-on VM (also called the gateway machine). You can install only one gateway of each type (personal mode or standard mode) on a machine.
Here are the key factors for determining where you'll install a gateway.
Tip
To avoid resource contention, don't install unrelated software on a gateway machine. The gateway machine should be fully dedicated to hosting the on-premises data gateway.
The gateway machine should have sufficient resources to handle the expected query workload.
Here are the key factors for determining gateway machine resources.
Tip
Validate the gateway machine resources by performing load testing. You can conduct this type of test by monitoring gateway machine health when performing dataset refreshes, and by simulating high concurrent usage of DirectQuery or live connection reports.
How you'll name the gateway and its data source connections is important. The name should make it straightforward for content creators to know what to connect to. To ensure gateways and data source connections have clear names, you should use a logical naming convention.
When you define your naming conventions, consider the following points.
Here are some examples of logical gateway names.
After making key decisions and preparation, the gateway owner installs the gateways and performs first-time setup.
Note
For information on how to download and install a gateway, see:
When installing and setting up gateways, consider the following factors.
Important
We recommend that you restrict tenant registration to only tenants within the organization. This step helps to improve gateway security because the default setting has no restriction on tenant registration.
Checklist - When preparing for and installing a gateway, key decisions and actions include:
After installing the gateways, you should then add data source connections. When adding these connections, you should also plan how you'll manage access to the gateway and its connections.
You must add the initial data source connections before you can use the gateway. You can add connections manually from the Power BI service or the Power Platform admin center, or programmatically with the Power BI REST APIs.
When adding connections, consider the following points.
Note
The data source name can be modified afterwards, but the server and database names can't be changed after they've been setup. To avoid errors, ensure that the data source information matches what will be used in Power BI Desktop.
Tip
To improve efficiency and accuracy, consider automating the creation of data source connections by using the Power BI REST APIs. In this case, we recommend including review and approval processes rather than automatically processing each request that creates or updates a connection.
After adding the initial data source connections, you should decide on how to manage access to both the gateway and its connections.
Content creators will need access to a gateway connection to successfully connect to a data source. User access to gateway connections is done for each connection, so consider who needs access to each gateway connection, and how you'll manage that access. You should manage access by using security roles for both gateways and connections.
Gateway roles let you control who can manage the gateway and its data source connections. These roles work similarly to workspace roles, allowing different permissions depending on the role. Using roles helps you to manage gateway access more effectively.
Tip
We recommend using security groups to manage role membership instead of individual accounts. That way, it's easier to manage users, particularly across multiple gateways. You can use the same security groups to manage other access control, like row-level security role membership and app audience membership.
Important
A user who just needs to use the gateway to connect to a data source doesn't need to belong to a gateway role. In this case, they'll only have the User connection role.
There are three gateway roles for managing an on-premises standard gateway.
Note
VNet gateways only support the Admin gateway role.
Data source connection roles let you control who can use, manage, and share connections. A user with a connection role doesn't need to belong to a gateway role.
There are three data source connection roles.
Tip
To prevent governance risk from oversharing, you should limit who can share gateways and connections to specific individuals who can accomplish this task effectively and responsibly.
After you've set up your gateway cluster, you should document it. You should document your gateways so that they're easy for content creators to find, and easy for gateway administrators to maintain. Consider storing gateway documentation in an accessible location, like your centralized portal of the relevant community of practice.
Consider documenting the following information.
Important
Ensure that you document your gateway recovery keys for standard mode gateways. These recovery keys are required should you ever need to recover or re-locate the gateway. Keep this information in a safe and secure place that's accessible to multiple trusted people in a central team. If you have an organizational password vault, that's an ideal location.
To ensure your gateways remain functional and perform well, you need to perform several tasks.
Note
The gateway owner must manually apply gateway updates to each gateway. For this reason, it's important to plan a process to update your gateways periodically.
Tip
When you work with the Power Query Online experience, the Power Query engine uses the latest version of Power Query that's available. However, when you're using a gateway to apply transformations, it uses the version that's installed on the gateway machine. To ensure a consistent user experience, it is important to keep your gateway machines up to date.
The remainder of this section describes how to update the gateway software.
It's important to keep your gateways up to date to avoid unexpected disruption, and also to benefit from the latest improvements. Nonetheless, updates can have unexpected consequences for your gateway performance and function. To avoid impacting business-critical solutions, you should first install gateway software updates to a development or test gateway (before they're applied to gateways that support production environments).
You can test gateways by first applying the update to gateways that support development and test environments.
Consider the following points when validating gateway updates.
Important
We strongly recommend that you test gateway updates on the development and test cluster before applying them to production. Testing updates is important, as there's no rollback process. As an alternative, before starting the update you can create a VM image, which is a complete copy of the file system structure and the data on the machine.
After validating the gateway update, you should apply the update to all gateways that support production environments. A gateway is unavailable while it's being updated, so you should be consistent in when and how you update your gateways.
Consider the following points about updating gateways.
For data source connections that require stored credentials, you might need to rotate the credentials regularly. For example, your organization might have a policy that requires frequent password resets. This practice is also useful when a key team member leaves the organization. To improve efficiency, you can use the Power BI REST APIs to update credentials.
Checklist - When managing data gateways, key decisions and actions include:
Gateways are a crucial part of Fabric data integration. To avoid disruption and mitigate risk, you should monitor and regularly audit the gateways in your tenant.
Monitoring gateways helps to:
The following table provides an overview of the typical issues you might encounter when managing a data gateway cluster, and how to monitor or investigate issues.
Potential issue | Type of issue | How to monitor or investigate the issue |
---|---|---|
Too many gateways | Governance | • Power Platform admin center • Gateway PowerShell cmdlets • Power BI REST APIs |
Gateway oversharing | Governance | • Power Platform admin center • Gateway PowerShell cmdlets • Power BI REST APIs • Power BI activity log |
Gateway is offline | Performance and availability | • Power Platform admin center • Gateway machine monitoring • Gateway logs |
Gateway errors | Performance and availability | • Gateway logs |
Query failures | Performance and availability | • Gateway logs • Gateway logs (additional logging) |
Slow refreshes or queries | Performance and availability | • Gateway machine monitoring • Windows event log • Windows performance counters • Gateway logs • Gateway logs (additional logging) • Server monitoring tools |
Note
For more information about auditing and monitoring gateways, see:
Tip
If you use Fabric capacity, the tools in Fabric can provide the ideal components for you to build and orchestrate an organizational gateway monitoring solution. For instance, you can:
You should regularly review how many gateways are installed on your tenant, and who installed them. You can monitor prevalence from the connections and gateways page of the Fabric portal, and the Power Platform admin center. Both views provide a concise, functional overview of all the gateways that you have access to. Administrators should review this information regularly.
Note
You can also programmatically retrieve a list of gateways by using the PowerShell cmdlets or by using the Power BI REST APIs. You can also identify gateway installation events by using the activity log.
Consider combining this information with aggregate analytics about the number and type of gateway data sources. You can present this information in consolidated, tenant-wide governance or auditing and monitoring reporting.
When monitoring gateway prevalence, focus attention on the following metrics.
Each gateway machine produces detailed logs that you can use to identify and resolve issues. These logs are a collection of detailed technical files stored on the gateway machine. You can also temporarily enable additional logs from the on-premises gateway app to collect more detailed information about queries and their timings.
To avoid introducing networking delays, we recommend that you allow the gateway logs to be written to the local gateway machine. However, you can choose to change the path where the logs are written with the gateway configuration files. You can also update how long the logs are retained. Always make a copy of the configuration files before you edit them.
Create a data integration solution to gather and consolidate the log files from each gateway machine so you can analyze the data. Ideally, this process should be automated and output to an analytical report to easily view all gateways and identify anomalies.
Tip
Consider using data alerts to notify gateway administrators and data source connection creators about anomalous activity for their gateways and connections, respectively. That way, they can take immediate corrective actions.
The health of your gateway is dependent on the health of its server. To avoid disruption, you should monitor gateway machines to detect when a machine isn't performing well or is offline.
Tip
Ensure that your gateway machine is added to any enterprise monitoring tools that your organization uses to monitor servers.
When issues arise with a gateway, you'll need to investigate and identify the problem. Typically, troubleshooting involves investigating the gateway logs described in the previous section, and testing various optimizations to see whether they resolve the issue.
Here are some common gateway optimizations.
Important
VNet gateways have a single hardware configuration, which can't be scaled or changed.
Note
For more information about optimizing and troubleshooting gateways, see:
Checklist – When monitoring gateways, key decisions and actions include:
For more considerations, actions, decision-making criteria, and recommendations to help you with Power BI implementation decisions, see Power BI implementation planning.
Events
Power BI DataViz World Championships
Feb 14, 4 PM - Mar 31, 4 PM
With 4 chances to enter, you could win a conference package and make it to the LIVE Grand Finale in Las Vegas
Learn moreTraining
Learning path
Solution Architect: Design Microsoft Power Platform solutions - Training
Learn how a solution architect designs solutions.
Documentation
On-premises data gateway - Power BI
This article provides an overview of the on-premises data gateway and its functionality in Microsoft cloud services.
Add or remove a gateway data source - Power BI
Learn how to add or remove data sources to an on-premises gateway in Power BI. Get tips for managing your data sources efficiently.
Install an on-premises data gateway
Learn how to install a gateway so you can connect to on-premises data.