Once more unto the breach

Developing your own PAW for cloud administration

John Rodriguez (MSFT) — Tue, 21 Jun 2016 11:21:25 GMT

In the Privileged Access Workstation guidance, we provide instructions on how to enable cloud services (it’s Phase 2, step 5, marked as optional). In that section we detail the use of the proxy.pac file, which includes a list of Microsoft Azure-related URLs. But what if you use another service, like Amazon Web Services or RackSpace? In this article, I’ll show you a very simple method to discover those URLs and add them to your own bypass list. Before I explain the method, I want to explain how we secure the system to allow access to trusted Internet sites without allowing access to untrusted ones. Our public guidance relies on the deployment of a proxy.pac file, which includes a list of the Microsoft Azure URLs, and instructs the browser to allow the user to browse to them. Any site which does not match those patterns is redirected to 127.0.0.2:8080 – in other words, it’s redirected to a non-existent location. However, there’s another way of achieving this same goal – you can bypass the proxy by adding sites to the local Intranet zone, as documented in the TechNet article, “Using Proxy Selection and Proxy Bypass Lists” (https://technet.microsoft.com/en-us/library/dd361953.aspx). The ProxyBypassList.txt file in the PAW content download includes all of the same (Microsoft Azure) properties as the proxy.pac file, and was written to work with the UserProxySettings.adm file. [The proxy.pac and ProxyBypassList.txt files are both available for download from the TechNet Gallery at https://aka.ms/pawcontent.] If you want to connect to a different cloud service, here’s how to generate your own ProxyBypassList.txt. Ideally you should complete all of these steps from a fully-configured PAW, but you can perform them on any system.

Close all other applications and services which may make Internet connections or queries.
Open your web browser of choice and enter the URL for your cloud property, but do not navigate to it yet.
Open an elevated command prompt and run the following command:
```
ipconfig /flushdns
```
This will clear the DNS Resolver Cache of any existing entries [caption id="attachment_175" align="alignnone" width="523"] Figure 1: Clear the DNS Resolver Cache before you connect to the cloud service.[/caption]
In your web browser, connect to your cloud service and, if necessary, provide your credentials to log into the portal.
In the command prompt, run the following command as many times as necessary (see Note below):
```
ipconfig /displaydns | find /v “:” | find “.” | find /v “Name does not exist.” | find /v “alpineskihouse.com” >> clouddnsnames.txt
```
Be sure to replace alpineskihouse.com with your internal domain name (not the cloud service name, but your internal DNS domain name). Here's what the output looks like when not redirected to the text file.

[caption id="attachment_185" align="alignnone" width="668"] Figure 2: Using the IPConfig command to capture and filter the list of URLs.[/caption]

Edit the temp.txt file to match the format in ProxyBypassList.txt (see below).

The first time you run the command in step 5, you'll generate a new clouddnsnames.txt which includes the appropriate entries from the (newly-repopulated) DNS resolver cache. This new file will include entries with periods, but not those which include a colon, the string “Name does not exist.” or your internal domain name. You will then need to edit the clouddnsnames.txt file to match the format of the ProxyBypassList.txt – remove all extra spaces, separate each entry with a semicolon, and add wildcards when appropriate to whitelist domains. Once you’ve done that, you can use the UserProxySettings.adm file to deploy your newly-generated list to all of your PAWs. Note: As you build your list of URLs, you will most likely need to issue the command in step 5 more than once (which is why we append to temp.txt, rather than simply overwrite it). This is not only because the time-to-live (TTL) values for many cloud service DNS entries are very short, but also because different URLs are accessed from different sections of the portal. We recommend that you perform all of the typical activities for managing your cloud properties (starting virtual machines, accessing blob storage, creating users, and so on) during your session to ensure that you capture all required DNS names. As you can see below, the URLs in the DNS Resolver Cache can differ from moment to moment - the two ipconfig commands below were run within 15 seconds of one another! [caption id="attachment_195" align="alignnone" width="670"] Figure 3: The list of URLs in the DNS Resolver Cache will vary depending on recent operations; make sure you check regularly to capture all appropriate URLs.[/caption] By following the instructions above, you can generate a ProxyBypassList.txt for the cloud service of your choice, and deploy it to your PAWs using the UserProxySettings.adm file. Secure cloud administration awaits! Acknowledgements: I want to thank Jared Poeppelman, who not only developed this procedure, but also provided the original ProxyBypassList.txt which we made available as part of the PAW content. I’d also like to thank Amy Hariharan Dang, who developed and contributed the UserProxySettings.adm which accompanies the ProxyBypassList. Thank you both!

Closing the Jump Server Security Gap with PAWs

John Rodriguez (MSFT) — Thu, 16 Jun 2016 11:19:18 GMT

When I’m out and about discussing Privileged Access Workstations with customers, I hear one question more than any other: Do we need to use PAWs if we use “jump servers”? The answer is a resounding YES! In this blog entry, I’ll explain why – the short version is that while jump servers can be valuable, they don’t actually increase your security as you might otherwise expect. But the resulting security gap can be easily closed with, yes, a Privileged Access Workstation. First, a definition – a jump server is, according to Wikipedia, “a (special-purpose) computer on a network typically used to manage devices in a separate security zone.” If you’ve read our guidance on Privileged Access Workstations, you may have noticed that we devote a few paragraphs to the jump server question. I’m going to expand on those paragraphs a bit here in order to really land the main point – that you need PAWs to make jump servers truly secure. Let’s start with a question: Why are jump servers vulnerable? Simply put, if you can’t guarantee the security of the source, you can’t guarantee the security of any system to which you connect from that source. It’s known as the clean source principle, and it’s at the very heart of many of Microsoft’s cybersecurity offerings (including, obviously, the Privileged Access Workstation). If you, as an end user, connect to the Internet or open email on a workstation, you increase the risk of that system being compromised – and once compromised, an attacker can compromise any other activity on that system via credential theft tools, keystroke loggers, or screen recording software. [It’s worth reinforcing this point – if the attacker compromises any administrative user on the workstation, s/he can easily install software that compromises any other user who logs onto that system.] If the attacker is on my workstation, and I connect to the jump server using Remote Desktop, that attacker now has potential access to everything I can do on that jump server – and to the systems connected to the jump server. This is true even if you configure the jump server to require multi-factor authentication – because MFA exposes reusable credentials on the source system, the attacker can compromise those and reuse them at will. [caption id="attachment_215" align="alignnone" width="876"] Figure 1: An attacker who compromises an end user workstation can hijack the jump server thanks to credential theft.[/caption] But all hope is not lost! The jump server model doesn’t provide additional security, but a PAW does. Imagine now that we connect to the jump server not from a general-purpose end user workstation, but from a PAW. Now, when I log in using my administrative credentials, I have a high assurance that an attacker cannot access them – and when I connect to the jump server using Remote Desktop, the attacker can’t compromise my credentials there either (because the attacker doesn’t have access to credentials which allow him or her to access that system). By isolating the credentials on a PAW, we render the attacker powerless to compromise them, and thus the connection to the jump server is protected. [caption id="attachment_225" align="alignnone" width="880"] Figure 2: By connecting to an administrative jump server via a PAW, the administrator avoids exposing privileged credentials to the attacker.[/caption] Does this mean that jump servers are redundant? Not at all! Although they don’t provide meaningful security assurances on their own, they extend and enhance the PAW model in two primary ways. Here’s how:

By deploying jump servers, you can install all required management software on it rather than the PAWs. This helps minimize the PAW footprint (maintaining high security), and reduces the number of systems where you must maintain and update management software. You’ll probably have fewer jump servers than PAWs, so jump servers can help reduce the operational cost of managing PAWs themselves.
Jump servers should be relatively static objects from a network perspective, unlike PAWs (which are often mobile devices, like tablets or laptops, which connect to different IP addresses at different times). If you know the IP addresses for your jump servers, you’ve just identified valid sources of administrative traffic. You can then either block administrative traffic from unauthorized IP addresses, generate network alerts for it, or both.

Jump servers do provide value, but to really get the most out of them, you need to combine them with other security measures – namely, PAWs. By securing all stages of the administrative path – from source to destination – you can ensure that your privileged credentials are never exposed to attackers. Addendum: One additional note about using Remote Desktop with jump servers – I’ve previously written about RDP RestrictedAdmin, and I want to reproduce my cautionary note about RDPRA here: As an aside, RDPRA is a poor fit in general for “Jump Servers” scenarios where users connect to an RDP server and from there connect to other resources. This is because credentials are never provided directly to the target system (jump server) and any additional connection you try to make from that jump server requires you to enter new credentials. To expand on this a bit – if everyone who connects to a jump server uses a PAW, we should have a high degree of confidence that the jump server itself is secure. If that’s the case, then there’s no additional risk in providing reusable credentials directly to the jump server. And if there were an attacker on the system and you used RDPRA, you’d need to provide them anyway to connect to other systems, which would expose them to an attacker on the jump server, thus eroding the value of RDPRA in the first place. So, in the end, using RDPRA with a jump server adds complexity but doesn't add any additional security - a good reason to simply use traditional RDP when connecting to a jump server from a PAW.

Why You Should Use RDP RestrictedAdmin

John Rodriguez (MSFT) — Tue, 14 Jun 2016 09:57:18 GMT

When Mark Simos and I documented Microsoft’s public guidance for deploying a Privileged Access Workstation, one of the features we leveraged is Remote Desktop’s /RestrictedAdmin feature (RDPRA for short). This feature, one of many introduced in the massive KB2871997 update (and its siblings), allows administrators to connect to remote systems without exposing their credentials to the target system. I can’t stress how important this change is – an administrator who connects using “normal” RDP exposes his or her credentials to the remote system with every connection. RDPRA, on the other hand, ensures that credentials aren’t exposed to the attacker on the remote computer being managed. RDPRA remains a hidden gem, so in this blog I want to demystify it a bit and reinforce why you too should use it any time you remotely manage a device (especially devices which are less secure than the one from which you’re connecting). Ultimately, I want everyone to use RDPRA every time you use Remote Desktop for administration (including connections to servers and workstations). [caption id="attachment_155" align="alignnone" width="289"] Figure 1: The modified RDP RestrictedAdmin title bar.[/caption] Remote Desktop has long been an administrator’s best friend, allowing interactive access even if the target system is on the other side of the world. Assuming that the target system allows remote access and the requesting user has been granted the right to log on via Remote Desktop, the remote user connects to the target system using a Type 10 Remote Interactive logon request across TCP 3389 (the default RDP port). [There’s a bit more to it if you require Network Level Authentication, but the basics are still the same.] If you peek into the local security agent (LSASS) with common attack tools on the target system, you’ll see that the remote user’s credentials now appear in memory. This is the source of the security risk with normal RDP – if the credentials are on the target system, they’re exposed to theft by an attacker on that system. [caption id="attachment_145" align="alignnone" width="300"] Figure 2: WCE output showing credentials captured during a normal (non-RDPRA) Remote Desktop session[/caption] Here’s an example of how an attacker would exploit this:

Alice is a user at Contoso, and she uses her workstation to browse the Internet and read email. She falls victim to a phishing attack and inadvertently downloads malware to her workstation.
Bob, the attacker, now has access to all identities on that system, but at this point Alice is the only person who’s logged into the workstation.
Bob wants the lure privileged users into logging into Alice’s workstation, so he deletes files on Alice’s system that will break Outlook.
The next morning, Alice can’t start Outlook, and calls the help desk for assistance. Alice is based in New York, but the help desk agent, Carlos, is based in Mumbai – too far for an in-person visit, obviously, but that distance is nothing to Remote Desktop!
Carlos logs into the workstation using RDP to troubleshoot the problem, and as a result his credentials now appear in LSASS memory.
Bob has been waiting for just this opportunity, and he grabs Carlos’ reusable credentials and uses them to connect to systems which only Carlos can access. The initial compromise has now turned into a wider breach.

[caption id="attachment_135" align="alignnone" width="739"] Figure 3: This attack timeline shows how an attacker can compromise credentials provided via traditional Remote Desktop.[/caption] Why does this work? From a technical perspective, Remote Desktop connections are functionally equivalent to those issued when an end user is sitting directly in front of the target system. A Type 10 Remote Interactive logon is indeed an interactive logon – the end user has authenticated to Active Directory and the local security agent (LSASS) on the target system now has a copy of that user’s credentials, which can be used by a malicious individual to impersonate the remotely-connected user. [For more on credential theft, please see Mark’s excellent video at https://aka.ms/credtheftdemo.] So if connecting remotely exposes our credentials like this, how do we securely administer remote systems? Enter RDPRA. When you connect using the /RestrictedAdmin switch, Remote Desktop uses a modified Remote Interactive logon through which the user is authenticated, but reusable credentials are never sent to the target system. Because the administrator signs on with network authentication, the target system receives only a Kerberos service ticket or the NTLM challenge responses, rather than the user’s actual credentials – which on the whole is a very good thing. In our example above, Carlos used normal RDP to connect, and thus exposed his credentials (username and password, or provided access to his smart card). If Carlos instead used RDPRA, neither his username/password combination nor derived credentials would ever appear in LSASS memory, and Bob wouldn’t be able to even see them, let alone steal them. It’s no accident that we recommend that administrators use RDPRA with Privileged Access Workstations. PAWs provide a safe starting point for privileged access, while RDPRA minimizes the exposure of the credentials on the targets. By using these together, I can have a very high degree of confidence that my privileged credentials are never exposed to attackers at the source (PAW) or at the target (RDPRA session). [caption id="attachment_125" align="alignnone" width="770"] Figure 4: This attack timeline shows how privileged credentials enable remote support without exposing privileged credentials to an attacker on the target workstation.[/caption] [As an aside, RDPRA is a poor fit in general for “Jump Servers” scenarios where users connect to an RDP server and from there connect to other resources. This is because credentials are never provided directly to the target system (jump server) and any additional connection you try to make from that jump server requires you to enter new credentials.] When we introduced RDPRA in KB2871997, it was initially enabled by default. After feedback from the wider security community, we changed this because many organizations use RDP for remote access to their environment over the internet and this increased their attack surface without notice. Because of this, you now have to enable RDPRA for workstations and servers (see this article for instructions). Some of those same security professionals who raised the aforementioned issue also raised a specific technical concern about how RDPRA “enables” attackers to combine RDP with credential reuse in order to access remote systems interactively in ways they couldn’t before. While technically accurate, those concerns are overblown in the real world. This new “attack” requires the attacker to have already stolen valid admin credentials, and these credentials can already be used in numerous other ways besides RDP (accessing Active Directory, modifying services, connecting via Remote Registry, and so on). The attacker could even crack the credential information to access the original passwords. Furthermore, to launch an RDPRA-based credential reuse attack, you need to use a special RDP client (for example, this one described by Portcullis Labs). Every new piece of software downloaded onto your systems is another opportunity to catch the bad guys at work – attackers can’t just reuse the default Remote Desktop client. Simply put, the additional protection from RDPRA (reduced exposure of privileged credentials) far outweighs the additional risk represented by the RDPRA credential reuse “attack” (minimal within the intranet environment). And when combined with other credential protection mechanisms (e.g. PAW), it’s extremely secure. RDPRA isn’t meant to be a complete security solution on its own, but it is a critical component of a complete credential theft mitigation program. Every administrator should include RDPRA in his or her standard toolkit because it’s the most secure way to remotely access and manage target systems. To see how to adopt RDPRA as part of your security standards, you can also check out Microsoft’s recommended Operational standards.

When requirements aren't really requirements

John Rodriguez (MSFT) — Mon, 13 Jun 2016 10:47:41 GMT

One of the primary roles of an architect is to develop solutions which meet business requirements. As a cybersecurity architect, I regularly have conversations with customers about their cybersecurity needs and how we might deploy specific operational processes or technical products to match those needs. But all too often, I find that the “requirements” aren’t really requirements at all – they’re actually solutions in their own right. In a nutshell, many IT professionals and business leaders confuse requirements with solutions. Here’s a great example – have you ever had someone tell you that his or her organization has a requirement to encrypt all USB drives? I’ve heard that countless times, and often it’s followed by a lengthy discussion about the difficulties of ensuring that USB drives are encrypted, whether or not they should include or exclude removable drives, how to differentiate between personal and business USB drives, and so on. These questions arise because the “requirement” is actually a solution – the underlying requirement is something similar to “we are required to encrypt all data in transit and at rest.” If we return to the requirement, you realize that there are other options – for example, you can encrypt files directly, rather than attempt to encrypt the underlying medium. Or, if the goal is not encryption but security against unauthorized access, perhaps digital rights management would be appropriate. In this example, the proposed solution (encrypt USB drives) might not be the best way of achieving the goal (of securing data at rest), and by identifying the real requirement, you can ensure that the customer receives the best solution possible. Whenever a customer is overly prescriptive on their requirements, or has given me other reason to believe that they haven’t thought through their requirements fully, I begin to ask very probing questions, like:

What happens if we don’t do this?
How could an adversary exploit non-compliance?

I also ask questions which might highlight the deficiencies of a proposed solution:

How would an attacker work around this proposed control?
Are there other controls which may work better than the proposed one?

This misidentification of requirements is more common than you might think. Here are a few others I often hear – how many have you heard in the last six months?

“We must back up our servers daily/weekly/etc.” – the requirement is actually to be able to recover in the event of a disaster. Backup is one means of doing so; others include data replication and availability groups.
“All connections to X server must use SSL” – the requirement is to protect data in transit. SSL protects only a subset of connections and has itself been supplanted by TLS. IPsec may be a better choice for internal network connections.
“Administrators must use smart cards for logon” – the requirement is for administrators to use multi-factor authentication. Smart cards are just one form of MFA; others include one-time tokens, phone authentication, and biometric authentication.

When you encounter these situations, drive conversations towards the true requirements and find the most effective way of achieving them. Always strive to balance cost, time, level of effort, impact to end users (among other factors) to ensure that you discover and identify the best way to meet the actual requirement. Your customers will benefit greatly from your diligence.

PAWs and Attack Graphs

John Rodriguez (MSFT) — Thu, 09 Jun 2016 15:32:12 GMT

In 2015 John Lambert penned a fantastic article on the concept of attack graphs – if you haven’t read it, I highly encourage you to do so. I recommend it to all of my customers as a way of illustrating one of the most important concepts in cyber defense – that of relationships. Broadly speaking, a computing environment exists to serve the purposes of the end users, and as long as they can achieve their primary business functions, they’re (usually) happy and tend to leave it at that. However, attackers aren’t concerned with what they’re supposed to do, they care about what they can do. That’s where the relationships come in – if an attacker determines that Workstation A can connect to Workstation B, which has a direct administrative connection to Server C, then the attacker will take advantage of that connection … even if the end users of Workstations A and B have no interest in that connection. End users – and security professionals – often miss or outright ignore these relationships, much to their cost in the event of a breach. While every organization should strongly consider limiting the connectivity between workstations (lateral traversal), it’s not an overnight job – it takes planning and effort to fully accomplish. However, you can gain “big wins” by carefully deploying Privileged Access Workstations (PAWs) to “break” the attack graph. The sample graph image below comes from John’s blog article that I referenced above. I’ll use it to illustrate the way you can use an attack graph to selectively deploy PAWs for immediate security wins. In this example, the attacker’s goal is the High Value Asset in the top right. The attacker has compromised an end user workstation and has parlayed that access into the further compromise of the Terminal Server at bottom left. Using relationship graphing software, the defenders have determined that the users U46 and U128 have access to workstations which are shared by U34 and U1, both of whom ultimately have access to the High Value Asset (HVA) Securing the HVA involves more than simply hardening that workstation further – it means breaking the graph by limiting where intersections occur. An intersection occurs when two or more accounts have access to the same resource – for example, M115 and M2821 are both intersections, because multiple end users log onto both. They’re really important intersections, too, because U1 and U34 have access to the HVA, and through their shared access to M115 and M2821, an attacker can work his or her way to that HVA. At this point, the attack graph has essentially become a roadmap for the attacker – but also for the defender. By identifying those critical users (U1 and U34), the defender can work with them to ensure that they only access the HVA using Privileged Access Workstations (PAWs). If we limit U1 and U34 to PAWs, then we’ve essentially severed the link between the Terminal Server and the HVA – there are no lines leading from the left half to the right half of the drawing below. This is really the essence of employing an attack graph in a defensive role – you learn about the relationships between objects which allow an attacker to move laterally in your environment, and once you’ve identified the key intersections, you then “remove” them. In this case, you remove an intersection by ensuring that the accounts with access to the HVAs only use specific systems for that access. It’s important to note that while PAWs were initially envisioned as protections for administrative access (as in, Active Directory administration), they’re equally valuable for protecting access to business HVAs like intellectual property, customer data, and financial records. Not everyone in an organization needs to use a PAW, but if you’re concerned with protecting your most precious digital possessions, consider deploying PAWs to secure their access.