Windows Subsystem for Linux

This blog has moved to the Windows Command Line blog

Craig Loewen — Thu, 03 Oct 2019 13:57:37 GMT

This blog has been archived, and is no longer active. If you would like to keep up with the latest Windows Subsystem for Linux news and blog posts please use the link below to visit its new location the: Windows Command Line blog.

Windows Command Line Blog - https://aka.ms/cliblog

File System Improvements to the Windows Subsystem for Linux

Jack Hammons — Tue, 18 Apr 2017 12:34:01 GMT

This is part of a series of blog posts on the Windows Subsystem for Linux (WSL). For background information you may want to read the architectural overview, introduction to pico processes, WSL system calls, and WSL file system blog posts. Posted on behalf of Sven Groot

Summary

In the latest Windows Insider build, the Windows Subsystem for Linux (WSL) now allows you to manually mount Windows drives using the DrvFs file system. Previously, WSL would automatically mount all fixed NTFS drives when you launch Bash, but there was no support for mounting additional storage like removable drives or network locations. Now, not only can you manually mount any drives on your system, we've also added support for other file systems such as FAT, as well as mounting network locations. This enables you to access any drive, including removable USB sticks or CDs, and any network location you can reach in Windows all from within WSL.

Mounting DrvFs

In order to mount a Windows drive using DrvFs, you can use the regular Linux mount command. For example, to mount a removable drive D: as /mnt/d directory, run the following commands: $ sudo mkdir /mnt/d $ sudo mount -t drvfs D: /mnt/d Now, you will be able to access the files of your D: drive under /mnt/d. When you wish to unmount the drive, for example so you can safely remove it, run the following command: $ sudo umount /mnt/d

Mounting network locations

When you wish to mount a network location, you can of course create a mapped network drive in Windows and mount that as indicated above. However, it's also possible to mount them directly using a UNC path: $ sudo mount -t drvfs '\\server\share' /mnt/share Note the single quotes around the UNC path; these are necessary to prevent the need to escape the backslashes. If you don't surround the UNC path with single quotes, you need to escape the backslashes by doubling them (e.g. \\\\server\\share). WSL does not have any way to specify which credentials to use to connect to a network share. If you need to use different credentials to connect to the server, specify them in Windows by navigating to the share in File Explorer, using the Windows Credential Manager, or the net use command. The net use command can be invoked from inside WSL (using net.exe use) via interop. Type net.exe help use for more information on how to use this command.

Volumes mounted on empty NTFS folders

If your system has any volumes that do not have drive letters but are instead mounted on an empty NTFS folder, you are now able to mount those as well. WSL only automounts volumes with drive letters, so up to this change volumes mounted on a directory could not be accessed. To now mount such a volume in WSL, simply use the path to its mount point: $ sudo mount -t drvfs 'C:\mountpoint' /mnt/myvolume Note that the path you specify must be a mount point; you cannot use an arbitrary directory as the root of a drvfs instance. If you wish to accomplish this, you can already do so using bind mounts.

DrvFs behavior for different file systems

The way drvfs behaves may be slightly different depending on the underlying file system. Certain features may not be available with all file systems. For example, the FAT file system is not case sensitive, and does not support hard links or symbolic links. With network file systems, DrvFs does not set the correct Linux permissions bits on a file; instead, all files are reported with full access (0777) and the only way to determine if you can actually access the file is by attempting to open it. Network file systems are also not case sensitive and do not support symbolic links.

Serial Support on the Windows Subsystem for Linux

Jack Hammons — Fri, 14 Apr 2017 13:43:47 GMT

Summary

In Windows Insider build #16176 WSL now allows access to serial devices (COM ports). This blog post will describe how to use this feature and the work that was involved in exposing this functionality within WSL. On Windows, COM ports are named COM1 through COM256. On Linux, serial devices can have arbitrary names but are typically character devices with a major number of 4 and a minor number from 64 to 256 represented by /dev/ttyS0 through /dev/ttyS191. By default init will populate this mapping using the mknod system call. Alternate mknod mappings can be created at runtime if needed. In WSL, the lxcore driver maps COM ports to Linux devices by the COM port number so /dev/ttyS is tied to COM. More specifically, the mapping is on the minor number, so minor number 65 (/dev/ttyS1) is COM1, 66 (/dev/ttyS2) is COM2, and so forth. Since pty, tty, and ttyS share a terminal library code base, ttyS will behave similarly but also support the following termios settings which are simply mapped to Windows serial driver ioctls:

Standard baud rates - B*
Stop bits - CSTOPB
Word length - CSIZE
Parity checking - PARENB, PARODD, CMSPAR, INPCK
Software flow control - IXON, IXOFF
Hardware flow control - CRTSCTS, CLOCAL
Control characters - VSTART, VSTOP, VEOF

To use your favorite serial tool\library in WSL, just map from the COM port of the device using device manager or the SERIALCOMM registry key (HKEY_LOCAL_MACHINE\HARDWARE\DEVICEMAP\SERIALCOMM). Figure 1. This diagram shows the basic architecture of WSL exposes Windows COM ports as serial devices in WSL.

Scenarios

The following scenarios were tested and confirmed to be working:

Programming pixhawk light controller - Thanks @lovettchris!
Connecting to Raspberry Pi 2 serial terminal using the cu tool
Transferring data on various serial configurations:
1. Hyper-V virtual COM port
2. FTDI USB to serial converter
3. Prolific USB to serial converter
4. Physical COM port

Please note that some serial drivers have known bugs. These issues exist on both native Windows and within WSL. The following areas are known to not work. We are tracking updates to future Windows Insider builds to address them.

Using legacy raw ioctls instead TCGETS or TCSETS* to configure and query the serial device
Screen and minicom depend on the above so they do not currently work as a serial terminal, but cu is an alternative.

Example

On a Windows 10 machine where a Raspberry Pi is connected on COM5 connect using the following steps:

sudo chmod 666 /dev/ttyS5 - This is required since cu changes it's uid which drops capabilities even when running as root. Not all serial programs have the same behavior.
stty -F /dev/ttyS5 -a . Optionally check your current serial settings before updating.
Set your serial settings, depending on your application this usually will be raw or sane. Either one seems to work fine with cu:
1. stty -F /dev/ttyS5 sane 9600
2. stty -F /dev/ttyS5 raw 9600 -echo -echoe -echok -echoctl -echoke -iexten -onlcr cs8 crtscts
cu -l /dev/ttyS5 -s 9600
Hit enter to refresh the cu console

Feedback

Please let us know on our Github about the specific serial scenarios that you are trying. Your feedback helps us prioritize what we should focus on next.

Testing the Windows Subsystem for Linux

Jack Hammons — Tue, 11 Apr 2017 13:06:46 GMT

Following the release of Windows Subsystem for Linux, the development team has created and released a series of technical deep dive blog posts. For a better understanding of the foundations of WSL check out the WSL Overview post. Thank you to Ben Hillis, Vikas Agrawal, and Sunil Muthuswamy for providing content and review.

Overview

The goal of Windows Subsystem for Linux (WSL) is to provide a command line developer environment identical to those on *nix systems without requiring a virtual machine. This primarily consists of access to command line utilities, native ELF64 binary execution, and support common developer frameworks like MEAN and LAMP web server stacks. Additionally, to make the system feel as native as possible, WSL exposes both the Windows file system and networking stack automatically. All of this is accomplished via a kernel level emulation layer that translates system calls from Linux to their corresponding NT counterpart. Where there is no direct NT counterpart WSL fills those gaps. WSL was initially released one year ago at //Build 2016 to an enthusiastic group of users. Since then, our active user count has continued to climb. At time of writing, WSL is used by over half a million people each month. With a community of this size we wanted to be transparent about what sort of quality should be expected. Shipping WSL with a beta tag was a no-brainer for us. We knew that the subsystem wasn't in a place where it would be suited for a production environment and wanted to be clear about what user's expectations should be. To us, the beta tag conveyed the message, "Some things will break and there will be gaps." As development progressed, we have looked for ways to quantify WSL's shortcomings and lay out an implementation road map for the missing features needed to reach parity with bare metal Linux. These efforts have focused on four main pillars: the Linux Test Project, scenario based testing, fuzzing, and feedback from our community.

LTP

An important tool in improving our compatibility has been the Linux Test Project (LTP). LTP is a jointly created repository of unit tests designed for validating the system call and file system interfaces of the Linux kernel. LTP has been invaluable in helping us verify the robustness and stability of our implementation by providing a standardized regression check as well as a road map for future development. LTP has told us a great deal about the completeness of WSL's system call implementation. We also publish the results of LTP on our Github.

LTP in the Anniversary Update

When the Anniversary Update shipped, our LTP pass rate was decent, but not great. We started using LTP relatively late in the development cycle and that didn’t leave much time to tune our system call implementations to improve the pass rate. The results below are using LTP version 20150420. System Calls

Passing	637
Failing	280
Skipped	144
Total	1061
Pass % (passing/(total - skipped))	69.45%

Filesystem

Passing	20
Failing	41
Total	61
Pass %	32.8%

LTP in the Creators Update

In the Creators Update WSL reports Linux kernel version 4.4. We have bumped our LTP version to 20160510 to match. In the below table, you’ll notice there’s an additional row not present in the LTP results from the Anniversary Update. The unimplemented designation means that the test is failing, but the system call is not implemented in WSL. This distinction is important to us because we implement system calls as-needed based on user input and telemetry. Many of the system calls that we have not implemented are not used or are used by a very narrow set of applications. Also included in the unimplemented set are 33 tests that use the /dev/loop device which is not something that we support yet. We’ve also re-categorized what it means to be a skipped test. Now every test in the skipped category is also skipped on native 64 bit Ubuntu. Most these tests are for the 16 bit versions of the system calls. The below results are using LTP version 20160510. System Calls

Passing	744
Failing	93
Unimplemented	171
Skipped	102
Total	1110
Pass % (not including unimplemented)	88.88%
Pass % (including unimplemented)	73.81%

Filesystem Of the failing filesystem tests the majority are due to missing support for the rt_sigqueueinfo system call.

Passing	52
Failing	9
Total	61
Pass %	85.24%

LTP Shortcomings

Unfortunately, LTP is not a one stop shop. Even if we could achieve 100% pass rate for applicable tests it wouldn't tell us everything. There are many system calls (primarily newer ones) that do not have coverage in LTP. Most LTP tests are rudimentary and don’t test interesting race conditions or the full range of arguments a system call supports. A very large percentage of LTP tests are negative variation tests that do little more than ensure that a given system call returns the correct error code when invalid input is provided. In these cases, the WSL team covers test gaps by writing our own unit tests. At time of writing the team has written over 150,000 lines of unit test code for systems calls and virtual files (/proc, /sys). Importantly, 100% LTP pass rate is not the gold standard. Many system calls that LTP covers are rarely used in practice and won’t affect the majority of users. Which brings us to scenario testing.

Scenario Testing

To improve the coverage of our testing we needed something else that stressed areas used by our scenarios of focus. For this, we turned to the unit tests that various open source projects use to validate their own product. We found that many of the frameworks we care about have extensive suites of unit tests. Interestingly, there are even a small number of tests that pass on WSL that did not pass on native Ubuntu 16.04.

Test Name	Version	WSL Failing Tests	WSL Pass %	Native	Native Pass %	Total Tests	Notes
Nginx	Nginx 1.4.6	0	100.00%	0	100.00%	99
Django	1.10x(master)	4	99.97%	4	99.97%	11776
Flask	0.11(master)	1	99.69%	1	99.69%	327
PIP (python 2.7)	Master	3	99.57%	3	99.57%	700	11 skipped on both
Grunt	Master	0	100.00%	0	100.00%	390
Gulp	Master	0	100.00%	0	100.00%	31
Express.js	4.x(master)	0	100.00%	0	100.00%	799
Bower	V1.8(master)	0	100.00%	0	100.00%	539	17 skipped on both
Json-server	Master	0	100.00%	0	100.00%	77
Coffescript	Master	0	100.00%	1	99.88%	822
Ember.js	Master	0	100.00%	0	100.00%	20642
Typescript	Master	0	100.00%	0	100.00%	52976
NVM	Master	1	99.01%	1	99.01%	101
Phantom.js	Master	12	94.50%	12	94.50%	218
Rails	Rails 5.0.0.1	0	100.00%	2	99.99%	14056
Rake	Master	0	100.00%	0	100.00%	573	1 skipped on WSL
RVM	V1.27.0	37	93.03%	37	93.03%	531
Sinatra	Master	0	100.00%	0	100.00%	2365
Sinatra	Stable	0	100.00%	0	100.00%	2354
JUNIT	Master	0	100.00%	0	100.00%	985	4 skipped on both
MAVEN	Master	0	100.00%	0	100.00%	494
STRUTS	Master	0	100.00%	0	100.00%	317
R	R-3.3.2	0	100.00%	0	100.00%	1088
PostgreSQL	postgresql-9.5.3	0	100.00%	0	100.00%	157
Cassandra	Master	0	100.00%	12	96.20%	316

While these tests give us a good idea of what's working they don't tell the whole story. Just because a framework’s unit tests pass at 100% does not guarantee that a user will never hit a bug in that framework. In the future, we will continue to build out the infrastructure for running these tests in a more automated fashion. Additionally, we will add more tests to this list as we identify top frameworks and tools. If you don't see your favorite on here, don't worry! We'll get there.

Stress Testing and Fuzzing

Another important part of the WSL testing story is stress and fuzz testing. We have leveraged Trinity for system call fuzz testing and have incorporated the test into our automated test process. For stress testing, we have relied on stress-ng. Both tools have helped us find critical bugs in hard-to-reach areas of our driver.

Community/GitHub

The final piece of our approach to improving WSL has been our GitHub and our incredible Windows Insider users. The tens of thousands of users previewing the Creators Update over the last 6 months have generated nearly 2000 issues that have been filed on everything from network connection problems to requests for more color support in the console. We look at every issue that is filed and make sure that we have a plan to address those that are actionable and applicable. This has helped us stay closely in sync with our community occasionally responding with fixes in a matter of weeks. If you are interested in receiving the latest WSL bits months before they are publicly available be sure to join the Windows Insider program.

Conclusion

We are thrilled with the progress that WSL has made in the past year. We also see a path forward to equally impactful features and fixes that will increase our compatibility and make WSL an even more compelling development tool. But we can’t do it without you! Continue to give us feedback on where you would like us to focus our efforts. Try your development stack on WSL. Let us know when you hit problems and help us to understand your scenarios. Join the Windows Insider program and file issues on our Github. Thank you for supporting us with your usage and your bugs. We’re excited about the future of WSL. Stay tuned!

WSL Networking

Jack Hammons — Tue, 08 Nov 2016 15:35:04 GMT

Posted on behalf of Sunil Muthuswamy

Disclaimer

The information presented in this blog post is as per the current design and is subject to change. Reader is expected to be familiar with the Overview of the Windows Subsystem and WSL System Calls blog posts.

Background

In this information age of “roam-anywhere, always connected” devices, networking plays a very important role. Users expect to be able to take advantage of complex capabilities and assume that they will be present at any point in time. This places a large burden on the networking stack to allow Internet access, data exchange, and state information to be communicated between distinct processes that live on the same device or a device across the world.

Overview

This post discusses networking within WSL. It explains how WSL configures networking within the subsystem, how it keeps the information up to date with any changes and the implementation of the various Linux\BSD socket address families. The design decisions were made with the specific goal of providing full binary compatibility¹ to Linux applications and to present the same networking view to Linux administrators (and users alike) within the WSL, as they are accustomed to.

Introduction

There are few key networking concepts involved within the operating systems², that are covered in this post; network interfaces, Domain Name Resolution Service (DNS) and sockets. Network interfaces can be thought of as an identifiable gateway that interconnects two or more endpoints with an established interface for communication. Network resources over the internet are usually identified using a domain name, such as www.microsoft.com . Though these names are more human readable, they are not well understood by the network systems. Under the hood, network identities have a different address format. For INET domains, an endpoint is identified using an IP address. DNS service provides the mapping between the more human-readable address format to the machine understood IP address. Without proper DNS resolution, Internet traffic would not reach anywhere. Sockets are set of APIs to communicate between two endpoints, which can be thought of as a conduit for communication.

Networking in Linux

This section will discuss how the various networking concepts that were discussed in the Introduction, applies and is available in Linux.

Network interfaces

The list of network interfaces that are available on a particular system can be accessed via couple of different ways in Linux, both through syscalls. The outdated, but, yet fully supported way is to use Socket IOCTL’s such as SIOCGIFNAME etc. The newer and recommended way that provides the same view, is through the NETLINK sockets, using the NETLINK_ROUTE family. All of the pertaining information about the various network interfaces is kept by the kernel and made available through the above syscalls.

Domain Name Resolution (DNS)

DNS is supported in Linux with the help of the resolver services configured using /etc/resolv.conf in combination with /etc/hosts. The hosts file contains a static list or map of hostnames to their corresponding IP address. While resolv.conf contains (amongst other things) a list of Domain Nameserver(s) that are capable of resolving a given hostname to its IP address. The resolver API’s encapsulates all of this and provides DNS service to the applications.

Sockets

Sockets is an API interface that allows inter process communication (IPC). The two endpoints that want to communicate with each other, open a socket at their end. Once a socket is bound to a given address, other sockets can discover it, depending on the scope of the address. For connection-less protocols such as datagram, this is sufficient for sending and receiving data. For connection-oriented sockets such as stream, a connection needs to be first established between the two peers, before data can be sent or received. Opening a socket requires the caller to provide the address family "AF" (also referred to as protocol family ‘PF’ or domain), the type (ex: datagram, stream etc.) and the protocol. Sockets are broadly categorized depending on their domain names. The type (and the protocol) helps identify sub-category of the socket. The most commonly used domains by *NIX applications is the AF_INET, AF_UNIX (also known as AF_LOCAL) and AF_NETLINK. AF_INET provides access to the Internet protocol. AF_UNIX is used for communicating between processes that live on the same system³. And, lastly, AF_NETLINK sockets are used for communicating between the user mode and the kernel. They can also be used for IPC between two user mode processes.

Networking in WSL

This section provides an overview of how the above Linux networking concepts are implemented within WSL for a full binary and interface compatibility.

Network interfaces and DNS

When the first instance of bash.exe is launched, the LXSS session manager service⁴ queries the list of network interfaces and the DNS servers from Windows. Using the LXBus, the service passes this information to the Linux Subsystem driver. The driver caches all the information about the various network interfaces locally. As for the DNS entries, the Linux Subsystem driver populates the resolv.conf with the list. The cached network interface information kept by the Linux Subsystem driver is accessible through the aforementioned socket IOCTL’s. During the first launch of bash.exe, it will also auto-generate⁵ the hosts file for the particular system. The LXSS session manager service also registers with Windows for notifications for any updates to the network interface (for example, moving from wireless to wired Ethernet), or to the DNS entries. Windows will notify the service of any change to the monitored network information, by calling the registered callback handler. When notified of any such change, the callback handler (part of the service), will use the same information flow mentioned above to notify the Linux Subsystem driver, which will update resolv.conf with the updated DNS entries. This way WSL can always keep the network components within the system up to date and in sync with Windows.

Sockets

Currently, WSL provides implementation for the AF_INET, AF_UNIX and AF_NETLINK address families. All of the socket implementation in WSL is provided in the kernel mode, from within the Linux Subsystem driver. This is essentially because all of the BSD socket API’s map one-to-one directly to syscalls⁶.

AF_INET (Internet domain)

As discussed previously, socket created with the AF_INET domain provide access to the information and services hosted on the Internet. Within the INET domain, there are few different supported socket types; namely the DGRAM, STREAM and RAW. DGRAM sockets, or, more commonly referred to as UDP sockets are connection-less sockets, with no reliability guarantees. STREAM sockets, or, more commonly referred to as TCP sockets are connection-oriented sockets, with some reliability and ordering guarantees provided by the protocol. RAW sockets support many different protocols, such as the ICMP (used by ping), and RAW which allows the protocol to be implemented entirely in the user space. Win32 has user-mode adaptation of the BSD sockets called Winsock. It would seem pretty straight forward to leverage Winsock for providing the socket implementation within WSL. But, as mentioned previously, all of the WSL socket implementation is in the kernel mode, in the WSL socket library (WslSocket.lib) (which is part of the Linux Subsystem driver). That rules out the possibility of using Winsock. Fortunately, NT has a kernel mode network programming interface, called as the Winsock Kernel (or more commonly, as WSK).

WSK

WSK is a publicly documented API set. It is a very thin layered, low-level NT API set that provides fast and easy access to the data from the TCP/IP driver, with little or no overhead of its own. The WSK API interface differs significantly from the BSD API’s, even though it uses the same underlying constructs such as “sockets”. As for example, WSK does not provide any buffering of data⁷. Another example would be the differences in the operating model of WSK and BSD sockets. WSK supports both synchronous and asynchronous modes of retrieving data from the TCP/IP driver, with the asynchronous mode being more efficient. BSD sockets also support synchronous and asynchronous I/O (though epoll), but their semantics are vastly different. WSL (*not* WSK) provides support for the BSD socket API’s by translating them to the WSK API’s and wherever needed (such as for data buffering) bridging the differences by having the necessary infrastructure and implementation within WSL. (see Figure 1 AF_INET sockets in WSL). The following sections will go into the details of each of the individual socket types such as TCP, UDP, and, explain how WSL implements them underneath.

Figure 1. AF_INET sockets in WSL

DGRAM or UDP socket type

As mentioned previously, UDP sockets are light weight, connection-less sockets, with no delivery guarantees. Once a socket has been created, it can be used to send data immediately. Receiving data requires other sockets to be able to identify it using its address. Once the socket binds to an address, then other sockets can send data to it without any further delay. Any UDP socket can send data to any UDP socket, as long it is able to identify (or locate) it using its address.

Figure 2. WSL INET UDP file context

When a WSL UDP application creates a BSD socket using the ‘socket’ syscall, the ‘WSL socket library’ creates a context (see Figure 2 WSL INET UDP file context) and attaches it to the file descriptor (using the VFS file object) that is returned by the syscall. As part of the same ‘socket’ syscall, the driver also creates a WSK UDP socket and stores it in the context. All further operations by the user-mode app on that socket will allow the driver to extract the associated context, and, the WSk socket within. Any data sent over the BSD UDP socket, is sent directly over on the WSK socket.

When the user mode application binds the BSD socket, the ‘WSL socket library’ binds the corresponding WSK socket and registers a WSK ‘receive from’ callback handler with WSK. Any time, data is available on the WSK socket to be read, WSK will call the registered handler/routine. The handler stores the data provided by the WSK (from TCP/IP) in the receive buffer, along with some metadata such as the size of the data (packet), ‘receive from’ address etc. When the application calls ‘recvfrom’ on the BSD socket, the socket library is able to satisfy the request using the data from the ‘receive buffer’.

STREAM or TCP socket type

As mentioned previously, TCP sockets are connection-oriented sockets, with some delivery guarantees. A connection needs to be first established between the two sockets, usually referred to as the ‘client’ and ‘server’ socket. The ‘server’ socket, binds to a well-known address and listens for incoming connection(s) using the ‘listen’ socket call. At this point, the ‘server’ socket is capable of accepting connections and clients can connect to it, using the ‘connect’ socket call. Once the connection is established, the server socket can then be used to accept connections using the ‘accept’ socket call. The accept call returns a new socket, that is connected to the client socket and can be used to send/receive data to/from the client socket, and vice versa. The server socket is then free to accept more incoming connections on the original server socket. The mechanism with which the ‘WSL socket library’ creates and manages TCP BSD sockets is very similar to that of the UDP BSD sockets, but is more involved. When a WSL TCP application creates a BSD TCP socket using the ‘socket’ syscall, the ‘WSL socket library’ creates a context (see Figure 3 WSL INET TCP file context) and attaches it to the file descriptor (using the VFS file object) that is returned by the syscall. As part of the same ‘socket’ syscall, the driver also creates a WSK TCP socket and stores it in the context. The one noticeable difference between how the ‘WSL socket library’ handles UDP and TCP sockets, is the point at which it registers for WSK callbacks. As we saw earlier, in the case of UDP, the callback was registered during bind. For TCP sockets, the WSK callbacks are not registered up until later because of the need to know whether the TCP socket will be used for accepting connections or send/receive (a TCP socket can only be either of the two). When the application calls the ‘listen’ socket call, the ‘WSL socket library’ registers its ‘WSK accept’ callback handler with WSK. For the send/recv socket, whenever an application calls ‘connect’, or, on all accepted sockets, the ‘WSL socket library’ will register the ‘WSK receive’ and ‘WSK disconnect’ callback handlers. With all the right callback handlers registered with WSK, the ‘WSL socket library’ is well equipped to deal with any event that is related to that socket.

Figure 3. WSL INET TCP file context

In the case of the ‘listening’ socket, whenever there is an incoming connection request on a socket, WSK will call the appropriate ‘WSK accept’ handler registered for that socket. The ‘WSL socket library’ can then decide whether to accept or reject that connection, based on the already accepted connection list. In the case where the connection is accepted, the accepted WSK socket is stored in the list of accepted sockets. Whenever the application calls ‘accept’, the ‘WSL socket library’ finds the next connection from the list, creates a new ‘WSL TCP file context’, stores the corresponding WSK socket within it, and, returns a new socket file descriptor to the user. The new file descriptor that is returned can be used for send/recv. For data transfer, once the connection is established, TCP sockets can send and receive data. In the case of send, when the WSL TCP application requests data to be sent over the socket, the ‘WSK socket library’ will queue the data in the ‘send buffer’ and log a pending request with WSK to send the data and return immediately, so that the send can complete asynchronously. The case of receive is similar to that of UDP. WSK will call the registered ‘WSK receive’ callback, whenever there is data to available on that socket. The ‘WSK socket library’ will buffer the incoming data in the internal ‘receive buffer’, and that data is now available to the user-mode TCP application. The application can receive the data using the ‘recv/recvmsg/read’ socket call.

RAW socket type

The case of ‘RAW’ socket is very similar to that of ‘UDP’ socket (see DGRAM or UDP socket type), because of the similarities in the data transfer protocol. The one major difference is that the underlying WSK socket type that is stored in the associated ‘file context’ is of the type, ‘RAW’. Currently, WSK only supports RAW sockets of the ‘ICMP’ protocol.

AF_UNIX or AF_LOCAL

AF_UNIX domain sockets are used for inter-process communication between processes that live within the same domain (or system). The ‘WSL socket library’ provides and manages the implementation for the AF_UNIX domain socket purely and wholly within the subsystem, without any involvement from WSK (see Figure 4 AF_UNIX sockets in WSL)

Figure 4 AF_UNIX sockets in WSL

AF_NETLINK

AF_NETLINK sockets can be used for communication between the kernel and user-mode processes, and also between multiple user-mode processes. AF_NETLINK supports multiple protocols for kernel and user-mode communication, including NETLINK_ROUTE, NETLINK_FIREWALL and NETLINK_KOBJECT_UEVENT. WSL has implemented support for user-mode calls to the kernel in the NETLINK_ROUTE protocol, which handles the querying and configuration of network parameters such as network interfaces, IP addresses and routing tables. The decision to prioritize NETLINK_ROUTE was based on targeted telemetry gathered from WSL users, which showed that NETLINK_ROUTE was the most commonly used Netlink protocol among the applications executed in WSL. Examples of Linux applications using NETLINK_ROUTE include ip, traceroute and whois. Support for AF_NETLINK sockets is implemented inside the Linux Subsystem driver. Specifically, the NETLINK_ROUTE protocol is implemented by calling the Windows NETIO APIs and translating the information provided by the NETIO APIs into the format expected by the NETLINK_ROUTE messages. The following table describes which NETLINK_ROUTE message types are currently supported in WSL, as well as the equivalent NETIO API used in the Linux Subsystem driver to implement its behavior, and the typical Linux user-mode utility usage. We are actively working to expand this table to more NETLINK_ROUTE message types.

NETLINK_ROUTE Message Type	Windows NETIO API	Linux User-mode Usage Example
RTM_GETLINK	GetIfTable2()	ip link
RTM_GETADDR	GetUnicastIpAddressTable()	ip addr show
RTM_GETROUTE*	GetIpForwardTable2()	ip route show

*Coming soon

Socket options

Linux provides plenty of socket options that are available to the user application through the set/getsockopt syscalls.

AF_INET socket options

The INET socket options are layered at different levels, following the OSI networking model. This allows the application to apply the socket option at different layer, such as the TCP (or UDP), IP or at the socket layer, providing high level of control to the application. WSL manages this by assigning clear ownership of the socket options. Some of the socket options such as the send/receive buffer sizes (SO_SNDBUF/RCVBUF), send/receive timeouts (SO_SNDTIMEO /SO_RCVTIMEO) are fully managed by the "WSK socket library". Most other socket options, including all of the TCP/IP socket options are applied to the WSK socket, using the WskControlSocket API, and the "WSK socket library" merely acts as a pass through for those options.

AF_UNIX socket options

All of the AF_UNIX socket options are fully managed by the "WSL socket library".

Credits

Many thanks to the Windows networking team for implementing the necessary features required to support networking in WSL, and for their guidance.

Footnotes

[1] From a networking perspective [2] In reference here to the Linux and Windows OS. [3] or in the same network namespace [4] Refer to the “WSL Components” diagram in the “Windows Subsystem for Linux Overview” blog post [5] Can be disabled by the user. [6] For details on how the syscall redirection works, refer to the blog post on ‘WSL System Calls’ [7] Which should not be confused with Winsock.

Sunil Muthuswamy and Seth Juarez explore networking on WSL

WSL Antivirus and Firewall Compatibility

Jack Hammons — Tue, 01 Nov 2016 15:56:59 GMT

Introduction

The purpose of this post is to enlighten antivirus (AV) companies to new security considerations for correctly interfacing with Bash on Ubuntu on Windows powered by the Windows Subsystem for Linux (WSL). This post will provide a background description of WSL and its underlying technologies including Pico processes. This, in combination with guidance surrounding new Pico process APIs, will provide the means necessary to prepare AV companies to properly support Pico Processes and WSL.

The Windows Subsystem for Linux

The Windows Subsystem for Linux (WSL) is a collection of components that enables native Linux ELF64 binaries to run on Windows. It was released April 1^st 2016 at the Microsoft //Build conference and included as an optional component in the Anniversary Update of Windows 10. It contains both user mode and kernel mode components and is primarily comprised of:

User mode session manager service that handles the Linux instance life cycle
Pico provider drivers (lxss.sys, lxcore.sys) that emulate a Linux kernel by translating Linux system calls
Pico processes that host the unmodified user mode Linux (e.g. /bin/bash)

Pico Processes

As part of Project Drawbridge, the Windows kernel introduced the concept of Pico processes and Pico drivers. Pico processes are OS processes without the trappings of OS services associated with subystems like a Win32 Process Environment Block (PEB). Furthermore, for a Pico process, system calls and user mode exceptions are dispatched to a paired driver. Pico processes and drivers provide the foundation for the Windows Subsystem for Linux, which runs native unmodified Linux binaries by loading executable ELF binaries into a Pico process’s address space and executes them atop a Linux-compatible layer of syscalls.

WSL File System Interactions

To bridge the gap between Linux and Windows WSL uses two different systems to provide users seamless access to their data. Lxfs allows access to Windows files by emulating full Linux behavior for the internal Linux file system within WSL. DrvFs allows full access to Windows drives and files on NTFS. DrvF enables some of the functionality of Linux file systems, such as case sensitivity and symbolic links, while still supporting interoperability with Windows.

Figure 1: WSL File System Architecture

Windows 10 WSL Antivirus/Firewall Deficiencies

Since WSL was released in the Windows Anniversary Update as a beta feature there was no work done to ensure smooth operation with external AV manufacturers. This has caused a variety of issues that has impacted user experience:

prevented users from accessing the internet
prevented installation of WSL
prevented access to certain directories
Caused unexpected file permissions behaviors within the Bash environment

Changes for Improved WSL Antivirus/Firewall Interactions

The Ps notifications exist so kernel mode drivers can run code when a state of the system changes. The expectation is that drivers use these callbacks to build up state about the system which is later used to perform some function. Vendors can use the notifications to develop firewalls or anti-virus software, but the APIs are generic enough that it could be applied to other scenarios. For the next release of Windows, we are planning to introduce the following changes. We feel that with these additions 3^rd party antivirus and firewall vendors should be able to properly interface with WSL without affecting existing threat detection logic.

Added the ability to query the subsystem type of processes and threads
Extended PS thread and process notifications for subsystems

Examples

Process Notify

Original registration code

Status = PsSetCreateProcessNotifyRoutineEx(Callback,FALSE);

New registration code

Type = PsCreateProcessNotifySubsystems;
Status = PsSetCreateProcessNotifyRoutineEx2(Type, Callback,FALSE);

Callback Example

void Callback (_In_ HANDLE ParentId, _In_ HANDLE ProcessId, _Inout_opt_ PPS_CREATE_NOTIFY_INFO CreateInfo) {
     if (CreateInfo->Flags.IsSubsystemProcess == 0) {
          /* Original callback code goes here */
     } else {
          Type = ProcessSubsystemInformation;
          Status = NtQueryInformationProcess(ProcessHandle, Type, &Subsystem, sizeof(Subsystem), NULL);
          if (Subsystem == SubsystemInformationTypeWSL) {
               /* New callback code to handle WSL processes goes here */
          }
     }
}

Thread Notify

Original registration code

Status = PsSetCreateThreadNotifyRoutine(Callback);

New registration code

Type = PsCreateThreadNotifySubsystems;
Status = PsSetCreateThreadNotifyRoutineEx(Type, Callback);

Callback Example

void Callback (_In_ HANDLE ProcssId, _In_ HANDLE ThreadId, _In_ BOOLEAN Create) {
     Type = ThreadSubsystemInformation;
     Status = NtQueryInformationThread(ThreadHandle, Type, &Subsystem, sizeof(Subsystem), NULL);
     if (Subsystem == SubsystemInformationTypeWin32) {
          /* Original callback code goes here */
     } else if (Subsystem == SubsystemInformationTypeWSL) {
          /* New callback code to handle WSL processes goes here */
     }
}

Links to relevant MSDN pages

Windows Kernel-Mode Process and Thread Manager Process and Thread Manager Routines PsSetCreateProcessNotifyRoutineEx

Windows and Ubuntu Interoperability

Ben Hillis — Wed, 19 Oct 2016 08:22:44 GMT

This is the fifth in a series of blog posts on the Windows Subsystem for Linux (WSL). For background information you may want to read the architectural overview, introduction to pico processes, WSL system calls, and WSL file system blog posts.

Seth Juarez and Ben Hillis talk about Windows and Ubuntu interoperability

Introduction

This post will cover the design and implementation of the top requested feature for the Windows Subsystem for Linux - interoperability with Win32 applications. Since the earliest days that our feature was available to Windows Insiders, our users have been crucial in pointing out issues and suggesting key scenarios that they wish to leverage using WSL. It has been a gratifying and humbling experience to have so many people actively using WSL and providing feedback. We appreciate all the bug reports and suggestions; they have been an invaluable tool to prioritize features for future versions of Windows. It did not take long for one such feature to bubble to the top - Interoperability with Win32 applications. First, let’s define what is meant by interoperability. In a broad sense, interoperability is the ability to mix and match NT and Linux binaries from within the same shell. For example, using bash.exe to navigate your file system and launching an NT graphical text editor from the current working directory. Another aspect is the ability for NT and Linux binaries to redirect their input and output. For example, filtering the output of a Windows command with the Linux grep binary. This functionality allows the Windows Subsystem for Linux to accomplish something that was previously not possible: seamlessly running your favorite native Windows and Linux tools from the shell of your choice. All of this functionality is available with Windows build 14951 and later.

Launching Processes in WSL

The following diagram describes how the various pieces fit together. When launching a process, there are a few pieces of state that need to be shared. The first is the current working directory. Second are the file objects that represent StandardInput, StandardOutput, and StandardError so output from the new process goes to the right place. For example, piping output to another process, redirecting output to a file, or making sure that the contents show up in the right console environment. The following sections will go into more details how this sharing is accomplished. Launching /bin/bash

bash.exe uses the internal CreateLxProcess COM API to talk to the LXSS Manager service.
The LXSS Manager service translates the current working directory to a WSL path and marshals NT handles provided by bash.exe representing StandardInput, StandardOutput, StandardError, and the console with LxCore.sys.
The LXSS Manager sends a message via LxBus to our custom /init daemon.
The init daemon parses the message and unmarshals StandardInput, StandardOutput, StandardError, and the console; setting them to file descriptors 0, 1, and 2 and creating a tty file descriptor.
/init forks and execs the /bin/bash binary and goes back to listen for the next create process message.

Launching an NT binary from within a WSL instance

/bin/bash forks and execs an NT binary. LxCore.sys finds a binfmt_misc registration that handles Windows PE images and invokes the interpreter (/init).
/init (running as the binfmt_misc interpreter) translates the current working directory and marshals file descriptors 0, 1, and 2 (representing stdin, stdout, and stderr) with LxCore.sys.
/init sends a create NT process message to bash.exe.
Bash.exe unmarshals the file descriptors, which creates NT handles, and calls CreateProcess specifying the NT handles as StandardInput, StandardOutput, and StandardError.

Under the covers – LxBus

To discuss the mechanism by which some of the higher-level goals are accomplished, we will need some details on WSL’s inter-process communication mechanism, LxBus. LxBus is the communication channel between NT and WSL processes (primarily the LXSS Manager service and /init). It is a socket like protocol that also allows sending messages and sharing state between NT and WSL processes. LxBus has a lot of functionality and will likely be the topic of a future blog post, but let’s talk about a few of the things it can do. Inside of a WSL instance, LxBus is accessed by opening the /dev/lxss device. This device is locked down to the /init daemon. There is also a /dev/lxssclient device which supports a subset of the functionality of the /dev/lxss device. On Windows, LxBus is accessed via the handle that represents the running WSL Instance. Once a file descriptor or handle to one of these devices is opened it supports several ioctls. The most relevant to our discussion are the “register server” and “connect to server” ioctls. A NT or WSL process that wishes to accept connections uses the “register server” ioctl to register a named LxBus server with the LxCore.sys driver. When the registration completes, a file descriptor or handle representing the server is returned to the caller. We call this object a LxBus ServerPort. You can think of the register server ioctl as the bind call in socket terminology. Once the server is registered, ServerPort objects support a “wait for connection” ioctl which functions similarly to a socket listen call. When a client wishes to connect to a registered server, it uses the “connect to server” ioctl which specifies the name of the server it wishes to connect. If the server is waiting for a connection, a connection is established between the NT and WSL process. The NT process receives a handle that represents the connection and the WSL process receives a file descriptor. These reference an underlying file object called a LxBus MessagePort. MessagePorts support read and write for message passing. They also implement several ioctls that can accomplish various goals, primarily involving sharing resources between NT and WSL processes. For example: One ioctl allows a NT process to share a handle with a WSL process. It is this ability that allows us to redirect the output of a WSL process back to Windows.

IO Redirection from bash.exe

Initially when we released WSL, bash.exe only supported console input and output. This was fine if you launched bash.exe and treated it like an ssh connection to a remote machine, but quickly became inadequate if you wanted to incorporate WSL into scripts or Win32 applications. Our users were quick to point this out, in fact the #2 issue on our GitHub reports this limitation. Coming soon to Windows Insiders, you'll be able pipe to and from WSL processes as well as use NT files as input or output. When launching a WSL process via the LxssManager’s COM interface, there are a variety of arguments. There are the obvious arguments like the path to the binary to be executed and the current working directory. Other arguments include the NT handles to be used as StandardInput, StandardOutput, and StandardError for the WSL process. These handles are passed up to the Lxss Manager service to be shared with the WSL process that is being created. We call the sharing of these handles marshalling. Marshalling is implemented as an ioctl on a LxBus MessagePort object that takes in a handle and returns a unique identifier. This identifier is then sent to the /init daemon along with the other create process parameters. Init parses this message and begins “unmarshalling” the handles so they can be used as input handles for the new WSL process. Unmarshaling is implemented as another MessagePort ioctl that takes the identifier and returns a file descriptor. Using this ioctl, /init unmarshals the three handles and calls the dup2 system call to replace file descriptors 0, 1, and 2. There is a fourth handle that is also marshalled, the console handle. This is used to create a tty device inside of a WSL instance. All the WSL processes spawned from a single bash.exe window will have the same tty device. Once all the handles are unmarshaled, /init uses the fork and exec system calls to create a child process. If you're familiar with bash, you probably know about the -c argument which allows you to supply a command string to run (instead of running bash in interactive mode). Bash.exe sends the command line arguments along to /bin/bash so invoking commands can be done in a similar way.

Launching Win32 applications from within WSL

At the time of posting the most recent Windows Insider build now supports the ability to launch Win32 binaries from within the bash shell! This functionality was so highly desired that Xilun, one of our amazing GitHub users, created his own open source project called cbwin to accomplish this goal. His project works well, but since the WSL team own the entire pipeline we've come up with an integrated solution that doesn't require a launcher binary inside WSL or an additional Windows service. Turns out getting rid of the launcher binary is something that native Linux is also able to do. Linux has a capability called binfmt_misc, that allows user-mode applications to register executable file formats that they can handle with the kernel. Java, Mono, and Wine are a few examples of applications that use this functionality to register file types with the Linux kernel. A registration string is colon-separated and has seven fields, some of them optional. To register a binfmt_misc interpreter a user-mode application writes registration string to the binfmt_misc register file (which is normally located in /proc/sys/fs/binfmt_misc/register). For example: sudo echo ":WSLInterop:M::MZ::/init:" > /proc/sys/fs/binfmt_misc/register WSLInterop - name of the registration. This must be unique. M - type of registration (either Magic Byte Array or Extension). MZ - Magic byte sequence for Windows PE images. /init - Full path to the interpreter. An interpreter is a script or binary on the system that knows how to handle the specified type of files. The exec system iterates over the list of registered binfmt_misc interpreters and looks for one that matches the file being executed. if a match is found the exec system call essentially puts the path to the interpreter in Argv[0] and shifts the rest of the arguments to the right by one (Argv[0] becomes Argv[1] and so on). If you're familiar with the "shebang" (#!) operator that you often see at the beginning of bash or Perl scripts this will probably sound familiar to you. Binfmt_misc is a more flexible way to accomplish the same goal without the limitation of having to add extra text to the header of a file, which makes it perfect for binaries. As you noticed in the above example, we're using /init as our binfmt_misc interpreter. In WSL, /init is a multi-purpose binary that was written by Microsoft and is shipped as a binary resource contained in LxssManager.dll. When /init is launched, it first checks its PID. If the PID is 1, /init will run in "daemon mode" where it essentially the Lxss Manger service’s endpoint inside a WSL instance. If the PID is not 1, /init runs as "binfmt_misc interpreter mode" which allows launching NT binaries. In the future if we begin running other Ubuntu daemons we will likely switch our daemon from replacing /sbin/init to simply being another daemon on the system. A couple of notes and caveats

NT processes will be launched with the same permission as the bash.exe window they are launched from.
NT binaries can only run out of DrvFs paths (for example: /mnt/c/Windows/System32).
Current working directory of launched NT processes will be inherited by NT processes if it is somewhere inside a DrvFs mount. Otherwise the NT process will inherit the current working directory that bash.exe was launched from.
In our initial implementation, NT environment variables and WSL environment variables are two disparate entities. This includes the $PATH environment variable. In development is a way to sharing of the user’s NT %PATH% environment variable with WSL processes – so stay tuned!
One of the first NT commands you'd probably think of trying is the Windows 'dir' command. Unlike Linux where '/bin/ls' is an actual binary, dir is a built-in command. So, if you want to run dir you'll have to run "/mnt/c/Windows/System32/cmd.exe /c dir". That's kind of a mouthful, but luckily bash allows you to create command aliases, for example: alias dir='/mnt/c/Windows/System32/cmd.exe /c dir'
/init being used as the binfmt_misc interpreter is currently how things are implemented but this is subject to change.

IO Redirection to Win32

Launching NT processes from within bash is great, but in order to support NT command-line utilities there needs to be a way for NT binaries to access the file descriptors that represent stdin, stdout, and stderr inside a WSL process. This was achieved by introducing a set of API’s to marshal VFS file to an NT process. Let's walk through how this works. If you're familiar with the CreateProcess API, you probably know that you can provide custom handles for StandardInput, StandardOutput, and StandardError. The problem is, we don't have a handle, we have a file descriptor inside of a WSL instance. Enter VFS file marshalling. A WSL process (the binfmt_misc interpreter) decides that it wants to marshal the stdin, stdout, and stderr file descriptors over to an NT process. It uses a special "marshal VFS file" ioctl on a LxBus MessagePort which takes in a file descriptor as an argument. The LxCore.sys driver handles this ioctl and uses the provided file descriptor to reference the underlying VFS file object. This object is then added onto a per MessagePort list of "marshalled VFS file objects". The ioctl call returns a unique token that identifies the VFS file object that was marshalled. The WSL process then sends this identifier over the MessagePort to the NT process vi an LxBus message. When the NT process (bash.exe) receives this message, it uses the "unmarshal VFS file" ioctl which takes the identifier as an argument. LxCore.sys looks up this identifier in the list of marshalled file objects, and if found, removes the entry from the list and returns an NT handle to the VFS file object to the caller. This handle can now be used as one of the standard handles for the CreateProcess API. Since the handle references a file object within a WSL instance, rundown protection is acquired for the duration of each read or write operation (to ensure that WSL instance specific state does not get torn down during an NT read or write).

Conclusion

This post has covered Windows Interop; the most highly requested feature of Windows Subsystem. Interoperability between NT and Linux processes is a feature that allows users of Windows Subsystem for Linux to mix and match NT and Linux binaries from within the same shell, all running natively on the NT kernel. We briefly covered LxBus, the underlying IPC mechanism which makes communicating between NT and WSL processes possible. Armed with this knowledge the post then covered how LxBus is used to allow sharing of NT and VFS file objects between NT and WSL processes. New features like this are greatly influenced by the feedback we get from the community via GitHub and User Voice, so please continue to make your voices heard! We at the Windows Subsystem for Linux team are excited to see how you use this functionality. Thanks for reading.

Bash on Ubuntu on Windows 10 Anniversary Update

Jack Hammons — Fri, 08 Jul 2016 13:59:12 GMT

Posted on behalf of Russ Alexander. It was just three months ago that we announced Bash on Ubuntu on Windows at Build 2016. After working in secret on the Windows Subsystem for Linux (WSL) for so long it was incredible to see everyone just excited as us for this project. A week later we released the subsystem to the Windows Insider community. Since then we have been overwhelmed with the response that we have received. News articles, blog posts and social media have fueled the excitement for WSL leading to new developer communities that have helped guide and refine our development priorities. It is the support and enthusiasm of our users which has helped us get to where we are today. We are very appreciative of all the bug reports, suggestions, critiques, and ideas which have been submitted. Thank you so much for your help. We look forward to your continued support. Our original plan was to keep WSL only within the Windows Insider program for the upcoming Windows update with the goal of gauging community feedback and refining quality. However, we have heard a strong and clear message that you would like us to ship WSL with the next Windows release. As a result, we plan to include Bash on Ubuntu on Windows as a beta feature in the upcoming Windows 10 Anniversary Update. Doing this will allow even more people to try WSL and we are excited to see what this new group of users will do. Moving forward we will be investigating new areas of interest while continuing to support the following scenarios:

GNU command line tools such as grep, sed, and awk
File system and symlink support within the WSL environment
Ability to run apt / apt-get for installs, updates, and package testing
Basic functionality for languages such as NodeJS/npm, Python, Perl
Command line tools such as Vim, Emacs, Git, and SSH

This official beta will be nearly identical to what is in the hands of Insiders today. After release, the beta will be serviced like any other Windows component. New enhancements and fixes will continue to roll out via our Windows Insider program which will remain the best place for the most recent builds and continuous updates. Users should continue voicing feature requests at the WSL User Voice page for prioritization, and report any issues to our Github page. As we continue to develop, new features and fixes we will be reflected on our MSDN documentation. There is still much work to be done and we have a lot of excitement for the journey ahead. We look forward to seeing all of the great things you will do and make with WSL.

WSL File System Support

Jack Hammons — Wed, 15 Jun 2016 11:19:31 GMT

This is the fourth in a series of blog posts on the Windows Subsystem for Linux (WSL). For background information you may want to read the architectural overview, introduction to pico processes and WSL system calls blog posts. Posted on behalf of Sven Groot.

Introduction

One of the key goals for the Windows Subsystem for Linux is to allow users to work with their files as they would on Linux, while giving full interoperability with files the user already has on their Windows machine. Unlike a virtual machine, where you have to use network shares or other solutions to share files between the host and guest OS, WSL has direct access to all your Windows drives to allow for easy interop. Windows file systems differ substantially from Linux file systems, and this post looks into how WSL bridges those two worlds.

File systems on Linux

Linux abstracts file systems operations through the Virtual File System (VFS), which provides both an interface for user mode programs to interact with the file system (through system calls such as open, read, chmod, stat, etc.) and an interface that file systems have to implement. This allows multiple file systems to coexist, providing the same operations and semantics, with VFS giving a single namespace view of all these file systems to the user. File systems are mounted on different directories in this namespace. For example, on a typical Linux system your hard drive may be mounted at the root, /, with directories such as /dev, /proc, /sys, and /mnt/cdrom all mounting different file systems which may be on different devices. Examples of file systems used on Linux include ext4, rfs, FAT, and others. VFS implements the various system calls for file system operations by using a number of data structures such as inodes, directory entries and files, and related callbacks that file systems must implement.

Inodes

The inode is the central data structure used in VFS. It represents a file system object such as a regular file, directory, symbolic link, etc. An inode contains information about the file type, size, permissions, last modified time, and other attributes. For many common Linux disk file systems such as ext4, the on-disk data structures used to represent file metadata directly correspond to the inode structure used by the Linux kernel. While an inode represents a file, it does not represent a file name. A single file may have multiple names, or hard links, but only one inode. File systems provide a lookup callback to VFS which is used to retrieve an inode for a particular file, based on the parent inode and the child name. File systems must implement a number of other inode operations such as chmod, stat, open, etc.

Directory entries

VFS uses a directory entry cache to represent your file system namespace. Directory entries only exist in memory, and contain a pointer to the inode for the file. For example, if you have a path like /home/user/foo, there is a directory entry for home, user, and foo, each with a pointer to an inode. Directory entries are cached for fast lookup, but if an entry is not yet in the cache, the inode lookup operation is used to retrieve the inode from the file system so a new directory entry can be created.

File objects

When an inode is opened, a file object is created for that file which keeps track of things like the file offset and whether the file was opened for read, write or both. File systems must provide a number of file operations such as read, write, sync, etc.

File descriptors

Applications refer to file objects through file descriptors. These are numeric values, unique to a process, that refer to any files the process has open. File descriptors can refer to other types of objects that provide a file-like interface in Linux, including ttys, sockets, and pipes. Multiple file descriptors can refer to the same file object, e.g. through use of the dup system call.

Special file types

Besides just regular files and directories, Linux supports a number of additional file types. These include device files, FIFOs, sockets, and symbolic links. Some of these files affect how paths are parsed. Symbolic links are special files that refer to a different file or directory, and following them is handled seamlessly by VFS. If you open the path /foo/bar/baz and bar is a symbolic link to /zed, then you will actually open /zed/baz instead. Similarly, a directory may be used as a mount point for another file system. In this case, when a path crosses this directory, all inode operations below the mount point go to the new file system.

Special and pseudo file systems

Linux uses a number of file systems that don’t read files from a disk. TmpFs is used as a temporary, in-memory file system, whose contents will not be persisted. ProcFs and SysFs both provide access to kernel information about processes, devices and drivers. These file systems do not have a disk, network or other device associated with them, and instead are virtualized by the kernel.

File systems on Windows

Windows generalizes all system resources into objects. These include not just files, but also things like threads, shared memory sections, and timers, just to name a few. All requests to open a file ultimately go through the Object Manager in the NT kernel, which routes the request through the I/O Manager to the correct file system driver. The interface that file system drivers implement in Windows is more generic and enforces fewer requirements. For example, there is no common inode structure or anything similar, nor is there a directory entry; instead, file system drivers such as ntfs.sys are responsible for resolving paths and opening file objects. File systems in Windows are typically mounted on drive letters like C:, D:, etc., although they can be mounted on directories in other file systems as well. These drive letters are actually a construct of Win32, and not something that the Object Manager directly deals with. The Object Manager keeps a namespace that looks similar to the Linux file system namespace, rooted in \, with file system volumes represented by device objects with paths like \Device\HarddiskVolume1. When you open a file using a path like C:\foo\bar, the Win32 CreateFile call translates this to an NT path of the form \DosDevice\C:\foo\bar, where \DosDevice\C: is actually a symbolic link to, for example, \Device\HarddiskVolume4. Therefore, the real full path to the file is actually \Device\HarddiskVolume4\foo\bar. The object manager resolves each component of the path, similar to how VFS would in Linux, until it encounters the device object. At this point, it forwards the request to the I/O manager, which creates an I/O Request Packet (IRP) with the remaining path, which it sends to the file system driver for the device.

File objects

When a file is opened, the object manager creates a file object for it. Instead of file descriptors, the object manager provides handles to file objects. Handles can actually refer to any object manager object, not just files. When you call a system call like NtReadFile (typically through the Win32 ReadFile function), the I/O manager again creates an IRP to send down to the file system driver for the file object to perform the request. Because there are no inodes or anything similar in NT, most operations on files in Windows require a file object.

Reparse points

Windows only supports two file types: regular files and directories. Both files and directories can be reparse points, which are special files that have a fixed header and a block of arbitrary data. The header includes a tag that identifies the type of reparse point, which must be handled by a file system filter driver, or for built-in reparse point types, the I/O manager itself. Reparse points are used to implement symbolic links and mount points. In these cases, the tag indicates that the reparse point is a symbolic link or mount, and the data associated with the reparse point contains the link target, or volume name for mount points. Reparse points can also be used for other functionality such as the placeholder files used by OneDrive in Windows 8.

Case sensitivity

Unlike Linux, Windows file systems are by default case preserving, but not case sensitive. In actuality, Windows and NTFS do support case sensitivity, but this behavior is not enabled by default.

File systems in WSL

The Windows Subsystem for Linux must translate various Linux file system operations into NT kernel operations. WSL must provide a place where Linux system files can exist, with all the functionality required for that including Linux permissions, symbolic links and other special files such as FIFOs; it must provide access to the Windows volumes on your system; and it must provide special file systems such as ProcFs. To facilitate this, WSL has a VFS component that is modeled after the VFS on Linux. The overall architecture is shown below.

When an application calls a system call, this is handled by the system call layer, which defines the various kernel entry points such as open, read, chmod, stat, etc. For these file-related system calls, the system call layer has very little functionality; it basically just forwards the call to VFS. For operations that use paths (such as open or stat), VFS resolves the path using a directory entry cache. If an entry is not in the cache, it calls into one of several file system plugins to create an inode for the entry. These plugins provide inode operations like lookup, chmod, and others, similar to the inode operations used by the Linux kernel. When a file is opened, VFS uses the file system’s inode open operation to create a file object, and returns a file descriptor for that file object. System calls operating on the file descriptor (such as read, write or sync) call file operations defined by the file systems. This system is deliberately very close to how Linux behaves, so WSL can support the same semantics. VFS defines several file system plugins: VolFs and DrvFs are used to represent files on disk, and the remainder are the in-memory file system TmpFs and pseudo file systems such as ProcFs, SysFs, and CgroupFs. VolFs and DrvFs are where Linux file systems meet Windows file systems. They are how WSL interacts with files on your disks, and serve two different purposes: VolFs is designed to provide full support for Linux file system features, and DrvFs is designed for interop with Windows.

Let’s look at these file systems in more detail.

VolFs

The primary file system used by WSL is VolFs. It is used to store the Linux system files, as well as the content of your Linux home directory. As such, VolFs supports most features the Linux VFS provides, including Linux permissions, symbolic links, FIFOs, sockets, and device files. VolFs is used to mount the VFS root directory, using %LocalAppData%\lxss\rootfs as the backing storage. In addition, a few additional VolFs mount points exist, most notably /root and /home which are mounted using %LocalAppData%\lxss\root and %LocalAppData%\lxss\home respectively. The reason for these separate mounts is that when you uninstall WSL, the home directories are not removed by default, so any personal files stored there will be preserved. Note that all these mount points use directories in your Windows user folder for storage. Each Windows user has their own WSL environment, and can therefore have Linux root privileges and install applications without affecting other Windows users.

Inodes and file objects

Since Windows has no related inode concept, VolFs must keep a handle to a Windows file object in an inode. When VFS requests a new inode using the lookup callback, VolFs uses the handle from the parent inode and the name of the child to perform a relative open and get a handle for the new inode. These handles are opened without any read/write access to the files, and can only be used for metadata requests. When a file is opened, VolFs creates a Linux file object that points to the inode. It also reopens the inode’s file handle with the requested read/write access and stores the new handle in the file object. This handle is then used to satisfy file operations like read and write.

Emulating Linux features

As discussed above, Linux diverges from Windows in several ways for file systems. VolFs must provide support for several Linux features that are not directly supported by Windows. Case sensitivity is handled by Windows itself. As mentioned earlier, Windows and NTFS actually support case sensitive operations, so VolFs simply requests the Object Manager to treat paths as case sensitive regardless of the global registry key controlling this behavior. Linux also supports nearly all characters as legal characters in file names. NT has more restrictions, where some characters are not allowed at all and others may have special meanings (such as ‘:’ denoting an alternate data stream). To support all Linux file names, VolFs escapes illegal characters in file names. Linux has some different semantics surrounding unlinking and renaming. Specifically, a file can be unlinked even if there are open file descriptors to the file. Similarly, a file can be overwritten as the target of a rename operation even if it’s still open. In Windows, if a file is requested to be deleted, it will only be deleted once the last handle to that file is closed, leaving the name visible in the file system until then. To support Linux unlink semantics, VolFs renames unlinked files to a hidden temporary directory before requesting deletion. Inodes in Linux have a number of attributes which don’t exist in Windows, including their owner and group, the file mode, and others. These attributes are stored in NTFS Extended Attributes associated with the files on disk. The following information is stored in the Extended Attributes:

Mode: this includes the file type (regular, symlink, FIFO, etc.) and the permission bits for the file.
Owner: the user ID and group ID of the Linux user and group that own the file.
Device ID: for device files, the device major and minor number of the device. Note that WSL currently does not allow users to create device files on VolFs.
File times: the file accessed, modified and changed times on Linux use a different format and granularity than on Windows, so these are also stored in the EAs.

In addition, if a file has any file capabilities, these are stored in an alternate data stream for the file. Note that WSL currently does not allow users to modify file capabilities for a file. The remaining inode attributes, such as inode number and file size, are derived from information kept by NTFS.

Interoperability with Windows

While VolFs files are stored in regular files on Windows in the directories mentioned above, interoperability with Windows is not supported. If a new file is added to one of these directories from Windows, it lacks the EAs needed by VolFs, so VolFs doesn’t know what to do with the file and simply ignores it. Many editors will also strip the EAs when saving an existing file, again making the file unusable in WSL. Additionally, since VFS caches directory entries, any modifications to those directories that are made from Windows while WSL is running may not be accurately reflected.

DrvFs

To facilitate interoperability with Windows, WSL uses the DrvFs file system. WSL automatically mounts all fixed drives with supported file systems under /mnt, such as /mnt/c, /mnt/d, etc. Currently, only NTFS and ReFS volumes are supported. DrvFs operates in a similar fashion as VolFs. When creating inodes and file objects, handles are opened to Windows files. However, in contrast to VolFs, DrvFs adheres to Windows rules (with a few exceptions, noted below). Windows permissions are used, only legal NTFS file names are allowed, and special file types such as FIFOs and sockets are not supported.

DrvFs permissions

Linux usually uses a simple permission model where a file allows read, write or execute access to either the owner of the file, the group, or everyone else. Windows instead uses Access Control Lists (ACLs) that specify complex access rules for each individual file and directory (Linux does also have the ability to use ACLs, but this is not currently supported in WSL). When opening a file in DrvFs, Windows permissions are used based on the token of the user that executed bash.exe. So in order to access files under C:\Windows, it’s not enough to use “sudo” in your bash environment, which gives you root privileges in WSL but does not alter your Windows user token. Instead, you would have to launch bash.exe elevated to gain the appropriate permissions. In order to give the user a hint about the permissions they have on files, DrvFs checks the effective permissions a user has on a file and converts those to read/write/execute bits, which can be seen for example when running “ls -l”. However, there is not always a one-to-one mapping; for example, Windows has separate permissions for the ability to create files or subdirectories in a directory. If the user has either of these permissions, DrvFs will report write access on the directory, while in fact some operations may still fail with access denied. Since your effective access to a file may differ depending on whether bash.exe was launched elevated or not, the file permissions shown in DrvFs will also change when switching between elevated and non-elevated bash instances. When calculating the effective access to a file, DrvFs takes the read-only attribute into account. A file with the read-only attribute set in Windows will show up in WSL as not having write permissions. Chmod can be used to set the read-only attribute (by removing all write permissions, e.g. “chmod a-w some_file”) or clear it (by adding any write permissions, e.g. “chmod u+w some_file”). This behavior is similar to the CIFS file system in Linux, which is used to access Windows SMB shares.

Case sensitivity

Since the support is there in Windows and NTFS, DrvFs supports case sensitive files. This means it’s possible to create two files whose name only differs by case in DrvFs. Note that many Windows applications may not be able to handle this situation, and may not be able to open one or both of the files. Case sensitivity is disabled on the root of your volumes, but is enabled everywhere else. So in order to use case sensitive files, do not attempt to create them under /mnt/c, but instead create a directory where you can create the files.

Symbolic links

While NT supports symbolic links, we could not rely on this support because symbolic links created by WSL may point to paths like /proc which have no meaning in Windows. Additionally, NT requires administrator privileges to create symbolic links. So, another solution had to be found. Unlike VolFs, we could not rely on EAs to indicate a file is a symbolic link in DrvFs. Instead, WSL uses a new type of reparse point to represent symbolic links. As a result, these links will work only inside WSL and cannot be resolved by other Windows components such as File Explorer or cmd.exe. Note that since ReFS lacks support for reparse points, it also doesn’t support symbolic links in WSL. NTFS however now has full symbolic link support in WSL.

Interoperability with Windows

Unlike VolFs, DrvFs does not store any additional information. Instead, all inode attributes are derived from information used in NT, by querying file attributes, effective permissions, and other information. DrvFs also disables directory entry caching to ensure it always presents the correct, up-to-date information even if a Windows process has modified the contents of a directory. As such, there is no restriction on what Windows processes can do with the files while DrvFs is operating on them. DrvFs also uses Windows delete semantics for files, so a file cannot be unlinked if there are any open file descriptors (or handles from Windows processes) to the file.

ProcFs and SysFs

Like in Linux, these special file systems do not show files that exist on disk, but instead represent information kept by the kernel about processes, threads, and devices. These files are dynamically generated when read. In some cases, the information for the files is kept entirely inside the lxcore.sys driver. In other cases, such as the CPU usage of a process, WSL queries the NT kernel for this information. However, there is no interaction here with Windows file systems.

Conclusion

WSL provides access to Windows files by emulating full Linux behavior for the internal Linux file system with VolFs, and by providing full access to Windows drives and files through DrvFs. As of this writing, DrvFs enables some of the functionality of Linux file systems, such as case sensitivity and symbolic links, while still supporting interoperability with Windows. In the future, we will continue to improve our support for Linux file system features, not only in VolFs but also in DrvFs. The goal is to reduce the number of scenarios that require you to stay in the VolFs mounts with all the limitations on interoperability that entails. These improvements are driven by the great feedback we get from the community on GitHub and User Voice to help us target the most important scenarios.

Sven Groot and Seth Juarez explore WSL file system support.

WSL System Calls

Jack Hammons — Wed, 08 Jun 2016 10:01:26 GMT

This is the third in a series of blog posts on the Windows Subsystem for Linux (WSL). For background information you may want to read the architectural overview and introduction to pico processes. Posted on behalf of Stephen Hufnagel.

System calls

WSL executes unmodified Linux ELF64 binaries by emulating a Linux kernel interface on top of the Windows NT kernel. One of the kernel interfaces that it exposes are system calls (syscalls). This post will dive into how syscalls are handled in WSL.

Overview

A syscall is a service provided by the kernel that can be called from user mode which typically handle device access requests or other privileged operations. As an example, a nodejs webserver would use syscalls to access files on disk, handle network requests, create processes\threads, and other operations. How a syscall is made depends on the operating system and the processor architecture, but it comes down to having an explicit contract between user mode and kernel mode called an Application Binary Interface (ABI). For most cases, making a syscall breaks down into 3 steps:

Marshall parameters – user mode puts the syscall parameters and number at locations defined by the ABI.
Special instruction – user mode uses a special processor instruction to transition to kernel mode for the syscall.
Handle the return – after the syscall is serviced, the kernel uses a special processor instruction to return to user mode and user mode checks the return value.

While the Linux kernel and Windows NT kernel follow the steps above, they differ in ABI so they are not compatible. Even if the Linux kernel and Windows NT kernel had the same ABI, they expose different syscalls that do not always map one to one. For example, the Linux kernel includes things like fork, open, and kill while the Windows NT kernel has the comparable NtCreateProcess, NtOpenFile, and NtTerminateProcess. The following sections will go into the specifics of making syscalls in the different environments.

Syscall mechanics on Linux x86_64

The calling convention for syscalls on Linux x86_64 follow the System V x86_64 ABI defined here. As an example, one way to call the getdents64 syscall directly from a C program would be to use the syscall wrapper:

Result = syscall(__NR_getdents64, Fd, Buffer, sizeof(Buffer));

To discuss the syscall calling convention it’s easiest to translate the line of code into pseudo assembly to show the 3 steps above:

mov rax, __NR_getdents64
mov rdi, Fd
mov rsi, Buffer
mov rdx, sizeof(Buffer)
syscall
cmp rax, 0xFFFFFFFFFFFFF001

First, let’s look at how the parameters are marshaled. Steps 1-4 move the syscall parameters into the registers specified by the calling convention. Then on step 5 the special syscall instruction is made to transition to kernel mode. Finally, on step 6, the return value is checked by user mode. Between steps 5 and 6 is where the Linux kernel handles the getdents syscall. When the syscall instruction is invoked, the processor performs a ring transition to kernel mode and starts executing at a specific function in kernel mode typically called the syscall dispatcher. As part of kernel initialization when the machine is booted, the kernel configures the processor to execute the syscall dispatcher with a specific environment when the syscall instruction is invoked. The first thing the Linux kernel will do in the syscall dispatcher is save off the user mode thread’s register state according to the ABI so that when the syscall returns the user mode program can continue executing with the expected register context. Then it will inspect the rax register to determine which syscall to perform and pass the registers off to the getdents syscall as parameters. Once the getdents syscall completes, the kernel restores the user mode register state, updates rax to contain the return value of the syscall, and uses another special instruction (usually sysret or iretq) that informs the processor to perform the ring transition back to user mode.

Syscall mechanics on NT amd64

The calling convention for syscalls on NT x64 follows the x64 calling convention described here. As an example, when the NtQueryDirectoryFile call is made we’ll have this simplified version of the call so it’s easier to compare against the getdents call above:

Status = NtQueryDirectoryFile(Foo, Bar, Baz);

The real NtQueryDirectoryFile API takes 11 parameters which would be strenuous for this example since it requires pushing arguments to the stack. Now let’s look at the disassembly side by side with the getdents case to see how the three steps are handled on NT compared to Linux:

Step	Getdents	NtQueryDirectoryFile
1	mov rax, __NR_getdents64	mov rax, #NtQueryDirectoryFile
2	mov rdi, Fd	mov rcx, Foo
3	mov rsi, Buffer	mov rdx, Bar
4	mov rdx, sizeof(Buffer)	mov r8, Baz
5	syscall	syscall
6	cmp rax, 0xFFFFFFFFFFFFF001	test eax, eax

For the marshalling parameters step, we see that NT also uses rax to hold the syscall number but differs in the registers used to pass the syscall parameters because it has a different ABI. For the special instruction step, we see syscall is also used since it is the preferred method on x64 for making syscalls. For the final step of checking the return, the code is slightly different because NTSTATUS failures are negative values whereas Linux failure codes fall into a specific range. Just like with the Linux kernel, between steps 5 and 6 is where the NT kernel handles the NtQueryDirectoryFile syscall. This part is pretty much identical to Linux with some small exceptions around the ABI related to which user mode registers are saved and which registers are passed to NtQueryDirectoryFile. Since NT syscalls follow the x64 calling convention, the kernel does not need to save off volatile registers since that was handled by the compiler emitting instructions before the syscall to save off any volatile registers that needed to be preserved.

Syscall mechanics on WSL

After looking at how syscalls are made on NT compared to Linux, we can see that there are only a few minor differences around the calling convention. Next we’ll take a look at how syscalls are made on WSL. The calling convention for syscalls on WSL follows the Linux description above since unmodified Linux ELF64 binaries are being executed. WSL includes kernel mode pico drivers (lxss.sys and lxcore.sys) that are responsible for handling Linux syscall requests in coordination with the NT kernel. The drivers do not contain code from the Linux kernel but are instead a clean room implementation of Linux-compatible kernel interfaces. Following the original getdents example, when the syscall instruction is made the NT kernel detects that the request came from a pico process by checking state in the process structure. Since the NT kernel does not know how to handle syscalls from pico processes it saves off the register state and forwards the request to the pico driver. The pico driver determines which Linux syscall is being invoked by inspecting the rax register and passes the parameters using the registers defined by the Linux syscall calling convention. After the pico driver has handled the syscall, it returns to NT which restores the register state, places the return value in rax, and invokes the sysret\iretq instruction to return to user mode.

WSL syscall examples

Where possible, lxss.sys translates the Linux syscall to the equivalent Windows NT call which in turn does the heavy lifting. Where there is no reasonable mapping lxss.sys must service the request directly. This section will discuss a few syscalls implemented by WSL and the interaction with NT. The Linux sched_yield syscall is an example that maps one to one with a NT syscall. When a Linux program makes the sched_yield syscall on WSL, the NT kernel hands the request off to lxss.sys which forwards the request directly to ZwYieldExecution which has similar semantics to the sched_yield syscall. While sched_yield is an example of a syscall that maps nicely to an existing NT syscall, not all syscalls have the same properties even when there is similar functionality on NT. Linux pipes are a good example of this case since NT also has support for pipes. However, the semantics of Linux pipes are different enough from NT pipes, that WSL could not use NT pipes to get a fully featured Linux pipe implementation. Instead, WSL implements Linux pipes directly but still uses NT functionality for primitives like synchronization and data structures. As a final example, the Linux fork syscall has no documented equivalent for Windows. When a fork syscall is made on WSL, lxss.sys does some of the initial work to prepare for copying the process. It then calls internal NT APIs to create the process with the correct semantics and create a thread in the process with an identical register context. Finally, it does some additional work to complete copying the process and resumes the new process so it can begin executing.

Conclusion

WSL handles Linux syscalls by coordinating between the NT kernel and a pico driver which contains a clean room implementation of the Linux syscall interface. As of this article, lxss.sys has ~235 of the Linux syscalls implemented with varying level of support. This support will continue to improve over time especially with the great feedback we get from the community.

Stephen Hufnagel and Seth Juarez explore how WSL redirects system calls.