I'm not a Klingon ( )

元年 vs 1年 (Gannen vs Ichinen) with the 令和 Reiwa Era

Shawn Steele (MSFT) — Wed, 10 Apr 2019 19:54:38 GMT

In my previous post I showed how to use the Registry to enable the new 令和 Reiwa Era for the Japanese Calendar. Some customers have asked how to also display the 元年 (Gannen) for the first year. For current versions of Windows, we have set a registry value to select 1年 (Ichinen). You can remove this registry value to enable the 元年 (Gannan). The value that is selecting the Ichinen behavior is:

[Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Nls\Calendars\Japanese]
"InitialEraYear"="1年"

So that value need to be removed for Gannen (元年) support. Note that part of the reason we left it as Ichinen (1年) was a concern for the ability for all applications to support the Gannen properly, if your applications have difficulty with the Gannen, you may need to reapply the above value. A command line to remove the value:

reg delete HKLM\SYSTEM\CurrentControlSet\Control\Nls\Calendars\Japanese /v InitialEraYear /f

If you want to add the Reiwa Era and enable the Gannen behavior with a single registry file, you can use this example:

Windows Registry Editor Version 5.00

; Remove any existing test data
[-HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Nls\Calendars\Japanese]

; Set first year to 元年 or 1年 (元年 is the default if not present)
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Nls\Calendars\Japanese]
"InitialEraYear"="元年"

; Restore the era data, including 令和 (Reiwa)
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Nls\Calendars\Japanese\Eras]
"1868 01 01"="明治_明_Meiji_M"
"1912 07 30"="大正_大_Taisho_T"
"1926 12 25"="昭和_昭_Showa_S"
"1989 01 08"="平成_平_Heisei_H"
"2019 05 01"="令和_令_Reiwa_R"

Enable-Reiwa-and-Gannen Again, note that some applications may not be able to properly handle the 元年 (Gannen), in which case the 1年 reg value at the top of this post would need to be re-added to the machine.

Setting/Testing the New Japanese Riewa Era in NLS

Shawn Steele (MSFT) — Mon, 01 Apr 2019 19:19:30 GMT

[Update: https://support.microsoft.com/en-us/help/4469068/summary-of-new-japanese-era-updates-kb4469068 lists the updates available for Windows to enable Reiwa support] I talked about the upcoming new Japanese Era, and today the new era name was announced! Since the era data is available in the registry on Windows, people can add the Reiwa Era information to the registry immediately. Though Reiwa doesn't start until May 1, administrators may want to test their software with the new Reiwa era. You can add an entry for 2019 05 01 to the registry, such as this:

Windows Registry Editor Version 5.0
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Nls\Calendars\Japanese\Eras]
"1868 01 01"="明治_明_Meiji_M"
"1912 07 30"="大正_大_Taisho_T"
"1926 12 25"="昭和_昭_Showa_S"
"1989 01 08"="平成_平_Heisei_H"
"2019 05 01"="令和_令_Reiwa_R"

Note that though much of NLS recognizes the era, it doesn't actually begin until May 1st. Additionally, some software on Windows does not depend on NLS, so the registry entry may not enable Reiwa for all applications. Since the era doesn't actually start until May 1st, some parts of the system won't show the 令和 era until May 1st. Other parts of the system may use the new era as soon as it's exposed in the registry. If you want to test the complete behavior after May 1st to ensure that your system behaves as expected in Reiwa, you may want to temporarily adjust your system clock to a date after May 1st for testing. Depending on your preference for Gannen or Ichinen, you may also want to see this post: https://blogs.msdn.microsoft.com/shawnste/2018/11/12/gannen-vs-ichinen/

Gannen vs Ichinen

Shawn Steele (MSFT) — Mon, 12 Nov 2018 00:00:19 GMT

What is 元年 (Gannen)?

With the upcoming new Japanese Calendar Era comes a new "year 1" for the new era. Traditionally the first year of an era is labeled "元年" (gannen), which differs from the numbered year "1年" (ichinen). "Gannen" can be loosely translated as "the first year of the era", presuming the era is already stated in the date. For example in the date "‎平成‎30‎年‎10月‎31‎日" - 平成‎ is the Era name (currently Heisei), 30 is the year number, "年" is "year", 10 is the month number (October), "月" is "month", 31 is the date, and "日" is "day". However, for the first year of an era, 元年 is preferred, so a similar date would look like 平成‎‎元年‎10月‎31‎日. Note that "gan" by itself doesn't have the context and needs the "nen" syllable, so something like 1/9/1 would remain numeric instead of 元/9/1. Additionally, gannen requires the context of an era, so 元年‎10月‎31‎日 would be underspecified; "First year of what?" The era name is required to make sense to readers.

元年 (Gannen) in Windows

Since 元年 is the preferred spelling of the first year, the latest Windows 10 Insider Builds 18272 and above will by default use the 元年 instead of 1年 at the beginning of each era (so long as the Era name and 年 character are included in the date format.) The gannen behavior is enabled by default in updated Windows.Globalization and Win32 NLS APIs. This provides that when the new era is used next year, then the year will be correctly displayed as 元年. Since this change my disrupt applications that haven't previously seen the 元年 (gannen) spelling, it can be enabled or disabled by a registry value. When enabled, date formats including the era name and year character will display the first year of the era as 平成. Applications using the Windows.Globalization date formatting APIs and the Win32 GetDateFormatA/W/Ex APIs will exhibit this behavior. To disable the gannen (元年) behavior and use ichinen (1年) instead, set this registry value:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Nls\Calendars\Japanese
"InitialEraYear"="1年"

元年 (Gannen) and Apps

With Windows returning the 元年 string for the first year of an era, applications that merely use the OS APIs for formatting should hopefully behave fine. However, apps that do their own parsing of the OS formatted strings or otherwise manipulate the year value may need to be updated to accept the 元年 as well as 1年.

Japanese Era Ligature Abbreviations

Shawn Steele (MSFT) — Tue, 14 Aug 2018 18:45:47 GMT

Since I've been doing so much investigation around the Japanese Era, I noticed that our abbreviation isn't really the best. We keep the era names in the registry with the era start date so that we can adapt when a new era happens: Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Nls\Calendars\Japanese\Eras] "1868 01 01"="明治_明_Meiji_M" "1912 07 30"="大正_大_Taisho_T" "1926 12 25"="昭和_昭_Showa_S" "1989 01 08"="平成_平_Heisei_H"

These entries are in the form "Japanese Era name, Japanese abbreviation, English name, English abbreviation." As you can see, the Japanese abbreviation in the registry is merely the first Kanji of the full name - which isn't really an era abbreviation used in Japan. In Japanese you use either the two-Kanji string: 平成, or a Kanji ligature, ㍻, or the English abbreviation H. A partial Kanji name isn't really an option. Fortunately this information is in the registry, so if you want to improve the Kanji abbreviations for your Japanese Era names, all you need to do is tweak the registry values a little.

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Nls\Calendars\Japanese\Eras] "1868 01 01"="明治_㍾_Meiji_M" "1912 07 30"="大正_㍽_Taisho_T" "1926 12 25"="昭和_㍼_Showa_S" "1989 01 08"="平成_㍻_Heisei_H"

Adding those values will update the abbreviated eras for your machine's Japanese Calendar. There's a .reg file below to do this for you (you'll need to be an administrator to change these values.) Updated-Japanese-Era-Abbreviations

Data Transformations & Data Security

Shawn Steele (MSFT) — Thu, 09 Aug 2018 02:57:01 GMT

Over a decade ago I posted "Code Pages and Security Issues" - which touches on an interesting case of security checks when the data is transformed. In short: If you run a security check on your data, and then do some sort of transformation on that data, then your data may no longer be secure. In math speak, security tests and transformations are not "communitive." Their order is important. In other words: Do whatever transformations you need to do, and THEN do your security check. There are a bajillion things we do in computing that cause data to be transformed to some other shape or view. Very few of those are truly 1:1 operations, and therein lies the problem. The things that my input represents may not end up being represented the same way in my output. Indeed, if I have an input and an output string, then I must be expecting that the information is being transformed! (otherwise why would I be calling the function?) There are lots of places that this can happen. Some of them are more obvious than others:

Codepages (what I focused on last time)
Escape and Quoted Sequences
Named Entities
Casing conversions
String Normalizations
Parsing and Tokenizing
XML or other markup
Clients and Servers
Allow/Deny Lists
Starting in the Middle
Are the Klingon?
Everything I forgot (or can't know)

Codepages Codepages (aka Encodings) aren't transitive. I tried to think of one that is transitive for this example, but even the UTF transforms have different spaces of illegal codepoints. For example, a high surrogate in UTF-16 cannot be encoded in UTF-8 or UTF-32 without a matching low surrogate. If a bad byte creeps into your sequence, either accidentally or maliciously, then you're at the whim of the decoder's error handling, which could be dropping the unknown byte(s), returning replacement character(s), or even allowing the illegal sequence if the library missed that test case. Every other codepage is far worse. The set of characters allowed in Windows-1252 and UTF-8 differs wildly. Fortunately UTF-8 is a superset, however some APIs allow "best fit" mappings or other behavior. The conversion is guaranteed not to round trip, but whether characters are dropped, replaced with ?, or best-fit can vary. Even the meaning of the various codepoints can change depending on the OS platform or library being used. Larger encodings tend to be worse at that. Some codepages are depend on state from previous bytes. Jumping into the middle of the stream can cause misinterpretation of the data, as can a lost byte or corrupted bit. Sometimes even breaking the input buffers can lead to boundary issues if a lead byte is at the end of one block and the trail byte at the beginning of another. Escape and Quoted Sequences There are a ton of ways to escape various sequences, particularly when trying to "tunnel" a larger character set through a smaller channel, like Unicode characters being forced into an ASCII mechanism. Or the % mechanism for URL queries or the & Unicode encoding for XML/HTML. If the security check doesn't decode these sequences, then it is trivial to sneak a malicious string through the escaped string and into the target data stream. Worse, it is easy for the security test to miss a detail of the conversion. I included quotes in here as a special type of escape sequence. They can be particularly troublesome for some environments. Named Entities These are sort of like a special escape sequence. & or other tokens representing a special character. I'm calling them out because they might be easy to miss when trying to pre-parse a sequence. They exact sequences supported can also vary wildly, so a sequence from one system may be misunderstood on another. Security checks can't just look for the start and end of the sequence, because who knows what the decoder does with bad sequences? Worse, sometimes these things can be nested, so an escape sequence might tokenize a named entity, which is then processed later (or vice versa) Casing Conversions Note that there are other conversions that act similarly to casing but might be unique to non-Latin languages, such as the Kana or width mappings in Japanese, or digit mappings for many scripts. Casing is particularly tricky as many systems have different casing tables. Additionally, the linguistic behavior of some casing operations can vary depending on the language (for example, the Turkish I). An attacker might be able to sneak a İ through as system where i is important if the linguistic operations differ. Client validations could use different "rules" than a server side, etc. If a system depends on, say, lower case matching, for sensitive strings (password?) and the attacker can sneak an upper case letter through one layer due to differing rules, they may be able to find a weakness in the system. String Normalization We often need to normalize strings into some sort of sane form so that the application can deal with them. This can be something like Unicode Normalization, or other manipulations that get a variety of inputs into a manageable form. Doing a ToLower() operation on a string before manipulating it is one kind of normalization. Removing escape sequences could be another. Rearranging tokens into a predefined order might be another. Unicode normalization provides known mappings from various sequences to other interesting (to an attacker) forms and is an easy target. If different layers handle the normalization differently, there may be an opportunity to sneak something through. This can be particularly troublesome retrofitting code paths that expected only ASCII data to now handle Unicode. Early ASCII checks on a Unicode string are trivial to bypass when Unicode normalization occurs later in the process. Since most of the interesting tokens we use when parsing are in the ASCII range (?, =, ., @, /, etc.) this can be a big gap, particularly Parsing and Tokenizing Eventually we probably have to figure out what that input "meant." Chopping up a string into it's component bits is a form of transformation. Of course, this is one that most applications eventually need to be able to handle at some point. If I chop up a string into 10 key/value pairs, my application is going to need some sort of assurance that those 10 keys/values are as expected. Tricks here could be to use escape sequences, abuse delimiters, buffer sizes, or other mechanisms to confuse the tokenizer. If I can manage to confuse the tokenizer, I may be able to cause malformed tokens or malformed values with interesting side effects. Typically an application is going to try to ensure that the stream to be parsed is sane prior to any parsing effort, but it shouldn't necessarily assume that any critical outputs are legal. Maybe I disallow "password" in the parser, but if that's a valid key in a key/value pair and someone can trick the parser, then my app could get confused. XML and Other Markup This is basically like parsing/tokenizing, but I divided it because this probably has more complex structure than a "simple" key/value query string. Here a misplaced escape token or mapping might be able to trick the markup into confusing the layers, perhaps injecting a parent that makes a critical block vanish into a hidden child. The richer the markup mechanism, the more opportunities there are to attack the surface, and if that markup is passed through multiple libraries, perhaps having different transformations, then someone may be able to take advantage of the discrepancy. Allow & Deny Access Lists A bulletin board I play with had some functionality like this, granting or denying permission based on lists of information. If someone can sneak an alternate form of a string past an "Deny Access" list, then they could gain unintended access. This could be as simple as alternate forms of bad-language in a post filtering list, or manipulation of a search index by providing unexpected forms with similar meanings. Starting in the Middle (or the beginning, or the end) Unfortunately, computers often have to deal with large data files and other objects and it can be inconvenient to manage the entire entity at once. Stateful encodings are particularly annoying as a switch at the beginning of a 20GB file could change the meaning of a block read at the end of the file. But this can even happen to simple structures. If I start looking at an HTTP query string from the end, I might stop when I reach a ? (presumably the beginning of the query string). Yet another parser might look from the beginning and assume that the query starts at an earlier ?. It Doesn't Have to Be Malicious Maybe the user's just trying to see if they can get a Klingon user name on your music service. Everything I Forgot (or Can't Know) I'm sure I missed some interesting examples where useful transformations can lead to interesting behavior. And obviously I don't know about all the cool applications out there that might have some brilliant protocol that connects their clients to their servers. But even then, if it has an input being transformed to some other output, it is likely subject to the same sorts of transformation problems that lead to security issues. It's important to ensure the security check is being done on the "right" form of the data. Reverse Engineering the Transformation Logic The solution to this transformation problem and the need for an early security check is fairly obvious: Make sure that the security check "understands" the transformation. Now, I want you to pause and take a moment to think about that. This is very appealing. If the security check can know what the transformation is going to do, then maybe we can get an early read on the security of the inputs before sending the stream on to the business logic of our system. (Kind of like a firewall I suppose, making sure that only good data gets through). But: Is reverse engineering the transformation realistic? Is my check going to be able to really do EXACTLY the same thing that the eventual transformation does? When that business logic does its transformation, it's going to call an API in a library or the system to help it. If there's a bad UTF-8 byte will it be dropped? Turned into an � or just left in to trip me up? Is the library an über-library that tries to undo % escapes, named entities and quotes all at the same time? In which order? Will a named % turn sneak through to be decoded as a % escape later? I do know that it is very difficult to predict how these transformation APIs behave exactly in all circumstances. Even then they may be updated to fix a bug. Standards, such as the UTF-8 encoding guidance, have changed over time as security weaknesses were discovered. You might have an older library - or be using a newer one. Conclusion I've encountered an endless stream of bugs from applications that are convinced they *MUST* do something very important *BEFORE* doing the transformation. I'm pretty sure that after jumping through a ton of hoops to try to get things to work, they still have troublesome edge cases. Try not to be that app. Hope this helps someone, Shawn

Using the Registry to Test the New Japanese Era on Windows

Shawn Steele (MSFT) — Tue, 07 Aug 2018 20:37:31 GMT

[Updated to include 令和 (Reiwa) instead of the test entry] Japanese Era Registry Key I've posted about using the registry to test for the Japanese Era before (Win7+) -- some folks were looking for a little more guidance, so I created a couple registry files to help out. Japanese calendar eras are enumerated in the registry, so adding a test era like the one below can help test your application's behavior when the era changes next year. Windows Registry Editor Version 5.00 [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Nls\Calendars\Japanese\Eras] "1868 01 01"="明治_明_Meiji_M" "1912 07 30"="大正_大_Taisho_T" "1926 12 25"="昭和_昭_Showa_S" "1989 01 08"="平成_平_Heisei_H" "2019 05 01"="令和_令_Reiwa_R" Selecting the Japanese Calendar To test the Japanese Calendar behavior, your machine has to be configured to use the Japanese Calendar. This can be done on any machine, it doesn't have to be a Japanese Language machine. For thorough testing you may want to install the Japanese Language Pack as well, but it isn't required for basic testing. In either case, the Japanese Calendar is only available to the Japanese locale.

1) Open intl.cpl (Win+r, intl.cpl, ) 2) Select "Japanese (Japan)" from the "Format" drop down 3) Select "Additional Settings" 4) Select the "Date" tab. 5) In the "Calendar Type" dropdown, select the 2nd calendar "和暦", which is the Japanese Calendar. (The 3 calendars listed are Gregorian (in Japanese), Japanese Calendar, and Gregorian (in English) calendars) 6) Select "Apply", "Ok" 7) Select "Apply", "Ok".

Your computer will now be configured to use the Japanese Calendar (and will reflect whichever eras are registered in the registry). Reg Files For Testing You can tweak the registry value(s) as described above to test your app or machine's behavior, but I've created a few .reg files that you can register to cover a couple of common test cases. The registry files are in a .zip downloadable from the link at the end of this post.

RestoreJapaneseEra.reg to restore to the no-new-era state when you're done testing.
AnticipatedJapaneseEra.reg which includes test date for the anticipated 1 May, 2019 transition date.
ActiveTestJapaneseEra using a date that has already past to test behavior on machines as if there were already a new era.

Restore After Testing Double click on the RestoreJapaneseEra.reg file to restore a machine to a state without a "new era". You probably want to use this after testing to avoid confusion. Note that this will also revert the entries for a machine that has an officially released update after the official transition happens - so you probably don't want to do this after the official era has been announced! Anticipated Era Transition Testing AnticipatedJapaneseEra.reg adds a test era with the anticipated 1 May, 2019 date. Add this to your registry to test the upcoming transition.

This is a test era and the test era names are not the "real" name of the new era, do not use them in a production enviornment!
Era names for this test are expected to be "新元号" or "新" (abbreviated) - which I think means something like "new era." The English form is "Test Era" and "E".
The transition date is the anticipated 1 May 2019 date. This can help test applications around the expected official transition date.
Since the test date is in the future, current dates won't show the new information.
The computer's clock may be advanced to dates after or around 1 May 2019 to test behavior at or after the official transition date.
Any API's that don't explicitly select an Era should continue to see era information appropriate for the current 平成 (Heisei) Era. (Until the computer's clock advances to 1 May 2019.)

Active Era Transition Testing Add ActiveTestJapaneseEra.reg to your registry to test with a new era start date of 1 May 2018 (a year prior to the anticipated date). This is helpful to test application behavior as if a new era already existed without requiring the machine's clock to be advanced to a future year.

Era names for this test are expected to be "？？" - Japanese Full Width Question Marks. (English Era name is ?????? for this test era)
?? is expected - This is to avoid any geopolitical/cultural issues around using a real word for the new era. It should still be clear in an application that the date is correctly showing the new era.
If you choose a string other than ??, please don't share any of that information externally.
Transition date is 1 May 2018, so any dates, including the current machine time, after that date should display as the new era.
Any APIs that don't explicitly select an Era should see display information appropriate for this test Era as it is the "current" era in this test configuration.

Notes on System Behavior When testing the Japanese Era behavior, please note that we are also in the process of testing and addressing issues that Japanese Era testing may uncover. The most recent Insider releases have the most reliable Era behavior. Test Registry Files For convenience, some test registry values are in the zip file below. There is one file to test the anticipated era start date, another to test with an already active new era with a past date, and the last file to restore the era entries after testing. JaEraTest

The Japanese Calendar's Y2K Moment

Shawn Steele (MSFT) — Thu, 12 Apr 2018 20:58:19 GMT

Japanese Calendar Eras

The Windows 10 Spring Release includes a placeholder for the era expected to begin on 1 May, 2019. That information is in a registry key that can be removed or edited in the event that a system's software misbehaves with this additional information. The Japanese Calendar has Japanese Era Names that coincide with the reign of the Emperor. For most of the modern age of computing that has been the Heisei era, however the Emperor is expected to step down on April 30, 2019. Which will bring about the beginning of a new era. Fortunately, this is a rare event, however it means that most software has not been tested to ensure that it will behave with an additional era. The magnitude of this event on computing systems using the Japanese Calendar may be similar to the Y2K event with the Gregorian Calendar. For the Y2K event, there was world-wide recognition of the upcoming change, resulting in governments and software vendors beginning to work on solutions for that problem several years before 1 Jan 2000. Even with that preparation many organizations encountered problems due to the millennial transition. After the era has changed it will be too late to test for compatibility problems. Therefore, the Windows 10 Spring Release includes a registry entry with placeholder information for the expected Era transition. This is intended to help users discover any software limitations around the expected change to the new era. Users are encouraged to ensure that their applications are well behaved before the actual era change. In the event that applications don't work as expected with the additional era, the registry value may be modified or removed as needed. Should problems be encountered, users are encouraged to contact their software provider(s) or otherwise investigate possible solutions. Next year we expect and official announcement of the new era and name, however well before May 2019 it will be too late to begin testing for computing issues. The registry value that controls this behavior may be modified by and administrator using regedit or other tools. It is located at Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Nls\Calendars\Japanese\Eras within the registry. In that key there are 5 string values for the eras Windows knows about. The last value "2019 05 01" contains the temporary information "？？_？_??????_?" That "2019 05 01" value may be removed or modified if need be to work around any software limitations. In the event that removal is required, we recommend that the problem not be ignored as the user will probably want the new era to be usable when the final announcement is made in 2019. When the final name is announced by the government, and update will be provided to replace the placeholder test strings with the official era name.

Things to Watch For

There are several interesting conditions that an additional era provides and current software may not expect.

Calendar controls may presume there is only one "current" era, and disallow navigation to a later or previous era. This is likely to pose complications around the transition date.
Applications may have stored future dates in the previous era expecting those to be parsed later. Should those dates represent times in the new era, use of the previous Era and year number may confuse parsing algorithms. An example would be a date in year 40 of the old era that is may be expected to be written using year 10 of the new era instead.
Future dates written using a new era name and year numbering mechanism may be difficult to compare with the same date written using the previous era name and larger year number. For example, year 32 of the old era may be the same date as year 2 of the new era.
Calendars may need to describe which Era a date belongs to, particularly if the Era changes between months or within a month.
Some algorithms attempting to count the years during a transition year may not consider the possibility of two partial Japanese Calendar years, in two different Calendars Eras, within the same Gregorian year.

Testing for New Japanese Era

Shawn Steele (MSFT) — Fri, 12 Jan 2018 19:57:31 GMT

Some time ago I mentioned that there's a registry key that has values admins can edit if they want to modify the Japanese Era data for the system. With the news that Emperor Akihito will be abdicating next year, people may want to check if their software works with an additional Era. Since a name of the new era hasn't been announced yet, I've been using a placeholder for testing. If you'd like to check out how your Windows system and software will behave with the additional Japanese calendar Era, you can add an entry for 2019 05 01 to the registry, something like this: Windows Registry Editor Version 5.00 [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Nls\Calendars\Japanese\Eras] "1868 01 01"="明治_明_Meiji_M" "1912 07 30"="大正_大_Taisho_T" "1926 12 25"="昭和_昭_Showa_S" "1989 01 08"="平成_平_Heisei_H" "2019 05 01"="？？_？_??????_?"

LCIDs are are really bad idea

Shawn Steele (MSFT) — Thu, 23 Nov 2017 19:00:25 GMT

Well, it's been 4 years since I posted about the deprecation of LCIDs, though I started discussing the topic over a decade ago. It's pretty much time to dump LCIDs. LCIDs were an OK idea when they were invented, though a couple quirks quickly broke their pattern. However on a modern Windows 10 machine they're not nearly enough information to describe their intended locale. In the past I've warned that there isn't enough room in the namespace for LCIDs for every locale, and then I warned about the restrictions that would prevent applications from seeing custom locales. And then may have mentioned that we've added a few esoteric locales without LCIDs. But in the modern Windows, half of the locales we ship in-box do not have assigned LCIDs. That means that applications depending on LCIDs are unable to accurately specify half the locales and languages on the box. Even worse, there are thousands of languages in the real world and hundreds of countries. Users that have correctly requested or tagged data with BCP-47 languages ids for those languages will be not be recognized by applications limited to LCIDs. That leads users to either mis-tag the data or mislead the computer about their desired settings. Which makes matching even more difficult. Additionally the LCID concept confuses other systems that depend on the standard names, making data interchange and requests with LCIDs awkward and error prone. Those systems have to do conversions and have little incentive to ensure that the LCID code path is perfect. With Windows 10, Windows will do it's best to provide information for whatever language tag the user or application has requested, meaning that there are millions (literally) of valid tags that the system recognizes. Less than two hundredths of a percent of those have assigned LCIDs. 0.02%. So: Please don't use LCIDs..

Windows Language Groups are Dead.

Shawn Steele (MSFT) — Tue, 21 Nov 2017 19:51:53 GMT

Well, they've been dead for a long time; I pretty much killed them in the middle of the XP time frame. Even then they were reasonably meaningless, but I finally added some "deprecated" tags to some of the APIs. Short story: please don't use language groups, they're silly. Longer version: Once upon a time computers didn't have a lot of storage. I mean, we were thrilled when denser floppy drives came out. (Not bigger, the bigger the floppy, the less data it stored). An 80MB (MB, not GB) hard drive was once considered big. NLS (national language support) is big. We need lots of data for the sorting tables, code page tables and other information, particularly for Chinese, Japanese & Korean, which have large character sets. One way to save the limited space, on say a German machine, is to not install all those other unused languages features. Originally Windows also didn't support as many locales/languages as it does today. So, everything was chopped up into "language groups". Applications could query to see if the language group (and locale) were "supported" or actually "installed" on a particular machine. If supported and not installed, they could prompt the user to install it if they wanted to use it. One of the first things I did in the Windows Globalization team was to say "gee, this isn't really a lot of data by modern standards, and apps keep getting confused by unavailable stuff, so why don't we just install everything all the time?" And so I went into the intl.cpl code and ripped out all that stuff allowing you to install (or uninstall) the various language groups. At that point in XPSP2 if I recall, language groups were effectively dead. But we still had the idea of "language groups" baked into some APIs, so as some new locales were added, we tried to pick groups for them, but the associations have always been somewhat loose and arbitrary. Eventually new locales just started getting assigned to "group 1", and now most of the locales on the system think they live in group 1. The language group architecture also depends on the Evil LCID concept. We can't assign groups to the newer no-LCID locales (which is most of the languages). That means that if you start asking which languages live in a group, your list is going to be woefully incomplete. Anyway: please don't use language groups, they're meaningless.