Windows 10 works incorrectly with Large memory pages
sourceforge.net
sourceforge.net
If you are a paying enterprise customer of Microsoft you can get very good support. I briefly worked for a company that ran their compute infrastructure on Windows (~2000 physical servers), in the month I worked there they had multiple tickets open with Microsoft. If I remember correctly, one was for excess memory usage and Microsoft dug through a memory dump to find the problem.
I hear Google has improved for paying enterprise customers, under Diane Greene's leadership, but they certainly burned some bridges in the past with poor support.
Public bug trackers are super helpful for professionals, but the lack of one isn't necessarily an indication of poor customer service.
[1] https://support.microsoft.com/en-us/assistedsupportproducts
At least, after I managed to get past the faux-IVR, the support staff was competent enough.
I use it every so often when I notice something peculiar. None of my feedbacks actually became so popular. Even then I know one of them has been fixed. The fix probably didn't come because I have pointed it out, but I was happy about it regardless.
I couldn't find a web-link to the Hub, so I think it is not entirely public, but surely available to the Windows 10 users.
[1] https://support.microsoft.com/en-us/help/4021566/windows-10-...
After adding one, I found out that when the display goes to sleep, Windows sees it as a disconnection, and 'helpfully' wakes up to reorganize all the windows onto one display. That wakes up the Display port monitor, which triggers another reorganization, and the cycle begins again.
Googling didn't get me too far obviously.
I ran into this after building a system and it was very frustrating. IIRC my problems were related to my network card and having virtualbox installed. Also turning off wake for my mouse helped (bumping the desk would turn on the computer).
Wasn't any device, like network adapter and power diagnostics returned nothing.
Solution: reset motherboard NVRAM. No more mystery wakeups!
Thanks svchost.exe!
I disabled updates, but it still does... something while I am sleeping.
1. The start menu latency is now so bad you have to forcibly wait 2-3 seconds before typing your search after hitting the start button. This is on a stacked E5 Xeon workstation class machine.
2. Search is crap. I can type "visual" and visual studio doesn't come up!?!?
Some of the problems I understand; “regedit” has to match exactly because it’s basically doing the “run” dialog box. But no idea why other stuff can’t be searched for.
An opposite case: https://bugs.chromium.org/p/chromium/issues/list
But as a non professional developer-windows user, I don't feel like it is my job to spend some time filling a detailed bug report with crash dump and reproducable steps. That's kind of the Micrisoft support/QA team offloading their job on their users.
This seems to affect a number of models from a number of OEMs that use the same "template".
In once case we got Scott Hanselman's attention via HN on an issue but the outcome was clearly that a major part of SCVMM's support for Linux was a hack job and mostly abandoned and their own assigned team member didn't understand the problem domain properly.
Chuck it on github for open source lip service then ignore it. Add telemetry on by default. Make it a paid service. Anything to hurt the customer's trust.
And I'm going to keep complaining loudly about this until it changes.
The googleproductforum.com (whatever it's called) which is usually the #1 result in Google is absolutely worthless.
I am not joking. It is infuriatingly worthless.
Igor is infamous for disabling virtually every setting. 7-Zip has no ASLR, compiler-stack checks, or anything what-so-ever. Igor refuses to use a modern VC++ (Visual Studio 2015 and later have FREE versions available for open-source code) that would solve a lot of security issues and bugs.
I'm not entirely sure if this is a Win10 problem, or if its a 7-Zip problem. Any complaints from the 7-Zip dev IMO will require a very careful eye: 7-Zip code is not necessarily in the cleanest state or using the best practices.
I don't know Igor, but if 7z is anything to go by then he probably uses VC--I'm guessing here--6 for the same reason Sean Barrett does: It's way faster and less bloated than modern VS. I'm also guessing he doesn't use those security features because they're largely ineffective and just complicate things and slow them down.
I'm sure many will argue the last point, but it's hard to imagine that that mindset doesn't play a role in 7-zip being as great a utility as it is.
> I don't know Igor, but if 7z is anything to go by then he probably uses VC--I'm guessing here--6 for the same reason Sean Barrett does: It's way faster and less bloated than modern VS.
Then they should learn how to download the SDK and learn to use the command line to properly compile code with proper ASLR and other such security features. Keep working in VC 6.0 if you want, but for the love of all things good please enable basic ASLR.
Its 2018. Its time to get with the program. Various solutions (or at least... mitigations) to buffer overflows and code execution bugs have been discovered in the last 20 years. Yes, VS 6 was released in 1998. My use of the phrase "20 years" is literal.
Its shameful that a dev of one of the most popular open source tools out there doesn't care about security.
--------------
In any case, Igor runs his dev environment from 20 years ago. If someone was compiling code with GCC 2.95 (released 2001, three years after his version of VS), the first response from Linus Torvalds would be "Dear lord, please upgrade your compiler. I'm not going to ensure compatibility with 18-year old tech".
> I'm also guessing he doesn't use those security features because they're largely ineffective and just complicate things and slow them down.
ASLR is ineffective? Really?
https://www.cisecurity.org/advisory/multiple-vulnerabilities...
https://landave.io/2018/01/7-zip-multiple-memory-corruptions...
The freaking executable doesn't have the freaking "NX Bit". We're talking about the most barebone basics of security here.
No ASLR. A lack of NX Bit. Pretty much any security feature discovered in the last 20 years is missing from 7zip. Its actually one of the worst offenders of security I've ever seen in 2018.
https://www.cvedetails.com/cve/CVE-2016-2334/
I mean, there are easy databases to look for these sorts of questions...
DEP would be nice, wouldn't it?
IIRC, Igor finally added DEP a few months ago (NXCOMPAT if you need to google the Internet Drama). But the fact that he was running this code without lol DEP until January of THIS YEAR makes me suspect his coding practices.
As per this other page:
https://landave.io/2018/01/7-zip-multiple-memory-corruptions...
> I have discussed this issue with Igor Pavlov and tried to convince him to enable all three flags. However, he refused to enable /DYNAMICBASE because he prefers to ship the binaries without relocation table to achieve a minimal binary size. Moreover, he doesn’t want to enable /GS, because it could affect the runtime as well as the binary size. At least he will try to enable /NXCOMPAT for the next release. Apparently, it is currently not enabled because 7-Zip is linked with an obsolete linker that doesn’t support the flag.
This is bad. Very very bad. This should not have been happening in 2018. He wants to save literally ~kilobytes of .exe space instead of use ASLR. Igor is not someone who seems to take security seriously.
----------------------
In any case, there are ways around DEP. Which is why ASLR exists, and other such flags. Enabling DEP brings 7-zip into the year 2003 (when x86-64 was first released by AMD) level of security, but there are other security threats that are addressed with these other compiler flags.
On this we agree. The question is, how seriously does security deserve to be taken in this case?
The evidence suggests not very seriously at all. As horrifically insecure as 7z is by your metrics, I can't think of a single incident of it ever ruining anyone's day, unlike a lot of modern software that manages to do that while supposedly being "secure" and functioning "correctly".
Well consider exactly what 7-zip is: its a plugin to explorer.exe that will always be running. Its embedded into your "right click" button and is running at all times.
Any code that takes over 7-zip can take over explorer.exe on your typical use of 7-zip. I personally think its position as a explorer.exe plugin demands some kind of security posture.
[HKEY_CLASSES_ROOT\7-Zip.7z\shell\open\command]
@="\"C:\\Program Files\\7-Zip\\7zFM.exe\" \"%1\""I can understand Igor like... not knowing about these features or whatever. But its been months / years since he was notified about security and yet he doesn't take any corrective steps.
I'm seriously considering to pay for WinRAR. No joke.
The one tool that I did find useful during this search though is https://github.com/ImminentFate/CompactGUI.
Apparently in Win10, Microsoft introduced a compact.exe commandline tool that transparently compresses files/folders. The above tool is a GUI for it. It's not a replacement for a proper archiver but it's a nice tool to compress things like games and other large programs that need to stay in place.
And what does "anything what-so-ever" mean? If he does static analysis on the code and runtime analysis (e.g. the valgrind type), I don't expect that Microsoft's runtime checking will do much more in practice.
None of what you point out really speaks about the quality of his code. Does he have a history of security issues and bugs that's significantly different from others?
Oh come on. The Linux Kernel also "lives and dies" based on its performance. Despite being one of the highest performance codebases ever written, Linux is still ASLR'd, Meltdown-patched, TLB-flushed and Reptoline-protected to all hell to mitigate against theoretical threats.
Web Browsers also live and die by performance numbers. Look at all of those Javascript benchmarks and shootouts that happen. And yet, web browsers are also taking advantage of the latest and greatest security features available on any OS they're deployed on.
Because security matters. And often, it matters more than performance.
7-Zip's DLLs / code are used as plugins everywhere. Not just in Windows 7z.exe, but also on Fedora Core, Ubuntu Linux, and more. There comes a point where we as a community have to recognize that security matters more than performance.
In most cases, there's a happy medium. I'm not necessarily saying that 7z needs to sandbox itself like Chrome, or needs to spin itself into a VM and isolate itself like MS Edge App Guard. Nor does it need to be Spectre-hardened with reptolines like the Linux kernel. All I'm saying that 7z should at least be compiled with the freaking DEP / NX flag and ASLR. Neither of these have major performance flaws and have HUGE security benefits. Is that seriously too much to ask?
Besides, ASLR + DEP/NX flags are the default compile option flags for the last 15 years. Most software uses these flags with no issues.
// A Windows bug exists where a VirtualAlloc call immediately after VirtualFree
// yields a page that has not been zeroed. The returned page is asynchronously
// zeroed a few milliseconds later, resulting in memory corruption. The same bug
// allows VirtualFree to return before the page has been unmapped.When you use Large Pages and you run out of contiguous 2MB chunks, what do you do then?
Unlike Linux, Windows actually guarantees its memory to anything that requested it. Windows does NOT ever "take back" memory and crash processes randomly (see Linux's OOM killer). But this guarantee has its own issue on Windows: important services who make requests for new bits of code will crash instead.
So Large Pages naturally will run out the longer a system runs. They are a limited resource: how often do you find a contiguous 2MB block when most programs request memory in 4kB blocks?? And the longer a system runs, the fewer 2MB blocks will exist.
I guess Linux handles the issue by making normal pool, large pool, and "huge" pool all separate. So you can run out of normal-pool but have lots of large-pool space remaining. But this has the disadvantage of being wasteful (Ex: 1GB Huge Pool permanently eats up 1GB that the smaller pools can't ever use).
------------
Ultimately, applications aren't supposed to use the OS-level memory allocator as if it were malloc / free. Because when fragmentation hits you in malloc/free, you mess up your own memory.
But if fragmentation hits you at the OS-level, you're basically screwing the entire system.
But, unless the analysis in the OP is wrong the problem stems from Windows handing out freed huge-pages again before zeroing them, so whichever program gets the page the second time might write data to it and then have it wiped a moment later, which sounds like a plain bug to me.
https://oracle-base.com/articles/linux/configuring-huge-page...
> That sounds like a good idea, but unfortunately Transparent HugePages don't play well with Oracle databases and are associated with node reboots in RAC installations and performance problems on both single instance and RAC installations. As a result Oracle recommends disabling Transparent HugePages on all servers running Oracle databases, as described in this MOS note.
-------------
Anyway, the very point of huge-pages is to gain ~3% to 5% faster program speed by keeping more of the memory-management units of the OS inside of the TLB-cache of the CPU.
Running a background thread to defragment the physical memory system of your computer WHILE your critical tasks are running... very most likely kills the performance benefit you were trying to get.
Linux's raw Hugepages work fine, but have their own set of drawbacks as I described in the previous post. For the moment, it seems like a better idea to use "normal" Linux Huge Pages than to use the THP. Indeed, I'm seeing reports that some people get 10s of millisecond pauses when their application does a 32-byte malloc with THP enabled (Linux Kernel decides to garbage collect + defragment + turn your pages into a Huge Page). A lot of programs don't expect a tiny malloc to cause such a major performance hit randomly (ie: video games).
https://groups.google.com/forum/#!topic/mechanical-sympathy/...
https://access.redhat.com/solutions/1560893
---------------------
Best practice (in both Linux AND Windows) is:
* Code your program to accept Large Pages if possible. But consider the failure case, and fall-back to normal 4kB pages if you fail.
* If performance is CRITICAL (ie: servers), then allocate the large page at the beginning of the program, and NEVER LET IT GO. Start the program early when the OS boots up to ensure that Large Pages are available.
* Linux's alternative is to have the human / SysAdmin manage large pages manually. Divy them out to the programs as needed. Which works just fine, although its a bit of a hassle.
(requires Feedback Hub App on Win 10 to view)