Intro to kernel debugging
blogs.msdn.microsoft.com
blogs.msdn.microsoft.com
It's also notable, that this year we're running Google Summer of Code[3] and hosting our own Radare Summer of Code[4]. And for the latter we're fundraising[5] the money to pay our accepted students.
[1] http://rada.re
[2] https://github.com/radare/radare2/blob/master/doc/windbg
[3] http://radare.today/posts/gsoc-midterm-2016/
[5] https://www.indiegogo.com/projects/radare-summer-of-code-201...
Seriously, thanks to the radare devs (if you are not one, xvilka, you talk like you are; in that case definitely thanks to you too). I am surprised in the era where people still people handsomely and brag about IDA Pro and scripts for it, I only learned about radare ironically from HN this year.
In the support scenarios, any dumps(can be up to multiple GBs of size as used by the process) collected and uploaded to the portal will be analyzed by a custom execution engine which is based on dbgeng.DLL .You can find a similar automated analysis with tool DebugDiag[1] which contains script for known issues such as crash,hang or high memory.
[1] https://www.microsoft.com/en-us/download/details.aspx?id=499...
I have been doing break-fix for Windows, *nix, and Mac systems for a long time. I have worshipped at the altar of Russinovich and Sysinternals gods for as long. I have wanted to be able to sit with Process Explorer, Process Monitor, and Windbg and know everything that goes wrong with a system from weird driver buginess to Word crash for reasons ignortant users can show me happening once a week but I cannot find evidence. I know the basics, but I really want to go up to the advanced levels with this. After years, Windows is still a black box to me, more than it should be after reading more than basic material, and that upsets me.
I see two problems here:
1) Where is good comprehensive material on the topic?
I see pages like this, but no unified material and training. On the Sysinternals book, I caved and bought the Bible, Sysinternals v6 back when it was current. I routinely show it for geek cred to other IT friends; few even know it exists. I met some more advanced consultants a month ago and learned of the Aaron Margosis companion (https://www.amazon.com/Windows-Sysinternals-Administrators-R...). But for the advanced stuff for WinDbg I know there are tier 2 customer engineers who know stuff inside out who post blogs like this, but find no really thorough books/material/courses I would gladly pay for to learn the basics of this skillset. Advice? Suggestions? I wish Julia Evans (jvns.ca) counterpart exists for the beauty of Windows debugging tools, but I have not found her/him/it yet.
2) Does tooling exist beyond the Sysinternals and debuggers I keep missing because I am uninformed? Help!?
On the point of Julia Evans, tooling in Windows is good, but I have not found much beyond the blessed debuggers and the gap between these debuggers+Sysinternals stack and that of the raw power tools of the Unix landscape of late: dtrace, strace, ltrace which go into crazy levels of introspection. I looked after reading this and stumbled upon maybe Dr. Memory but have not had time to learn it. Do you have counterpoints or suggestions?
Again, I would love to learn fu like Julia Evans in the Unix space and begin a love affair with the Windows side of debugging tools, but the landscape is more confusing. I would seriously love to hear from you.
Thanks so much for pointing us to DebugDiag. I have never found this and I am checking it out now. Very exciting!
UPDATE: I am reading the Code Machine posts listed in another comment, but it seems that Advanced Windows Debugging book might be the start of what I am looking for. Would love to know if you have other suggestions.
Many troubleshooting steps in that blog also covers DebugDiag .DebugDiag is an amazing tool which can help you collect dumps automatically for crash,high memory,IIS slow response etc.
Another tool which can help you profile your applications easily is PerfView https://channel9.msdn.com/Series/PerfView-Tutorial .
Perfview is built on the (Windows Event tracing) ETW infrastructure windows have .You can think of it as Lttng or perf_events linux equivalent but built into everything windows have. I think process monitor is also built on ETW. After debugging ( which is not required for lot of scenarios) the most powerfull thing in troubleshooting any issues in Windows is ETW traces IMHO.So understanding and analyzing ETW will help you troubleshoot a lot of issues.
https://blogs.msdn.microsoft.com/ntdebugging/2009/08/27/part...
XPERF is built to consume ETW traces https://blogs.msdn.microsoft.com/ntdebugging/tag/XPERF/ https://blogs.technet.microsoft.com/askpfeplat/2013/03/22/tr...
Also Advanced Windows debugging[1] and Advanced .NET debugging[2] to be very good for learning debugging in windows land.
[1] http://www.amazon.in/Advanced-Debugging-Addison-Wesley-Micro... [2]https://www.amazon.com/Advanced-NET-Debugging-Mario-Hewardt/...
Also, the dead comment appears to have alluded to the 71-660. Once other things settle down, I am diving into Sysinternals and these books when I had put them aside.
Seriously, thanks a bunch!
BTW, in case you don't know, Windows provides "public symbols" that has things like function names but lack things like local variable names.
For usermode problems WER would do a first level analysis on the client by looking at the crashing thread's exception record (1) or in the case of a user mode hang it would do something much more sophisticated called wait chain traversal (2). It would then generate a signature for the issue and report it to the backend via a single (and cheap) http/https request. The backend would then increment a count for that particular signature and if needed, would ask the client for more information about the issue. Among other things it could ask the client to capture a dump of the crashing process and even a multiprocess dump in the case of hang that involved multiple processes (process A waiting for process B waiting for process C etc). The client would do the necessary collection, prompt the user if necessary (depending on the user's consent settings) and eventually send it off. The 2nd level collection was configurable via a internal web portal - teams could look up their top signatures and request for more information about it. They'd eventually get notified once the extra information was collected - for high frequency issues, the turnaround was very quick.
Kernel mode issues aka bluescreens were treated slightly differently. The client wasn't able to do first-level analysis on kernel mode crashes. Instead it would always send at atleast a minidump of the bluescreen to a backend service where a pool of servers would run the !analyze windbg extension in realtime on the minidump.
(1) https://msdn.microsoft.com/en-us/library/aa260334(v=vs.60).a... (2) https://msdn.microsoft.com/en-us/library/windows/desktop/ms6...
[1]http://www.codemachine.com/articles.html
[2]http://advancedwindowsdebugging.com -is a very good book for user mode debugging
I used to have an RS-232 cable in my office, for Kernel debugging. Before VM prevalence the debugged and the debugee would be two physical machines, linked with a physical cable over their COM ports. Once the VPC/VirtualServer came out with COM redirect and the debugger support for it, I never looked back.