As we just entered 2024, let me wish you peace, happiness, and positivity in the new year. And, of course, many successes in your troubleshooting endeavors :)
]]>Thank you for being a wtrace.net subscriber and until the next time!
PS. If you need to interact with native libraries on Windows, I highly recommend checking the CsWin32 project. It helped me tremendously in generating the C# bindings for the Detours library. In case you are interested, I described the whole process in a post on my blog.
There were also changes to the way how comon manages breakpoints. They are now public and visible in the user session. Their command field has a comment explaining a given breakpoint purpose, for example: * [comon] function breakpoint (name: RPCRT4!DllGetClassObject)
.
I also made some changes to the site. The tools documentation was moved to READMEs in their repositories, and I created a brand new Articles section for various troubleshooting tutorials. COM troubleshooting is the first one published, so check it out if you are interested in this subject.
]]>11:42:41.7551 winver (9620.13744) RPC/ClientCallStart 'fb8a0729-2d04-4658-be93-27b4ad553fac (lsapolicylookup) [0]'
And in the summary view, we would see the number of calls to a given RPC procedure:
--------------------------------
RPC
--------------------------------
fb8a0729-2d04-4658-be93-27b4ad553fac (lsapolicylookup) [5] calls: 4
One thing that was missing in this output was the procedure name. Until now, I recommended using RpcView to decode it. RpcView is an excellent tool, but its binaries are not always available, and you may not have time (or possibility) to build it. Fortunately, some time ago James Forshaw added support for RPC servers to his excellent NtApiDotNet library. Querying RPC information is only a tiny part of the library, but it is all I needed for wtrace. Additionally, you may call some of the NtApiDotNet functions in PowerShell by using the NtObjectManager package. Check out this post on the Google Zero Project blog if you’re interested.
Moving back to wtrace, thanks to NtApiDotNet, wtrace 3.3 will try to resolve RPC procedure names in the summary view. I will later describe how it’s done, but first, let’s have a look at the final result. Instead of seeing:
--------------------------------
RPC
--------------------------------
fb8a0729-2d04-4658-be93-27b4ad553fac (lsapolicylookup) [5] calls: 4
We will now see:
--------------------------------
RPC (client calls)
--------------------------------
fb8a0729-2d04-4658-be93-27b4ad553fac (ncalrpc:[lsapolicylookup]) [5]{LsaLookuprGetDomainInfo} calls: 2
As you may have noticed, RPC calls are now split between server and client calls, and wtrace prints the procedure name (in curly braces) next to its number. Displaying meaningful procedure names requires access to the debugging symbols. If you have the _NT_SYMBOL_PATH
environment variable set (I highly recommend configuring it), wtrace will use it. Otherwise, you need to set debugging symbols path through the --symbols
parameter, for example:
wtrace.exe --symbols="SRV*C:\symbols\*https://msdl.microsoft.com/download/symbols" -v notepad.exe
The procedure resolution works only for local RPC servers. Apart from RPC parsing improvements, wtrace 3.3 includes minor fixes and optimizations, so give it a try the next time you need to trace something in the system.
If you’re willing to learn a bit more about how RPC name resolution works, the next section is for you :) Otherwise, thank you for using wtrace and if you’d like to keep in touch, please subscribe to the wtrace newsletter
The payload of the RPC ETW event gives us enough data to build a binding name (constructBinding
method in the wtrace RPC event handler). With the binding name, we may query its published interfaces using the RpcEndpointMapper.QueryEndpointsForBinding
method from NtApiDotNet. Under the hood, it calls RpcMgmtInqIfIds
. For each retrieved interface, NtApiDotNet builds a RpcEndpoint
object by calling a family of RpcMgmtEpEltInq*
functions, providing the interface UUID and version. In the next step, we need to learn which process hosts a given endpoint. This part is a bit complicated and depends on the protocol used in the communication. Check the _factories
dictionary in the RpcClientTransportFactory
class to learn how NtApiDotNet constructs various connection points. When we know the process ID, it’s time to search through process modules, looking for NDR structures. The RpcServer.ParsePeFile
method parses a DLL/EXE file and, among many other things, retrieves information about the RPC interface procedure addresses. With the symbols available, we can then decode their names. As you can imagine, resolving every RPC binding is a time-consuming process. I do that asynchronously in background threads (RpcResolver
class), but it could happen, especially when the trace session is short, that wtrace will require some additional time to finish resolving the queued endpoint names.
The !cometa command parses COM metadata found in the registry or standalone TLB/DLL files. Thanks to the metadata, comon will decode CLSIDs and IIDs to their human-friendly names. You may also query the metadata with the showc or showi subcommands.
Here is a short snippet of a debugging session with comon:
0:000> .load c:\Users\me\Debugging Tools for Windows\extensions-x86\comon.dll
[comon] Opening an existing metadata database from 'C:\Users\me\AppData\Local\Temp\cometa.db3'.
0:000> !comon attach
[comon] COM monitor enabled for process 0.
0:000> !cometa showi {59644217-3E52-4202-BA49-F473590CC61A}
Found: {59644217-3E52-4202-BA49-F473590CC61A} (IGameObject)
Methods:
- [0] QueryInterface
- [1] AddRef
- [2] Release
- [3] get_Name
- [4] get_Minerals
- [5] get_BuildTime
Registered VTables for IID:
- Module: protoss (32-bit), CLSID: {EFF8970E-C50F-45E0-9284-291CE5A6F771} (Protoss Probe), VTable offset: 0x37854
- Module: protoss (32-bit), CLSID: {F5353C58-CFD9-4204-8D92-D274C7578B53} (Protoss Nexus), VTable offset: 0x37808
0:000> sxe ld:protoss.dll
0:000> g
...
ModLoad: 70290000 702d4000 c:\Users\me\repos\protoss-com-example\Debug\protoss.dll
ntdll!NtMapViewOfSection+0xc:
774a4e7c c22800 ret 28h
0:000> !cobp {F5353C58-CFD9-4204-8D92-D274C7578B53} {59644217-3E52-4202-BA49-F473590CC61A} get_Name
0:000> bl
8 e Disable Clear 702a1505 0001 (0001) 0:**** protoss!ILT+1280(?get_NameNexusUAGJPAPA_WZ) "..."
0:000> g
...
[comon] 0:000 [IUnknown::QueryInterface] CLSID: {EFF8970E-C50F-45E0-9284-291CE5A6F771} (Protoss Probe), IID: {00000001-0000-0000-C000-000000000046} (IClassFactory)
[comon] 0:000 [IUnknown::QueryInterface] CLSID: {EFF8970E-C50F-45E0-9284-291CE5A6F771} (Protoss Probe), IID: {00000001-0000-0000-C000-000000000046} (IClassFactory)
[comon] 0:000 [combase!CoGetClassObject] CLSID: {EFF8970E-C50F-45E0-9284-291CE5A6F771} (Protoss Probe), IID: {00000001-0000-0000-C000-000000000046} (IClassFactory)
[comon] 0:000 [IUnknown::QueryInterface] CLSID: {EFF8970E-C50F-45E0-9284-291CE5A6F771} (Protoss Probe), IID: {246A22D5-CF02-44B2-BF09-AAB95A34E0CF} (IProbe)
[comon] 0:000 [IUnknown::QueryInterface] CLSID: {EFF8970E-C50F-45E0-9284-291CE5A6F771} (Protoss Probe), IID: {59644217-3E52-4202-BA49-F473590CC61A} (IGameObject)
[comon] 0:000 [IUnknown::QueryInterface] CLSID: {F5353C58-CFD9-4204-8D92-D274C7578B53} (Protoss Nexus), IID: {59644217-3E52-4202-BA49-F473590CC61A} (IGameObject)
== Method get_Name [3] called on a COM object (CLSID: {59644217-3E52-4202-BA49-F473590CC61A} (IGameObject), IID {F5353C58-CFD9-4204-8D92-D274C7578B53} (Protoss Nexus)) ==
0:000> k
# ChildEBP RetAddr
00 0095f540 004e482a protoss!ILT+1280(?get_NameNexusUAGJPAPA_WZ)
If you’re working with COM (or debugging Windows applications), I believe that comon will benefit your work. So comon, give it a try 😊 The latest binaries are on the release page and the documentation is here.
]]>The reasoning for creating dotnet-wtrace was that I could not find any tool that will show me .NET trace events in real-time. Until now, we could only record the trace (using dotnet-trace or PerfView) and later analyze it. Of course, this approach works in all situations, but sometimes it could be a bit tedious, especially for easy-to-diagnose bugs, such as swallowed exceptions or networking problems. Dotnet-wtrace should help in such cases, quickly pointing you to the error details. It is not meant to replace dotnet-trace or PerfView, but rather help in the initial phases of diagnosing problems. That’s why it does not simply dump all event details but preprocess them and extracts only the most interesting bits, making the output human-readable. Dotnet-wtrace includes multiple event handlers, each one of them handling a specific event category (such as GC, network, loader, or ASP.NET Core events). An example output looks as follows:
As in wtrace, you may choose the handlers and specify event filters through the command-line options. Please have a look at the documentation to learn more.
I hope I convinced you to give dotnet-wtrace a try. You may install it as one of the dotnet tools:
dotnet tool install -g dotnet-wtrace
Or download the precompiled binaries from its repository release page.
Happy tracing and until the next time! 🧐️
]]>One of the most significant changes is the possibility to collect traces system-wide. If you don’t provide a file path or PID, wtrace will trace all the processes. To keep the number of trace events acceptable, consider using one of the extensive filtering options (a new feature, too!).
You may also choose the event handlers for each session. The sensible default set includes process, file, RPC, and TCP handlers. The 3.0 version introduces a Registry event handler, so if you enable it, you may trace Registry operations with wtrace! I plan to add handlers for less common event types in future releases, too.
The summary section got a new view that displays a process tree. When tracing system-wide or system-only, the tree includes all the running processes. In other modes, you will see the parent process and all its descendants.
The missing file paths are no longer a prevalent issue. And wtrace can finally run in a Windows container (thanks to updates in the TraceEvent library).
Unfortunately, I needed to drop support for ALPC and PowerShell events. In the previous versions of wtrace, I tried to match ALPC connections with the RPC ones, but it never worked reliably. Similarly, the PowerShell event handler had much to improve. I want to revive those handlers, but I need to be sure that they present accurate data. And that requires some more research.
Finally, wtrace has its homepage now: https://wtrace.net, and I hope that soon new utilities will join its Tools section.
Get the new version from the release page and start (w)tracing! 😃
]]>