One such not-so-subtle bug (application hang) is particularly nasty, and is described fairly well here. Distilled down, the application hangs, usually when the computer comes out of sleep mode, unlocks, or another similar event occurs. The hang happens in the firing of an event handler, called from “SystemEvents.OnUserPreferenceChanged”. The cause is that OnUserPreferenceChanged is trying to be super nice, and invoke the event in the appropriate context for each event subscriber. That means that if a Control subscribes to the event, the handler will be called on the UI thread. If user code (on a background thread) subscribes, the handler will be called on an arbitrary thread, etc. The problem occurs when, at some point in the past, a Form or Control was created on a background thread. The creation of the control “installed” a WindowsFormsSynchronizationContext as the current SynchronizationContext (this is default behavior). When OnUserPreferenceChanged attempts to Send (Invoke) to the appropriate thread context, it hangs because the WindowsFormsSynchronizationContext is on the wrong thread, and thus has no message pump with which to process messages.

Blah blah blah. Right now you’re thinking “Whatever dude, I’m a web developer, man, I’m just trying to fix this bug in that other jerk’s code.” Fair enough, you can read the linked post for more gory details. What you care about is the hard part. Well, hard for web developers anyway. By the time the app hangs, it’s too late to find out where the problem occurred. In a medium-to-large application, how do you find the control that was created on the wrong thread? Here’s how to do it in a matter of seconds.

1. Name your UI thread. If you’re not already doing this, it’s a good idea in general. I like to call mine “UI”, personally. In your “static void Main()” add this single line of code:

Thread.CurrentThread.Name = "UI";

2. Set a breakpoint deep in the bowels of the BCL. What we want to do is cause our application to break when a WindowsFormsSynchronizationContext gets assigned to the current thread. I cracked open Reflector to look at the constructor for WindowsFormsSynchronizationContext:

And I see that there’s a call to “Application.ThreadContext.FromCurrent()”. Close enough for government work, it’ll do for what I want. I didn’t feel like figuring out how to specify a constructor call when setting a breakpoint. (If you know how, leave a comment so we can all learn!) Add a breakpoint to that method call. In Visual Studio, go to the “Debug” > “New Breakpoint” > “Break at Function…” menu. In the “Function” area type the full path to the function: System.Windows.Forms.Application.ThreadContext.FromCurrent()

When you hit “OK” you’ll get a warning about IntelliSense not finding the specified location. Hit “Yes” to set the breakpoint anyway.

3. Run your application. Depending on what version of Visual Studio you’re running, and whether you’ve got source-level debugging for the Framework turned on, you’ll either get the breakpoint on some code, or you’ll get this message box:

It doesn’t matter, you can press “OK” or “Show Disassembly”, whatever floats your boat. Go to the “Call Stack” debugging window, right-click on the red breakpoint circle located at the top of the stack frame, and select “Breakpoint” > “Filter…”

4. Add a filter to this breakpoint so that it only breaks when the current threads’ name isn’t “UI”.

After you hit “OK”, continue running your application in the debugger. The first time a WindowsFormsSynchronizationContext is created and it’s not on the UI thread, BAM. There’s your problem, and there’s your stack trace allowing you to find the bad code.

This worked like a champ for me today, hopefully you have as much success with it too.

I got some good feedback on the utility, along with some additional feature requests that ran outside the bounds of what the symstore utility offered. That lead me to upgrade my project to include a full-fledged (well, almost) symstore replacement, plus the GUI manager utility.

I’ve published the project tonight on CodePlex as "PSSymbolStore", and I’ve released the initial 0.1 beta release.

So what did I add and what did I change?

PowerShell

PSSymbolStore is now founded on a set of PowerShell cmdlets that replace and extend symstore’s functionality. (Almost: I need some feedback about a few symstore features and how they might work in the PowerShell world. If you’re a symstore wizard – ahem, John – drop me a line.) The initial release includes the following cmdlets, along with full help, including examples:

• Add-Symbols – Adds symbols to the symbol store
• Get-Transaction – Retrieves some or all of the transactions from a symbol store
• Remove-Transaction – Deletes transaction(s) from a symbol store
• Lock-Transaction – Locks transaction(s), preventing them from being deleted with the Remove-Transaction cmdlet
• Unlock-Transaction – Unlocks transaction(s), allowing them to be deleted with the Remove-Transaction cmdlet

Symbol Store Manager

The Symbol Store Manager (formerly "Symbol Server Transaction Manager") is basically the same from a UI perspective, but has been massively gutted. The internal changes (MVP pattern, PowerShell hosting, etc.) are all for the better as they fully enable unit testing, which is coming in a future release. I’m looking for feedback on the UI – what I can improve, etc. It’s really basic right now. I had started adding a search feature using ShuffleText’s Highlight fuzzy search library, but didn’t have time to finish it the way I wanted so I’ve pushed it out to a future release.

vNext

So what’s coming next? I’ve already mentioned unit tests and searching. I’m also kicking around some UI enhancements. Aside from those features, and finishing up some "paperwork" like installs, I don’t currently have a strong feature list. If you have ideas, please either post them as a comment here, or even better on the codeplex project site. For example: would this be more useful as an MMC snap-in instead of a standalone application?

Download PowerShell if you haven’t already, and give PSSymbolStore 0.1 Beta a whirl – I hope it’s useful!

It’s easy to upload your builds to the symbol server using the command line. You execute a command like:

symstore add /r /f d:\builds\*.* /s \\symbol\ProductSymbols /t "Product Name" /v "Version string" /c "Comment"

(For full usage information run “symstore” with no arguments.)

The not-so-easy part of this process comes when managing previously uploaded versions of your symbols. John Robbins suggested (and who am I to resist a challenge) that it would be great if somebody wrote a tool with some sort of UI that allowed users to select previous “transactions” and delete them. The only way to do that right now is by manually scanning the “history.txt” file located on the server’s share (in an “000admin” subdirectory) and pulling out the transaction id to delete by using in a “symstore del” command.

Way, way too manual.

So, dear reader, I give you my response to John’s challenge: Symbol Server Transaction Manager!

UPDATE 1/4/2008: I’ve done a huge overhaul of the app, and released it as open source on CodePlex as the PSSymbolStore project. Download the latest release there.

It makes the process much easier, I hope. Simply start the app, and press CTRL+O to open a symbol store. Note that in all my examples I’m using a local symbol store, “C:\symbols” — but in most cases you’ll be specifying a network path such as “\\symbols\ProductSymbols”.

Once you’ve opened a symbol store, you’ll see a list of existing transactions. Any transactions that have previously been deleted will not be shown. You can sort by any of the columns to help find the transactions you are looking for, such as all transactions older than 2 weeks.

At this point, it’s as easy as highlighting the transactions you want and pressing “Delete”. Or you can right-click one or more items and use the context menu. If you haven’t refreshed your view (I didn’t code it to auto-refresh because of how long the operation might take), and try to re-delete a transaction, you’ll see an error message in red. In fact, any error will be shown in red. If you come across an error condition that shows up in blue (which indicates success), let me know.

Here’s an example of successfully deleting two transactions:

And here’s an example of trying to immediately re-delete those same transactions (bad):

You can press the “Refresh” button to update the view to match the server:

That’s about it, pretty simple stuff, eh? There is one thing worthy of a special note. Because deleting transactions can be a fairly long-running operation, you can cancel it. The “Delete” button changes to “Cancel” if an operation is running. If you click “Cancel”, it will kill the currently executing instance of SYMSTORE, orphaning any remaining files in the transaction, and it won’t process any more transactions, if you had selected more than one. That’s just something to be aware of. Eventually maybe I’ll be smarter about cleaning up the files, but for now, cancel at your own risk.

I welcome any feedback, if you find this tool useful, let me know! I’m happy to add any enhancements you can think of, too.

There are a couple of gotchas that have to be configured for this to work. I was scratching my head wondering why this didn’t work for me immediately, here’s why.

First, make sure the “Just My Code” setting is turned OFF in the Visual Studio settings (Tools -> Options -> Debugging). In fact, just leave this setting off all the time — it sucks.

Secondly, make sure you have a symbol path configured so you can load the appropriate symbols. You must have symbols for the “source-less” code! The symbol path/server is specified via an environment variable named “_NT_SYMBOL_PATH“. If you don’t have a symbol server configured (more on that in a later post) your environment variable will probably look like this: (see the previous link for the exact syntax)

_NT_SYMBOL_PATH = SRV*C:\Symbols*http://msdl.microsoft.com/download/symbols

You can also configure symbols from within Visual Studio in the Tools -> Options -> Debugging -> Symbols configuration:

Now, start debugging your application (have it break on the first instruction of the app). This is especially important if Visual Studio hasn’t yet pulled down the symbols from the symbol server. Bring up the breakpoint window in Visual Studio (CTRL+B) and type in enough detail for the resolver to find the method/property you’re looking for. If you want to break on a property, be sure to use the method syntax for the property: “get_[propertyname]” or “set_[propertyname]“.

Hit OK, and, if you had “Use Intellisense to verify the function name” selected, press “Yes”:

At this point, if your code calls the method or property, the debugger will break on it and take you to the Disassembly view. You can use the Call Stack window to traverse up to whatever point you care about. Notice also that in this example, “child” breakpoints were automatically set on all the overloaded methods for System.Console.WriteLine, automatically.

This is an awesome trick, one that can save you loads of time! Thanks John!

It was an absolute pleasure to be under his tutelage, if even it was “virtual” and he’s already forgotten my name. I do find that courses like this serve to not only help develop my specific skills in a particular area, but it really expands my understanding of .NET in general. I highly recommend the course, and it’s a bargain compared to in-house or off-site training. The online venue worked very well, with the only downside being that John moves so fast (he’s good!) that it was hard to get questions asked during their context

He presented a lot of undocumented tips and tricks, many of which were extremely cool. I’ll be posting a few of them in the near future, with full credit and respect to John as the source. He kept mentioning that he relies on these undocumented tips to make a living, but I have a sneaking suspicion that he’ll do just fine even if I give away one or two here for free.

If I wanted to sum up the training, I would say there are two important concepts to embrace when it comes to debugging:

1. Write fewer bugs. (Duh!) There are a lot of “best practices” that can be followed to help reduce or catch bugs in the development cycle instead of in a production environment.
2. Do everything possible to gather as much information as possible about the environment the moment a bug occurs. You need to know the state of the system, and anything else that can give you “context”. Data that can be “explored” is best (like memory dumps, etc).

Oh, and with the training (and, I suspect, with John’s latest book) you get literally a TON of code: tools, helpers, snippets, visual studio add-ins, etc. If you don’t take the class, at least buy the book (probably a must-read anyway, but I haven’t read it yet) to get all this great stuff–if it was freely available, it would be at the top of Hanselman’s tools list, I’m sure.

Thanks John!