Kelly Leahy (technical)

Azure Fast-Deploy

2011-08-24T19:58:00.000-07:00

We have been building a web system on Windows Azure for quite a while now in my company, and are very familiar with the 20 minute development cycle of redeploying an Azure Role. Recently, the Windows Azure team released a streamlining process for doing web deployments to Azure (for development purposes only, not production deployments). Andy Cross talks about it here: http://blog.bareweb.eu/2011/04/using-web-deploy-in-azure-sdk-1-4-1/

The azure team blog talks about it here.

We tried to use Web Deploy, but we don't do things from Visual Studio. We instead do everything from the command line for our automation (since we do a TON of automation). As such, we tried for a while (to no avail) to get Web Deploy to work from the command line (tried for a while for us is measured in minutes not days, we don't have much patience for crappy tooling on our team).

So... we built our own.

It works like this - we have a very simple console app that we built in C# that looks at a particular blob storage account and tries to download a new blob from there if a new blob exists. In our case, we've named the container "w-deploy-newbits" and the blob name is "WebRole.zip". This zip file is built on our build machine by doing a msbuild /t:Publish on our ccproj in order to build a deployable version of our website (_PublishedWebsites\WebRole in our build folder). We zip this file and upload it to the appropriate cloud location from the desktop as part of our build script (powershell with psake), and when the "NewBits" console app sees that new zip, it redeploys.

We have been using this technique for our profiling of our app (for performance) so that we can quickly make changes, redeploy, reprofile, and repeat the process. It has worked VERY well for this. We have no intention of using this for production releases or anything of that matter and would never recommend such a thing.

I'm going to try to make the code available in github, but it will probably take me some time to do so, as I need to remove anything sensitive to our own application setup and I'm really busy right now. However, if you want more details about how we did this, please feel free to contact me via comments or my email address.

Setting up branches to always rebase in git

2011-02-20T21:56:00.000-08:00

So, today at Harmony Hackathon, several people asked me to post the way to set up your branches in git to always rebase. I know this is a controversial thing in git, if you disagree with me on why you might want to rebase, I don't care. I like it and all I'm doing in this post is telling people how to do it if they want to. Don't do it if you're afraid of rebase, you don't want rebase, or you are afraid your friends will shun you for using rebases by default. Don't tell me if you don't like rebase, because I don't care ;)

Ok... I'm going to tell you how to set up rebase to ALWAYS rebase on ALL repos. If you don't want this for all repos, then you can do this on a per-repo basis by doing all the config without "--system". Also, you can use "--global" instead of "--system" if you want to ensure that other people that use your machine aren't affected by the setting (nobody else uses my machines, so I don't care and do all my config at the --system level).

So, to set up all "master" branches to do rebase by default - do the following:

git config --system branch.master.rebase true

To set up all "new" remote tracking branches to be created with rebase by default, do:

git config --system branch.autosetuprebase always

If you already have remote tracking branches, then I recommend you make sure you're current, delete the local branches, and recreate them (1.7.x does this automatically if you use the branch name to check out the remote branch, so make sure you're using Git 1.7.x or later for that hawtness (as darkxanthos would say)).

Resharper: Please indent my shit by 1000 miles, please! kthxbye

2011-02-20T21:21:00.000-08:00

So... Today, while at the Harmony Hackathon, I took a few minutes of yak shaving time to figure out the location of the setting that had all of us at the hackathon cursing resharper from time to time. If you want to see an example of why we were cursing, see a screenshot below from visual studio for one of the methods we were working on.

Resharper has a setting that tells it to indent the crap out of array and object initializers, and the default is "please resharper, indent my shit off the screen so far to the right that I run out of scrollbar space if I have more than two nested lambdas or objects". Today we took to rectifying this problem in our configuration.

The setting is under formatting style / other / other / indent array, object and collection initializers. You can see it in the screenshot below. Uncheck it and stop bitching about resharper moving your shit 1000 miles to the right ;)

Not your mama’s sort algorithm

2010-03-18T02:14:00.001-07:00

So, not all that long ago, we were profiling our application, and I noticed that a particular part of our system was consuming a huge amount of time sorting arrays of 300,000 to 1M items. I was a little surprised to see that this was taking an inordinate amount of time but it shouldn’t have been so surprising to me, since the keys by which we were sorting these arrays are somewhat complex and so comparisons are nontrivial.

The arrays we were sorting were lists of keys (and their associated data), and we were sorting them in a lexicographic ordering that was based on an elsewhere defined sort order (that was defined by the user). The keys are themselves a list of id-value pairs where the ‘id’ is a short integer, and the value is a string. So, as an example, consider the problem of sorting the following list in lexicographic order by the key ids 3, 4, 0 (in that order):

(0 = “UL1”, 3 = “M”, 4 = “S”)
(3 = “F”, 0 = “UL4”, 4 = “S”)
(4 = “S”, 3 = “M”, 0 = “UL2”)
(4 = “N”, 3 = “F”)

When looking at the data, a few things stand out. Firstly, the keys aren’t all in the same “internal order”. This is because the keys come from various different places in the system and are created at various different times, and are not sorted because the cost of doing so proved to be a performance issue. Secondly, not all of the keys have all of the key ids (0, 3, and 4). This is a feature of our system where not all keys require all ids. Keys cannot overlap, but a missing ‘id’ is a wildcard, and allows searches to match that key with any value for that id.

When doing a comparison between these keys, we were using a IComparer implementation that looks like the following code snippet. The IKeyInstance interface is the interface for the objects that contain one of the rows of the above list of data. GetValue returns the string associated with the id passed as its argument (or null if the value doesn’t exist in the key).

   1: public class KeyInstanceSortedComparer : IComparer<IKeyInstance>   2: {   3:     private readonly short[] _KeyIdSortOrder;   4:     5:     public KeyInstanceSortedComparer(IEnumerable<short> keyIdSortOrder)   6:     {   7:         _KeyIdSortOrder = keyIdSortOrder.ToArray();   8:     }   9:    10:     public int Compare(IKeyInstance keyInstanceA, IKeyInstance keyInstanceB)  11:     {  12:         foreach (var keyId in _KeyIdSortOrder)  13:         {  14:             var keyValueA = keyInstanceA.GetValue(keyId);  15:             var keyValueB = keyInstanceB.GetValue(keyId);  16:    17:             var compareResult = string.Compare(  18:                 keyValueA.ToValue(),   19:                 keyValueB.ToValue());  20:             if (compareResult != 0)  21:                 return compareResult;  22:         }  23:         return 0;  24:     }  25: }

The _KeyIdSortOrder field is the sort order for the particular sort operation we are performing when using this comparer. The ToValue method is an extension method that returns the value’s string if it’s non-null, or null otherwise. It’s pretty easy to see from this snippet that this code implements a lexicographic sort on the IKeyInstance implementers.

Now, given the fact that the IKeyInstance implementers did not sort their id-value pairs internally, GetValue has to do a linear search of the key instance in order to find the appropriate value. Additionally, the comparator must compare as many keyId’s values as it encounters until it either reaches the end of the list of id’s to compare, or it finds an id for which the two keys have different values. The keys are never larger than about 25 ids in any key, and they are typically more like 2 – 8 id-value pairs per key. Also, much of the memory consumed in the runtime footprint of our application was at one point held by these IKeyInstance implementers or their associated id-value pairs, so every effort to keep their size down must be made.

One of the optimizations we made to reduce the size of these key instances was to manufacture the strings used as “values” for the id-value pairs from a factory that coalesced duplicates so that we only ever had one instance of a given value in memory at any point in time. Another optimization was to do the same with id-value pairs themselves. These two combined optimizations had a profound impact on the memory footprint of our application, due to the amount of duplication present in the data, and the limited amount of uniqueness in the key’s values and id-value pairs.

Anyway, to get back to the sorting, I spent some time looking at the sort performance and realized that no matter what I did, I couldn’t really improve the performance of the sort much, so long as we were continuing to do all these comparisons. The problem is, the comparisons are simply expensive due to the nature of GetValue in IKeyInstance and the fact that we weren’t sorting the entries. We could have used Dictionary in the key instances, but that would have taken up too much memory for our taste. We could have used a sorted list of id/value pairs, but when there are only typically 2-8 columns in the key, binary search would likely only be slightly faster than a linear search, and may have even been slower.

So…. I started looking for ways to parallelize the algorithm. One of the other developers on my team – Robert Ream – was looking into parallel implementations of quicksort and mergesort to try to figure out how we could improve the performance of our sorting, since we have a few processors at our disposal on our typical users’ machines. I also asked him to look into ways of making the algorithm cancellable and restartable so that we could ‘stop’ the algorithm in progress, serialize its state to disk, and move on to another item in the UI without losing the work that had already been done. While this was a very interesting problem to solve, it turned out that trying to get rid of the comparisons would bear much more tasty fruit.

Thinking outside the box

In the process of looking for ways to reduce the number of comparisons, I started talking to Robert about which of the algorithms we knew about had less comparisons (mergesort / quicksort / etc.). After some discussion, we really didn’t end up anywhere useful, so Robert continued working on his parallel quicksort implementation. In the meantime, the weekend came, and I decided to spend a couple hours researching what else could be done. I pulled out Knuth’s TAOCP Vol 3 and started reading. In it, I came across his description of radix sort. I read his description, which didn’t lead me to any great revelations (since he basically said that it doesn’t typically perform that much better than the other algorithms), but I was intrigued by the fact that it didn’t use comparisons.

I then continued searching for information, reading Cormen, and a couple of other algorithm books I have around the house, and then did some searching online. I came across several publications by Stefan Nilsson on the subject of radix sort, and read some of them. It turns out that after reading a few of these I started to think about our problem a bit more and try to consider whether it could be framed in a structure that allowed us to apply radix sort. Still, the complete lack of comparisons was the motivating factor for me, given my knowledge of the fact that the comparisons were more than 95% of the time spent in our sorting code.

After a few hours of thought on the subject on Friday, I decided Saturday afternoon to investigate a bit in code. So, if you don’t have any background on radix sort, the basic idea is this – instead of using comparisons, you must have numerical values that can be assigned to the ‘values’ of your key elements being sorted, and the assignment must be in the same order as your sort order. For instance, if you want to sort a bunch of strings, the values are the ASCII values of the characters (or at least their ASCII sort collation order number). Radix sort takes advantage of this fixed space to allow you to do array assignments and simple linked list operations instead of doing comparisons. Radix sort performs best if you have a large ‘alphabet’ of values to work with and your keys are either non-overlapping, or don’t have a very large number of duplicates.

There are several variations of radix sort algorithms - the two classifications are MSD and LSD approaches. I used the MSD approach since it was easier to understand how it applied to my problem, and I originally thought I had varying length keys so it seemed like it would be more appropriate (LSD approach seems to work better with fixed-length keys).

In order to apply radix sort to my problem, I needed to be able to assign a value to each key-value in my key. Since I already had a factory that maintained the invariant that there was only one instance of each key-value in memory at any given point in time, and in order to do this, the factory needed to track these objects, I decided to provide the ability for the factory to assign a ‘sort order’ value to each of the key-values. An integer was big enough, and I could put some intelligence in the caching factory to have it only perform the sort when requested AND necessary (due to a change having occurred to the list of key-values).

Given these sort order values, I could define an “alphabet” of 0..N where N is the number of values in existence and the highest sort order value, 1 is the lowest, and 0 represents a missing id in the key. Then, I could logically think of my keys as looking like ordered M-tuples of integers (where M is the number of IDs in my sort order specification, and the maximum number of id-value pairs to be found in any key). The integers in these M-tuples were the sort order values from the key-value list, and the tuples were logically sorted in the appropriate order for doing the lexicographic sort.

Then, the keys above could (fictitiously) look something like (in the sort order 3, 4, 0):

(2, 4, 5) //(0 = “UL1”, 3 = “M”, 4 = “S”)

(1, 4, 7) //(3 = “F”, 0 = “UL4”, 4 = “S”)

(2, 4, 6) //(4 = “S”, 3 = “M”, 0 = “UL2”)

(1, 3, 0) //(4 = “N”, 3 = “F”)

with the sort order values being:

(1 = F, 2 = M, 3 = N, 4 = S, 5 = UL1, 6 = UL2, 7 = UL4)

and the prescribed sort order is, of course

(1, 3, 0)

(1, 4, 7)

(2, 4, 5)

(2, 4, 6)

Implementation of Radix Sort

So, armed with this approach of assigning ordinal positions to the values in my keys, and the logical definition of a key as given above, I was in a position to apply radix sort (which requires a desired lexicographic ordering and known ‘cardinal’ values for the letters of the alphabet you are sorting). My implementation breaks into three pieces – an ‘inner’ algorithm (in my case bucketsort), a sorter, and a way to track the unfinished portions of the array being sorted.

The basic idea of my implementation is that you start with the full array as a single ‘unsorted’ range. The algorithm loops as long as there is work to do, by removing a single unsorted range from a queue, processing that unsorted range, and then moving on to the next step. Once there is no more work to do, the array is sorted.

The algorithm requires an ‘inner’ sort algorithm, since radix sort processes one ‘digit’ at a time from the source alphabet. In my case, radix sort will process one key-value column at a time, starting from left to right. When processing a column, it will use a working array that has one linked list for each possible value of the digit. For instance, we would have a working array (of linked lists) of length 8 (since there are 7 distinct key values, plus one ‘missing’ key value). The sort will walk linearly through the list putting items into the ‘bucket’ that corresponds to the value for the first column. Then, after it has assigned all items to their appropriate buckets, it will put those items back into the array in the order in which they appear in the array and their linked list. At this point, the array is sorted by the first column, and the work remaining to be processed is the list of buckets of length 2 or more. The process is iteratively applied to each column, ignoring the values of previous columns, but when applying for a column, it is only applied to the subset being currently processed (i.e. after splitting buckets 1 and 2, the ‘remaining work’ will be the ranges (0, 2) and (2, 2), where the tuples are (start, length).

The work of splitting items into their buckets is mostly done by my ‘RadixSortBucketer’ class, given here:

   1: public class RadixSortBucketer   2: {   3:     private readonly LinkedList<int>[] _Buckets;   4:     5:     public RadixSortBucketer(int bucketCount)   6:     {   7:         _Buckets = new LinkedList<int>[bucketCount];   8:     }   9:    10:     public void AddItem(int bucket, int item)  11:     {  12:         var list = _Buckets[bucket];  13:         if(list == null)  14:             _Buckets[bucket] = list = new LinkedList<int>();  15:         list.AddLast(item);  16:     }  17:    18:     public void SaveItemsTo(  19:         LinkedList<RadixSortUnfinishedRange> unsortedRanges,   20:         int[] workArray,   21:         int startOffset)  22:     {  23:         var targetIndex = startOffset;  24:         foreach(var list in _Buckets.Where(b => b != null))  25:         {  26:             if(list.Count > 1)  27:                 unsortedRanges.AddLast(  28:                     new RadixSortUnfinishedRange(  29:                         targetIndex,   30:                         list.Count));  31:             foreach(var item in list)  32:                 workArray[targetIndex++] = item;  33:         }  34:     }  35:    36:     public void Clear()  37:     {  38:         for(var i = 0; i < _Buckets.Length; i++)  39:             _Buckets[i] = null;  40:     }  41: }

The array of LinkedList<int> is the buckets, the LinkedList<int> in each bucket is (possibly empty) list of indices that fall into that bucket. The unsorted range object looks like:

   1: public class RadixSortUnfinishedRange   2: {   3:     public readonly int Start;   4:     public readonly int Count;   5:     6:     public RadixSortUnfinishedRange(int start, int count)   7:     {   8:         Start = start;   9:         Count = count;  10:     }  11: }

and the overall algorithm looks like:

   1: public class RadixSorter   2: {   3:     public delegate int ValueFunctionDelegate(int level, int item);   4:     5:     private readonly RadixSortBucketer _Bucketer;   6:     private readonly int[] _WorkArray;   7:     private readonly ValueFunctionDelegate _ValueFunction;   8:     private readonly int _LevelCount;   9:    10:     public RadixSorter(int[] arrayToSort, int bucketCount,   11:         int levelCount, ValueFunctionDelegate valueFunction)  12:     {  13:         _WorkArray = arrayToSort;  14:         _Bucketer = new RadixSortBucketer(bucketCount);  15:         _LevelCount = levelCount;  16:         _ValueFunction = valueFunction;  17:     }  18:    19:     public void Sort()  20:     {  21:         var currentLevel = 0;  22:         var currentUnsortedRanges = new LinkedList<RadixSortUnfinishedRange>();  23:         currentUnsortedRanges.AddLast(  24:             new RadixSortUnfinishedRange(0, _WorkArray.Length));  25:         while(currentUnsortedRanges.Count > 0)  26:         {  27:             var nextUnsortedRanges = new LinkedList<RadixSortUnfinishedRange>();  28:             foreach(var range in currentUnsortedRanges)  29:             {  30:                 _Bucketer.Clear();  31:    32:                 for(int i = range.Start, j = 0; j < range.Count; i++,j++)  33:                     _Bucketer.AddItem(  34:                         _ValueFunction(currentLevel, _WorkArray[i]),  35:                         _WorkArray[i]);  36:    37:                 _Bucketer.SaveItemsTo(  38:                     nextUnsortedRanges,  39:                     _WorkArray,  40:                     range.Start);  41:             }  42:    43:             currentUnsortedRanges = nextUnsortedRanges;  44:             currentLevel++;  45:             if(currentLevel >= _LevelCount)  46:                 break;  47:         }  48:     }  49: }

The only ‘special’ part of this is that I have defined a delegate to allow this to be parameterized. The delegate is a value function that returns the integral value corresponding to a particular item and ‘level’ (level is like the ‘current column’ value). In my particular problem, the value function is a delegate that uses the ‘level’ to determine which ‘value’ to ask for, and returns 0 if there is no value, or the sort order of the value if the value is not null. Item is the index into the original array to use when getting values for the algorithm.

Conclusion?

So, what was the end result of my experimentation, you ask? Well, the sort that used to take 17 seconds or so, now takes under 1 second, and much less if it doesn’t include the cost of assigning the sort orders to the key values (which doesn’t happen for each sort, it only happens for the first sort after a change to the key-value cache – which is pretty infrequent). Could my implementation be more efficient – absolutely. However, I’m a strong believer in “write it the way that makes the most sense to you, and then only make it ‘more efficient’ if profiling shows it to need to be more efficient” – this is basically the philosophy of avoiding ‘premature optimization’. Of course, I don’t condone ‘premature pessimisation’ either, so you shouldn’t choose the wrong algorithm or data structure for the job, but you also shouldn’t micro-optimize every bit of code you write, as you’re writing it.

I hope you find this description useful – I tried to give details about how & why I did what I did, instead of just ‘what’ I did. Hopefully that makes this an interesting read and provides some insights that just posting the code wouldn’t provide.

Weak Events

2010-03-16T22:14:00.001-07:00

A while back I was having a discussion with my friend Tim Erickson (@imerickson) about problems surrounding GC and events in WPF. Specifically, we were talking about problems with ICommand (WPF) implementations. I mentioned to him that we have and use our own weak eventing pattern on my team. I promised to write a blog post about it, but I’ve been slow doing so. This is my attempt to make good on my promise. Hopefully it’s not too awful.

The problem that weak events are meant to solve is one in which you an event source (the thing raising the events) that has a potentially longer lifetime than some of the corresponding event sinks (the things receiving the events). In that case, normal .NET events (delegates) will cause the event source to keep the event sinks alive, unless some explicit event removal code is executed on the sinks to tell them to remove their event handlers from the source. In many cases this is relatively easy to do, but in many it is very inconvenient to do so and the programmer would rather not “know” when the removal is necessary.

As an example, in our application, we have an editor grid that has bindings that act as an event source, while the grid acts as an event sink. BTW, when I say “bindings” here, think of them as a ‘data source’ for the grid, not ‘bindings’ in the WPF sense. These bindings are not really a problem, as the bindings are created and “attached” to the grid, and when they are “detached” from the grid, the grid removes any event handlers it has attached to the bindings. In this case, we don’t need weak events. However, underlying the grid bindings is a view model object (that is sort of an amalgamation of a view model and a data model object, but that’s an argument for another day) that has events to notify its subscribers if any of the contents of the VM have changed from anywhere in the UI.

In our UI, we can have multiple views on the same exact model object (think multiple views in MS Excel or MS Word) that each need to be notified if any one of the views causes a change in the model. In this case, our model is an event source, and our bindings are an event sink. Here’s the problem – our model lives “forever” (at least it lives as long as our application is ‘running’), yet the grid bindings get recreated when the user changes their view, moves to another item, opens a new view, or any number of other changes to the state of the UI. In effect, we create MANY of these grid bindings objects throughout a single run of our application, and we wish to make no effort to track when they are no longer “in use”, since doing so is rather difficult and would litter our code everywhere with code to handle this tracking.

However, if we ignore the tracking, and just let the grid bindings “get released” by the grid, etc., the events that were hooked by the bindings from the model in order to get notifications of changes in the model will never get unhooked. This will lead to the model “keeping alive” a huge number of these bindings objects and never releasing them, effectively creating a “memory leak” until the model is shut down. Therefore, we had a choice – either hook the events and try to force appropriate shutdown in all the places it was needed, or find another solution that didn’t involve the bindings hooking events directly on the model.

Since we have total control over both the control and the underlying model, we had a lot of flexibility in our choice of solutions to this dilemma. I investigated the .NET (WPF) model for weak events, as well as several other methods people had used on the net. I even found a description from the venerable Jeffrey Richter in his CLR via C# book (and found it to have errors, as a matter of fact). The WPF model is very general and is the way to go if you are building a control and don’t have control over the model that will be hooked to your control (or want to play nice in the WPF data binding way), but it leaves much to be desired in terms of the programming model.

The WPF weak event model is centered around the Microsoft.Windows.IWeakEventListener interface and the Microsoft.Windows.WeakEventManager base class. For a discussion of some of the details, see http://blogs.msdn.com/nathannesbit/archive/2009/05/28/weakeventmanager-and-iweakeventlistener.aspx

For the most part, I agree with Nathan’s assessment of this class. I do, however, believe that if you are writing controls to be consumed by applications out of your control, you effectively MUST follow this model (not mine or anyone else’s), since this is the model that is used throughout WPF. While you could implement your own variation of this model, you still need to apply the general concepts that it does in any solution you come up with (i.e. the ‘endpoint’ providing the ‘weakness’ in the events is the event sink, not the event source).

For our application, we have adopted the ‘opposite’ pattern – that is, that the weak events should be sourced from the event source as weak, rather than having an intermediary in the weak event manager that handles turning ‘strong’ events from the model into ‘weak’ events (listeners) on the event sink. We like this model because it hides the complexity of dealing with the ‘weakness’ of the events behind the facade of the event handler attachment points and ‘invocations’ in the model. Our model basically works as follows: the event source provides a custom event implementation, and when sinks subscribe to this event, they are added to a weak event “multicast delegate-like” object that is private to the event source. The subscribers don’t know that they are hooking to a weak event. On the other hand, the model raises events using this “special weak event object” and the attached event sinks receive these messages if they are still alive.

There is one small wrinkle to our model. Event handlers attached to weak events in this manner MUST be kept around by something else, or else they will get collected and will cease to receive events. This isn’t a problem for us, since it’s exactly what we’d like to see happen, but there can be problems if one isn’t careful, as it will seem like events stop getting ‘caught’ by their intended targets, and you won’t know why.

So, now on to the details. First off, we have a ‘WeakDelegate’ class, that is responsible for being a weak version of the Delegate class built in to the .NET framework. It looks like the following:

   1: /// <summary>   2: /// A "weak reference" version of a delegate.  This class is used    3: /// by the MulticastWeakDelegate type to support "weak events"    4: /// pattern in the system.   5: /// </summary>   6: /// <typeparam name="T">The type of delegate to return.</typeparam>   7: public class WeakDelegate<T>   8:     where T : class   9: {  10:     private readonly WeakReference _Target;  11:     private readonly MethodInfo _Method;  12:    13:     /// <summary>  14:     /// Constructs a weak delegate from the provided delegate.  15:     /// </summary>  16:     /// <param name="source"></param>  17:     public WeakDelegate(Delegate source)  18:     {  19:         if (source == null)  20:             throw new ArgumentNullException("source");  21:         if (source.Target == null)  22:             throw new ArgumentException(  23:                 "Source points to a null target.  "   24:                 + "This is not usable as a weak reference.",   25:                 "source");  26:    27:         if (source.GetType() != typeof(T))  28:             throw new ArgumentException(  29:                 "Source is not the proper delegate type.  "  30:                 + "The delegate type must match the type passed "  31:                 + "as the type parameter 'T'",  32:                 "source");  33:         _Target = new WeakReference(source.Target);  34:         _Method = source.Method;  35:     }  36:    37:     /// <summary>  38:     /// Returns the contents of the WeakDelegate as a proper   39:     /// Delegate object, or NULL if the  40:     /// delegate's target has been collected already.  41:     /// </summary>  42:     public T Delegate  43:     {  44:         get  45:         {  46:             var obj = _Target.Target;  47:             if (obj == null)  48:                 return null;  49:             return (T)(object)System.Delegate.CreateDelegate(  50:                 typeof(T), _Target.Target, _Method);  51:         }  52:     }  53:    54:     /// <summary>  55:     /// This property identifies if the delegate target has   56:     /// already been collected (false if so).  Note: there is a   57:     /// possible race condition with this property, so be sure to   58:     /// check the return value of the Delegate property for null   59:     /// even if IsAlive returned true.  This is just a quick way to   60:     /// identify as "definitely dead", it cannot be trusted not to   61:     /// change between calling it and calling the Delegate method!  62:     /// </summary>  63:     public bool IsAlive  64:     {  65:         get { return _Target.IsAlive; }  66:     }  67:    68:     /// <summary>  69:     /// Checks whether the contents of this weak delegate is the   70:     /// same as the delegate passed as a parameter.  If it   71:     /// contains the same target & method combination, then the   72:     /// function returns true.  If the contained "delegate" has   73:     /// already been collected, or doesn't match the passed   74:     /// delegate, it returns false.  75:     /// </summary>  76:     /// <param name="other">The delegate to test against.</param>  77:     /// <returns>True if the delegate is the same as the   78:     /// target/method contained by this weak delegate, False if   79:     /// the weak delegate has already been collected or if the   80:     /// target/method differs.</returns>  81:     public bool IsSameAs(Delegate other)  82:     {  83:         return ReferenceEquals(other.Target, _Target.Target)  84:                && other.Method.Equals(_Method);  85:     }  86: }

The weak delegate has support for “wrapping” a normal .NET delegate (it, of course, copies the contents of the delegate and throws away the underlying delegate so that it can hold a weak reference to the target of the original delegate). It also has support for asking about the validity of the weak reference, getting the delegate back (or null if it’s not alive anymore), and comparing this delegate to others (used in the ‘remove’ handler of events).

To implement weak events, we use these weak delegates in a similar way to the MulticastDelegate class’ use in C# (the class that is used for ‘event’ implementation). For this, we have a MulticastWeakEvent generic class that allows us to get as close as possible to declaring an ‘event’ in C#, but the language won’t let us get quite there. Here is our MulticastWeakEvent implementation:

   1: /// <summary>   2: /// MulticastWeakEvent is the "weak event" pattern support object.  This class    3: /// allows classes that wish to provide weak event registration to do so.  One    4: /// of these delegate classes must be created for each event, and the class    5: /// must be given the event type (typically EventHandler&lt;TEventArgs&gt;) and    6: /// the type of the event arguments.   7: /// </summary>   8: /// <typeparam name="TEventType">The type of event to support.  This should    9: /// normally be the .NET 2.0 EventHandler&lt;TEventArgs&gt; type.</typeparam>  10: /// <typeparam name="TEventArgs">The type of the event arguments object to use   11: /// in calls to this event.</typeparam>  12: public class MulticastWeakEvent<TEventType, TEventArgs>  13:     where TEventType : class  14:     where TEventArgs : EventArgs  15: {  16:     private readonly List<WeakDelegate<TEventType>> _List;  17:     private readonly Action<TEventType, object, TEventArgs> _InvokeAction;  18:    19:     private static Delegate _AsDelegate(TEventType handler)  20:     {  21:         return (Delegate)(object)handler;  22:     }  23:    24:     /// <summary>  25:     /// Constructs a new weak event delegate.  The action should simply call   26:     /// the delegate with the appropriate arguments.  For instance, the   27:     /// action should simply be a delegate that looks like   28:     /// <code>(d,s,e) =&gt; d(s,e)</code>.  29:     /// </summary>  30:     /// <param name="invokeAction">A delegate that invokes its first argument   31:     /// (the event delegate) passing the second and third arguments as the   32:     /// arguments to the invocation.</param>  33:     public MulticastWeakEvent(Action<TEventType, object, TEventArgs> invokeAction)  34:     {  35:         if (!typeof(Delegate).IsAssignableFrom(typeof(TEventType)))  36:             throw new InvalidOperationException(  37:                 "TEventType must be a delegate type in MulticastWeakDelegate");  38:    39:         _List = new List<WeakDelegate<TEventType>>();  40:         _InvokeAction = invokeAction;  41:     }  42:    43:     /// <summary>  44:     /// Invokes the event for all listeners, one at a time.  45:     /// </summary>  46:     /// <param name="sender">The sender to pass to the invocation.</param>  47:     /// <param name="e">The event args for the invocation.</param>  48:     public void Invoke(object sender, TEventArgs e)  49:     {  50:         lock (_List)  51:         {  52:             _List.RemoveAll(eh => !eh.IsAlive);  53:             _List.ForEach(  54:                 eh =>  55:                 {  56:                     var target = eh.Delegate;  57:                     if (target != null)  58:                         _InvokeAction(target, sender, e);  59:                 });  60:         }  61:     }  62:    63:     /// <summary>  64:     /// Adds a handler to the weak event invocation list.  65:     /// </summary>  66:     /// <param name="handler">The handler delegate.</param>  67:     public void AddHandler(TEventType handler)  68:     {  69:         lock (_List)  70:         {  71:             _List.RemoveAll(eh => !eh.IsAlive);  72:             _List.Add(new WeakDelegate<TEventType>(_AsDelegate(handler)));  73:         }  74:     }  75:    76:     /// <summary>  77:     /// Removes a handler from the weak event invocation list (and compacts   78:     /// the list).  79:     /// </summary>  80:     /// <param name="handler">The handler delegate.</param>  81:     public void RemoveHandler(TEventType handler)  82:     {  83:         lock (_List)  84:             _List.RemoveAll(  85:                 eh => (!eh.IsAlive) || eh.IsSameAs(_AsDelegate(handler)));  86:     }  87: }  88:    89: public class MulticastWeakEvent : MulticastWeakEvent<EventHandler, EventArgs>  90: {  91:     public MulticastWeakEvent(): base((d, s, e) => d(s, e))  92:     {  93:     }  94: }

Now, given this class, we can show some examples on how to expose weak events. Consider a class that has a “NameChanged” event of type EventHandler<NameChangeEventArgs>. We can declare a subclass of MulticastWeakEvent (not strictly necessary, but helps with readability) as:

   1: public class NameChangeEventArgs: EventArgs   2: {   3:     public readonly string OldName;   4:     public readonly string NewName;   5:     6:     public NameChangeEventArgs(string oldName, string newName)   7:     {   8:         OldName = oldName;   9:         NewName = newName;  10:     }  11: }  12:    13: public class NameChangedEvent:   14:     MulticastWeakEvent<EventHandler<NameChangeEventArgs>, NameChangeEventArgs>  15: {  16:     public NameChangedEvent()  17:         : base((d, s, e) => d(s, e))  18:     {  19:     }  20: }

The main reason for declaring a special type for one of these events is that it helps avoid the silly (d,s,e) => d(s, e) crap that we’re forced to do so that we can avoid reflection costs in our code. I couldn’t find a way around this without using reflection, so if someone has any ideas I’d be happy to hear about them (but don’t be surprised if I’m skeptical, since I fought this problem for a very long time).

Now, to create a model object that raises events of the type NameChanged (transparently), we can do the following:

   1: public interface IPerson   2: {   3:     string Name { get; }   4:     event EventHandler<NameChangeEventArgs> NameChanged;   5: }   6:     7: public class Person: IPerson   8: {   9:     private string _Name;  10:     private readonly NameChangedEvent _NameChanged = new NameChangedEvent();  11:    12:     public Person(string nameAtBirth)  13:     {  14:         _Name = nameAtBirth;  15:     }  16:    17:     public string Name  18:     {  19:         get  20:         {  21:             return _Name;  22:         }  23:         set  24:         {  25:             if(_Name == value)  26:                 return;  27:             var oldName = _Name;  28:             _Name = value;  29:             _NameChanged.Invoke(  30:                 this,   31:                 new NameChangeEventArgs(oldName, _Name));  32:         }  33:     }  34:    35:     public event EventHandler<NameChangeEventArgs> NameChanged  36:     {  37:         add { _NameChanged.AddHandler(value); }  38:         remove { _NameChanged.RemoveHandler(value); }  39:     }  40: }

I think this programming model is pretty clean, and it works well for what we’re doing. The only caveat, as I said earlier, is that you absolutely must understand how GC works when working with these, or else you’ll have some very weird bugs related to event sinks disappearing in the night ;) Allow me to give an example of the “event sink disappearing” problem in case you haven’t figured out what I’m referring to yet. Consider the following code:

   1: public class PersonWatcher   2: {   3:     private readonly List<IPerson> _PeopleWithNameChanges = new List<IPerson>();   4:     5:     public IEnumerable<IPerson> GetPeopleWithNameChanges()   6:     {   7:         return _PeopleWithNameChanges;   8:     }   9:    10:     private class Listener  11:     {  12:         private readonly List<IPerson> _TargetList;  13:    14:         public Listener(List<IPerson> targetList)  15:         {  16:             _TargetList = targetList;  17:         }  18:    19:         public void HandleNameChanged(object sender, NameChangeEventArgs args)  20:         {  21:             _TargetList.Add((IPerson)sender);  22:         }  23:     }  24:    25:     public void ListenToPerson(Person personToListenTo)  26:     {  27:         var listener = new Listener(_PeopleWithNameChanges);  28:         personToListenTo.NameChanged += listener.HandleNameChanged;  29:     }  30: }

The code above (while only an example) could, or at least some variation of it could very well arise in ‘real life’. One might think they are decoupling the ‘listening’ aspect from the tracking aspect of the problem. The problem with this is that since the Person sources ‘weak’ events, there are no objects that are ‘live’ that are holding strong references to the ‘listener’ instances created on line 27 of the snippet. This means that as soon as the GC decides to do so, the listener will get collected, and you won’t receive the events you expected to receive.

One way around this is for something ‘live’ to hold on to the listeners until their work is done. For instance, if the “PersonWatcher” class held on to a list of all its listeners, this would keep them alive. Of course, this has the other problem of requiring you to release them at some point in order to allow them to get GCed, but that may not be such a big deal, considering you might know better when to do that release than you know when to ‘detach’ the event listeners. In this particular example, I suspect it’s a wash.

Anyway, I hope you enjoyed this brief tour of our island of code surrounding weak references. It has served us well thus far, even with the caveats I keep mentioning. Until next time – happy coding!

Creating a remote branch in Git

2009-12-18T18:36:00.000-08:00

I just had to do this today, and I can't believe how unintuitive it is... At least I found this post

Binding to SelectedItems in WPF ListBox.

2009-02-22T15:59:00.000-08:00

Here's a very interesting approach to solving the "can't bind to selected items" issue with the WPF ListBox control. I'll definitely want to look into this more later, as I haven't had a chance to yet. I'm still fighting issues getting CheckedItems initialized on my CheckListBox control that I've written and it suffers from the same issue as SelectedItems in ListBox.

Alex's blog has the approach I'm referring to. Check it out.

Holy Crap! HyperV versus nVidia drivers - everyone loses!

2009-02-21T23:11:00.001-08:00

Wow, I've been trying to figure out why WPF apps are soooo slow on my machine for months now. I just chalked it up to Server 2008 being a server OS and having all the goodies for fast rendering turned off, or something like that.

Today, I stumbled across a post on a forum about problems with HyperV causing video and other apps that "use" video to be really slow on startup and run very sluggish, causing problems with audio and other side effects. It turns out that many (most) video drivers do not play well with HyperV (or HyperV doesn't play well with most video drivers - I'll let MS & the vendors argue over which is the correct statement, I'm staying out of the middle). As such, when running HyperV, the machine can be very sluggish when apps do anything that accesses the video hardware (obviously something that VS2008, VLC media player, and WPF applications do a lot of).

Finally, I bit the bullet and decided to uninstall HyperV and after doing so, my apps immediately run way faster. I'm not sure exactly how much faster, but I know that when running my UITestHarness app (part of the project I work on), the app would start up rather quickly, but when clicking on a button that shows a WPF window (the app is WinForms mainly), it would take a good 15 seconds to show the window. Now, those windows show immediately (less than 1 second). I find this rather amazing that there are such perf differences between HyperV and non HyperV configs. Oh well, hopefully they'll fix them soon, or I'll find a video card that doesn't have problems (reportedly ATI cards are often in better shape), 'cause I really like HyperV!

IDisposable and Garbage Collection

2009-01-30T21:16:00.001-08:00

I've had several discussions recently with friends and associates about garbage collection, IDisposable, and the disposal pattern recommended by Microsoft. These discussions have brought to light many misconceptions, a few incorrect statements, and some good and bad advice. In particular, the 'final straw' that led me to write this blog post is the post by Jeff Atwood (@CodingHorror) on his blog, and the response by Jeff Tucker on his blog (Agilology).

I tend to lean more towards agreeing with Jeff Atwood (@CodingHorror)'s post, in that I believe that calling Dispose is absolutely an optimization. That said, it's an optimization that in some cases simply should not be optional for all but those people who understand the precise implications of not doing so. For instance, SqlConnection. It's absolutely best practice to dispose of SqlConnection as soon as you can do so in applications that might use connection pooling. On the other hand, there are resources such as FileStream, DataSet, and the many WaitHandle-derived types that you may want to dispose of early, but it's absolutely an optimization (either in terms of memory footprint or resource contention) to do so early, not a requirement.

Among the things that can be said about IDisposable and GC, there are a few that I want to get out of the way first. First of all, the GC does not care about IDisposable at all! It is simply not aware of whether your class implements IDisposable or not. That said, many classes that implement IDisposable also implement a finalizer (discussed later), of which the GC is intimately aware.

Garbage Collection

First, before I go too far, let me make a brief description (simplified, of course) of the garbage collector and the process it follows to collect your unused objects. Basically put, you go along happily creating new objects from the managed heap, and the GC follows behind you and computes which of those objects are no longer 'reachable' from your program and cleans up after you.

The GC divides the garbage collection process into stages, called "generations". All objects start their lifetime in generation 0. Objects that "survive" a GC in generation 0 move to generation 1. Those that survive generation 1 move to generation 2. Earlier generations are collected more frequently than later generations.

The collection pass is actually two passes - the marking pass, and the reclamation (collection) pass. The marking pass is where the GC goes through all GC roots in the application and marks 'reachable' objects and all objects reachable from them as "live". The collection pass is where the GC goes through all objects in the generation(s) being collected and frees those objects that aren't being collected (it may also relocate objects to compact memory if it decides this is useful).

There's a bit more complexity to it when finalizers are involved. When an 'unmarked' object to be collected contains a finalizer that hasn't been suppressed using GC.SuppressFinalize(this) (presumably in IDisposable.Dispose) then the object is not freed, but rather is moved from the finalization queue to the freachable queue - and is marked (and all objects reachable from it are also marked). Obviously the computation is done in a way that makes sense, not iteratively as it sounds from my description, but that complexity is not important to our discussion.

For more information on the GC, I recommend Jeffrey Richter's CLR via C#, an excellent book on this and many other topics related to advanced .NET programming. However, I would caution you against reading his "Weak Events" example in that chapter as it is wrong (my next blog post will describe why and how to detect the invalidity of his approach). Also, there are several blogs that are good for GC and other debugging bits, such as Maoni's or Tess's blogs. A good start might be Tess's post here. (special thanks to my buddy Rich Lang for his tips on resources to recommend).

IDisposable and its uses

There are several different reasons people might use IDisposable. Of them, there are two that are the most popular and probably the only ones that "normal" developers should ever put into action. The first, most obvious, use of IDisposable is when your object needs to own 'unmanaged' resources, either directly or indirectly. For instance, if you're writing a .NET class that manages some resource that you obtain via a P/Invoke call to some unmanaged library and you need to "free" or "release" that resource when your object is no longer being used. In this case (direct ownership of unmanaged resources), IDisposable is not strictly necessary, but a finalizer is absolutely necessary. If the object you're trying to manage is a Win32 handle (closed by CloseHandle), you should probably look at the SafeWaitHandle and SafeFileHandle classes, as well as the SafeHandleMinusOneIsInvalid and SafeHandleZeroOrMinusOneIsInvalid classes.

The other popular use of IDisposable is for RAII (a C++ concept - "resource acquisition is initialization", whereby a resource is acquired as a constructor call, and released when it goes out of scope - i.e. a using statement body). An example of this usage is the TransactionScope object, where you acquire the transaction by "newing up" a TransactionScope object (in a C# "using"), and you release it when you exit the using statement.

I'll take these two uses in turn.

Resource Ownership

I call the first of the two use cases for IDisposable "Resource Ownership" as your object is the consumer of some resource either directly or indirectly and should free those resources when applicable. There are two forms of resource ownership, direct and indirect. Direct is, as it sounds, when your object has direct ownership over a resource. If the resource is unmanaged (that's really what we're talking about here), you must implement a finalizer for your object, and in that finalizer you should dispose of the resource. Also, since your finalizer is only executed when your object is collected, it's generally a good idea to give users of your object the opportunity to release the resource 'early'. For this reason, you implement IDisposable and the disposal pattern (described below).

If, on the other hand, you only have indirect ownership of unmanaged resources, you don't need a finalizer. Instead, you should only provide the IDisposable interface and implement the disposal pattern. If you provide a finalizer when it isn't needed, you will, in effect, be delaying the GC cleanup of your object unnecessarily, since objects with finalizers survive at least two collections after they are freed, if not more.

RAII in C#

I refer to the second of the two cases for IDisposable as "Resource Acquisition". The canonical example for this is TransactionScope, in my mind, but another example is a Mutex acquisition class (not to be confused with the FCL's Mutex class) or any other similar class. These classes can make your code much easier to maintain and read if used properly, but can introduce some very difficult to detect bugs if used improperly, so use them with caution. For this use of IDisposable, you aren't really using IDisposable because you own resources, but rather because you're building a class that should have acquire/release semantics and the syntax for doing so with C#'s "using" statement is very nice and clean.

There are several places where this pattern is used, TransactionScope is the one that comes to mind for me in the FCL, but Oren (Ayende Rahien) uses this pattern in Rhino Mocks and you see it in several other frameworks. Jeffrey Richter describes it in his book in the chapter on memory management, and makes the recommendation that if you are building a library to be used by others outside of your production code, you should make your RAII (my name, not his) objects reference types, but if you are using them only internally, they can be made very efficient through the use of value types that implement IDisposable. If you are writing these types for libraries, you should take great care to ensure that they free the resources they own at most once, and should think very carefully about whether these types need a finalizer (I believe the jury is out on this one, but I'd say they do and should implement the disposal pattern just as if they were managing a unmanaged resource).

It should be noted that while I call this "RAII in C#" it really isn't quite the same as RAII in C++, since in C++ destructors are guaranteed to be called when the object goes out of scope, whereas there's really nothing in the C# language / compiler that requires you put these objects in a "using" block, and thus there's nothing that guarantees that their Dispose() method gets called automatically if you choose not to use the using construct.

The disposal pattern and its canonical implementation

Many resources describe the canonical implementation of IDisposable via the disposal pattern, so I'm not going to go into excruciating detail here. I'll describe the basics of the pattern and refer the reader to other sources for exact details. The core idea of the disposal pattern is that there are two ways in which you might want to cleanup after your object: finalization and explicit disposal via IDisposable.Dispose. If your object is one that should have a finalizer, then the disposal pattern should absolutely be followed. If you don't have a finalizer, then you don't strictly need to follow the disposal pattern (often this is the case for RAII applications that don't acquire resources that will deadlock the application if they aren't released, or those that are guaranteed to be properly used - because you're writing both the object and all code that uses it). Even so, you're probably best to follow the disposal pattern every time you implement IDisposable and just leave the parts of the pattern empty that don't apply to your particular application.

The basic rules of the disposal pattern are:

finalizers should not refer to other managed objects during finalization, since those objects may have already had their finalizers called.
IDisposable.Dispose() should call IDisposable.Dispose on any objects owned by the implementing object that are IDisposable.
IDisposable.Dispose() and the finalizer should BOTH free any unmanaged resources owned by the object.
IDisposable.Dispose() should call GC.SupressFinalize(this) to mark the work of the finalizer as being already done.
if your objects have shared state, then the finalizers should have code to guarantee that two finalizers being called at the same time is thread-safe.
finalizers should not assume they are being called on any particular thread - therefore they cannot access TLS (thread-local storage) in any way, shape, or form!
calling IDisposable.Dispose() shouldn't throw an exception if called more than once.
method calls to any methods other than disposal methods (or the finalizer) should throw ObjectDisposedException if Dispose (or the finalizer) has already been called.

Microsoft's recommended approach for implementing the disposal pattern is to have a non-public (protected) virtual (unless your class is sealed) method called Dispose on your object that takes a boolean argument called "disposing". This method should be called both by the finalizer and by IDisposable.Dispose, and if you implement a "Close" convenience method or some other method that does the same thing as IDisposable.Dispose, that should also call this single-argument version of Dispose. In this Dispose method, your class should free any unmanaged resources, and if "disposing" is true, should also call Dispose on any IDisposable members of your class. It should also set a flag so you know to throw ObjectDisposedException when any of your other methods or properties are accessed. Finally, the Dispose method should call GC.SuppressFinalize(this) to notify the GC that the finalizer need not be called. Then, the object should implement the finalizer as a call to Dispose with disposing = false, and IDisposable.Dispose as a call to Dispose with disposing = true.

Sample code is as follows:

   1: class MyClass: IDisposable   2: {   3:   [DllImport(...)] // assume this is correctly specified.   4:   private extern void Free(IntPtr handle);   5:     6:   private IntPtr _UnmanagedThing;   7:   private FileStream _LogFile;   8:     9:   private bool _IsDisposed;  10:    11:   public MyClass(FileStream logFile, IntPtr unmanagedThing)  12:   {  13:     // check arguments and don't allow finalizer if they aren't valid.  14:     GC.SupressFinalize(this);  15:     if(logFile == null)  16:       throw new ArgumentNullException("logFile");  17:     if(unmanagedThing == IntPtr.Zero)  18:       throw new ArgumentException("unmanaged thing is invalid!",  19:                                   "unmanagedThing");  20:     GC.ReRegisterForFinalize(this);  21:    22:     _UnmanagedThing = unmanagedThing;  23:     _LogFile = logFile;  24:   }  25:    26:   protected virtual void Dispose(bool disposing)  27:   {  28:     // we can skip doing anything if it's already been done.  29:     if(_IsDisposed)  30:       return;  31:    32:     if(disposing)  33:     {  34:       // dispose of managed resources here, since we  35:       //   were called from IDisposable.Dispose()  36:       _LogFile.Dispose();  37:    38:       // make sure we know that we're disposed for other calls.  39:       _IsDisposed = true;  40:     }  41:    42:     // free unmanaged resources in either case (IDisposable or  43:     //   Finalize) and make sure the finalizer doesn't get called  44:     //   later by the GC.  45:     Free(_UnmanagedThing);  46:     GC.SuppressFinalize(this);  47:   }  48:    49:   public void Dispose() { Dispose(true); }  50:   ~MyClass() { Dispose(false); }  51:    52:   public void DoSomething()  53:   {  54:     // some function not related to disposal of the object,  55:     //   but requiring valid state...  56:     if(_IsDisposed)  57:       throw new ObjectDisposedException();  58:   }  59: }

Again, if you are working with a handle that is typical of Win32, you should look at the classes mentioned above (SafeHandle and it's family of derived-classes) as they do much of the work for you. You should also probably have a look at CriticalFinalizerObject as well, and the MSDN topic "Safe Handles and Critical Finalization", especially if you expect your code to be run in a hosted environment other than traditional .NET applications (i.e. IIS7, COM+, SQL Server, etc.).

So what's the point?

So, now, as I reread the beginning of my blog post, I wonder - what was the point I was trying to make? Well, it's basically this - the use of IDisposable to dispose of objects early is an optimization, assuming the disposal pattern was correctly implemented by author of the objects you are calling. That is, unless you are calling RAII-style objects, in which case forgetting to call Dispose and not using these objects in a using block could be disastrous to your program. On the other hand, there are several cases where it's extremely important to dispose of objects as soon as you are done with them, for instance when dealing with SqlConnection.

On the other hand, there are plenty of objects that are fine to allow GC to collect them and "finalize" them, and there is little or no perf impact to doing so (possibly even a positive impact of not forcing early cleanup). As a for-instance, consider a managed class that wraps an unmanaged resource that is not subject to contention (like some sort of unmanaged object in a library that is using the C/C++ heap to allocate these objects). If you create a large number of these objects, but are not at risk of running out of memory, it can be much faster to allow the GC to collect these objects (through finalization) than having your code call Dispose on all of them and forcing early cleanup (forcing the application to incur the cost of freeing this unmanaged memory on your user threads instead of the finalizer thread).

As with any of my posts and most of the advice on the CLR in general, the most important takeaway from this blog post should be "learn the details and use your own judgement". Happy coding.

Detecting if other instances of your app are running.

2009-01-23T00:21:00.001-08:00

I got a question from a friend of mine the other day about why Process.GetProcesses() was returning an error when used under Terminal Services by a user that doesn't have the Debug Programs privilege. It was because Process.GetProcesses() was trying to return the list of ALL processes running on the machine (including those not started by the current user). This worked just fine when a single user was logged into the TS server, but when more than one user was logged in it failed. My friend was very confused by this, until I told him that non-admin users need the Debug Programs privilege in order to open the process token of a process not created in their session.

Of course, his next question was "how do I get Process.GetProcesses() to return only the processes accessible to the current user's session?". At that point I took a step back and asked "why?".

In fact, what he was trying to do was to detect whether another instance of his application was running, in order to gracefully tell the user that only one instance of the program can run at a time. I asked him why he wasn't using a Mutex and the answer surprised me, just a bit. He said "we were worried about what would happen if the process unexpectedly exited". My next question was yet another "why?".

Apparently there's some confusion about how Mutexes work in Windows, as well as how the Mutex method that I've used for about 10 years to detect multiple instances of an app works. I'm not sure where I first learned this technique from, but I've been using it since my Win32 programming days and it's worked like a charm ever since then. I've now used it in several reincarnations (from VB6, Delphi, C++, and now .NET) and it's never failed me.

The first misconception that almost everyone has when hearing about this approach is that it has something to do with Mutexes (at least in the way they are used in multithreaded programs). In fact, any Kernel named object could be used for this approach, including even a "delete on close" file. However, I've always used mutexes because that's what was recommended to me, and I don't think it makes much difference since you never actually 'wait' on the mutex so it's not really important what type of object it is. The idea of this technique is not to use a mutex, but to use the 'kernel namespace'. Essentially, by creating a mutex in the naming scope appropriate for your purposes, you are 'registering' that name with the OS. Then, if you were the first one to register the name, you continue, otherwise you show your message and quit.

The pattern looks like this:

bool createdNew;
MutexSecurity ms = new MutexSecurity();
MutexAccessRule mar = new MutexAccessRule(
  new SecurityIdentifier(WellKnownSidType.WorldSid, null),
  MutexRights.FullControl,
  AccessControlType.Allow);
ms.AddAccessRule(mar);
Mutex m = new Mutex(
  false,
  @"Global\somenameuniquetomyapplication",
  out createdNew,
  ms);
if(!createdNew)
{
  // show a message that only one instance is allowed and exit
  return;
}

where the first part (the MutexSecurity stuff) guarantees that anybody can access the mutex (the security probably doesn't need FullControl, but I didn't want to think about what it really wants, so I left it at that, no real security risk here since the only thing that could happen is other apps could grab hold of this mutex and use it for their own purposes, but that's not really going to do much). The point of this is that the other instances of your app may not be started by the same user - this guarantees that only one app per machine is startable (because I'm using the Global namespace for my mutex name). If, on the other hand, you want a per-session limit of a single instance (rather than per-machine) you could use the local namespace (change Global\ to Local\), which is local to each TS session.

The second part of that mess is the mutex creation code. This code (as written) will create the mutex if it doesn't exist or return the existing one if one already exists with that name (and you have permissions to open it). If you created it (because it didn't exist), then 'createdNew' will be set to True. Otherwise, it will be false. Either way, you get a valid Mutex object.

Then, you should keep this Mutex object alive for the lifetime of your application (i.e. save "m" in a variable that has the same lifetime as your application so that it doesn't get GCed and disappear on you). A good way to do this is to make it a local variable in your Program.cs's main function. When your application is done shutting down, you can close the mutex either as soon as you think it's ok for other instances to start, or let Windows reclaim the mutex for you.

Generally, I put this code in a separate library function that I can call from everywhere. That function generally looks something like the following (in .NET):

/// <summary>
/// Attempts to create an application isolation mutex, and
/// return it to the caller.  If the caller isn't the first
/// to create the mutex (i.e. it already exists) then we
/// return null to indicate that the caller "lost".
/// </summary>
/// <param name="objectName">The name to use for the mutex,
///   should begin with Global\ if you want per-machine
///   isolation, or Local\ if you want per-session isolation.
///   If you want per-machine/per-user isolation (slightly
///   different from per-session) then you should mangle the
///   mutex name by putting the username in it somewhere.</param>
/// <returns>Null if the mutex already existed, or the mutex
///   if it was created by this function.  You should keep the
///   mutex in scope somewhere until you are ready to release
///   the isolation.  You shouldn't use this mutex for locking
///   or anything else - just forget it's a mutex altogether.
/// </returns>
public IDisposable GetAppIsolationHandle(string objectName)
{
  // setup the mutex security settings.
  var ms = new MutexSecurity();
  var sidWorld =
      new SecurityIdentifier(WellKnownSidType.WorldSid, null);
  var mar = new MutexAccessRule(
      sidWorld,
      MutexRights.FullControl,
      AccessControlType.Allow);
  ms.AddAccessRule(mar);

  // create the mutex and return it if it's "ours".
  bool createdNew;
  var mutex = new Mutex(
      false,
      objectName,
      out createdNew,
      ms);
  if (createdNew)
      return mutex;

  // return null if the mutex isn't "ours".
  return null;
}

Then, you can call this code in your Program.cs as follows:

var isolationHandle
  = GetAppIsolationHandle(@"Global\MyApp");
if (isolationHandle == null)
{
  MessageBox.Show("Sorry, only one at a time!");
  return;
}
using(isolationHandle)
{
  Application.EnableVisualStyles();
  Application.Run(new Form());
}

which, of course, looks really nice (at least in my opinion). If you had special "shutdown" stuff to do after Run returns, you can put that outside the using block (assuming it doesn't need to be isolated).

Getting the message count from an MSMQ Queue

2008-11-02T23:33:00.001-08:00

While at KaizenConf (http://www.kaizenconf.com) today, I was attending a session held by Chris Patterson (twitter: @PhatBoyG) & Dru Sellers (twitter @drusellers) on ESBs and MassTransit, their ESB implementation.

During the session, Dru complained that there wasn't a good way to get the message count from an MSMQ queue. Of course, I was required to take that as a challenge, since that's the kind of guy I am :). I found that as of the 3.0 API for MSMQ (apparently, that's the XP / Server 2003 vintage), there's a few ways to get the message count for a queue.

The available methods (that I found) for asking for the count of messages in a Queue were:

Call MQMgmtGetInfo API to query the queue for the PROPID_MGMT_QUEUE_MESSAGE_COUNT property.
Load up a MSMQManagement COM object, call it's Init method to associate with a Queue, and then ask for it's MessageCount property value.

It appears that the first method (which is an API call) is actually just a proxy for the second, so I'm not going to talk about it. Calling the COM object from .NET is much easier than calling the API anyway, since it's not exactly a 'pretty' API for P/Invoke purposes.

Since I'm not really interested in investing a lot of time in this blog post, I'm just going to paste the code here and let you do with it as you please... Here goes...

var path = @".\Private$\foo";
MessageQueue mq = MessageQueue.Exists(path) 
                ? new MessageQueue(path) 
                : MessageQueue.Create(path);
try
{
    // try to insert a few items into the queue...
    mq.Send("foo");
    mq.Send("bar");
    mq.Send("foo");

    var msmqMgmt = new MSMQManagement();
    object machine = null; // mq.MachineName;
    object queuename = mq.Path;
    object formatname = null; //mq.FormatName;
    msmqMgmt.Init(ref machine, ref queuename, 
        ref formatname);
    int messageCount = msmqMgmt.MessageCount;

    MessageBox.Show(string.Format("Queue has {0} items", 
        messageCount));
}
finally
{
    string path2 = mq.Path;
    mq.Close();
    MessageQueue.Delete(path2);
}

Of course, this code requires a reference to the COM type library - namely the "Microsoft Message Queue 3.0 Object Library" on the COM list in VS2008 when you have MSMQ installed on your dev box.

I had some weird problems trying to test this code on my machine, hence the commented machinename and formatname. I think the problem was probably related more to the configuration of my machine than it was the code. I suspect, however, that there may be some complexities that require you to specify machine name, queue name, and format name differently depending on whether you are working with a local queue or a remote one.

I found that for a local queue, the easy way to reference it was the code snippet above (don't specify machine, don't specify format name, supply the "path"). For a remote queue, I suspect that it will be easier to pass the machine name, the format name, and omit (pass null for) the path. Note, the API states that you should NOT pass both the format name and the path name, or it will give an exception.

As always, if you have questions regarding this code, please don't hesitate to contact me via the comments or my email.

.NET wish list...

2008-11-01T10:24:00.000-07:00

Generics that can support 'text-like' replacement at runtime, where you can basically say that "I know that type T actually has a method called Foo with the signature bool Foo(int) but may not implement some particular interface (since you may not control the implementation of T)". I'd like to be able to call Foo from my generic class even if I can't change T to implement an interface that supports Foo. I'm thinking something like:
```
public static void DoSomething(T target)
    where T: class having bool Foo(int)
{
    if(target.Foo(0))
        Console.WriteLine("Woohoo");
}
```
A way of marking objects that MUST be used in a 'using' expression (i.e. they only make sense there) and having the C# compiler enforce it. For instance, an attribute would work for me (similar to how 'FlagsAttribute' indicates special semantics on enums). My reasoning for this is that I'd like to use IDisposable for some C++-style RAII-like stuff, but there's no way to guarantee that the objects are used correctly.
A way of injecting simple code / hooking "before" and "after" property notifications on 'automatic properties' in C#. For instance, if I do an automatic property, I'd love to be able to say "any time this changes, call this method", or something like that. It could be useful for INotifyPropertyChanged, but also for other things as well.

Building a WPF Grid Control (Part 2 of ?)

2008-08-09T00:23:00.001-07:00

So, last time (Building a WPF Grid Control (Part 1 of ?)) we described some of the data binding structures of our WPF grid control. I'm not going to provide implementations of those interfaces yet, instead concentrating on some of the interesting bits of the control's implementation. For the most part, building the control comes down to several bits of functionality - scrolling, rendering, mouse support, keyboard support, and data binding. We've already seen the data binding part, so next I want to concentrate on scrolling.

Building scrolling into the UI is not terribly difficult, and for the most part, the implementation can be done without actually dealing with the data binding interfaces at all. However, there are a few details that we need from the interfaces in order to have a go at scrolling - namely, we need to know how big the fixed and scrolling regions need to be. Therefore, we need a dummy implementation of IDimensionMetrics that give us some sizes to play with. Since we currently won't have fixed rows or fixed columns, and all the grid cares about for laying out the regions is the TotalSpace member of this interface, we should be able to get away with a really dumb implementation.

So, let's implement IDimensionMetrics as:

public class ReallyDumbMetrics : IDimensionMetrics
{
    private double _TotalSpace;

    public ReallyDumbMetrics(double totalSpace)
    {
        _TotalSpace = totalSpace;
    }

    #region IDimensionMetrics Members

    public double GetSpace(int index)
    {
        throw new NotImplementedException();
    }

    public void SetSpace(int index, double space)
    {
        throw new NotImplementedException();
    }

    public double GetStart(int index)
    {
        throw new NotImplementedException();
    }

    public double TotalSpace
    {
        get { return _TotalSpace; }
    }

    public int Count
    {
        get { throw new NotImplementedException(); }
    }

    public event EventHandler SpaceChanged;

    #endregion
}

I just realized that I forgot to describe the IGridBindings interface in my last post. This interface is very simple and is just a container for the various pieces of the bindings (selection info, dimensions, region render info) that allows a single point of binding for the grid control. The interface is defined as:

public interface IGridBindings
{
    IRegionRenderInfo TopLeftRenderInfo { get; }
    IRegionRenderInfo HScrollRenderInfo { get; }
    IRegionRenderInfo VScrollRenderInfo { get; }
    IRegionRenderInfo HVScrollRenderInfo { get; }

    IDimensionMetrics FixedRowMetrics { get; }
    IDimensionMetrics FixedColMetrics { get; }
    IDimensionMetrics ScrollingRowMetrics { get; }
    IDimensionMetrics ScrollingColMetrics { get; }

    ISelectionInfo SelectionInfo { get; }

    void Reorder(int[] oldPositions);
}

Since we'll be starting on our grid control now, it would be nice to have a IGridBindings implementation that will allow us to start working with the binding interfaces of the grid. So, let's go with a dummy implementation of GridBindings that uses our ReallyDumbMetrics implementation above. Here goes:

public class ReallyDumbGridBindings : IGridBindings
{
    ReallyDumbMetrics _FixedRowMetrics;
    ReallyDumbMetrics _FixedColMetrics;
    ReallyDumbMetrics _ScrollingRowMetrics;
    ReallyDumbMetrics _ScrollingColMetrics;

    public ReallyDumbGridBindings(double fixedRowSize, double fixedColSize, double scrollRowSize, double scrollColSize)
    {
        _FixedRowMetrics = new ReallyDumbMetrics(fixedRowSize);
        _FixedColMetrics = new ReallyDumbMetrics(fixedColSize);
        _ScrollingRowMetrics = new ReallyDumbMetrics(scrollRowSize);
        _ScrollingColMetrics = new ReallyDumbMetrics(scrollColSize);
    }

    #region IGridBindings Members

    public IRegionRenderInfo TopLeftRenderInfo
    {
        get { throw new NotImplementedException(); }
    }

    public IRegionRenderInfo HScrollRenderInfo
    {
        get { throw new NotImplementedException(); }
    }

    public IRegionRenderInfo VScrollRenderInfo
    {
        get { throw new NotImplementedException(); }
    }

    public IRegionRenderInfo HVScrollRenderInfo
    {
        get { throw new NotImplementedException(); }
    }

    public IDimensionMetrics FixedRowMetrics
    {
        get { return _FixedRowMetrics; }
    }

    public IDimensionMetrics FixedColMetrics
    {
        get { return _FixedColMetrics; }
    }

    public IDimensionMetrics ScrollingRowMetrics
    {
        get { return _ScrollingRowMetrics; }
    }

    public IDimensionMetrics ScrollingColMetrics
    {
        get { return _ScrollingColMetrics; }
    }

    public ISelectionInfo SelectionInfo
    {
        get { throw new NotImplementedException(); }
    }

    public void Reorder(int[] oldPositions)
    {
        throw new NotImplementedException();
    }

    #endregion
}

Ok... I think we're ready to write some Grid code...

Region Implementation

Let's start with a simple implementation of a GridRegion base class that will provide the functionality for the four cell-containing regions in the grid UI. This class will derive from FrameworkElement just as our Grid does, and will participate in the layout system as usual (measure and arrange passes). For now, to avoid getting bogged down in the details of rendering rows and columns, we'll make it just draw big red ellipses in the regions.

Let's start with the GridRegion. We can begin with the following class:

internal class GridRegion: FrameworkElement
{
    public GridRegion()
    {
    }
}

Now, for any FrameworkElement, we need to support Measure and Arrange layout passes. We also need to support the dimension metrics in order to obtain the sizes for our control. The dimension metrics additions to GridRegion will be:

private IDimensionMetrics _RowMetrics;
private IDimensionMetrics _ColMetrics;

private void _ReplaceMetrics(ref IDimensionMetrics target, IDimensionMetrics source)
{
    if (target != null)
        target.SpaceChanged -= Metrics_SpaceChanged;
    target = source;
    if (target != null)
        target.SpaceChanged += Metrics_SpaceChanged;
}

public void SetBindings(IDimensionMetrics rowMetrics, IDimensionMetrics colMetrics)
{
    _ReplaceMetrics(ref _RowMetrics, rowMetrics);
    _ReplaceMetrics(ref _ColMetrics, colMetrics);
    InvalidateMeasure();
    InvalidateVisual();
}

void Metrics_SpaceChanged(object sender, EventArgs e)
{
    InvalidateMeasure();
    InvalidateVisual();
}

The basic idea here is the SetBindings method, used by the EditorGrid (that owns the GridRegion objects) to initialize the RowMetrics and ColMetrics properties. Each time the Bindings property of the EditorGrid is changed (it's the property that holds the IGridBindings interface reference) the grid will call SetBindings on each of the four regions.

We also need to support the layout and arrange passes for our control. Our container (the grid) will decide the layout of our control, all we need to do is request as much space as it is willing to give us (by returning availableSize from the MeasureOverride method as follows).

protected override Size MeasureOverride(Size availableSize)
{
    return availableSize;
}

This same functionality could possibly be achieved in another way, but this was the easiest way that I found. I suspect that setting the alignment properties to 'stretch' might have worked, but it didn't seem to (or at least I don't remember it working when I thought I tried it).

Grid Implementation

Ok, we can now start implementing the grid control itself. Let's start with this class, similar to the GridRegion we just completed:

public class EditorGrid: FrameworkElement
{
    GridRegion _TopLeftNonScroll;
    GridRegion _HScrollRegion;
    GridRegion _VScrollRegion;
    GridRegion _HVScrollRegion;

    public EditorGrid()
    {
        _TopLeftNonScroll = new GridRegion();
        _HScrollRegion = new GridRegion();
        _VScrollRegion = new GridRegion();
        _HVScrollRegion = new GridRegion();
    }

    private IGridBindings _Bindings;
    public IGridBindings Bindings
    {
        get { return _Bindings; }
        set
        {
            if (_Bindings != value)
            {
                if (_Bindings != null)
                {
                    _Bindings.FixedColMetrics.SpaceChanged -= Metrics_SpaceChanged;
                    _Bindings.FixedRowMetrics.SpaceChanged -= Metrics_SpaceChanged;
                    _Bindings.ScrollingColMetrics.SpaceChanged -= Metrics_SpaceChanged;
                    _Bindings.ScrollingRowMetrics.SpaceChanged -= Metrics_SpaceChanged;
                }
                _Bindings = value;
                if (_Bindings != null)
                {
                    _Bindings.FixedColMetrics.SpaceChanged += Metrics_SpaceChanged;
                    _Bindings.FixedRowMetrics.SpaceChanged += Metrics_SpaceChanged;
                    _Bindings.ScrollingColMetrics.SpaceChanged += Metrics_SpaceChanged;
                    _Bindings.ScrollingRowMetrics.SpaceChanged += Metrics_SpaceChanged;
                }
                _TopLeftNonScroll.SetBindings(_Bindings.FixedRowMetrics, _Bindings.FixedColMetrics);
                _HScrollRegion.SetBindings(_Bindings.FixedRowMetrics, _Bindings.ScrollingColMetrics);
                _VScrollRegion.SetBindings(_Bindings.ScrollingRowMetrics, _Bindings.FixedColMetrics);
                _HVScrollRegion.SetBindings(_Bindings.ScrollingRowMetrics, _Bindings.ScrollingColMetrics);

                InvalidateMeasure();
                InvalidateVisual();
            }
        }
    }

    void Metrics_SpaceChanged(object sender, EventArgs e)
    {
        InvalidateMeasure();
        InvalidateVisual();
    }
}

As you can see, the bindings on the regions are set to different metrics depending on their locations in the grid. The TopLeftNonScroll region uses 'Fixed' metrics for both rows and columns, the HScrollRegion uses 'Scrolling' for columns and 'Fixed' for rows, and so on.

We now need to discuss the layout of the children of the editor grid. We also need to add scrollbars and the other non-cellular regions. For the moment, we'll ignore the other regions and the scrollbars. Let's just get the cellular regions in place first. In order to do all the layout stuff, we'll create a helper class that will make things easier for us. I'll call this class LayoutMetrics and define it as follows:

internal class LayoutMetrics
{
    public Rect vscroll_rect;
    public Rect hscroll_rect;
    public Rect hscrollR_rect;
    public Rect vscrollR_rect;
    public Rect hvscrollR_rect;
    public Rect topleft_rect;
    public Rect topright_rect;
    public Rect botleft_rect;
    public Rect botright_rect;

    public LayoutMetrics(Size size, double vscroll_width, double hscroll_height, double fixedRowHeight, double fixedColWidth)
    {
        vscroll_rect =
            new Rect(size.Width - vscroll_width,
                fixedRowHeight,
                vscroll_width,
                Math.Max(0, size.Height - hscroll_height - fixedRowHeight));

        hscroll_rect =
            new Rect(fixedColWidth,
                size.Height - hscroll_height,
                Math.Max(0, size.Width - vscroll_width - fixedColWidth),
                hscroll_height);

        hscrollR_rect = hscroll_rect;
        hscrollR_rect.Y = 0;
        hscrollR_rect.Height = fixedRowHeight + 1;

        vscrollR_rect = vscroll_rect;
        vscrollR_rect.X = 0;
        vscrollR_rect.Width = fixedColWidth + 1;

        hvscrollR_rect =
            new Rect(hscroll_rect.X, vscroll_rect.Y,
                hscroll_rect.Width, vscroll_rect.Height);

        topleft_rect =
            new Rect(new Size(fixedColWidth + 1, fixedRowHeight + 1));
        topright_rect =
            new Rect(vscroll_rect.X, 0, vscroll_rect.Width, hscrollR_rect.Height - 1);
        botleft_rect =
            new Rect(0, hscroll_rect.Y, vscrollR_rect.Width - 1, hscroll_rect.Height);
        botright_rect =
            new Rect(vscroll_rect.X, hscroll_rect.Y, vscroll_rect.Width, hscroll_rect.Height);
    }
}

The main idea of this class is to break the space occupied by the Grid into the component rectangles. For now, we'll supply some dummy values for the sizes of the horizontal and vertical scrollbars. Now, given this class, we can implement our 'measure' and 'arrange' methods for the WPF layout system. We do so (on our EditorGrid class) as follows:

protected override Size MeasureOverride(Size availableSize)
{
    // for now, fake the sizes of the scroll bars just to reserve some space.
    LayoutMetrics m = new LayoutMetrics(availableSize,
        14, // vscroll_width
        14, // hscroll_width
        _Bindings.FixedRowMetrics.TotalSpace,
        _Bindings.FixedColMetrics.TotalSpace);

    _TopLeftNonScroll.Measure(m.topleft_rect.Size);
    _HScrollRegion.Measure(m.hscrollR_rect.Size);
    _VScrollRegion.Measure(m.vscrollR_rect.Size);
    _HVScrollRegion.Measure(m.hvscrollR_rect.Size);

    return availableSize;
}

protected override Size ArrangeOverride(Size finalSize)
{
    // for now, fake the sizes of the scroll bars just to reserve some space.
    LayoutMetrics m = new LayoutMetrics(finalSize,
        14, // vscroll_width
        14, // hscroll_width
        _Bindings.FixedRowMetrics.TotalSpace,
        _Bindings.FixedColMetrics.TotalSpace);

    _TopLeftNonScroll.Arrange(m.topleft_rect);
    _HScrollRegion.Arrange(m.hscrollR_rect);
    _VScrollRegion.Arrange(m.vscrollR_rect);
    _HVScrollRegion.Arrange(m.hvscrollR_rect);

    return finalSize;
}

Now, we need to add visual tree support to our grid control. In order to do this, we need a few features - first, we need to add the regions to the visual tree by calling AddVisualChild on our grid visual. Second, we need to override the 'render list' method & property GetVisualChild and VisualChildrenCount respectively. The first (calling AddVisualChild) we do by adding the following lines to the constructor (after the creation of the regions):

AddVisualChild(_TopLeftNonScroll);
AddVisualChild(_HScrollRegion);
AddVisualChild(_VScrollRegion);
AddVisualChild(_HVScrollRegion);

Now that we have those lines in the constructor, we need to implement the 'rendering' functionality. The easiest way to do this is either with a VisualCollection, or since in our case we have a fixed list, just an array of Visuals. The GetVisualChild method must return visuals in the order in which they should be rendered, and we want our regions to render in the following order: HVScrollRegion, HScrollRegion, VScrollRegion, TopLeftNonScroll. We will add a field to EditorGrid class that is an array of visuals (Visual[]) called _Visuals, and initialize it in the constructor (after the four lines above) as follows:

_Visuals = new Visual[]
{
    _HVScrollRegion,
    _HScrollRegion,
    _VScrollRegion,
    _TopLeftNonScroll
};

Additionally, we need to implement the GetVisualChild method and VisualChildrenCount property as follows:

protected override Visual GetVisualChild(int index)
{
    return _Visuals[index];
}

protected override int VisualChildrenCount
{
    get { return _Visuals.Length; }
}

We now need to implement rendering in our GridRegion and then we'll have something we can start messing with. Here's the implementation of the OnRender method for the GridRegion control.

protected override void OnRender(DrawingContext drawingContext)
{
    if (_RowMetrics == null  _ColMetrics == null)
        return;

    double xmid = _ColMetrics.TotalSpace / 2;
    double ymid = _RowMetrics.TotalSpace / 2;

    drawingContext.PushClip(new RectangleGeometry(new Rect(RenderSize)));
    drawingContext.DrawEllipse(Brushes.Red, null, new Point(xmid, ymid), xmid, ymid);
    drawingContext.Pop();
}

Next time, we'll work on getting some scrolling features working.

Building a WPF Grid Control (Part 1 of ?)

2008-08-01T13:21:00.001-07:00

So, some of you may know that I'm building a Grid control for our MG-ALFA application. We started by looking at the WPF grids out there, and found that none of them really fit our needs. There were several issues with the controls on the market. Much of what we wanted was very 'simple', as far as look & feel (like a traditional grid), yet all the WPF grids on the market seemed to be focused on 'pretty'. Also, we needed to be able to customize several very specific features - for instance, we wanted dragging of columns for reordering, fixed rows & columns, and the ability to easily transpose the grid.

None of these features were easy to come by in existing controls, and in order to get an of them, we would have to heavily customize the controls out of the box. The customization would be 'on top' of the control, so there wasn't a good way to tie it to our data model, and transpose was the killer feature. In order to get transpose, we would have had to write some really nasty code and do some really unpleasant things with databinding. If we didn't do those nasty things, we'd have to use 'unbound' mode on the controls, which would lead to really unpleasant code to keep the grid in sync with the data changes.

Finally, after a ton of investigation, we decided we'd be better off just writing our own control and building a truly custom data model for the grid, rather than trying to force fit an existing control to our problem. We were very skeptical about the amount of time it would take to build a grid that had the features we needed, but I was pretty sure it would be less than a few weeks, and it turns out I was mostly right about that. I'm going to try to describe the design of the control, from the ground up, in a series of blog posts, but I hope you'll ask questions if you want more details, as I undoubtedly won't cover everything.

So, first, let's talk about the basic design of the grid, and the features we required. The grid is built on WPF mostly using visual layer programming to do the rendering. It supports fixed rows, fixed columns, and has row and column headings. Visually it looks very much like MS Excel. The UX is also intended to be very much like Excel, except with some modifications that are specific to our domain needs (and no support for formulas).

What to derive from?

The first decision in any custom control development for WPF is to decide which of the multitude of classes you should derive from. The class hierarchy is:

Visual
UIElement
FrameworkElement
Control
...

Visual is very rudimentary, and basically provides only the ability to manage and participate in the visual tree. It is possible to build components at the 'Visual' level, but relatively difficult, and they must be built from other components that are at a higher level (at least UIElement). This is because the only thing a Visual can effectively do is contain other Visuals, and perform hit testing. There are a few Visual-derived classes that can be used by control designers (DrawingVisual, ContainerVisual), but these aren't really useful as base classes (maybe ContainerVisual, but certainly not DrawingVisual).

UIElement is effectively the lowest level class that a control designer might want to derive from. It provides basic layout, event handling, focus support, and rendering features. To provide code for rendering a UIElement, you must override OnRender. You also will want to override MeasureCore and ArrangeCore to participate in the layout process. You may also want to override HitTestCore to provide sophisticated hit testing for your control, especially if you have a non-rectangular area (our control will be rectangular and covered by other child controls, so we don't really need to mess with HitTestCore).

FrameworkElement is really the 'entry point' into WPF framework-level programming. Much of the core functionality for rendering is introduced in the UIElement class, but FrameworkElement builds on these features and provides some core implementation that makes it easier for you to implement the layout methods (i.e. it handles things like HorizontalAlignment, VerticalAlignment, Width, Height, etc., so you don't have to write the tedious code to make these work in your MeasureCore and ArrangeCore implementations). It also provides the core functionality needed for data binding. We'll derive our grid control from FrameworkElement, because it's the lowest class we can derive from without making a ton of extra work for ourselves, and it's the highest class we can effectively derive from without introducing features we don't want.

Control introduces style and templating support. Since we specifically don't want the XAML user to be able to customize the control template and styles for our grid (we want very specific control over how things are rendered), and don't need the flexibility that styles and templates provide, we don't want Control. However, it should be noted that if you want to build the 'best' control, from a flexibility standpoint, you probably do want to provide these features and use the 'recommended' approach of deriving from something at the Control or higher levels of the inheritance tree.

Visual Layout

Our grid control looks like the screenshot below (currently). It is still a work in progress, and that's why those ugly orange sections are there, and why the fixed rows look kinda funny (no gridlines, green background, etc.).

Obviously the data I've been working with is dummy data, generated by my data source for my benefit during development.

My first step in building the control was to design the layout of the control, in terms of separate 'regions' of the grid, based on their scrolling nature, and based on the relative positions of the scrolling regions. In the picture below, I've labeled the 9 independent regions of the grid control.

The regions, in left-to-right (top-to-bottom) order are:

TopLeftNonScroll - the fixed row/fixed column intersection, including the "select all" box.
HScrollRegion - the horizontal-only scrolling region (fixed rows, scrolling columns).
TopRightNonScroll - the area above the vertical scroll bar, that doesn't scroll and will eventually house buttons or some other visual cue / support.
VScrollRegion - the vertical-only scrolling region (fixed columns, scrolling rows).
HVScrollRegion - the 'data' section of the grid. This area scrolls both directions, and is made up of the scrolling rows / scrolling columns intersection.
VScrollBar - the vertical scroll bar (a ScrollBar control with its Orientation set to Orientation.Vertical)
BottomLeftNonScroll - the non scrolling area to the left of the horizontal scroll bar, will eventually be another place for buttons, etc.
HScrollBar - the horizontal scroll bar (a ScrollBar control with its Orientation set to Orientation.Horizontal).
BottomRightNonScroll - this will likely just be a gray 'dead-zone' so the scroll bars don't look stupid.

For the most part, except for scrolling, the HScrollRegion, TopLeftNonScroll, VScrollRegion, and HVScrollRegion have the same UI / UX, so I've combined much of the functionality into a single base class called "GridRegion". It has some parameterized options (like which directions it can scroll), but for the most part the code is all shared and used from this class. The derived classes are generally pretty small, and are just responsible for 'customizing' the GridRegion functionality.

The other regions are currently implemented just as canvases (except the ScrollBars, of course).

Data Binding model

My data binding model has several parts. The entire model is really a 'view' from the standpoint of MVP-like design patterns (at least in my assessment it is). The real data model is specific to the application. The Data Binding model supported by the grid has bindings for the selection and active / anchor cells, the row & column sizes, and the contents of the cells (including render flags and other special items).

In order to simplify the design and implementation, I've separated the binding into several objects based on the way the regions break up the grid. The major breakdown is between dimension metrics, render info, and selection support.

Dimension Metrics (sizes of rows / columns)

For the dimensions of the rows and columns, I've defined an interface called IDimensionMetrics that allows management of and provides information about the sizes of rows or columns. It is defined as follows.

public interface IDimensionMetrics
{
    double GetSpace(int index);
    void SetSpace(int index, double space);
    double GetStart(int index);
    double TotalSpace { get; }
    int Count { get; }

    event EventHandler SpaceChanged;
}

Each object that implements IDimensionMetrics only represents a single group of rows or columns. There are 4 implementations of IDimensionMetrics for a single set of grid bindings. The columns are broken into "fixed" and "scrolling", as are the rows (for a total of 4 separate groups).

In order to provide some additional features for IDimensionMetrics without requiring that all implementers implement these features, I've used some extension methods to implement common algorithms based on IDimensionMetrics. The extension class is as follows.

internal static class DimensionExtensions
{
    public static double GetEnd(this IDimensionMetrics metrics, int index)
    {
        return metrics.GetStart(index) + metrics.GetSpace(index);
    }

    public static int HitTestNoSizing(this IDimensionMetrics metrics, double v)
    {
        for (int i = 0; i < metrics.Count; i++)
        {
            double ofs = v - metrics.GetStart(i);
            if (ofs >= 0 && ofs <= metrics.GetSpace(i))
                return i;
        }
        return -1;
    }

    public static int HitTestWithSizing(this IDimensionMetrics metrics, double v, out bool overSizingGrip)
    {
        for (int i = 0; i < metrics.Count; i++)
        {
            double ofs = v - metrics.GetStart(i);
            double space = metrics.GetSpace(i);
            if (ofs.InNeighborhood(space, 3))
            {
                overSizingGrip = true;
                return i;
            }
            else if (ofs.InBetween(0, space, DoubleExtensions.EndpointInclusionMode.LeftInclusive))
            {
                overSizingGrip = false;
                return i;
            }
        }
        overSizingGrip = false;
        return -1;
    }

    private delegate Rect CVR_GetRect();
    private delegate void CVR_UpdateRect(double space);

    public static void ComputeVisibleRange(this IDimensionMetrics metrics, Rect visibleRect, Direction direction, out int first, out int second)
    {
        if(direction != Direction.Horizontal && direction != Direction.Vertical)
            throw new ArgumentException("direction must be horizontal (columns) or vertical (rows)", "direction");

        int min = metrics.Count;
        int max = -1;

        var initializeRect = direction == Direction.Vertical ?
            (CVR_GetRect)(() => new Rect(visibleRect.Left, 0, visibleRect.Width, 0))
            : (CVR_GetRect)(() => new Rect(0, visibleRect.Top, 0, visibleRect.Height));

        Rect rngRect = initializeRect();

        var updateRectSize = direction == Direction.Vertical ?
            (CVR_UpdateRect)((double space) => rngRect.Height = space)
            : (CVR_UpdateRect)((double space) => rngRect.Width = space);

        var updateRectPos = direction == Direction.Vertical ?
            (CVR_UpdateRect)((double space) => rngRect.Y += space)
            : (CVR_UpdateRect)((double space) => rngRect.X += space);

        for (int i = 0; i < metrics.Count; i++)
        {
            double space = metrics.GetSpace(i);
            updateRectSize(space);

            if (rngRect.IntersectsWith(visibleRect))
            {
                min = Math.Min(i, min);
                max = Math.Max(i, max);
            }

            updateRectPos(space);
        }

        first = min;
        second = max;
    }
}

In IDimensionMetrics, GetStart gives the starting position of a row or column, and GetSpace gives the space that it occupies. The extension method GetEnd returns the result of GetStart + GetSpace for a given column. TotalSpace is the sum of all GetSpace values for all rows/columns in the IDimensionMetrics. It could have also been computed as an extension method, but I decided it would be better as a property so that the IDimensionMetrics implementer could precompute it.

The HitTestNoSizing and HitTestWithSizing extension methods help determine which column or row a given point is over (for mouse hit testing). The former ignores sizing grips, while the latter will indicate the proper position for a sizing grip (currently hardcoded to a neighborhood of 3 device-independent pixels on each side of the sizing line).

Selection Info

First of all, my grid only supports selection of rows / columns / cells in the scrolling area of the grid. For that reason, I have a single SelectionInfo binding for all regions, and the columns / rows used by the SelectionInfo members (FirstCol, FirstRow, LastCol, LastRow, etc.) are relative to the top left corner of the data region (HVScrollRegion). If I needed support for selecting within the fixed rows and columns, then there would be some additional complexity in my code but it could be supported. I think of the fixed rows and fixed columns as essentially being 'extended' headers, so it doesn't make sense to select them, or have an active cell in these regions (just as it doesn't make sense to be able to make the active cell be the 'C' header in the C column of Excel!).

As with my other stuff, selection info has an interface that exposes the functionality required by the system - ISelectionInfo, defined as follows.

public enum SelectionType
{
    None,
    Rows,
    Columns,
    Cells,
    All,
}

public interface ISelectionInfo
{
    SelectionType SelectionType { get; }
    int FirstRow { get; }
    int FirstCol { get; }
    int LastRow { get; }
    int LastCol { get; }

    int AnchorRow { get; }
    int AnchorCol { get; }

    int ActiveRow { get; }
    int ActiveCol { get; }

    void SelectColumn(int column, bool extend);
    void SelectRow(int row, bool extend);
    void SelectCell(int row, int column, bool extend);
    void SelectAll();
    void Clear();

    int MaxRow { get; }
    int MaxCol { get; }

    event EventHandler SelectionChanged;
}

I'm still not sure whether this interface will remain the same forever, I might change it to act more like Excel (i.e. removing the distinction of SelectionType and just making the different selection rendering be handled by comparing FirstRow/FirstCol with 0, and LastRow/LastCol with MaxRow/MaxCol). The interface is pretty self-explanatory, except for AnchorXXX and ActiveXXX. ActiveXXX is used to track where the keyboard has you on a keyboard-based selection extension (i.e. you hold shift and move around with the keyboard). AnchorXXX is used to track the starting cell for the selection. When moving away from a selection (without the shift key held), AnchorXXX is the position from which you start. This is counterintuitive to me, but it's how Excel works so I've replicated it.

Notice that ISelectionInfo is just the 'keeper' of the selection, and provides some methods for modifying the selection, but it doesn't have anything to do with the keyboard or the mouse. The support for modifying the selection via keyboard or mouse is isolated in the KeyboardManager and MouseManager classes, discussed in a later post from this series.

Render Info

Within each of the regions, we need to be able to obtain and change the cell text, get the render flags (i.e. is it selected, is it a special cell, etc.), get the text alignment, and some other special info for the grid region. The interfaces involved are IRenderInfo and IButtonInfo. The applicable definitions are as follows.

[Flags]
public enum CellRenderFlags
{
    None = 0,
    TopLeft = 1,
    ColHeader = 2,
    RowHeader = 4,
    FixedCol = 8,
    FixedRow = 16,
    Active = 32,
    Selected = 64,
    Hover = 128,
    Anchor = 256,
}

static class CellRenderFlagsExtensions
{
    public static bool Contains(this CellRenderFlags target, CellRenderFlags flag)
    {
        return (target & flag) == flag;
    }
}


public interface IRegionRenderInfo
{
    CellRenderFlags GetCellFlags(int row, int col);
    string GetCellText(int row, int col);
    void SetCellText(int row, int col, string text);
    TextAlignment GetTextAlignment(int row, int col);
   
    Orientation ButtonOrientation { get; }
    IButtonInfo GetButtonInfo(int index);
    int ButtonCount { get; }

    event EventHandler ResetRenderInfo;
}

public interface IButtonInfo
{
    ... <removed for brevity> ...
}

We will ignore IButtonInfo and the associated bits from the IRenderInfo interface for now and focus on the other IRenderInfo information. The cell render flags are various pieces of info that allow the CellRenderer class (discussed in a later post) to do its work. The CellRenderer also needs to know the text alignment for the cell and the cell text itself. The SetCellText method is provided for the in-place editor that allows modification of the cell text.

This concludes our discussion of the data binding interfaces used by the grid.

MSDN Articles Online (.chm)

2008-06-12T11:24:00.001-07:00

Glenn Block (@gblock on Twitter) just posted an tweet that mentioned the existence of these. I had no idea they were there and I'm sure others didn't as well - so here you go: http://is.gd/vZE

Enjoy!

Extension Methods are way cool!

2008-06-08T00:16:00.001-07:00

OK. So everyone's heard of LINQ by now. Most everyone has even heard of some of the cool features of C# 3.0 (lambdas), but in my mind, the coolest - extension methods - largely goes unnoticed. Extension methods are the plumbing on which LINQ and some of the other cool features in the C# 3.0 libraries are implemented. They are, in my opinion, the best feature C# has introduced since Generics, and are possibly one of the best features added to traditional languages EVER!

Consider the following - you have a class that someone else wrote. On their class, they've provided a public interface for doing all of the things you need, but there are several additional things that you've implemented (as a separate utility set of functions) that it would be nice to add to the class' public interface. Unfortunately, the class is marked 'sealed', or it is the base of a large hierarchy of classes that you simply can't add your functionality to (since you can't cause classes in a vendors library to derive from your 'new' version of their base class).

Extension methods to the rescue - all you need to do is declare a static class in your library (which you probably already have called 'StringUtils' or something like that :)), and provide some static methods on it that use the new 'this' keyword on their first argument. Magically, the compiler will then 'add' this method to all items that have a type that is compatible with the type you have in the 'this-marked' argument.

For example:

public static class StringUtils
{
    public static string RemoveAll(this string s, params string[] args)
    {
        string ret = s;
        foreach(string sremove in args)
            ret = ret.Replace(sremove, string.Empty);
    }
}

by the way, of course I know this is the most horrible way to implement this function - it's just an example so don't tell me how crappy my code is or that I should be using StringBuilder or yada yada yada...!

The point of this is that after declaring such a function, all objects of type 'string' syntactically receive a member called 'RemoveAll' that has a single 'params' argument. This is VERY cool.

The coolest thing about this - you can also do it for interfaces, enums, and various other types that you can't possibly provide "code" for in a more traditional way.

What else?

Much of the code that I write on a day-to-day basis works with tree-based data structures. Some of these structures can get very complicated and much of my unit test code needs to do asserts over a large part of a tree (after performing some complex operation). As a for instance, consider an expression parser. Such a parser would presumably build an AST for the expression it's given and return that AST for further processing. ASTs for all but the most simple expressions can get very tedious to 'check' for validity when writing a parser.

I've recently begun using extension methods for by base node class to help with my unit testing. I directly put the unit test 'asserts' into the extension methods, and these extension methods are in NO WAY suitable to exist in the library that is being tested (why on earth would I want to have all this extra junk in my library just to support unit tests). As a matter of fact, my libraries even target .NET 3.0 (C# 2.0) rather than .NET 3.5, C# 3.0. However, that doesn't stop me from being able to use extension methods in my unit testing code (which doesn't get deployed to my clients, so I don't require them to have 3.5, I just have to have it on my dev machine and build machine).

Here's a simple example of how some of my unit testing code looks:

[Test]
public void FormulaTests2()
{
    PrimaryLexer l = new PrimaryLexer();
    StringReaderAdapter sra = new StringReaderAdapter("a / (b + c)", 0);
    InforceScriptLexerFilter lf = new InforceScriptLexerFilter(sra, l);
    InforceScriptSemanticParser sp = new InforceScriptSemanticParser(lf);

    RootFormula rf = sp.Parse();

    // look for 'a' and '/'
    rf.Body.Is<binaryop>()
        .OperatorIs(InforceScriptTokenId.Slash)
            .Left.Is<invokeop>()
                .IdRef.Is<idreference>()
                    .NameIs("a");
    // look for 'b' and '+'
    rf.Body.Is<binaryop>()
        .Right.Is<binaryop>()
            .OperatorIs(InforceScriptTokenId.Plus)
                .Left.Is<invokeop>()
                    .IdRef.Is<idreference>()
                        .NameIs("b");
    // look for 'c'
    rf.Body.Is<binaryop>()
        .Right.Is<binaryop>()
            .Right.Is<invokeop>()
                .IdRef.Is<idreference>()
                    .NameIs("c");
}

In order to make all this possible, I defined a few extension methods:

internal static class TreeAssertions
{
    public static T Is<T>(this ExpressionBase node) where T : ExpressionBase
    {
        Assert.IsInstanceOfType(typeof(T), node, "wrong node type");
        return (T)node;
    }

    public static IdReference NameIs(this IdReference node, string name)
    {
        Assert.AreEqual(name, node.Id.Name, "names don't match");
        return node;
    }

    public static BinaryOp OperatorIs(this BinaryOp node, InforceScriptTokenId op)
    {
        Assert.AreEqual(op, node.OperatorTokenId);
        return node;
    }
}

As you can see, the 'Is' test checks the type of a node, and then returns the node, so I can continue checking other things for the same node (assuming it 'passed' the check). The same is true for the 'NameIs' and 'OperatorIs' checks. This sort of programming is generally referred to (I think) as 'Literate Programming' - a technique for which the venerable D. Knuth is given the credit. However, in order to do this sort of thing in the past, I'd have needed to put all these methods on my base class for the tree nodes, something that would have absolutely been the 'wrong' thing to do (since this test code should not be part of the library-proper). (By the way, I think this style is now being referred to as 'fluent interfaces' in programming circles).

I can't wait to see what else I can find to use these methods for. I've already found it to be an amazing benefit to my productivity and the readability of my tests.

CI / TeamCity is Seriously COOL!

2008-06-05T21:00:00.001-07:00

OK... I'll be the first to admit that I'm just getting into Agile processes and I'm still a bit skeptical. At first, I thought (CI = Continuous Integration) 'CI builds - is it really worthwhile?'. Now, I've got a TeamCity site & build agent up and going, and I'm totally SOLD!

Here are the benefits as I see them for our situation:

We know almost immediately when someone broke the build (they know too!)
We have better checkin quality now that people are tired of getting those 'compilation failed' emails.
We always have a source to go to for a 'current' build - no need to get the sources and build on your own machine, or go ask a 'build master' to get you a build.
We have other 'automation' points that we can hook into when we're ready to move on to bigger & better methods.

As an example of #4, I hope to soon have our NUnit tests running as part of an automated build. I also think we can have automated installer builds going if we wanted to. And, best of all, by virtue of TeamCity's ability to 'watch' our source control server for updates, and it's ability to run any arbitrary command line, NAnt, or MSBuild (or many more) task in response to those updates, the sky is the limit!

I can't wait to get more 'good stuff' implemented on TeamCity.

The Day of Bugs

2008-06-04T00:49:00.001-07:00

Ok, I can honestly say that today was one of the weirder days that I've had in a long while. I don't know about others, but I can say with confidence that I've never personally identified a bug in Visual Studio in my career. I've seen plenty of them mentioned by other folks, I've seen 'features' that I'd be inclined to call a bug (but could be interpreted either way), but I've never really found a bug myself.

Today, I found two. I guess it's a case of 'when it rains, it pours'. One of them was known long before I 'found' it, but obviously not known to me. The other, I'm pretty confident, is still unknown to 'everyone'.

Bug #1 - Dynamic Version vs. BAML.

Ok... So we've done a fair bit of playing with WPF on my project, and we've done some custom control development (user controls) in WPF for use in our application. WPF is very convenient for being able to prototype and design the UX/UI of something without being bogged down by all the crap you have to do to customize WinForms (our users seem to never like the 'way it is'). I can say that I feel pretty comfortable that my skills with WPF, while not the best on the block, are probably up there along with most of the folks currently doing WPF development. I've done a ton of data binding work, and feel pretty confident that I know most of the tricks there - especially thanks to the wonderful work of Bea Costa!

So, I was very puzzled when one of our developers started having issues with running our application's forms that use one of my user controls. The user control was pretty simple - it was a list of names that had alternating highlights for the rows (one row was white, one was gray, etc.). It did a few other things, but mostly that was the gist of it. This is one of the simpler controls we have. Anyway, the weird thing was that the 'bug' that we kept seeing only appeared when running the debug build of our application, and it only appeared when our UI was being used from the APL application. It never appeared in release builds, and it never appeared when running debug mode in our UI test harness.

So, naturally, I looked first at the APL runtime, thinking it was a bad install on this dev's machine. We then took his build and his APL workspace and ran it on my machine. To my surprise, it crashed on my machine too. Then, we tried running one of my builds on his machine - it worked (also to my surprise!). So then, I concluded it was a problem with his machine.

Two days later, after he had gotten some other work done and managed to uninstall all of .NET 2.0 through 3.5, VS2005 and VS2008, and then reinstall all of them (carefully in order), he tried it again. BOOM! It still didn't work. I brought him my old laptop, and I had IT set it up for him to be able to use it instead of his desktop, thinking we'd be rebuilding his desktop from scratch. All the while, still being puzzled by the fact that the behavior ran around to different machines and environments and was so skittish.

Later that day, he came over to my desk and told me the problem started appearing on his release builds too. I thought, "oh great - a viral bug!". He then said that the problem also started appearing on my builds. At this point, I thought - "ok, there's gotta be something else going on here".

The bad part about this bug was this - whenever you ran the application, it would look like it wanted to pop up an error dialog, in fact it would show the thread exception dialog (System.Windows.Forms.ThreadExceptionDialog) briefly (actually several of them on top of each other), but then the application would disappear before you could do anything. Apparently, looking back, the problem was on one of WPF's "special" threads and APL apparently doesn't react very nicely to the .NET AppDomain having threads other than the main UI thread throw exceptions.

Finally recognizing that I might be able to do something about this, I went into the code and added an exception handler with a plain-old message box in it (e.ToString()). Looking at the exception text, I saw that it said something about a XAML parse error and that my ValueConverter couldn't be loaded (I had a ValueConverter as a static resource in my XAML for getting the backcolor brush for doing the highlighting). This error message pointed me to: Rob Relyea's blog post (along with several MSDN forums posts). The only thing was that his post didn't apply completely to my issue. But, the workarounds did. It turned out that I was using AssemblyVersion(1.0.*) in my files (which I really like for our 'in dev' work), but it was causing problems. It seems that the reason the bug was so 'fleeting' was that there must have been a timing issue on the fourth bit of the version number (the revision), since it's based on a timestamp.

Apparently, my computer is too fast (most of the time), so I didn't see this bug on my builds, just on my colleagues'! As I said, this bug has been known about for a long time, and while I'm not thrilled with the workaround, it's there and working, so I'll live with it.

BTW, I spent nearly 4 days chasing this bug, off and on (my colleague did most of the legwork). It wasn't much fun, being that we couldn't get an error message for the first 3 of those.

Bug #2 - VS2008 STL vs. NUnit, C++/CLI, and ME!

So, I spent the last 4 days chasing what I though must have been a bug in our code, only to figure out that I'm pretty sure it's a bug in MS's STL implementation (for Debug builds). However, this bug is really hard to find (though it, at least, is VERY consistent - i.e. happens every time and is very reproducible).

First, some background on our design. Our application consists of two major pieces, a UI, and a backend calculation engine. The UI is written in .NET 3.0/3.5 (C#), and the backend is written in C++ (native). However, these systems need to be able to share a common file format. To meet this need, we developed a generic file library in native C++ code that can be used to read/write files for our system. The files are basically like MS's structured storage, but with the features and interfaces that we desired for our system (along with a design oriented towards meeting our required performance characteristics).

All of our file structures are build upon this file library, along with several other native libraries that support it and some of the other 'shared' features. We also have a generic 'key' library (this is a domain concept for us - you can think of it as a property bag with some special 'matching' features).

In order to support using these libraries from both C++ and C# code, we decided we'd write the implementations in native C++ using traditional object-oriented design principles (many of which were lifted from my C#/Java experiences), and then write a thin C++/CLI (the managed C++ language) wrapper around this library using the IJW (it just works) interop supported by C++/CLI. Then, our C# clients would call into the C++/CLI managed library (not even knowing that it's implemented in C++) just as they would call into managed C# code, but be able to use the underlying data structures and implementation of the native libraries. I think this design is pretty elegant, and we solved quite a few interesting issues when developing it. We've been using it now for some time and it's working quite well.

So... now, the bug. Just recently we needed to add a new feature to the 'key' library. This library is very simple and the feature was also quite simple. I added it to the native code, added it in the managed C++/CLI library, and added the C# unit test code to exercise it. By convention, we only run our unit tests in Release mode, unless we're debugging them, since we only want to take the time to test in one build environment and it makes most sense to test the bits that are going out the door...

Anyway, so I tested in release mode, the tests all passed, and I checked in. I was happy and I went home for the day. The next day, I happened to be compiling in Debug mode and I decided to run what I was working on. I had forgotten that my startup project in VS2008 was set to the unit tests for the 'key' library, so the unit tests ran instead of what I was intending to test. To my surprise, the unit tests for the 'key' library 'exploded' (they didn't fail, they caused an AccessViolationException that was caught by VS2008 and popped up in the debugger!). The AV was showing up on the destructor call for one of our unmanaged C++ objects (native library).

To make matters worse, the bug only showed up when the finalizer was called for the class, and even weirder, only showed up when the variables were not deterministically destroyed (i.e. using IDispose and 'using'). Since my unit test code wasn't using 'using' anywhere, I saw the 'explosion'. But, I saw it only when the finalizer was called (much later than the offending code, obviously). I invested some time writing code to track which object was the culprit, and after figuring out which (using 'value numbering', a technique I use a lot in single-threaded debugging of applications with lots of object instances and no unique identifiers in them), I followed the code. It was not at all obvious why there was a problem. In fact, I looked at it for several days and couldn't figure out what the problem was.

I then posted here, and called my Arch. Evangelist at MS and talked with him about it, and still couldn't figure out what it is (the spoiler is the last post in the thread). I finally had to resort to the 'commenting' technique to track down the bug. I first commented out all unit tests and started adding them back in one by one. Once I found which unit test failed, I commented out the entire test and started adding code back in block by block. Once I found the offending block, I looked down into the C++ code and STILL couldn't find anything wrong with it.

At that point, I decided I needed to think outside the box. I looked at what was different between the various method calls that worked and didn't work, and decided that the throwing of the exception might be a problem. First, I removed all the code in the offending method. Now, my test failed, but it didn't explode. So, I put the code back, and looked at the exception more thoroughly - I decided to move it to the head of the function. That also caused my test to fail, but it didn't explode. So then I concentrated on the code before the exception (in the original function). I kept saying to myself - there's NOTHING wrong with this code! (you can see the code in the MSDN post). Finally, I thought - "it has to be this code, so let's assume it's broken and figure out when and why". I then restructured the code (as described in the MSDN post) and determined that it was absolutely something in the 'begin()' or 'end()' STL vector calls. There's no way I messed that up - that's their code. Voilà - bug #2.

For this one, I'm going to have to figure out how to submit it to MS. I'm sure nobody's seen this one (at least as far as I can tell from searching).

Whew! Looking forward (hopefully) to not having very many more of those days!

Redistributing C++ runtime components

2008-05-31T00:11:00.001-07:00

Ok. I've had to look for this several times over the years. This time, I finally decided to just post it on my blog so I can find it again later :)

Here it is

Hopefully it helps you sometime too...

StL gave Oakwood 1/2 Million Dollars???

2008-05-29T20:30:00.000-07:00

What the hell?

Article in recent STL Business Journal:

"St. Louis development fund invests in Oakwood Systems.Oakwood Systems Group Inc. received a $500,000 investment for expansion from the St. L Business Development Fund, which invests in St. L based companies that are unable to access sufficient senior debt to finance organic growth or acquisition.Creve Coeur MO based technology consulting firm Oakwood Systems is one of the fastest growing private companies in the St. Louis area, according to Business Journal research. The company reported $7.5 million in fiscal 2006 revenue, a 92.3 percent increase from fiscal 2004 revenue of $3.9 million. The firm has 85 employees and an office in Nashville, TN."

If this isn't the funniest thing I've heard in a long time - it's gotta be up there. Of course this will only be funny to those of you who have heard my stories about my time at Oakwood and the goings on there.

Patching from a single .exe

2008-05-26T23:25:00.000-07:00

We've been doing some work lately to try to figure out a good way to make our software patches go out the door as a single .exe. We aren't using MSI for our patches because we need to be able to do some things that MSI can't do (i.e. install older patches over newer versions, verify the current system version before installing, always install files, regardless of whether they are have been modified by the user or not, etc.).

So, I was thinking about different approaches. As I saw it, there were:

Package everything in a zip, require users to download the zip and extract it into a temp folder, and run a .exe inside the zip that does the patch. Then require the user to delete the folder on their own.
Build an .EXE that has resources in it for the files that were to be included in the patch - have the .exe extract these resources to disk in the proper locations and perform the other logic necessary.
Build a generic packaging process, package stub, and 'package runner' that can solve the problem as well as other similar problems should we ever want to package anything else as a single .EXE (besides patches).

I didn't like (1) for the UX - I think it's pretty nasty UX-wise. I didn't like (2) because there are several problems - (a) the person who would be building the patches doesn't really work in VS, and I don't want them compiling the code. I also don't want to write code to 'package' the resources - it's a pain. (b) I already know one other case where we will need to package stuff into a single .exe for deployment that isn't the same as our patch process, so I'd have to build a similar project again - I hate duplicate code...

I decided to go down the path of (3). I knew from my Win32 days that the .EXE loader will be perfectly fine with an .EXE with 'junk' concatenated to the end of it. I've done this several times in the past - the PE format doesn't have any problem with 'extra' bits at the end of the file. So, I decided to go with the idea of a stub .EXE with a 'package' tacked on the end.

The next question was - what should my package support? I decided that for me a package would be two things - a list of files, and an entry assembly (that gets run by the package stub upon loading). The package would be a 'manifest' (an XML file that lists the package contents), along with a stream of bytes for the package data. For my needs, it was sufficient if all packages contained (1) a list of files, (2) an entry assembly, (3) an argument to pass to the entry assembly on running it, and (4) a list of assemblies upon which the entry assembly depended. The entry assembly would be loaded and executed by the package stub (.EXE) upon running the .EXE.

So, the file looks like:

Stub .EXE

Package Data Bits

Package Manifest

Package Trailer

Where the data bits are GZip compressed streams of bits corresponding to the items listed in the manifest, and the Package Trailer is a 'header' (trailer actually) that is of fixed length and identifies this file as a valid package (i.e. has a identifying 'magic number') and contains offsets of the various items within the file (i.e. the location of the first byte of data bits, and the length of the manifest).

Since my support code for this functionality is in a class library (I called it 'Packaging'), the class library is embedded in the stub .exe as a resource. The stub performs the following steps when it is run:

load the resource "Packaging" and call Assembly.Load on the byte array.
set up a 'resolver' to resolve this assembly for the other assemblies that might be loaded by this package.
unpackage the dependencies of the entry assembly and load them as well
unpackage the entry assembly
look for a 'Package' class in the global namespace of the entry assembly, and cast it to an IPackage (defined in Packaging assembly)
call 'Execute' method on the IPackage from (5), passing an IPackageHost interface that provides functionality for reading the contents of the package and interacting with the stub.

(2) is an event handler attached to the AppDomain.CurrentDomain.AssemblyResolve event.

The IPackageHost is an interface that allows the IPackage.Execute implementation to gain access to the entries (and bits) in the manifest. This host interface is provided by the stub, but can also be implemented by other code in order to do testing, extract packages, run them separately, etc. In fact, my Packaging library provides the implementation of IPackageHost via a PackageHost class that is used by the stub .EXE to provide the necessary support to the entry assembly's IPackage implementation.

If you want more details on how this all works, give me a call, or shoot me an email - I'll be happy to provide more details.

Another great ALT.NET event!

2008-05-24T23:34:00.001-07:00

Yet another great ALT.NET event today. Thanks Glenn for the conversation in the car on the way there (and congrats / good luck on your new job on MEF), thanks Justin for the space, and thanks everyone else for the great conversations.

Hopefully people are interested in coming to my place in July or early Aug for one.

I'll try to be more diligent about blogging some of my recent dev work. I think it's pretty interesting and I haven't found similar content elsewhere.

Coolest weekend ever.

2008-04-21T09:51:00.000-07:00

Ok... the weather sucked - it was snowing here - but otherwise this weekend was awesome. I got to meet a lot of great people, and some real top minds in the field. Who did I meet? Among others - Brad Abrams (.NETfx team, coauthor of framework guidelines book), Jeremy Miller (the StructureMap guy), Glenn Block (P&P guy), Scott Hanselman (goes without introduction), Martin Fowler (the 'Refactoring' book guy), Charlie Poole (the NUnit guy) ... the list goes on (no offense please if I met you and didn't include you in this list).

I had a wonderful lunch conversation with Brad for about 20-30 minutes over lunch. It's amazing to have someone like Brad come sit down next to you, introduce himself, and then share a one-on-one conversation with you. It'd be like Fisher Black coming and sitting down next to an actuary and introducing himself and asking about your work. It's very hard to not feel intimidated in such a situation, but not only was Brad not intimidating, but he's a really genuinely nice guy to talk to. Hopefully I'll have the luck of meeting him again and having more conversations. hehehe... maybe he'll ask me if I want a job someday :) I can dream, right?

All in all, this is the best work-related event I've ever been to, and it's probably the best weekend of my working life thus far. I definitely consider the 30 minutes I spend with Brad to be the best 30 minutes of my career to this point.

If you ever hear of an 'open spaces' event, I highly recommend you jump on the opportunity to attend. Also, if you ever want to go to ALT.NET events, hopefully I'll see you there.

SSCLI code browser

2008-03-20T16:54:00.000-07:00

Ok... This is pretty cool!

http://labs.developerfusion.co.uk/SourceViewer/view/SSCLI/

Interop, Interop, and more Interop

2008-02-25T10:54:00.000-08:00

Well, we're sure doing our fair share of WinForms and WPF interop! We've really jumped on the WPF bandwagon, but we really like the functionality of our application and aren't willing to give up major parts of it in order to go 'all' WPF.

For instance, our Syntax Editor control (from http://www.actiprosoftware.com) is only available currently as a WinForms control. Also, my management really likes the Infragistics Grid's functionality that we're using for our selection grid in our formula browser tab. In addition, we really like our tabbed MDI style of windowing, and our tool windows (pinnable tool windows like in VS2005).

Unfortunately, there are several ways in which WinForms doesn't play nice with WPF. Basically, the problem is that when WinForms draws on a window, it 'owns' that space on the screen. However, when WPF draws on a window, it is using DirectX compositing. This makes lots of things in WPF very nice, but in order to allow WinForms controls to 'live' on a WPF window, they must have total control over the space where they 'live'. This means that the WPF controls can not render 'on top of' the WinForms control. This is very well documented behavior and basically the only way the WPF team could figure out to make this work (as far as I can tell).

What does it mean for us? A whole lot of interop. Basically, we want to use WPF everywhere we can and would love to be able to use it for our whole application. Currently, we can't use it for the whole app, and so we must use it where we can and hope for the best later. However, tool windows have this neat 'auto-hide' feature with them. We really liked the WPF 'SandDock' tab & dock controls from DivElements. However, since our WinForms grid is in the main client area of our form layout, we couldn't use a WPF docking control, since when you drug the dockable windows around, or when they tried to 'flyout' from their 'auto-hide' position, they would do so UNDER the WinForms control! That sucks, but it's the only way it can reasonably work.

What's that mean? Our docking controls can only be WinForms controls until we get rid of all the WinForms controls that a flyout would want to fly over (and replace them with WPF equivalents). Now, since our docking and tabs controls (they're the same control for us, since we've switched to Actipro's UIStudio) have to be hosted in WinForms, and you can't have a WinForms host inside a WPF host inside a WinForms host (the interop facilities of WPF only allow WF inside WPF or WPF inside WF, but not 'double' nesting), we MUST use WinForms for our main application (this sucks since it means we lose a bunch of the nice features of WPF, like commands, command bindings, routed events, etc.).

This also means that EVERY WPF control in our app must be hosted inside a WPF element host (interop) control on the WinForms form.

Yuck. That's a hell of a lot of interop.

Portrait Mode

2008-02-20T21:47:00.001-08:00

So, have you guys heard of "Portrait mode" for monitors? If not, you should really check these out - here's an example of what it is:

http://reviews.cnet.com/4520-7610_7-5084364-3.html

It turns out that portrait mode is actually a feature of your video card, not your monitor. What makes it a "feature" of the monitor is simply that a monitor that supports portrait mode must have a stand that allows you to rotate the monitor.

So... it turns out that the Dell 20" Widescreen LCD panel that I've had on my desk in the Milliman office for 6 months now is actually a 'portrait-capable' monitor. Of course I had to hire a new employee in order to figure that out :)

Of course, after setting up my monitor at the office to do portrait mode, I decided I needed to check to see if my Samsung SyncMaster 225BW supported it (my home monitor). After doing a web search, it seemed that it didn't, so I did some investigating of my own. This monitor is absolutely awesome, and this is the very first design 'flaw' I've seen with it so far. However, it turns to be a weirdly long-sighted and simultaneously short-sighted design!

The stand on the 225BW is removable, and has a square mount. Inside the back of the monitor, the metal substructure has four mounting studs. Woohoo! I say, it seems like I can rotate this thing after all. Unfortunately, there are two mounting ears on the stand that fit into some holes in the plastic case in order to ensure that the monitor doesn't pull away from the stand when tilting it (it only mounts with two screws - the top ones).

So, long story short, I pulled out some cutting tools and cut myself a couple of holes in the plastic case (it's backed by the metal substructure, which I verified before cutting so that I wouldn't damage any components). Voila! - I now have a portrait mode monitor! I can fit 351 code lines in visual studio on one screen given my current toolbar settings (and the error list/bottom windows unpinned) - WAY COOL! I can taste the productivity already!

Collections library for .NET

2008-02-11T14:29:00.001-08:00

Here's a neat collections library somebody just mentioned in one of the newsgroups I frequent. I haven't had a chance to check it out much yet, but it seems very promising.

http://www.itu.dk/research/c5/

Interesting Blog Post - SKU driven development

2008-02-09T22:56:00.000-08:00

Jeff has a very good post here. Mostly, I like the bulleted list in the post, I think it's one of the best lists of 'must read topics' I've ever seen in one place.

http://www.moserware.com/2008/02/sku-driven-development.html

BNF in code?

2008-02-08T09:32:00.000-08:00

Ok... this is pretty cool.

http://www.codeplex.com/irony

I was very surprised this works. I wonder how expressive you can really be with this, but it's an interesting idea.

NETfx source code...

2008-02-06T15:53:00.001-08:00

Get it while it lasts!

http://www.sellsbrothers.com/news/showTopic.aspx?ixTopic=2169

Chris blogs about a tool that will let you get the whole distro in one fell swoop!

VS2008 and .NETfx source code!!!

2008-02-02T21:22:00.000-08:00

Check out this post! I haven't tried it yet, but I'm really hoping it works.

http://blogs.msdn.com/sburke/archive/2008/01/16/configuring-visual-studio-to-debug-net-framework-source-code.aspx

Job Opening

2008-01-31T21:57:00.000-08:00

We have a job opening for a C# developer on our MG-ALFA development team. If you're interested, contact me soon.

Executing code in another app domain.

2008-01-31T20:05:00.000-08:00

Recently, we found out that the Infragistics controls we're using have some bugs in them that cause them to never be garbage collected once you've used them in your code. There doesn't seem to be any resolution short of asking them to fix the bugs, as the bugs are related to their code adding event handlers for receiving notification of changes in the office 2007 themes container (their internal class).

Anyway, we figured out that there isn't any workaround for this short of modifying and recompiling their code or waiting for them to fix their bugs. Since we don't have time for that, and it represents a HUGE memory leak in our application (each time our form is opened and closed, the UI sucks up 30-50MB of additional memory!)

So, our solution, you ask? We first built an out of process approach to executing our UI via IDispatch implemented in a out-of-process .EXE COM server. This was a fair amount of work and was a really cool project in and of itself. If you're interested in it, let me know and I'll tell you how it works. Unfortunately, due to changes in windows rules for window activation and some plumbing issues, it turns out that the out-of-process approach won't activate the window when it starts up, and short of doing some really nasty hack, there isn't an easy fix for that.

When we figured the out-of-process solution wasn't really the best answer for our problems, we decided to investigate further. Once we determined that the leak was on our .NET side of the fence (our .NET UI is called by a Dyalog APL-based user interface via their .NET interop support), we decided to investigate the .NET side a bit more. That's when we found that it was a leak caused by the Infragistics controls.

I did a proof of concept that created a separate AppDomain, called our NUI, and then shutdown the app domain. This seems to fix our memory leak problem as all memory associated with the app domain is freed (at least it seems to be). It turned out to be much harder to get this working than I had expected. In theory it's quite easy, but we have several deployment issues that make it very difficult (we can't put our DLLs in the GAC, and our application DLLs are not (and can't be) in the same folder as the 'application base' (because the base application that creates the main app domain is Dyalog.exe or DyalogRT.exe and we don't want this in our application folder).

The process goes something like this:
1) create an app domain
2) load our assembly into the domain
3) create a serializable object with our arguments for the call to the assembly
4) call to the assembly (serializing the arguments) (blocking call)
5) the assembly returns from the call, returning the results in a serializable object
6) the calling code deserializes the object and continues as it normally would.

The problems in this are: (2) finding the assembly; (3) making the serialized object's assembly available to the target assembly; (4) getting a reference to a MarshalByRefObject to make the call against; (5) loading the results objects assembly in the source app domain.

The main issues are loading the assemblies on the caller's side, rather than the target side, amazingly! It seems like this should be the easy part, but for some reason the remoting infrastructure isn't smart enough to realize that the assemblies are ALREADY LOADED!

Anyway, the basics are like this:

create an object that derives from MarshalByRefObject. Put this in an assembly that you don't mind having loaded in both app domains.
define a method on this object that does the work you want done.
make sure that the objects you pass to and from the method are marked Serializable (and can be serialized and deserialized - you can test this by using the BinaryFormatter to write and read the file to/from a stream).
create a method somewhere in your main app domain that does the following:
1. packages the parameters into the serializable objects
2. creates an app domain using AppDomain.CreateDomain. We also pass a different appbase that is based on our assemblies' locations.
3. sets up an event handler on the current app domain's AssemblyResolve event (this is only necessary if you can't get the app domain to properly resolve your assembly and instead your object comes back only as MarshalByRefObject, instead of what you expected).
4. call CreateInstanceAndUnwrap on the target app domain, asking it to create an instance of your MarshalByRefObject derived object.
5. cast the object from CreateInstanceAndUnwrap to your target object type.
6. Call your method.

Put code in the method handler (if you need it) for your AssemblyResolve event that iterates through the loaded assemblies and returns the one that was requested, if it matches by name. ResolveEventArgs.Name should be matched against Assembly.FullName. My code looks like:

static Assembly CurrentDomain_AssemblyResolve(object sender, ResolveEventArgs args)
{
    Debug.WriteLine("Attempting to resolve assembly: "
        + args.Name);
    // search the loaded assemblies for this one.
    foreach (Assembly assy
                in AppDomain.CurrentDomain.GetAssemblies())
        if (assy.FullName == args.Name)
            return assy;
    return Assembly.GetExecutingAssembly().FullName
        == args.Name
        ? Assembly.GetExecutingAssembly() : null;
}

Write code to unload the AppDomain (call AppDomain.Unload(...)) (I do this in a 'finally' that follows a 'try' started immediately after the app domain is created)

This seems to do the trick. Let me know if you have any problems.

Here's where I got some of my information:

http://blogs.msdn.com/suzcook/archive/2003/06/12/57169.aspx
http://blogs.msdn.com/suzcook/archive/2003/05/29/57120.aspx
http://www.codeguru.com/forum/showthread.php?t=398030
http://blogs.msdn.com/suzcook/archive/2003/05/29/57143.aspx
http://blogs.msdn.com/suzcook/archive/2003/06/13/57180.aspx

By the way, if you've never come across it, Suzanne's blog is a GREAT source of information.

WPF Data Binding in code

2008-01-24T21:25:00.000-08:00

Recently, I was working on a WPF prototype for our application. Unfortunately, we need some features in a grid control and some other controls that won't allow us to use the WPF grids available on the market. For this reason, we need to use the Infragistics grid (the same one we've used in the past), the WinForms version.

Unfortunately, we also need to have docking (VS 2005 style) and tabbed MDI. Since docking has flyout controls, and WPF z-order doesn't work right with WinForms, we need to host our application in Windows Forms framework.

When you have a windows forms control in a WPF layout, the windows forms control (in the WindowsFormsHost control of WPF) doesn't participate in the z-order the way you might expect. Unfortunately, windows HWNDs don't support layering like WPF (DirectX) does, instead each HWND owns its part of the screen (except for areas overlapped by overlapping windows). Transparency in HWNDs is done by 'hacks', not builtin support.
Due to this structure, winforms controls must be rendered 'on top' of WPF content, when you host winforms in WPF windows. This means my flyout windows will show up "under" the winforms window, rather than on top of it like you'd expect. Anyway, the only way to get around this is to host WPF controls in WinForms rather than the other way around.

One of the real benefits of using WPF is the XAML markup and the ability to do data binding so easily in your layouts. Unfortunately, at first glance, this seems to be lost when you switch to WPF controls on top of a WinForms layout. Don't fear, all is not lost.

Fortunately, even though you can't use XAML markup to define your data binding, you can programmatically add bindings to your winforms layout. For instance, we have a infragistics grid (a winforms control) that lists the 'items' in our browser that people can choose to look at. On the other hand, we have WPF controls that react to which item is selected in the browser grid. It'd be nice if we could bind the WPF controls to a property on the other control so that they are automatically updated when necessary.

The way I did this was to create a DependencyObject that contains my properties for the grid that I'd like to bind to - in particular one of them was CurrentItem. The properties are DependencyProperty-based. Now, I needed one-way binding to update my WPF controls when the current item changed in the grid, but just in case, I set up my code to support two way binding, so that if I chose to set the value of the property from other parts of my code, I could do so safely and have everything update properly.

So, now I have a DO with a DP. When I register the DP, I created a update method so that I could trigger code when the property was changed - this is done by adding a delegate to the UIPropertyMetadata constructor passed in the DependencyProperty.Register call. To expose this DO to my clients (I've called it BindingInterfaceObject), I make a property on my UserControl (that contains the Infragistics Grid) called BindingInterface and create an instance of the BindingInterfaceObject in the constructor of the UserControl. I give it access to the UserControl's private members by making it a nested class and passing the 'this' pointer to its constructor.

In order to bind to it, I need to set bindings on my WPF controls. I do this by creating a Binding object (System.Windows.Data.Binding) with the name of the property in BindingInterfaceObject that I care about (in my case CurrentItem) and pass this binding object to the SetBinding method on the WPF controls.

Voila! I now have WPF controls binding to changes in a non-WPF control. The only thing I have to do now is set the CurrentItem property whenever the appropriate event occurs on the grid. I do this by handling those events in my UserControl and setting the BindingInterface.CurrentItem as appropriate.

I can provide sample code if you're interested. Just ask.

Raymond Chen - "The Old New Thing" (best book ever)

2007-10-30T14:05:00.001-07:00

This is the most interesting non-technical (mostly) computer book I've ever seen! You owe it to yourself to check out Raymond Chen's blog (here) and buy his new book "The Old New Thing". It's great - I'm either laughing or totally nose down entrenched in reading this book - it's so interesting and at times funny too.

To save you having to look it up, it's a book on the historical development of windows and the 'why' in windows, rather than the 'how'. There are some really great insights here, especially about UI design, but there are also some amazing tidbits that explain questions that have been floating around in my head for some time (like what's the deal with GlobalAlloc, LocalAlloc, and all that crap!).

Anyway, check it out - you can buy it by going to his blog (here) and clicking the book image in the upper right corner.

Away for a while...

2007-10-19T22:48:00.000-07:00

So, I've been away for a while. Sorry about that. I've got a bunch of new topics I'd like to write about, so many I can't choose which to do first.

1) OpenXML - this is way cool!
2) More about attributes and their use in .NET - building a binary file system engine.
3) WiX
4) .NET 3.0 System.IO.Packaging and it's use for custom apps...
5) .NET 2.0, 3.0, 3.5 relationships and compatibility between them...
6) DLLs, dependencies, and best practices for COM objects that use non-COM dll's for inprocess use.
7) Benefits of MS Gold Partnership.
8) MG-ALFA planned improvements (at least the ones I can talk about)...

Not sure - any preferences - please email me...

Joel on Software

2007-03-20T17:43:00.000-07:00

Good book, been recommended to me several times in the past, but I finally bought it with the 20% discount here at SD West 2007's book store...

The details are "Joel on Software" by "Joel Spolsky"... Get a copy yourself. Also, check out Joel's web site at http://www.joelonsoftware.com

Playing with attributes

2007-01-26T08:50:00.000-08:00

I'm sure by now you've seen all sorts of .NET framework attributes that can be applied to various pieces of code. The most notable of these in my code are FlagsAttribute (for enums), and the XML serialization / Web Services ones (XmlElementAttribute, XmlAttributeAttribute, etc., and WebServiceAttribute). These are very nice and unleash all kinds of built-in behavior in the framework, but did you know you can make your own attributes and use them for your own purposes? That's what this blog entry is all about!

For me, one of the most common problems I face is one in which I would like to use an enumeration for a class member that has a list of choices. Of course, this is a no-brainer to use an enum for this, but often I want to go one step further. Often, I'd like to limit the enum's choices for the class member, based on the value of one or more of the other class members. This problem is very common in heirarchies where you have a shared base class that you want to support a member in that can have values that aren't always applicable to the object.

Let's say for the sake of argument that you have a class A with members x, y, and z. The z member should be a choice of z1, z2, or z3, but each of these choices is only available when the following conditions hold:

Condition		Availability
x > 0, y anything		z1 z2 available, z3 not
x = 0, y > 0		z1 available, z2, z3 not
x = 0, y < 0		z2 available, z1, z3 not
x < 0, y anything		z3 available, z1, z2 not

We can see that these rules can be distilled into a separate "test function" for each value of z. For z1, it's available if x>0 (x == 0 && y>0). For z2, it's available if x>0 (x == 0 && y<0). For z3, it's available if x<0 only.

Now, I crashed my compiler in the process of trying to use anonymous delegates to express these conditions. Unfortunately, that doesn't seem to work. However, we can package these three tests into a helper class that provides static methods to perform these tests. For instance, let's make a class ZValueTests as follows:

  public static class ZValueTests
  {
    public static bool TestForZ1(int x, int y)
    {
      return x>0  (x == 0 && y>0);
    }

    public static bool TestForZ2(int x, int y)
    {
      return x>0  (x == 0 && y<0);
    }

    public static bool TestForZ3(int x, int y)
    {
      return x<0;
    }
  }

Now that we have these tests, we can use them in an attribute that we apply to each of our enum values. Let's first define our attribute class. We'll call it ValidForAttribute, and it must derive from the Attribute class to be known by the compiler. We'll have to use reflection to call the test functions, so this will be a fun demo in many ways! Here's our attribute declaration.

  [AttributeUsage(AttributeTargets.Field)]
  public class ValidForAttribute : Attribute
  {
    private SR.MethodInfo _Test;

    public ValidForAttribute(
         Type TestClass,
         string MethodName)
    {
      SR.MethodInfo mi
        = TestClass.GetMethod(MethodName);
      // assertions omitted for brevity
      _Test = mi;
    }

    public bool Test(int x, int y)
    {
      return (bool)_Test.Invoke(
        null,
        new object[] { x, y });
    }
  }

As you can see from the declaration, we'll need to specify our attribute using the type of the class that implements our tests, and the name of the method on that class. This is because we can't use non-constant expressions in the attribute initializer (a pretty big limitation if you ask me), so we can't even use delegate creation expressions, much less anonymous methods! Anyway, this works, it's just a little less pleasant than I would have liked.

Now, we need to declare our actual Enum. You'll see it will be very easy and will look something like the following:

  public enum ZValue
  {
    [ValidFor(typeof(ZValueTests), "TestForZ1")]
    z1,
    [ValidFor(typeof(ZValueTests), "TestForZ2")]
    z2,
    [ValidFor(typeof(ZValueTests), "TestForZ3")]
    z3
  }

This is very pleasant as it very clearly defines the test functions that apply to each z value. Of course, one unpleasant thing is that the types of these functions cannot be checked until runtime (that's what the asserts that are omitted from the code in the attribute do). I would recommend even throwing an exception if the function isn't declared right from the constructor of the attribute, but that may be a bit harsh.

Now, I've shown how you can declare the enums, attribute class, and test class - but what we really want to see is the validation in action, right? So, here we go. If we declare the class A as follows:

public class A {
  ZValue _z;

  bool ValidateZ(int x, int y, ZValue z) {
    SR.FieldInfo fi = typeof(ZValue)
      .GetField(Enum.GetName(
        typeof(ZValue), z));
    // test all attrs (ValidForAttribute)
    if(fi != null)
      foreach(ValidForAttribute vfa
           in fi.GetCustomAttributes(
                 typeof(ValidForAttribute),
                 true))
        if (vfa.Test(x, y))
          return true;
    // ret false if enum not found,
    //   or no attrs found returning true
    return false;
  }

  public int x, y;
  public ZValue z
  {
    get { return _z; }
    set {
      if(ValidateZ(x, y, value))
        _z = value;
      else
        throw new Exception(
          string.Format(
            "Invalid z: ({0}, {1}, {2})",
            x, y, z));
    }
  }
}

then the code will validate a new value of z against x and y before setting the internal field (_z) to that value. An exception will be raised if the value is invalid. This may seem like a lot of work, instead of just validating z inside the z "set" accessor. That's true, it is. However, the real advantage to this is that the validation code can be used in other ways. For instance, consider if we wanted a list of valid values for z (for a UI, for instance) given a particular value of x and y. We can write the following code:

public static ZValue[] ValidZs(int x, int y) {
  List<ZValue> zv = new List<ZValue>(
    (IEnumerable<ZValue>)
      Enum.GetValues(typeof(ZValue)));
  zv.RemoveAll(new Predicate<ZValue>(
    delegate(ZValue v)
    {
      SR.FieldInfo fi = typeof(ZValue)
        .GetField(Enum.GetName(
                    typeof(ZValue),
                    v
      ));
      if (fi != null)
        foreach (ValidForAttribute vfa
             in fi.GetCustomAttributes(
                 typeof(ValidForAttribute),
                 true))
          if (vfa.Test(x, y))
            return false;
      return true;
    }
  ));
}

Hope you found this interesting! Email with questions, if you have them!

Anonymous Delegates in C#

2007-01-06T19:10:00.000-08:00

A friend asked me recently what features other than Generics I really liked in C# 2.0. I mentioned that the anonymous delegate is one of those features. So, you might ask, what is an anonymous delegate? First of all, for those of you who don't know what delegates are, they are the C# equivalent of a function pointer, sortof.

A delegate is a strongly typed method pointer, that stores both the method reference and the object reference on which the method is defined. Delegates are declared with the syntax:

[access] delegate [return type]
   [delegate name] ( [param signatures] )

where [access] is an access modifier (optional) - public, private, protected, or internal, return type is the return type of methods that will be pointed to by the delegate (or void), and the other items are relatively self-explanatory.

Now. That's how a delegate "type" is declared. However, anonymous delegates aren't really delegate types - they're delegate instances. Let's say we have a delegate called BinaryIntegerOperation declared as:

public delegate int
  BinaryIntegerOperation (int a, int b);

This declaration gives us a delegate type called BinaryIntegerOperation. If we want to create a delegate instance, we need to pass the name of a function with the appropriate signature to this delegate's constructor during a invocation of the new operator. Let's assume that we have a class declared as:

public class Adder {
  public int Add(int x1, int x2) {
    return x1 + x2;
  }
}

Let's also assume that we have some instance of the Adder class, referenced by a variable called myAdder, within the context of the code we care about. The code to construct a delegate instance of BinaryIntegerOperation using the Add method of our myAdder object instance would look like:

...
BinaryIntegerOperation op
  = new BinaryIntegerOperation(
    myAdder.Add);
...

and the code to invoke this delegate would look like:

int result = op(value1, value2);

(just like a normal method call).

Now, let's say that instead of declaring an Adder class, we just wanted to be able to pass some simple implementation of the BinaryIntegerOperation delegate's method signature to the delegate's constructor so that we could use it in a context that needs a BinaryIntegerOperation instance. We can do the following, instead of declaring a Adder object altogether:

BinaryIntegerOperation op
  = new BinaryIntegerOperation(
    delegate (int a, int b) {
      return a + b;
    });

As you can see from this code, the code for the delegate instance is actually placed inline using the delegate keyword, along with the parameter signatures of the method. C# will automatically create a class, instance, and method for us, so we don't have to do so ourselves.

You might say, what is the advantage of this? I use this feature very extensively, often in code dealing with list filtering or searching.

Let's say you have a list of integers, and you want to determine if any of them is larger than a particular value (say 10).

Consider the code (list is List<int>)

bool bigger = false;
foreach(int item in list) {
  if(item > 10) {
    bigger = true;
    break;
  }
}

you could instead write it as:

bool bigger =
  -1 != list.FindIndex(
     new Predicate<int>(
       delegate (int x) {
         return x > 10;
       }));

where, again, list is List<int>.

The ClickOnce reference (on MSDN)

2007-01-05T19:32:00.000-08:00

Just in case you want to learn a bit about ClickOnce yourself, from the source, you can do so on the MSDN site here: http://msdn2.microsoft.com/en-us/library/6ae39a7c(VS.80).aspx

Clickonce Deployment Tidbits

2007-01-05T15:03:00.000-08:00

Ok... First, I'm going to talk about CD-ROM based initial installations, 'cause that's what I know best. The scenario where this install type works best is as follows:

Application prerequisites are quite large (SQL Server, stuff like that...)
Application updates should be delivered via the web
Application should be available offline (see the "Offline as well" option in Install Mode and Settings under Publish properties of your VS2005 project).
Application may have custom prerequisites (another article in itself, coming soon).

We typically configure our applications so that they don't have a deployment web page, but either way is fine (this option is in the Options... dialog). For the purposes of discussion, I'm going to assume the program name is MyApp and that the only files deployed with the application will be the app.config and the app exe.

To set up the application to publish to CD-ROM or DVD-ROM, you should choose a publish location that is a file path on your local machine. We typically use c:\deployments\... or something like that. For the purposes of discussion, I'm using c:\deploy\MyApp\v1.0.0.0 as my first deployment location. You'll want to have a version number in your deployment location so that you don't overwrite older deployments when creating new ones.

So, for version 1.0 of our application, we'll set the deployment location to c:\deploy\MyApp\v1.0.0.0. It's tempting to put something in the "Installation URL" but we won't for building the initial CDs to deploy from. It's also tempting to let visual studio automatically increment the revision with each publish. However, we prefer to have total control over the version number used for publish and for the assembly version (stored in the AssemblyInfo.cs file). There are actually a ton of different version numbers floating around in .NET applications. In our simple case here, there's the Deployment version, the Assembly version, and the Assembly file version. We like to keep all of them in sync. Things can get very confusing otherwise. I'm not going to talk much about the significance of each different version number here, unless requested to do so, so make a request if you want more details about them.

So, on to the publish. We're looking to publish our application to a CD, but also to be able to update it from the web. Let's say you have a web server available (we'll use localhost for these examples). If you create a site (or virtual directory) called MyApp on your IIS install, it will be easier to follow my examples. I'm going to use the path http://localhost/MyApp/updates as my updates deployment path. So, let's click on the Updates... button and see what we see.

In the updates dialog, we obviously want "Application should check for updates" checked, since that's the whole point of this discussion. I would choose "Before the application starts" for the when, since the other alternative is very strange. For now, leave the "minimum required version" blank. For the update location, let's put our http://localhost/MyApp/updates path.

Close the dialog and click publish now. After a few moments, you should see an explorer window pop up with the deployment files in it. Let's look at those files for a moment. There should be a setup.exe - it's the setup bootstrapper that knows how to install your prerequisites and get the ball rolling on the installation. There's also a couple of .application files. The unversioned one and the versioned one are identical (in contents).

These files can be copied to a CD-ROM or DVD-ROM and installed from there, by clicking the Setup.exe. However, we also need to set up the path for updates too. Let's talk about how to deploy the updates to a web server. Whereever you placed your virtual folder (physically) on your web server, you need to copy the .application files and the MyApp_1_0_0_0 folder to that location. In my case, http://localhost/MyApp refers to a "C:\MyApp Deployment" folder on my local hard drive. Since I want the files to be in the "./updates" path relative to this virtual folder, I need a "updates" folder in this folder. I then need to copy those files to the "updates" folder. I should make sure the permissions on those files and folders are appropriate for the users I wish to be able to access this application (normally anonymous access is needed if you want to deploy the application to anyone that wants it).

Once you've copied those files and set up those folders, you should be good to go for your first CD-based ClickOnce deployment. Run the setup.exe from the CD to install the application. If you run Setup.exe the app should install (after warning you about security) and then immediately run itself. You'll see that the app should also appear in your start menu and in the add/remove programs control panel applet.

Now, let's update the application and see a new version deployed. In my case, the original application didn't do anything and was simply a blank form (from the new application wizard). To make version 1.1, I'm going to add a label to the form with some text so I can tell I've got the new version. Once you've done that, deploy again, but to a v1.1.0.0 folder, and change the deployment version (and assembly versions in AssemblyInfo.cs) first. Do this by going to the publish properties (in the project properties) and setting the major and minor publish versions as appropriate. Now publish to the new folder. Once the publish is done, you should see the explorer window pop up. Copy the .application files and the MyApp_1_1_0_0 folder to the updates folder in your web server again. This will replace the MyApp.application that was put there for version 1.0.

Now, let's see the application update itself. Click the application from the start menu and see what happens. The application should ask you if you want to update. DON'T click SKIP! It won't ask you again for a long time if you do (I'm not sure how long). If you don't want your users to be able to skip updates, set a required version equal to your current version each time you deploy (in the Updates... dialog of the Publish settings).

Anyway, you've just done your first ClickOnce deployment. More tidbits to come.