John's Coding Cave

Analyzing website memory dumps

Wed, 11 Sep 2019 00:00:00 +0000

When debugging a production website, Application Insights sometimes is not enough, and you need more powerful tools.

Going through a memory dump is both a blessing and a curse. To be fairly honest, it is mostly a curse. Today one of our services experienced multiple “OutOfMemoryExceptions”. Instead of doing what we mostly do, restarting the hell out of that server, we were a bit smarter and asked Azure to give us a memory dump. The memory dump led us to a specific web request causing a quasi-memory-leak by putting lots of objects (530 038 to be exact) in memory. What follows is how we did it.

Step one: Getting the memory dump

Identifying the correct instance

Our service runs on multiple instances, and we needed the memory dump for that specific instance. You can find the right instance easily by going to the live metrics stream of the connected Application Insights.

In case of this screenshot, the instance name is RD0003FF22484C, but that will be different for your case. There are other ways of detecting what instance is the naughty one. For example, you can use the application insights search option, or any other way to find the instance name.

Requesting the memory dump

You should request the memory dump while the mischief is happening. A memory dump is a snapshot of threads and memory of the running web app and does not contain historical data. Getting a memory dump when the error is no longer happening is pointless.

To request the memory dump, you need to open the web app diagnostic tools. To do this, you open your Web App page in Azure, click “Diagnose and solve problems”, and click “Diagnostic Tools.”

The App Service Diagnostics Tools and Resources house a few handy resources. The ones I use most are the “Collect Memory Dump”, “Collect .NET Profiler Trace” and “Advanced Application Restart”. All of these tools prove to be very handy from time to time, and one should keep them in mind at all times when your application is doing the dirty thing.

As you can guess, the tool we are going to use is going to be the “Collect Memory Dump” one. After we successfully created a memory dump, we can restart only the one specific bogus instance using “Advanced Application Restart”. Doing so limits the impact of the restart.

Starting the memory dump is as easy as selecting the instance you want the dump for, clicking “Collect Memory Dump” and wait until all three steps are done. In mode, you can opt to skip the “Analyze Data” step; I tend not to do it because all inputs of data analysis are helpful.

Creation of a memory dump can take a while, depending on how much memory your application is munching. In my case, the dump was about 3GB big and took quite a while to create.

You can open the “Azure analysis” on the right of the screen. The dumps themselves show on the right. For our next steps, we are going to need the dump on our hard drive, so download away.

Step two: getting more tools

For my analysis today, I made use of two tools. The first tool was JetBrains dotMemory, and the other one is WinDbg. Lastly, I’ve downloaded two extensions: one called SOSex and another called MEX

My experience with these tools is that dotMemory is a friendly general analysis tool to pinpoint problems generally. With WinDbg, you can dig deep into the data, but it is hard to have a general overview of what is happening.

Below is a screenshot of both dotMemory and WinDbg and the difference in the user interface is very clear.

To be specific: dotMemory is on the left of the red line, WinDbg on the right.

Step 3: General analysis with dotMemory

To open our memory dump, click “Import Dump”, select the correct file, and click “Open”.

It will show some screens and do some initial analytics. After you went through them you should see something like this:

It was quite obvious there was a problem. There is 450.49 MB worth of string data in memory. Another problem that pops up is that three objects in memory hold about 415.96MB of data. That can’t be a coincidence.

By clicking the “3xEnumerable…”, followed by opening one of those three, I got the data above. In the list there are 530 038 items, accounting to more than 146Mb of data for a single API call. Adding the knowledge that at the time this memory dump was taken, there were three of those lists; there could be only one conclusion. This list must be the cause of the problem.

Sadly dotMemory did not allow me to pinpoint what call caused this situation, so a deeper dive had to be taken, straight into the deep waters of WinDbg.

Step 4: The deep waters of WinDbg

Part 1: Downloading SOS.dll from Azure Web App

Before we can start debugging, we need the SOS tools. There is a SOS.dll on your machine, but sadly it’s the wrong one. We need the one from the Azure Web App.

The dll can be downloaded very easily (ahem). Here is how:

Go into the kudu site of your web app
Open the debug console (CMD) and type following command: pushd D:\Windows\Microsoft.NET to go to the hidden folder
Go into the 32 or 64bit folder of “Framework”, depending on your needs (trial and error is the way I do this)
In that folder, open the “v4.0.30319” folder
Download sos.dll from this folder, and save it somewhere convenient and not too deep in the folder tree on your local machine. Do remember where you saved it”

See, that was not hard at all, very user-friendly. Thanks Microsoft!

Part two: loading all the things

First of all, it is time to start WinDbg. When you have this open (and you use the GUI version of it), start with opening the memory dump by dragging the file into the WinDbg screen.

If opened, your screen will look a bit like above. You type commands in the command bar, and you see the results above that. And now the fun begins! .load C:\Users\john.verbiest\Downloads\SOS(1).dll type this command to load the SOS tools from Azure (replace the path with the correct one, and yes, it starts with a dot). If all is good, you will see no response at all. .load C:\Users\john.verbiest\Downloads\sosex_32\sosex.dll type this command to load the extensions. Replace the path, and if you did it all correctly, you get no response.

Ready to dive? Let’s go!

Look at that heap!

If you want to debug something that went wrong with the heap, it might be a good idea to get a summary of how the heap looks. If you execute !dumpheap -stat you will get an ordered list of all the types that are in memory. The list is ordered by size ascending, so the more memory a specific type uses, the lower it will show. In our case, the end of the list looked like this:

      MT    Count    TotalSize Class Name
9d953db8    56181      2696688 <Unloaded Type>
051c24bc    72780      2911200 NHibernate.Engine.EntityKey
8b50c27c    56184      2921568 NHibernate.Proxy.DefaultLazyInitializer
21410cc8    54141      2921816 System.Int32[]
216086e4    63677      3820620 System.Reflection.RuntimeMethodInfo
216a60c0   260011      4160176 System.Collections.DictionaryEntry
1ff44d20   174422      4337152 System.String[]
79aa8eac        5      6360516 hannibal.HBBusiness.Queries.AuteurTool.BeheerEKitContent.IFicheLesMateriaalDto[]
061ea28c   208428      7705356 System.Object[]
1ff46fcc    60768      9108281 System.Byte[]
7984edfc   193914     12410496 hannibal.HBBusiness.Queries.AuteurTool.BeheerEKitContent.FicheFolioMateriaalDto
7984f06c   172152     18592416 hannibal.HBBusiness.Queries.AuteurTool.BeheerEKitContent.FicheLesMateriaalDto
794d8638   402027     28945944 hannibal.HBBusiness.Queries.Reporting.AgendaOverzicht.Dtos.MaterialenFlatDTO
7984ec10  1206213     82022484 hannibal.HBBusiness.Queries.AuteurTool.BeheerEKitContent.FicheOwnMaterialDto
064fcaa8   608533     87955734      Free
061efd98  5831585    505948476 System.String
Total 11400232 objects
Fragmented blocks larger than 0.5 MB:
    Addr     Size      Followed by
d5ddf8e0   11.5MB         d6953548 System.Threading.OverlappedData
de0e77b4    5.4MB         de6583b4 System.Byte[]
de6a5830    1.1MB         de7c4630 System.String
de7c6488    5.0MB         decbb76c System.Byte[]

Again we see in this view that there are way too many strings in the system. But furthermore, the next few types are viral as well, counting more than one million of type “FicheOwnMaterialDto”. (The first column of that list is the Method Table, where the methods on that type are stored - we don’t need this at the moment, but you can use it to see compiled code). If you click on the link that method table represents you will get a list of the instances of that specific type. As it was 1.2 million instances, it took me quite some time to get the list.

Clicking a random instance (the first column) made WinDbg execute !DumpObj /d d260197c. This command shows the object located at address d260197c. It gave me this:

Name:        hannibal.HBBusiness.Queries.AuteurTool.BeheerEKitContent.FicheOwnMaterialDto
MethodTable: 7984ec10
EEClass:     7981eb7c
Size:        68(0x44) bytes
File:        D:\local\Temporary ASP.NET Files\scoodle\a9371f3b\c1a73863\assembly\dl3\40209bed\00ca9cb9_e266d501\hannibal.HBBusiness.dll
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
211dcc10  4000f10       20          System.Guid  1 instance d260199c <Id>k__BackingField
061efd98  4000f11        4        System.String  0 instance ae09aaec <Naam>k__BackingField
061efd98  4000f12        8        System.String  0 instance 00000000 <Url>k__BackingField
211dcc10  4000f13       30          System.Guid  1 instance d26019ac <FicheId>k__BackingField
061efd98  4000f14        c        System.String  0 instance ae09ab10 <MateriaalType>k__BackingField
061eaa48  4000f15       1c       System.Boolean  1 instance        0 <CanHaveFile>k__BackingField
7b082380  4000f16       10 ...in.Common.Bestand  0 instance 00000000 <Bestand>k__BackingField
061efd98  4000f17       14        System.String  0 instance 00000000 <Verwijzing>k__BackingField
061efd98  4000f18       18        System.String  0 instance ae09ab30 <Beschrijving>k__BackingField

Handy as this data could be, I actually needed the parent of this object. To get this I used !gcroot d260197c. This will look for the root object of this one. Result:

Thread 1b84:
    4fcbecfc 4700f9db System.Linq.Enumerable+<SelectManyIterator>d__17`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].MoveNext()
        esi: 
            ->  0884b810 System.Linq.Enumerable+<SelectManyIterator>d__17`2[[hannibal.HBBusiness.Queries.AuteurTool.BeheerEKitContent.IFicheLesMateriaalDto, hannibal.HBBusiness],[hannibal.HBBusiness.Queries.Reporting.AgendaOverzicht.Dtos.MaterialenFlatDTO, hannibal.HBBusiness]]
            ->  18aacb70 hannibal.HBBusiness.Queries.AuteurTool.BeheerEKitContent.IFicheLesMateriaalDto[]
            ->  d260197c hannibal.HBBusiness.Queries.AuteurTool.BeheerEKitContent.FicheOwnMaterialDto

Found 1 unique roots (run '!GCRoot -all' to see all roots).

This gave me something I really needed: the thread id. Lets put WinDbg into that thread: ~~[1b84]s. Now that we are in the correct thread, we can do more with the CLR. How about getting the current stack? !clrstack gives the following:

OS Thread Id: 0x1b84 (52)
Child SP       IP Call Site
4fcbe968 73b68423 [HelperMethodFrame: 4fcbe968] 
4fcbe9dc 216b0e7a System.String.ConcatArray(System.String[], Int32)
4fcbe9f8 23ea737c System.String.Concat(System.Object[])
4fcbea14 8b55979c NHibernate.Engine.Cascade.CascadeOn(NHibernate.Persister.Entity.IEntityPersister, System.Object, System.Object)
4fcbea58 8b5594ad NHibernate.Event.Default.AbstractFlushingEventListener.CascadeOnFlush(NHibernate.Event.IEventSource, NHibernate.Persister.Entity.IEntityPersister, System.Object, System.Object)
4fcbea8c 8b5590f7 NHibernate.Event.Default.AbstractFlushingEventListener.PrepareEntityFlushes(NHibernate.Event.IEventSource)
 ....... omitted .......
4fcbf15c 4e152882 System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
4fcbf198 4d513b05 System.Web.HttpApplication.ExecuteStepImpl(IExecutionStep)
4fcbf1ac 4d513854 System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)
4fcbf1ec 4d5105fc System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception)
4fcbf2c8 4ddcfd7d System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback)
4fcbf2e0 4ddca879 System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, System.Web.HttpContext)
4fcbf324 4ddc9224 System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
4fcbf328 4ddc8a08 [InlinedCallFrame: 4fcbf328] 
4fcbf420 4ddc8a08 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
4fcbf4f8 0537f17a [ContextTransitionFrame: 4fcbf4f8] 

this is a hell of a lot of data you should not read, but it shows you have the full stack trace of this thread and something to start working with. More useful (and with more data) is the following command: !clrstack -l -p. This will give you the same stack, but with all locals (-l) and all properties (-p) included. I will only show the most interesting one below:

4fcbee7c ad198f5c hannibal.HBWeb.Areas.Agenda.Controllers.AgendaAfdrukController.AgendaOverzicht(hannibal.HBBusiness.Queries.Reporting.AgendaOverzicht.AgendaOverzichtReportDataQuery)
    PARAMETERS:
        this (<CLR reg>) = 0x0833340c
        dataQuery = <no data>
    LOCALS:
        <no data>
        <no data>

This is the controller of the api and is a good starting point to know more. Right now I’m looking for the URI that was used to get here.

Clicking on the address next to the “this” parameter, we open the object and can start exploring the memory space.

Name:        hannibal.HBWeb.Areas.Agenda.Controllers.AgendaAfdrukController
MethodTable: 7914e310
EEClass:     7913c3a4
Size:        72(0x48) bytes
File:        D:\local\Temporary ASP.NET Files\scoodle\a9371f3b\c1a73863\assembly\dl3\0736780f\00bc75b2_e266d501\hannibal.HBWeb.dll
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
8b135040  4000153        4 ...c.SingleEntryGate  0 instance 083334d0 _executeWasCalledGate
98d061f0  4000154        8 ...iewDataDictionary  0 instance 00000000 _dynamicViewDataDictionary
79377bd8  4000155        c ...empDataDictionary  0 instance 0833362c _tempDataDictionary
061eaa48  4000156       1c       System.Boolean  1 instance        1 _validateRequest
79377ca8  4000157       10 ...vc.IValueProvider  0 instance 0833539c _valueProvider
79377e8c  4000158       14 ...iewDataDictionary  0 instance ae030e64 _viewDataDictionary
73747220  4000159       18 ...ControllerContext  0 instance 08333584 <ControllerContext>k__BackingField
8b134fc4  40001e6       20 ...sync.AsyncManager  0 instance 08333454 _asyncManager
793777e0  40001e7       24 ...vc.IActionInvoker  0 instance 08333758 _actionInvoker
74d9cc24  40001e8       28 ...lBinderDictionary  0 instance 00000000 _binders
4d4f6170  40001e9       2c ...g.RouteCollection  0 instance 00000000 _routeCollection
79377844  40001ea       30 ...ITempDataProvider  0 instance 08333668 _tempDataProvider
760efb44  40001eb       34 ...wEngineCollection  0 instance 00000000 _viewEngineCollection
76159534  40001ec       38 ...ependencyResolver  0 instance 00000000 _resolver
793779e8  40001ed       3c ...Web.Mvc.UrlHelper  0 instance 0833359c <Url>k__BackingField
061e6780  40001e4      138        System.Object  0   static 0fd021d8 _executeTag
061e6780  40001e5      13c        System.Object  0   static 0fd021e4 _executeCoreTag
760ed9ac  4000250       40 ....IQueryDispatcher  0 instance 0fca4938 queryDispatcher

When exploring these waters, it is important to note the difference between value and reference types. Everything you will see here are the value of a type, or the reference to the reference type. For example, you can see the value of the boolean value type as 1 (_validateRequest), but the _executeWasCalledgate is an object (reference type), so only the reference is shown here. You can, however, open the object by clicking the address.

To give myself a reference on how to proceed: opening the UrlHelper instance got me to this magic:

Name:        System.Web.Mvc.ControllerContext
MethodTable: 73747220
EEClass:     7375c9e8
Size:        24(0x18) bytes
File:        D:\local\Temporary ASP.NET Files\scoodle\a9371f3b\c1a73863\assembly\dl3\5f898403\00d8c3c0_e266d501\System.Web.Mvc.dll
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
4701d3ac  40000d3        4 ...b.HttpContextBase  0 instance 16dbebe0 _httpContext
4837d548  40000d4        8 ...ng.RequestContext  0 instance 16dbed78 _requestContext
4837d650  40000d5        c ...Routing.RouteData  0 instance 16dbec40 _routeData
761501f8  40000d6       10 ...vc.ControllerBase  0 instance 0833340c <Controller>k__BackingField

And from here on you shall explore on your own. I managed to find the url and the parameters, and reproduce the issue this way.

Debugging with MEX

Everything above was before I’ve discovered MEX. With MEX it was much easier to get the data I’ve wanted. Mex adds a lot of missing functionality to debug .net code. Loading can be done by doing .load C:\windbg\x86\mex.dll or wherever you have installed mex into. A quick overview of its features:

!aspxpagesext

This awesome call gives you insights of all ASPX requests the server was doing at the time of the memory dump.

   Address  Completed Timeout Time (secs) ThreadId ReturnCode Verb Url
   06ce6788       yes   19200                             302 GET  <Url>
   06982728       yes   19200                             302 GET  <Url>
   06d589d8       yes   19200                             302 POST <Url>
   06e8b504       yes    7200                             304 GET  <Url>
SL 07a6cc44        no   19200        1148                 200 GET  <Url>
   07a707a4       yes    7200                             304 GET  <Url>
AS 0a9254c8        no       0       14478                 200 GET  <Url>
   0c1c0724       yes    7200                             304 GET  <Url>
84 contexts found (84 displayed).
   AS == possibly async request
   SL == session locked

I omitted some requests (this page is getting pretty long already), but you get the picture. Clicking through the addresses gives you nice overviews of the objects using !mex.DisplayObj

!tasktriage

This will list all the active tasks, and their status. Applying -t will only show the tasks for the current thread and -v will also list finished tasks.

Address    Target     Status                         Method                                                               Exceptions
====================================================================================================================================
cce51f64 |   <null> | TASK_STATE_RAN_TO_COMPLETION | <null>                                                             |     <none>
12949428 | 1294938c | TASK_STATE_DELEGATE_INVOKED  | Microsoft.ApplicationInsights.Channel.InMemoryTransmitter.Runner() |     <none>
db9c29f4 |   <null> | TASK_STATE_RAN_TO_COMPLETION | <null>                                                             |     <none>
0a962918 | 0a96276c | TASK_STATE_DELEGATE_INVOKED  | Microsoft.ApplicationInsights.Channel.InMemoryTransmitter.Runner() |     <none>
e3e01224 |   <null> | TASK_STATE_RAN_TO_COMPLETION | <null>                                                             |     <none>
e3e01140 |   <null> | TASK_STATE_RAN_TO_COMPLETION | <null>                                                             |     <none>
e3e19d10 |   <null> | TASK_STATE_RAN_TO_COMPLETION | <null>                                                             |     <none>
f2795fa8 |   <null> | TASK_STATE_RAN_TO_COMPLETION | <null>                                                             |     <none>
0ac27d5c | 0ac27bb0 | TASK_STATE_DELEGATE_INVOKED  | Microsoft.ApplicationInsights.Channel.InMemoryTransmitter.Runner() |     <none>
0ad31350 | 0ad311a4 | TASK_STATE_DELEGATE_INVOKED  | Microsoft.ApplicationInsights.Channel.InMemoryTransmitter.Runner() |     <none>
0b755f94 | 0b755eec | TASK_STATE_DELEGATE_INVOKED  | Microsoft.ApplicationInsights.Channel.InMemoryTransmitter.Runner() |     <none>
f27cd34c |   <null> | TASK_STATE_RAN_TO_COMPLETION | <null>                                                             |     <none>
f27cd268 |   <null> | TASK_STATE_RAN_TO_COMPLETION | <null>                                                             |     <none>
f27e7f4c |   <null> | TASK_STATE_RAN_TO_COMPLETION | <null>                                                             |     <none>
====================================================================================================================================
Address    Target     Status                         Method                                                               Exceptions

!dumpaspnetsession

With this command you are able to see the content of the session. However if you run session with Redis (as we do) you only get the session id and will get to be creative to get the rest.

0:042> !dumpaspnetsession -ctx 0x07594dcc
Failed Reading .NET heap object at 12f23258. Heap may be incomplete.
Could not find session 'ddasv1trqtdyn1wirt4teenx'

!mex.DisplayObj

Is an extended version of !dumpobject, with better deserialization and awesome views of objects.

0:052> !do 0x08331110
Name:        System.Collections.Hashtable
MethodTable: 213c08a8
EEClass:     211e8a6c
Size:        52(0x34) bytes
File:        D:\Windows\Microsoft.Net\assembly\GAC_32\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
21411630  4001821        4 ...ashtable+bucket[]  0 instance 08331fcc buckets
061ec9b0  4001822       18         System.Int32  1 instance       33 count
061ec9b0  4001823       1c         System.Int32  1 instance       10 occupancy
061ec9b0  4001824       20         System.Int32  1 instance       64 loadsize
061edd50  4001825       24        System.Single  1 instance 0.720000 loadFactor
061ec9b0  4001826       28         System.Int32  1 instance       37 version
061eaa48  4001827       2c       System.Boolean  1 instance        0 isWriterInProgress
061e947c  4001828        8 ...tions.ICollection  0 instance 00000000 keys
061e947c  4001829        c ...tions.ICollection  0 instance 00000000 values
1ff488bc  400182a       10 ...IEqualityComparer  0 instance 0f39ae78 _keycomparer
061e6780  400182b       14        System.Object  0 instance 00000000 _syncRoot

Versus: 0:052> !DisplayObj 0x08331110 [raw] 08331110 System.Collections.Hashtable Entries: 3

Key: 08331c64 “schooldatabase” [14] (System.String) Value: 08331ccc (System.Collections.Specialized.NameObjectCollectionBase+NameObjectEntry)

Key: 08331f54 “displayNickname” [15] (System.String) Value: 08331fbc (System.Collections.Specialized.NameObjectCollectionBase+NameObjectEntry)

Key: 083312b4 “vestiging_id” [12] (System.String) Value: 08331314 (System.Collections.Specialized.NameObjectCollectionBase+NameObjectEntry)

Others

You can get a list of all the commands via !mex.help

Reading Redis Cache Values

Mex has support for reading InProc session data built-in. However it is not uncommon to put session data in a Redis Cache to support multiple machines. To read the sessiondata you will have to navigate a bit. Be sure to open your objects with !Mex.DisplayObj

Navigate to the HTTPContext of the call you want to debug
Open the object within _sessionStateModule
Open the object within _rqSessionState
Open the object within _sessionItems
Open the object within innerCollection
Open the object within _entriesTable -> This is the list of key/value pairs for the session
Open the object within Value, then again Value
Open the object within _serializedvalue
Read the serialized (binary) value

Now what was the problem anyway?

We reasonably quickly found what we were looking for. There were 1 974 306 items of 4 specific types in memory, taking up 135MB. Because of this, the number of strings was going through the roof: 5 831 585 string taking up 505Mb of the stack. Knowing this process runs in a 32-bit environment, the stack only had 85Mb of free memory.

By doing a gcroot on one of the instances of one of the four types, I was able to find the thread these objects originated from. The stack trace of this thread allowed me to open the HttpContext. From there, we needed to identify the call that led to this HttpContext. This can be found in the HttpRequest object in this HttpContext, together with the form data. For the session data, I’ve used the steps for Redis above.

After we identified the call, the session variables and the request body, we were able to reproduce the issue. For some reason, we noticed an intensive memory increase when we did the call. After some debugging, we found out there were some edge cases when a SQL query omitted a where clause. With no where clause the complete table got loaded into memory (and it was a massive table). Fixing this issue had a substantial positive impact on performance.

Final words

When your application gets to a state where it does no longer behave as expected, a memory dump can get useful. It is only helpful if you can make the dump during the error state of the application. When you are investigating high CPU usage, I would suggest other techniques such as a Trace Log.

I made this list of tools and techniques primarily for myself, so I can use it whenever an application goes bonkers on me. I will extend this blog post as I learn more about this topic.

Commands Cheat Sheet

Command	Action
.load	loading tools
!clrstack	collapsed version of the CLR stack of current thread
!clrstack -l -p	CLR stack with locals and properties of current thread
~~[1b84]s	Changing the current thread
!dumpheap -stat	statistics about the heap (# of types and usage)
!locks	finds the locks in the system
!VerifyHeap	validates managed heap integrity
!vmstat	shows info about the virtual memory
!dumpdomain	dumps out the applications domain
!threadpool	dumps out information on the clr thread pool
!gcroot	dumps out the reference chain of the object
!strings	dumps out all the strings
!threads	dumps out all threads and their last exception
!tasktriage	MEX - dump out all running tasks
!dumpaspnetsession	MEX - dump out aspnet session data (in Proc Only)
!aspxpagesext	MEX - dump out all aspnet running calls
!address -summary	dumps a summary of the memory usage

Maintainable Unit Testing

Wed, 03 Jul 2019 00:00:00 +0000

I’m a big fan of Test Driven Development. I try to push my enthusiasm to my colleagues, but they usually find reasons not to do it. One of these reasons is maintainability, and for a while, I could not disagree with them.

After a visit to Techorama in Antwerp in 2017, I found some insights in unit testing that changed the way I write code and unit tests. Here’s my story.

Defining the problem

When you write code the TDD way, you follow these steps:

Write a (failing) unit test for something your Unit has to do
Write the implementation in your Unit, with the least amount of effort
Refactor and repeat

Following these steps leads to hundreds of Unit Tests throughout your code base. And this is good until you have to change something.

Issue 1: Dependency Change

Imagine this: you have a great class with 20 useful methods, thoroughly tested, and you want to add some functionality. At the moment, your class looks a little like this:

public class AwesomeClass : IAwesomeClass
{
    private readonly IDependency1 _dep1;
    private readonly IDependency2 _dep2;
    private readonly IDependency3 _dep3;
    private readonly IDependency4 _dep4;

    public AwesomeClass(IDependency1 dep1, IDependency2 dep2, IDependency3 dep3, IDependency4 dep4)
    {
        _dep1 = dep1;
        _dep2 = dep2;
        _dep3 = dep3;
        _dep4 = dep4;
    }

    public string DoAThing(string withThis, int withThat)
    {
        var someResult = _dep1.Result(withThis);
        someResult += withThat;
        return someResult.ToString();
    }

    public string DoAnotherThing(string withThis, int withThat)
    {
        var someResult = _dep2.Result(withThis);
        someResult += withThat;
        return someResult.ToString();
    }
    
    // And many more thing to come
}

And a typical test looks like this:

using FakeItEasy;
using FluentAssertions;
using Xunit;

// Fluff omitted

[Fact]
public void DoAThing_WhenCalled_ShouldCallDependency1ForResult()
{
    // Arrange
    var dep1 = A.Fake<IDependency1>();
    var dep2 = A.Fake<IDependency2>();
    var dep3 = A.Fake<IDependency3>();
    var dep4 = A.Fake<IDependency4>();
    var withThis = "Some Text";
    var withThat = 3;

    var sut = new AwesomeClass(dep1, dep2, dep3, dep4);

    // Act
    sut.DoAThing(withThis, withThat);

    // Assert
    A.CallTo(() => dep1.Result(withThis)).MustHaveHappenedOnceExactly();
}

// Fluff omitted

When you want to implement the new feature, however, you need to add a new dependency in the constructor of the Class, and now TDD Hell emerges. Instantly you have all your old tests broken, and you can start fixing these one by one. Fixing all your tests is a tedious, dangerous, and, above all, boring job. We’re developers: we don’t like boring work. We need to address this.

Issue 2: Focus on what matters

Take a look at this test again:

using FakeItEasy;
using FluentAssertions;
using Xunit;

// Fluff omitted

[Fact]
public void DoAThing_WhenCalled_ShouldCallDependency1ForResult()
{
    // Arrange
    var dep1 = A.Fake<IDependency1>();
    var dep2 = A.Fake<IDependency2>();
    var dep3 = A.Fake<IDependency3>();
    var dep4 = A.Fake<IDependency4>();
    var withThis = "Some Text";
    var withThat = 3;

    var sut = new AwesomeClass(dep1, dep2, dep3, dep4);

    // Act
    sut.DoAThing(withThis, withThat);

    // Assert
    A.CallTo(() => dep1.Result(withThis)).MustHaveHappenedOnceExactly();
}

// Fluff omitted

As you can see, there are 7 lines of code in the “arrange” phase, 1 line of code in the “act” phase, and one line of code in the “assert phase”. A test that is heavy on the “arrange” phase is harder to read. When you want to know the meaning of the test only by looking at the code, some questions may arise.

What are dependencies 2 to 4 used for?
Is it essential that the value of “withThis” is “Some Text” specifically?
Is it necessary that the value of “withThat” is “3” specifically?
Will the test fail if I change any of the previous values?

To address these issues, I made some ground rules for myself:

Tests should be as short as possible
If the value of some variable is not important, it should be clear.
If the value of a variable is important, set it in the “arrange” phase.

These rules produce short unit tests, where it is clear to see what is essential to run this test and what this test is actually about. However, how do we do this?

Introducing another way of testing

Library 1: Autofixture

If the value of some specific variable is not crucial for a test, It should be random. To make this happen, You can use a library called AutoFixture. This library can generate random values for whatever concrete type you ask. Even complex and nested types are no issue for AutoFixture.

[Fact]
public void Lib1Autofixture()
{
    // Arrange
    var dep1 = A.Fake<IDependency1>();
    var dep2 = A.Fake<IDependency2>();
    var dep3 = A.Fake<IDependency3>();
    var dep4 = A.Fake<IDependency4>();
    var fixture = new Fixture();
    var withThis = fixture.Create<string>();
    var withThat = fixture.Create<int>();

    var sut = new AwesomeClass(dep1, dep2, dep3, dep4);

    // Act
    sut.DoAThing(withThis, withThat);

    // Assert
    A.CallTo(() => dep1.Result(withThis)).MustHaveHappenedOnceExactly();
}

Library 2: Autofixture.Xunit2

With Autofixture.Xunit2, you can inject random values for concrete classes in your method parameters. No need to create a Fixture manually and use it. Using the Theory attribute you instruct Xunit to request some data from the AuthoData Attribute.

[Theory, AutoData]
public void Lib2AutofixtureXunit2(string withThis, int withThat)
{
    // Arrange
    var dep1 = A.Fake<IDependency1>();
    var dep2 = A.Fake<IDependency2>();
    var dep3 = A.Fake<IDependency3>();
    var dep4 = A.Fake<IDependency4>();

    var sut = new AwesomeClass(dep1, dep2, dep3, dep4);

    // Act
    sut.DoAThing(withThis, withThat);

    // Assert
    A.CallTo(() => dep1.Result(withThis)).MustHaveHappenedOnceExactly();
}

As you can see, the unit test starts to get cleaner and is becoming more readable.

Library 3: AutoFixture.AutoFakeItEasy

If you were to ask for an instance of the “AwesomeClass” in your test method parameters, you would get an exception. AutoFixture is designed to only create implementations of concrete classes, but the constructor of the “AwesomeClass” needs 4 implementations of interfaces. However, we can customize AutoFixture so that it’s capable of providing implementations for interfaces via the AutoFakeItEasy library.

To facilitate this, we have to create a new attribute, inheriting from the AutoData attribute. I like to call this my “UnitTestAttribute” as all this thing does is provide implementations for unit tests.

/// <summary>
/// This is the base test attribute, used in almost all the in-memory unit tests.
/// Whenever a special configuration is required (example: for implementation tests)
/// you can use the static CreateBasicFixture as a base.
/// </summary>
public class UnitTestAttribute : AutoDataAttribute
{
    public UnitTestAttribute() : base(CreateBasicFixture)
    { /* Leave empty */}

    /// <summary>
    /// Creates the fixture for a basic in-memory unit test, to be re-used by other TestAttributes
    /// </summary>
    /// <returns>Configured fixture</returns>
    // ReSharper disable once MemberCanBePrivate.Global
    public static IFixture CreateBasicFixture()
    {
        var fixture = new Fixture().Customize(new AutoFakeItEasyCustomization());
        return fixture;
    }
}

The magic happens in the new Fixture().Customize(new AutoFakeItEasyCustomization()) part of the code. It instructs AutoFixture to customize its inner workings and request a fake implementation of an interface whenever it runs into one to create an object. Instead of throwing an exception, it now uses FakeItEasy to build a mock instance of that object.

Our test now looks like this:

[Theory, UnitTest]
public void Lib2AutofixtureAutoFakeItEasy(
    [Frozen] IDependency1 dep1, 
    string withThis, 
    int withThat, 
    AwesomeClass sut)
{
    // Arrange no longer needed:
    // no test specific setup is required for this test

    // Act
    sut.DoAThing(withThis, withThat);

    // Assert
    A.CallTo(() => dep1.Result(withThis)).MustHaveHappenedOnceExactly();
}

Few things to note:

First, there is no need for an “arrange” phase. Everything that required setup is of no real importance to the scope of this test and is not included. Now our test is very readable.

Secondly, we need the dependency dep1 in our “assert” phase of the test. We can request this in our method parameters. To make sure the instance requested by the method is the same as the instance injected in the AwesomeClass, we add the attribute [Frozen] to it. Frozen works as follows: when the test gets started, the UnitTest attribute runs through all the method parameters one by one. Each time it needs an object, it requests AutoFixture to create one. However, when the [Frozen] attribute is attached to a parameter, it requests a new instance one more time and keeps using that instance for the remainder of the test. When it is time to create the AwesomeClass, and it needs an implementation of the IDependency1 type, it uses the same one as the one requested as a parameter.

It is vital to note that the order of your parameters does count. Always place the parameters with the [Frozen] attribute first, so they surely get injected in the right places.

TLDR: The UnitTest attribute magically builds all you ever need. When you want to interfere with the interfaces about to be injected, request this as a parameter and add the [Frozen] tag to it. Place this parameter before the parameter containing the object to be injected.

Example with a specific setup

[Theory, UnitTest]
public void DoAThing_WhenCalled_ExampleWithSetup(
    [Frozen] IDependency1 dep1,
    AwesomeClass sut)
{
    // Arrange
    var withThis = "<SomeUserInputThatWillLookLikeThis>";
    var withThat = 3; // Also a specific case
    
    // Setup the interface to respond with the value expected from that 
    // interface when calling it.
    A.CallTo(() => dep1.Result(withThis)).Returns(5);
    
    // Act
    var result = sut.DoAThing(withThis, withThat);

    // Assert
    result.Should().Be("8");
}

In this example, you can see 2 stages in our “arrange” phase. The first stage sets the input parameters to a test-specific value. Doing a setup like this shows the reader of your test clearly that those values are important.

The second stage is setting up the behavior of the fake interface. It instructs that object that whenever a call to “Result” is made with the value of “withThis”, it should return the value 5.

Packages used in these examples

To create a setup like this, you need some nuget packages. These can be installed in Visual Studio with the following package manager commands:

Install-Package xunit
Install-Package xunit.runner.visualstudio
Install-Package FakeItEasy
Install-Package AutoFixture
Install-Package AutoFixture.Xunit2
Install-Package AutoFixture.AutoFakeItEasy
Install-Package FluentAssertions

If you don’t like the result.Should().Be("8"); type of assertions, you can skip the FluentAssertions Library.

That’s all folks

Whenever I need to write unit tests, I always use this setup. It gives me lots of tranquility to know I’m able to change my dependencies without a headache. The bonus of small, readable unit tests is an absolute win for me as well. I hope these insights and code patterns can be useful to you as well.

In another blog post, I will go deeper into this system, and explain how you can customize the fixture to create standard functionality for each unit test without repeating yourself.

You can find the source of these examples on my Github page.

My first babysteps in static code analysis with nDepend

Mon, 13 May 2019 00:00:00 +0000

A few weeks ago, I was asked to look into static code analysis, mainly with the tool nDepend. I’ve never had any experience with static code analysis, and was fairly curious about what this was all about.

In this blogpost I will be talking about what static code analysis is, what the difference is with dynamic code analysis and when to use it. After that I will be covering some metrics and conclusions you can get from a static code analysis tool and I will be ending my post with a small overview of what nDepend has to offer.

Static code analysis

What is it?

You might not know it, but in most IDE’s we already use static code analysis all the time. However it is wrapped into a nice package known as IDE features.

As you can see in the image above, I can get 4 kinds of static code analysis from this little snippet of code in Visual Studio alone.

On top we have codelens, informing us how many times and where the function is used
The blue squiggly line indicates we have a local variable that does not comply to the local naming conventions
The red squiggly line indicates we should expect a build error on that line
The slightly dimmed code below tells us that code is currently unreacheable

Static vs Dynamic code analysis

We can define the difference between static and dynamic code analysis as follows:

Static code analysis	Dynamic code analysis
Code is not running	Code is in a running state
Analysing code itself	Analysing features
For obvious problems & code smells	For everything else
Examples: Code style & smells, metrics, …	Examples: Memory & CPU pressure, debugging, …

In short: Static code analysis focusses mostly on code quality in the domain of maintainability & readability. Dynamic code analysis focusses on features and performance.

Static code metrics

In this chapter I will focus on the four metrics I think are the most important ones for your code. I might blog about some new metrics later, but in my humble opinion these are key for your code.

Cyclomatic Complexity

Quite a mouthful as a word. When this metric is measured, it will count every if, else, switch, goto, for, while and every other known code-flow-changing keyword. This count results in the Cyclomatic Complexity.

To make it a little easier to understand: Cyclomatic complexity measures the amount of code paths in a method.

It is a good practice to keep your complexity under 10. Methods between 10 and 20 can exists but should be rather rare. Methods with a complexity of 20 and above should be refactored on the earliest convenience.

When you request the metric to be displayed, it shows up like this:

The first time I saw this one, I was like: what am I looking at? Ain’t nobody can read that. But I got it figured out, and this is how it works:

Reading an nDpend metrics chart

Settings on top and to the right

On the top you will find the following:

Item	What it controls
Level	The smallest part of the chart you will see, and subsequently the metrics that are available
Size	The metric controlling the size of the boxes
Color	The metric controlling the color of the boxes

On the right you will find a slider and some minor settings, controlling what level of the Color-selected-metric correspondents with what color.

Boxes, boxes, boxes

Inside the blue square you see the biggest kind of box in the graph. Surrounded by a thick yellow line (remember: never eat yellow lines) you will find all code within an assembly.

For this example specifically, the size setting has been set to lines of code. This means the size of the box represents the amount of lines in the code, relative to all lines of code in the solution.

This time the blue square surrounds a namespace. It’s not so clear to see, but the namespace is being surrounded by a thin, non-edible yellow line. The size is again a representation of the amount of lines of code.

The next step in this graph is evidently a class. They are found by looking at the slightly darker borders making a square. And you guessed it correctly, again the size is dependant on the lines of code.

The final and smallest part in the chart is a method. They should be hard to see in an overview this large, but this particulary codebase has some big (read: huge) methods.

All these levels together give you a nice overview on how the distribution of lines of code is in your solution.

Color

Going back to the cyclomatic complexity graph without any blue boxes. When we look at the colour settings you see that I have selected the “IL Cyclomatic Complexity” as a metric for the color metrics. On the left you can see that I’ve set the green bar at 10 and the red bar at 20.

This results in all methods having a complexity equal or less than 10 to be green. Methods with a complexity between 10 and 20 are a shade of green, yellow or red-ish. All methods above 20 are red in this example. As you can see there are quite some complex methods in the system, with the bulk of them on the bottom left corner in the XSL assembly.

Diving to your method

Whenever you click a “box” in the graph, you get redirected to the method representing that box. The example above has a cyclomatic complexity of 723 (only part of the method is displayed here, the full method is around 500 lines long).

Lines of code

The first rule of functions is that they should be small. The second rule of functions is that they should be smaller than that.

“Uncle Bob” - Robert C. Martin

Uncle bob also specifies some numbers regarding this. However he does state that these numbers are based only on his gutfeeling and his experience. He states that functions should not be a 100 lines long. In fact they should hardly be 20 lines long.

This is the same codebase as the previous example. However the size of the boxes is dependent on it’s rank now. Rank is a metric based on the Google Rank system where methods and classes that are used more, are bigger, and the least used are the smallest. The color represents the number of lines of code. From this metric you can see that the code ranked highest is actually not that bad in regards of lines of code. Almost all of them are beneath 20 lines of code, a few are above 50 and some rare ones are above 100.

Instability vs abstractness

This is actually a metric in the scope of the assembly. It produces a nice graph where you can get some meaningfull insights very quick. But first I need to introduce you to stability and abstractness themselves.

Stability

In the context of code analysis, stability means how stable code should be, and not how stable code really is. You can also describe stability as “how difficult and risky is it to change something about this code”.

When talking about stability, it actually boils down to this: if something in your code is referenced a lot, it is hard to change. Ever tried removing a parameter in a method referenced 100 times? That method is stable code.

When talking about assemblies, stable assemblies mean that this particular assembly has lots of incoming references.

Abstractness

Abstractness is a bit easier to explain: it’s a number between 0 and 1. A zero means the assembly contains only implementations. An assembly gets rewared a 1 when all the types within the assembly are only interfaces or abstract classes.

Stability vs Abstractness

Combining the two together, you can get the following graph:

When designing your application, you have to keep instability and abstractness in mind. A well designed appliction will have stable abstract types and unstable implementations of those types. Combining these two gives you the best chance of creating maintainable code. This is however theroretical, and the image above shows you how well your code behaves in this aspect.

In the middle you have the green zone. This is the place you want your assembly to be. It goes from the fully abstract stable assemblies in the top left to the unstable but implemented zone on the bottom right.

Next to the green zone you will find the orange “Danger zones”. These are assemblies that are moving in the wrong direction, but are not that bad (yet).

In the top right corner you will find the “Zone of Uselessness”. Assemblies ending up here are abstractions that are not very usefull. For example: An assembly with all kinds of interfaces but those interfaces are used only in one other assembly.

The bottom right corner is the worst place to find your assembly in. The “Zone of Pain” is the place where you have code that is referenced a lot, but all are implementations instead of abstractions. The zone of pain is the place where you find yourself in fixing a bug because of a code change you did not expect to affect that many things.

The main thing you are measuring here is if your software depends on interfaces instead of implementations. Having this balance in order makes your life a lot easier.

Coupling & Cohesion

When designing software, you should take the following into account:

Software should have low coupling and high cohesion

But what does this actually mean?

Coupling

Methods and classes are coupled when one uses a part of the other. For example when you have a line of code in your class doing this:

var test = new DateTime(2019, 01, 01);

Your class is now coupled with the DateTime object. Any changes to the DateTime object will reflect changes in your code, and this could break your code. Thankfully the DateTime object is a very stable .net framework object that is very unlikely to change.

In your own code however you want your classes to be coupled to very little other classes. It’s much better to be coupled to an interface than an implementation. (see SOLID principles).

When measuring coupling from a certain class, there are 2 types:

Afferent coupling: number of entities (methods/classes/assemblies) that are dependend on this entity
Efferent coupling: number of entities (methods/classes/assemblies) that this entity depends upon

In both cases, you want that number to be as low as possible to keep your maintainability as high as possible.

Cohesion

A class is cohesive if everything that the class needs to do its job exists within the class. This is a very hard thing to measure, so they made a metric that might do the trick a bit.

The LCOM (Lack of cohesion) is a number between 0 and 1. A class containing only methods using all of the internal fields gets rewarded with a nice 0. When none of the methods in the class use any of the internal fields, it gets a nice 1.

A result below 0.5 is considered a good result. Above 0.7 there might be a small problem and above 0.8 you’re not very cohesive. Keep this number as low as possible as well.

Coupling & Cohesion together

This time the box size is the amount of types using the specific class. The color is the lack of cohesion of the methods in that class. The class on the top left could be a class that is badly designed, because it’s functionality is not contained within the specific class.

nDepend added values

For the creation of this blog, I’ve been using nDepend. nDepend is a tool, integrated in Visual Studio or integrated in the buildserver, that can be used for much more than what I’ve described here.

More data

All the data of your code is being stored in nDepend and can be queried via linq queries. This gives you the ability to quickly query your code. In the example I query the code, requesting the top 50 methods with the highest Cyclomatic Complexity.

Rules

However, you can do more: from these queries you are able to create rules. For example, you can create a rule that enforces all methods containing more than 100 lines of code to have a cyclomatic complexity of maximum 10 or the method name to not contain your first name.

You can also define rules as critical, this is particulary of use if you use the build server integration.

Build Server step

When you configure nDepend as a build step in your pipeline, you can get a nice graph and overview of how your code is evolving over time. You can detect pitfalls and bad code practices earlier, when it is still easy to overcome them.

The coolest thing however is when you integrate the build server in a setup with critical rules, these are really enforced. Breaking a critical rule in your code will trigger a build step failure and will stop that code from ever entering production.

Final thoughts

When I was first asked to venture on a journey through static code analysis, I had no idea what to expect. Even after the first two hours of looking at the charts, data and controls, it had me confused. After a while I got the hang of it and now I really see the added benefits of having static code analysis.

It enables you to find code smells and potential errors
It enables you to get to know code you have never seen before
It enables you and other developers to get to know best practices
It enables you to have control about the amount of “bad code” you allow in the software

nDepend can be a real help here. There are other products on the market, even free ones but I have no experience with those. Maybe later?