Debugging Godot Assembly Unloading

If you’ve used Godot C# .NET development before, you might be familiar with an error like this:

  ERROR: modules/mono/mono_gd/gd_mono.cpp:749 - .NET: Failed to unload assemblies. Please check https://github.com/godotengine/godot/issues/78513 for more information.
  ERROR: modules/mono/csharp_script.cpp:674 - Condition "managed_callable->delegate_handle.value == nullptr" is true. Continuing.
  ERROR: modules/mono/csharp_script.cpp:674 - Condition "managed_callable->delegate_handle.value == nullptr" is true. Continuing.
  ERROR: modules/mono/csharp_script.cpp:674 - Condition "managed_callable->delegate_handle.value == nullptr" is true. Continuing.
  ERROR: modules/mono/mono_gd/gd_mono.cpp:749 - .NET: Failed to unload assemblies. Please check https://github.com/godotengine/godot/issues/78513 for more information.
  ERROR: modules/mono/csharp_script.cpp:674 - Condition "managed_callable->delegate_handle.value == nullptr" is true. Continuing.
  ERROR: modules/mono/csharp_script.cpp:674 - Condition "managed_callable->delegate_handle.value == nullptr" is true. Continuing.
  ERROR: modules/mono/csharp_script.cpp:674 - Condition "managed_callable->delegate_handle.value == nullptr" is true. Continuing.
  ERROR: modules/mono/mono_gd/gd_mono.cpp:749 - .NET: Failed to unload assemblies. Please check https://github.com/godotengine/godot/issues/78513 for more information.
  ERROR: modules/mono/mono_gd/gd_mono.cpp:733 - .NET: Giving up on assembly reloading. Please restart the editor if unloading was failing.
--- Debugging process stopped ---

This can happen when you build and/or when you run your game. Often times this happens when you are using a [Tool] or third party libraries, though sometimes it just feels random.

If you’re like me, it’s not even clear how to begin debugging something like this (beyond some crude and cumbersome bisection perhaps?)

Thankfully, I just found a github comment by someone smarter than me who gave us a huge lead which finally helped me understand the problem a lot better.

If you immediately understand that github comment, then this article probably won’t be of much use to you and you can go ahead and close now.

Otherwise, I’m about show exactly how I used that information to debug my problem, and how you might do the same. This is a huge level-up for your debugging in general and will hopefully equip you to be able to handle these issues yourself in the future.

The Debugging Process

Setup WinDbg

The first step is to install WinDbg, which is a nice program for debugging Windows programs.

windbg.png

I’d never used WinDbg before, and haven’t really done much debugging like this in the past, so right away I was feeling skeptical and overwhelmed–worried that it would take a million years to understand. Fortunately that wasn’t the case!

Next, I needed to install a DLL that enables debugging .NET programs, called dotnet-sos. Again, I’d never heard of this. This is what Claude has to say about it:

Dotnet-sos (SOS stands for “Son of Strike”) is a debugger extension that helps developers troubleshoot .NET Core and .NET applications. It provides commands that let you examine the internals of .NET applications when debugging, particularly useful for investigating crashes, memory issues, and other complex problems.

In order to install it, I had to open PowerShell: PS> dotnet tool install -g dotnet-sos and then PS> dotnet-sos install

Once I did that, it will provided a .load command which to use inside of WinDbg. We’ll get to that in a bit.

Attach the debugger

Now with WinDbg open, I opened the Godot project separately. Once I did that, I attached WinDbg to the Godot application.

In WinDbg, go to File > Attach to Process. Then search for Godot and click “Attach” windbg-search.png

The debugger then looked like this: windbg-active.png

I didn’t really understand everything I was looking at but it seemed like it was working correctly. For a moment I was worried that Godot was frozen / unresponsive. Then I realized it’s because WinDbg has grabbed it and enabled a breakpoint (i.e. paused it) so it was all good.

At this point, I loaded my sos DLL in the WinDbg command console: .load C:\Users\wyatt\.dotnet\sos\sos.dll

Once it loaded, then I went ahead and clicked the big green “Go” button in the top left to proceed.

Finally, I once again made Godot trip the dreaded error:

  ERROR: modules/mono/mono_gd/gd_mono.cpp:749 - .NET: Failed to unload assemblies. Please check https://github.com/godotengine/godot/issues/78513 for more information.

This time, I’m no longer afraid. I’m confronting my fears. That’s right. I’m PISSED.

Wielding the the WinDbg program, I ran the command:

!dumpheap -type LoaderAllocator

Which gave some output:

0:056> !dumpheap -type LoaderAllocator
Loading extension C:\Program Files\WindowsApps\Microsoft.WinDbg_1.2504.15001.0_x64__8wekyb3d8bbwe\amd64\winext\sos\extensions\Microsoft.Diagnostics.DataContractReader.dll
Loading extension C:\Program Files\WindowsApps\Microsoft.WinDbg_1.2504.15001.0_x64__8wekyb3d8bbwe\amd64\winext\sos\extensions\Microsoft.Diagnostics.DataContractReader.Extension.dll
Loading extension C:\Program Files\WindowsApps\Microsoft.WinDbg_1.2504.15001.0_x64__8wekyb3d8bbwe\amd64\winext\sos\extensions\Microsoft.Diagnostics.DebuggerCommands.dll
         Address               MT           Size
    01ba2700a638     7ffa68ab8ac8             48 
    01ba2700a668     7ffa68ab8bf0             24 

Statistics:
          MT Count TotalSize Class Name
7ffa68ab8bf0     1        24 System.Reflection.LoaderAllocatorScout
7ffa68ab8ac8     1        48 System.Reflection.LoaderAllocator
Total 2 objects, 72 bytes

To be honest with you, this means very little to me. I vaguely understand it, at least enough to infer that the most important part was the “Address” column with those two hex values:

         Address
    01ba2700a638 
    01ba2700a668

Then I ran the gcroot command. Don’t ask me what it does. I could research it, but that would waste valuable brain space that can be filled with more important things like more Japanese kanji (I’ve been studying the language extensively lately, just in case you wanted to know. You didn’t, but I just told you anyway.)

0:056> !gcroot  01ba2700a638
Caching GC roots, this may take a while.
Subsequent runs of this command will be faster.

HandleTable:
    000001ba17c713a8 (strong handle)
          -> 01ba25004018     System.Object[] 
          -> 01ba2702d530     System.Collections.Concurrent.ConcurrentDictionary<System.Runtime.Loader.AssemblyLoadContext, System.Collections.Concurrent.ConcurrentDictionary<System.Runtime.InteropServices.GCHandle, System.Object>> (static variable: System.Runtime.CompilerServices.ConditionalWeakTable<System.Runtime.Loader.AssemblyLoadContext, System.Object>._alcsBeingUnloaded)
          -> 01ba2702d8e8     System.Collections.Concurrent.ConcurrentDictionary<System.Runtime.Loader.AssemblyLoadContext, System.Collections.Concurrent.ConcurrentDictionary<System.Runtime.InteropServices.GCHandle, System.Object>>+Tables 
          -> 01ba2702d7a8     System.Collections.Concurrent.ConcurrentDictionary<System.Runtime.Loader.AssemblyLoadContext, System.Collections.Concurrent.ConcurrentDictionary<System.Runtime.InteropServices.GCHandle, System.Object>>+VolatileNode[] 
          -> 01ba27036a68     System.Collections.Concurrent.ConcurrentDictionary<System.Runtime.Loader.AssemblyLoadContext, System.Collections.Concurrent.ConcurrentDictionary<System.Runtime.InteropServices.GCHandle, System.Object>>+Node 
          -> 01ba27036848     System.Collections.Concurrent.ConcurrentDictionary<System.Runtime.InteropServices.GCHandle, System.Object> 
          -> 01ba270a7c00     System.Collections.Concurrent.ConcurrentDictionary<System.Runtime.InteropServices.GCHandle, System.Object>+Tables 
          -> 01ba270a7888     System.Collections.Concurrent.ConcurrentDictionary<System.Runtime.InteropServices.GCHandle, System.Object>+VolatileNode[] 
          -> 01ba270f2cd8     System.Collections.Concurrent.ConcurrentDictionary<System.Runtime.InteropServices.GCHandle, System.Object>+Node 
          -> 01ba270f2c98     System.Func<Nebula.WorldRunner, Godot.ENetPacketPeer, Nebula.Serialization.HLBuffer, Godot.GodotObject, Godot.GodotObject> 
          -> 01ba270f2c80     Nebula.Serialization.ProtocolRegistry+<>c__DisplayClass31_0 
          -> 01ba270c3288     System.Reflection.RuntimeMethodInfo 
          -> 01ba270c3020     System.RuntimeType+RuntimeTypeCache 
          -> 01ba270e8390     System.RuntimeType+RuntimeTypeCache+MemberInfoCache<System.RuntimeType> 
          -> 01ba270e83c8     System.RuntimeType[] 
          -> 01ba2709b0b8     System.RuntimeType 
          -> 01ba2700a638     System.Reflection.LoaderAllocator 

Found 1 unique roots.

Now this is something I’m more familiar with! To me it looks like some kind of call stack / stack trace. Sure enough, I can see two parts that are particularly relevant.

First, the label at the top (strong handle).

This jumped out at me because if you run Godot in a terminal, it can provide extra debug logs, and part of the logs it gave me in the terminal (which don’t appear in the app itself) was this:

Failed to unload assemblies. Possible causes: Strong GC handles, running threads, etc.

“Strong GC handles”. So I had pretty good evidence that this was at least one thing causing the assembly unload failure.

Second, I saw this section of the stack:

          -> 01ba270f2c98     System.Func<Nebula.WorldRunner, Godot.ENetPacketPeer, Nebula.Serialization.HLBuffer, Godot.GodotObject, Godot.GodotObject> 
          -> 01ba270f2c80     Nebula.Serialization.ProtocolRegistry+<>c__DisplayClass31_0

That’s my code! Now that I knew WinDbg was accusing my code of being the problem, I promptly exited everything and reported WinDbg as faulty to Microsoft, swearing about how bad their products are… (winky emoji)

After that, I opened up my code and went to the part where I know the System.Func<Nebula.WorldRunner, Godot.ENetPacketPeer, Nebula.Serialization.HLBuffer, Godot.GodotObject, Godot.GodotObject> type was defined.

MyDict[someKey] = Callable.From((WorldRunner currentWorld, NetPeer peer, HLBuffer buffer, GodotObject initialObject) => method.Invoke(null, [currentWorld, peer, buffer, initialObject]) as GodotObject);

The exact details of that code doesn’t really matter. The important thing to note is that Callable was being saved in a Dictionary of a [Tool]. The method variable is actually a C# MethodInfo type, which allows me to call a static method on C# types via reflection, without knowing the exact class in advance.

I went to Claude (not a sponsor) and asked it why this might cause assembly unload failure. It explained in that sweet, slow, and patient way (perfect for a dumb dumb like me to understand) that the type was being stored in a dictionary, and it is a strong handle, so all references have to be cleared in order to dispose the object.

If you don’t already know, GodotObjects all implement IDisposable which means that you just need to put your cleanup tasks in there:

protected override void Dispose(bool disposing)
{
	MyDict.Clear();
	base.Dispose(disposing);
}

(I just put that function on the Node that had MyDict and the above Callable assignment code.)

Yup, that’s it! The error completely disappeared!

Surely it won’t come back in some other form in the future…

The End…?

(Ominous outro music)