Subject: LoadLibrary fails: How to debug in this situation
From: [email-masked] (root)

30 Mar 1997 02:46:53 -0800
Sender: [email-masked]
Approved: [email-masked]
Distribution: cygnus
Message-ID: <[email-masked]>
Original-To: [email-masked]
Content-Type: text
Original-Sender: [email-masked]

This is a general receipt for users that want to debug a dll that will
not load with "LoadLibrary". I thought that this could be useful for some
people in the list. Matt Calder asked explicitely about it.

If you do not need any technical info about this, just skip this message.

Explanation of symptoms:
-----------------------

If the dll doesn't load, this means that one of the LibMains of that dll
returned 'False' or crashed.

This is not obvious to many people: For instance, when you make a 
LoadLibrary of some dll you built using cygnus.dll runtime you are
executing:
1. The LibMain of all dlls that are used by cygnus.dll
1. The LibMain of cygnus.dll
2. The LibMain of yourdll.dll

The LoadLibrary call will return TRUE if ALL of those LibMain's suceed. If
any of those fails, it will return FALSE.

Receipt for debugging in this situation:
---------------------------------------
Now, a FALSE return value from 'Load Library' is not very useful. Try to
get more information with 'GetLastError' first, to avoid trivial errors like
for instance 'Path not found', that would make you waste time when the
error is just in some typo at the name you pass to LoadLibrary.

If that is not the case then do:

1) Compile the cygnus dll in your system with debug info.
   In the cygnus system compile with -g turned on, and all optimizations off.
   This is surely not a trivial task, but explaining this is beyond the scope
   of this answer. Look at the FAQs etc. It can be done, but since I do not
   use cygnus's tools I have never done it.
   Maybe cygnus.dll has already the debug info in there. In that case skip 
   this.

2) At the startup of cygnus's LibMain write as the very first instruction:
	_asm( int $3);  // or whatever the syntax for putting an assembler
                        // instruction is under gcc.
   It is fundamental that this interrupt be planted at the entry point
   of the dll, that could be different from the declared entry point in C.

  To disambiguate this, dump the executable of the dll with 'objdump'
  of cygnus or an equivalent tool, and find the entry point.Disassemble the
  code around that entry point and verify that an 'int 3' instruction is
  there.

  The rationale of this point is: Debugging the startup of a dll is very
  difficult. I do not think that something like gdb is up to the task.
  This interrupt FORCES the system to call the debugger at the startup of
  the dll. You should be sure that either:

  gdb is declared in your registry as the default debugger.
  gdb is already debugging this process.

To ensure that gdb is loaded you could make a tiny executable that the only
thing it does is to call 'LoadLibrary'. The load that with gdb.

  Be careful never to load this modified dll if there is no
  debugger present. Personally I have always avoided things like 'gdb', so
  I do not remember if it can be told to debug a dll startup. In case that
  is possible, all this is unnecessary.

If gdb works as a good debugger, it should intercept the interrupt you
planted in the LibMain, and you can follow the initialization of the dll 
and see why it fails. 

In case gdb fails, you can use MSVC's debugger, and follow the program under 
disassembly window. This is not a very interesting perspective...

The problem with cygnus tools is that they are incompatible with all other
programs running under windows, so that they can't be debugged with standard 
tools. This leads them into an isolated corner...

Good luck!

-- 
Jacob Navia	Logiciels/Informatique
41 rue Maurice Ravel			Tel 01 48.23.51.44
93430 Villetaneuse 			Fax 01 48.23.95.39
France
-
For help on using this list, send a message to
"[email-masked]" with one line of text: "help".