X-Authentication-Warning: delorie.com: mail set sender to djgpp-bounces using -f X-Recipient: djgpp AT delorie DOT com Message-ID: <561101C4.7040701@gmx.de> Date: Sun, 04 Oct 2015 12:39:00 +0200 From: "Juan Manuel Guerrero (juan DOT guerrero AT gmx DOT de) [via djgpp AT delorie DOT com]" User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; de; rv:1.9.2.13) Gecko/20101206 SUSE/3.1.7 Thunderbird/3.1.7 MIME-Version: 1.0 To: djgpp AT delorie DOT com Subject: Re: dlclose not removing dependency dxes References:

<952a68b4-223b-4222-b456-35514bb8b7eb AT googlegroups DOT com> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Provags-ID: V03:K0:7PK+6B3NkAeSWVyg7lp59EigiiK3QaqVt+KaHOil7boISh/sVvQ UT7LAmbThS5B2SzqBmy6P9DfiM0QHnKSbT9ku4dJvwy9hO9uVSe7zj/yndeOTjM/MLKARjg 0lECK09CbbXEptARE+yEIs34+VtOjhVxuWJ2wqEDIz8TRyxCjEQsnXAKT+c+I24P1LRVJAv MjXOkinMe4je+A0GjW+hg== X-UI-Out-Filterresults: notjunk:1;V01:K0:GFYFzkRnyng=:CreYK9nOyYl2onVA9l1Q3E N69njXEQ/meFm+lbd8ygFUHok3iBL73t7D2Ifwdn2PrWmVKWtKrXUA3YjaTLk3ddK+nRM3Oul 8U0XgMZ9Z0PD0tDJMp3wFXASOBJcIzeUBmY94c/8AVRiUV892bOSrtTqcnl8L97cZMdlgQDmu Ev7vdTrLYzEYx6SPdMNzsMbjNT9drjVLSItaUxeTDACEsuw/C08yt/8p2+GL+3ish/YDASL/B akGhEPwK1XZGvmOO9I9XC45NWt4klzRZh6pZ4+fivPUzWnOiJI5m0MIjfijG+qLACWh9tE+07 XgnQmXLClEBl8FhAXBvbsVagjm56dO7FE1fF5xUr0kNx+vSUea8z14/zaoQKpFLgyGud5m7Yc t5pm6F2xzEirT4Kj8mgd8OkEC9HgZSNZZiR1TR0R+hUCBKtO4RsIzB+daIScRHSpos/KQChne 47SRdNXjeh53LFcf4YEb1sCaLTaHx+2rVyaaJBJXqytnXPzHvbs31MU7ywCMIJy7mwJpkATnN F1Js1FktTunDIEdKqabgE3wzVk6WBUGZPgaUk3F0iCgCWl7GNtIE3izyUAmd2kMpIC45eJBK4 sDl6OroNry1p02XVmioECfUTNrWCl+7K2GwwEAp9D+Poy94lSNoWZN4y5VobOwBLuy6XAmHIr XXDB1Ch7CuteAGIejWgWjPkMn8Dn3jxyPWF/cDGlBto7ztv7c/TY7MOmm5WKewxJa37qXg8hT umuEOFvc+FyyHoUnJ4NC+Pqf2DReSiQXeIBhuc+RxtA9/hpFKTCIa76JIiBrCQkdL8C2sAoGi K301WAu Reply-To: djgpp AT delorie DOT com Am 01.10.2015 15:56, schrieb Ozkan Sezer (sezeroz AT gmail DOT com) [via djgpp AT delorie DOT com]: > On 10/1/15, Juan Manuel Guerrero (juan DOT guerrero AT gmx DOT de) [via > djgpp AT delorie DOT com] wrote: >> On Thursday, October 1, 2015 at 9:35:24 AM UTC+2, Ozkan Sezer >> (sezeroz AT gmail DOT com) [via djgpp AT delorie DOT com] wrote: >>> On 9/27/15, Ozkan Sezer wrote: >>>> On 9/26/15, Juan Manuel Guerrero (juan DOT guerrero AT gmx DOT de) [via >>>> djgpp AT delorie DOT com] wrote: >>>>> On Friday, September 25, 2015 at 7:21:47 PM UTC+2, Ozkan Sezer >>>>> (sezeroz AT gmail DOT com) [via djgpp AT delorie DOT com] wrote: >>>>>> On 9/25/15, Ozkan Sezer wrote: >>>>>>> AFAICS, dlclose()ing of a dxe doesn't remove its dependency dxes >>>>>>> along >>>>>>> with it, which will result in unnecessarily occupied memory which >>>>>>> may >>>>>>> prove fatal upon multiple dlopen()/dlclose() of a dxe with deps. >>>>>>> This >>>>>>> needs addressing. >>>>>>> >>>>>> >>>>>> My last argument was inaccurate and misleading. Here's better: >>>>>> >>>>>> One has a.dxe and b.dxe; a.dxe depends on b.dxe. Do dlopen a.dxe >>>>>> and b.dxe is opened implicitly. Do dlcose a.dxe, and b.dxe stays >>>>>> open still occupying its memory. The memory occupied by the unused >>>>>> b.dxe might be needed by the app but will be unavailable to it. >>>>>> Further dlopen calls for a.dxe will increment the refcount for b.dxe >>>>>> which never gets decremented. >>>>> >>>>> >>>>> Can you provide a minimal sample code to demonstrate the issue? >>>>> >>>>> Regards, >>>>> Juan M. Guerrero >>>>> >>>> >>>> ======================================= >>>> >>>> $ cat 12.c >>>> extern int my_func2(); >>>> int my_func1() { >>>> return my_func2(); >>>> } >>>> $ gcc -c 12.c >>>> $ dxe3gen -o a.dxe -P b.dxe -U 12.o >>>> $ dxe3gen --show-exp a.dxe >>>> _my_func1 >>>> $ dxe3gen --show-dep a.dxe >>>> b.dxe >>>> >>>> ======================================= >>>> >>>> $ cat 13.c >>>> int my_func2() { >>>> return -1; >>>> } >>>> $ gcc -c 13.c >>>> $ dxe3gen -o b.dxe 13.o >>>> $ dxe3gen --show-exp b.dxe >>>> _my_func2 >>>> >>>> ======================================= >>>> >>>> $ cat 11.c >>>> #include >>>> #include >>>> >>>> void *my_dxe1; >>>> int (*my_f)(); >>>> >>>> int main (void) { >>>> my_dxe1 = dlopen("a.dxe",0); >>>> if (!my_dxe1) { >>>> printf("dlopen() failed\n"); >>>> return 1; >>>> } >>>> dlclose(my_dxe1); >>>> >>>> my_f = dlsym(RTLD_DEFAULT,"_my_func1"); >>>> if(!my_f) { >>>> printf("a.dxe not loaded\n"); >>>> } >>>> else { >>>> printf("a.dxe:my_func1(): %p\n",my_f); >>>> } >>>> >>>> my_f = dlsym(RTLD_DEFAULT,"_my_func2"); >>>> if(!my_f) { >>>> printf("b.dxe not loaded\n"); >>>> } >>>> else { >>>> printf("b.dxe:my_func2(): %p\n",my_f); >>>> printf("b.dxe:my_func2() returns: %d\n",my_f()); >>>> } >>>> >>>> return 0; >>>> } >>>> >>>> $ gcc -Wall -W 11.c >>>> >>>> ======================================= >>>> >>>> $ a.exe> a.log >>>> $ cat a.log >>>> a.dxe not loaded >>>> b.dxe:my_func2(): 95830 >>>> b.dxe:my_func2() returns: -1 >>>> >>> >>> PING? Any ideas? (Sorry that I brought the issue >>> but have no patch for it myself.) >> >> >> An inspection of your sample code and dldemo.cpp shows that it seems to be >> that it is the user's responsibility to explicitly dlclose every dlopen'ed >> modules. > > The thing is, the dependency dxe, i.e. b.dxe in my example, > is implicitly dlopen()ed and I won't ever have the handle for it. > >> This is what an inspection of dldemo.cpp shows. >> This may not be what you expected from unix but I assume that no DJGPP >> developer has really promised that the behaviour would be identical to unix >> behaviour. >> Neither less I will inspect the issue during week-end but I do not promise >> that a fix will be possible. >> > > Thanks. > > -- > O.S. Below is a patch that will implement ref-counting for implicitly loaded modules by other ones. Please not that this ref-counting only works for dlopen/dlclose. There may be other loading mechanisms that suffer from the same issue and that will not be fixed by this modification. The patch is intended for the main branch but should work on v2_05_1 as well. Regards, Juan M. Guerrero 2015-10-03 Juan Manuel Guerrero * djgpp/src/libc/dxe/dlopen.c: Added reference counting support for implicitly loaded DXE modules by dlopen. diff -aprNU5 djgpp.orig/src/libc/dxe/dlopen.c djgpp/src/libc/dxe/dlopen.c --- djgpp.orig/src/libc/dxe/dlopen.c 2015-10-03 15:18:20 -29907 +++ djgpp/src/libc/dxe/dlopen.c 2015-10-04 11:40:00 -29907 @@ -36,14 +36,16 @@ #ifndef ELOOP #define ELOOP EMLINK #endif +#define __DEBUG_DXE3__ 0 /* Prints struct __dxe_handle members. Always commited as 0. */ + /* private stack */ -typedef struct stk_node { +typedef struct __stk_node { const char *name; - struct stk_node *next; + struct __stk_node *next; } stk_node; /* Exported symbols table entry */ typedef struct { @@ -57,22 +59,29 @@ typedef struct unsigned short n_rel_relocs; unsigned short n_abs_relocs; char name [1]; /* expanded as needed */ } __attribute__((packed)) unres_table_entry; +/* This is a linked list of implizitly loaded dxe modules. */ +typedef struct __linked_list { + struct __dxe_handle *handle; /* Pointer to next implicitly open module by this module */ + struct __linked_list *next; /* Next entry in list */ +} dxe_handle_list; + /* This is the private dxe_h structure */ typedef struct __dxe_handle { - struct __dxe_handle *next; /* Pointer to next module in chain */ - char fname[FILENAME_MAX]; /* Full module pathname */ - int mode; /* Module open mode */ - int inuse; /* In-use counter */ - int n_exp_syms; /* Number of entries in export table */ - exp_table_entry **exp_table; /* Exported symbols table */ - char *header; /* The resident portion of header */ - char *section; /* code+data+bss section */ - long _fini; /* finalization */ + struct __dxe_handle *next; /* Pointer to next module in chain */ + char fname[FILENAME_MAX]; /* Full module pathname */ + int mode; /* Module open mode */ + int in_use; /* In-use counter */ + int n_exp_syms; /* Number of entries in export table */ + exp_table_entry **exp_table; /* Exported symbols table */ + char *header; /* The resident portion of header */ + char *section; /* code+data+bss section */ + long _fini; /* Finalization */ + dxe_handle_list *implicit_handle_list; /* Linked list of implicitly open module by this module or NULL */ } dxe_handle, *dxe_h; /* Last-resort symbol resolver */ void *(*_dlsymresolver) (const char *symname) = NULL; /* Last-error unresolved symbol count */ @@ -110,10 +119,12 @@ void *dlopen(const char *filename, int m char *discardable; stk_node *node; static stk_node *stk_top = NULL; + dxe_handle_list *implicitly_open_handle_list; + #ifndef __GNUC__ static int cleanup = 0; #endif _dl_unresolved_count = 0; @@ -180,11 +191,11 @@ found: /* Look through the loaded modules list */ for (cur = dxe_chain; cur; cur = cur->next) if (!strcmp(realfn, cur->fname)) { - cur->inuse++; + cur->in_use++; return cur; } fh = OPEN(filename, OPENFLAGS); if (fh < 0) return NULL; @@ -202,13 +213,14 @@ found: return NULL; } /* O.k, fill the module handle structure */ strcpy(dxe.fname, realfn); - dxe.inuse = 1; + dxe.in_use = 1; dxe.mode = mode; dxe.n_exp_syms = dxehdr.n_exp_syms; + dxe.implicit_handle_list = NULL; /* Read DXE tables and the data section */ hdrsize = dxehdr.symbol_offset - sizeof(dxehdr); discardsize = dxehdr.dep_size + dxehdr.unres_size + dxehdr.nrelocs * sizeof(long); if ((dxe.header = malloc(hdrsize + dxehdr.sec_size)) == NULL) @@ -240,11 +252,11 @@ found: /* Fill the unfilled portion of code+data+bss segment with zeros */ memset(dxe.section + dxehdr.sec_f_size, 0, dxehdr.sec_size - dxehdr.sec_f_size); /* Load the dependencies */ scan = discardable; - for (i = 0; i < dxehdr.n_deps; i++) + for (implicitly_open_handle_list = NULL, i = 0; i < dxehdr.n_deps; i++) { stk_node tmp; tmp.name = realfn; tmp.next = stk_top; stk_top = &tmp; @@ -252,14 +264,35 @@ found: if (dlopen(scan, RTLD_GLOBAL) == NULL) { stk_top = tmp.next; goto unrecoverable; } + else + { + dxe_handle_list *next_handle_list; + + stk_top = tmp.next; - stk_top = tmp.next; + scan = strchr(scan, 0) + 1; - scan = strchr(scan, 0) + 1; + /* Register all implicitly open modules by this one. */ + if ((next_handle_list = malloc(sizeof(dxe_handle_list))) == NULL) + { + errno = ENOMEM; + goto midwayerror; + } + next_handle->handle = dxe_chain; + next_handle->next = NULL; + + if (implicitly_open_handle_list) + { + implicitly_open_handle_list->next = next_handle_list; + implicitly_open_handle_list = implicitly_open_handle_list->next; + } + else + dxe.implicit_handle_list = implicitly_open_handle_list = next_handle_list; + } } /* Allright, now we're ready to resolve all unresolved symbols */ _dl_unresolved_count = dxehdr.n_unres_syms; _dl_unresolved_symbol[0] = 0; @@ -325,10 +358,50 @@ found: errno = ENOMEM; goto midwayerror; } memcpy(dxe_chain, &dxe, sizeof(dxe_handle)); +#if defined(__DEBUG_DXE3__) && __DEBUG_DXE3__ == 1 + { + FILE *f = fopen("c:/tmp/dxe_chain.txt", "a"); + + if (f) + { + fprintf(f, "dxe_chain : 0x%p\n" + " next : 0x%p\n" + " fname : %s\n" + " mode : %s\n" + " in_use : %d\n" + " n_exp_syms : %d\n" + " exp_table : 0x%p\n", + dxe_chain, dxe_chain->next, dxe_chain->fname, + dxe_chain->mode == RTLD_LAZY ? "RTLD_LAZY" : + dxe_chain->mode == RTLD_NOW ? "RTLD_NOW" : + dxe_chain->mode == RTLD_LOCAL ? "RTLD_LOCAL" : + dxe_chain->mode == RTLD_GLOBAL ? "RTLD_GLOBAL" : "unknown", + dxe_chain->in_use, dxe_chain->n_exp_syms, dxe_chain->exp_table); + for (i = 0; i < dxe_chain->n_exp_syms; i++) + fprintf(f, " exp_table[%d]->offset : %ld\n" + " exp_table[%d]->name : %s\n", + i, dxe_chain->exp_table[i]->offset, i, dxe_chain->exp_table[i]->name); + fprintf(f, " header : 0x%p\n" + " section : 0x%p\n" + " _fini : %ld\n", + dxe_chain->header, dxe_chain->section, dxe_chain->_fini); + if ((implicitly_open_handle_list = dxe_chain->implicit_handle_list)) + for (; implicitly_open_handle_list; implicitly_open_handle_list = implicitly_open_handle_list->next) + fprintf(f, " implicitly_open_handle_list: 0x%p\n" + " handle : 0x%p\n" + " handle->fname : %s\n\n", + implicitly_open_handle_list, implicitly_open_handle_list->handle, implicitly_open_handle_list->handle->fname); + else + fprintf(f, " implicitly_open_handle_list: 0x00000000\n\n"); + fclose(f); + } + } +#endif + #ifndef __GNUC__ if (!cleanup) { cleanup = !0; atexit(_closeall); @@ -348,11 +421,11 @@ midwayerror: int dlclose(void *dxe) { if (!dxe) return -1; - if (--((dxe_h)dxe)->inuse) + if (--((dxe_h)dxe)->in_use) return 0; /* finalization */ if (((dxe_h)dxe)->_fini != -1) ((void (*) (void))(((dxe_h)dxe)->_fini + (long)((dxe_h)dxe)->section))(); @@ -361,10 +434,21 @@ int dlclose(void *dxe) { dxe_h *cur; for (cur = &dxe_chain; *cur; cur = &(*cur)->next) if (*cur == dxe) { + /* Un-register all implicitly loaded modules by this module. */ + dxe_handle_list *implicitly_open_handle_list = (*cur)->implicit_handle_list; + while (implicitly_open_handle_list) + { + dxe_handle_list *next_handle_list = implicitly_open_handle->next; + + dlclose(implicitly_open_handle_list->handle); + free(implicitly_open_handle_list); + implicitly_open_handle_list = next_handle_list; + } + *cur = ((dxe_h)dxe)->next; break; } }