experchange > fortran

steve kargl (01-18-19, 05:37 AM)
gah4 wrote:

[..]
> random_seed_i4 or random_seed_i8, and I was looking for one
> actually called random_seed. (Assuming the above were
> called from somewhere else.)


There is name mangling that occurs. If you call random_seed
with an integer(4) (yes, I know 4 is non-standard), the frontend
generates a call to random_seed_i4.

> So, yes, even if someone understands C, it isn't so easy
> to actually find in the source.


Assuming you have gcc code,

find gcc/libgfortran -name \*.c | xargs grep random_seed

> As for the actual question, one might assume that asking
> for the SIZE, doesn't do anything else, such as initialize
> the generator. Not that I think there is anything wrong
> with doing it, but it isn't so obvious.


It has to be allocated at some point. If you simply do

program foo
call random_number(x)
end program foo

gfortran sees a NULL pointer for the internal state,
and does the required allocation and initialization of
state.

If you do

program foo
call random_seed()
end program foo

gfortran sees a NULL pointer for the internal state,
and does the required allocation and initialization of
state.

If you do

program foo
integer, allocatable :: seeds
integer n
call random_seed(size=n) ! allocate and initialize state
allocate(seeds(n))
call random_seed(get=seeds) ! only copies out internal states
end program foo
gah4 (01-18-19, 06:32 AM)
On Thursday, January 17, 2019 at 7:37:19 PM UTC-8, steve kargl wrote:

(snip, I wrote)

> > OK, I found it now. It seems that they are called either
> > random_seed_i4 or random_seed_i8, and I was looking for one
> > actually called random_seed. (Assuming the above were
> > called from somewhere else.)


> There is name mangling that occurs. If you call random_seed
> with an integer(4) (yes, I know 4 is non-standard), the frontend
> generates a call to random_seed_i4.


Obvious once you know it.

I think is was slightly more expecting the first routine
to be Fortran, and that would call the C routines. Even though
I know about name mangling, I wasn't looking for it.

> > So, yes, even if someone understands C, it isn't so easy
> > to actually find in the source.


> Assuming you have gcc code,


> find gcc/libgfortran -name \*.c | xargs grep random_seed


As I noted, they are in github, and can be found individually
without downloading them. But it helps to know the directory
structure.

> > As for the actual question, one might assume that asking
> > for the SIZE, doesn't do anything else, such as initialize
> > the generator. Not that I think there is anything wrong
> > with doing it, but it isn't so obvious.


> It has to be allocated at some point. If you simply do


> program foo
> call random_number(x)
> end program foo


> gfortran sees a NULL pointer for the internal state,
> and does the required allocation and initialization of
> state.


Yes.

Presumably programs that use RANDOM_SEED(SIZE=..)
and no other random calls are rare, so no need to
optimize that case. But it is interesting to know.
steve kargl (01-18-19, 07:21 AM)
gah4 wrote:

> On Thursday, January 17, 2019 at 7:37:19 PM UTC-8, steve kargl wrote:
> (snip, I wrote)
> Obvious once you know it.
> I think is was slightly more expecting the first routine
> to be Fortran, and that would call the C routines. Even though
> I know about name mangling, I wasn't looking for it.


Why incur unnecessary function call overhead. Almost all gfortran
routines (and I suspect most compilers) do not do an indirect call
sequence. Note, this isn't name mangling in the sense of
C++ name mangling. This is more along the lines of setting
up the function call to the correct intrinsic subprogram.

> As I noted, they are in github, and can be found individually
> without downloading them. But it helps to know the directory
> structure.
> Yes.
> Presumably programs that use RANDOM_SEED(SIZE=..)
> and no other random calls are rare, so no need to
> optimize that case. But it is interesting to know.


Optimization has nothing do to here. The allocation
occurs exactly once. The Fortran standard allows all of the
above programs, so random_seed and random_number
needs to be prepared to set up internal state under all
circumstances.
Ron Shepard (01-18-19, 09:18 AM)
On 1/17/19 7:47 PM, steve kargl wrote:
> The source code for random_seed() is available for anyone to read.
> There is a leak to the extent that if your installed gfortran can use
> pthreads, then the first call to random_seed() will cause it to allocate
> memory for the internal state of the PRNG. As this is the global state
> of the PRNG, it is released when the program exits. If your gfortran
> is installed where pthreads is not available, there is no leak as the
> internal is stored in static memory.


Does the allocation of memory for the internal state (in the case that
pthreads is active) count as "lost" memory to valgrind? It seems to
report "definitely" and "possibly" lost memory, depending on how it is
started up. So far, I am focusing on the "defintetly lost" row of output.

$.02 -Ron Shepard
gah4 (01-19-19, 03:25 AM)
On Thursday, January 17, 2019 at 9:21:42 PM UTC-8, steve kargl wrote:

(snip)

> > Presumably programs that use RANDOM_SEED(SIZE=..)
> > and no other random calls are rare, so no need to
> > optimize that case. But it is interesting to know.


> Optimization has nothing do to here. The allocation
> occurs exactly once. The Fortran standard allows all of the
> above programs, so random_seed and random_number
> needs to be prepared to set up internal state under all
> circumstances.


But note that the OP's program doesn't generate any
random numbers, and doesn't even extract a seed.

As written, the program doesn't actually need a seed!

What I was suggesting, is that there is no need to optimize
for this rare case of not needing a seed.

Now, there might be other routines that allow one to determine
a SIZE, and that also might be used in a similar way.
steve kargl (01-19-19, 06:09 AM)
gah4 wrote:

> On Thursday, January 17, 2019 at 9:21:42 PM UTC-8, steve kargl wrote:
> (snip)
> But note that the OP's program doesn't generate any
> random numbers, and doesn't even extract a seed.


I really should have ignored this thread, Sigh.

> As written, the program doesn't actually need a seed!


So? What real program calls random_seed() with size and
never calls it again? Really?

> What I was suggesting, is that there is no need to optimize
> for this rare case of not needing a seed.


Argh, that's exactly what I showed.

> Now, there might be other routines that allow one to determine
> a SIZE, and that also might be used in a similar way.


And the sky might be blue.
Ron Shepard (01-19-19, 09:23 AM)
On 1/18/19 10:09 PM, steve kargl wrote:
> gah4 wrote:
> I really should have ignored this thread, Sigh.
> So? What real program calls random_seed() with size and
> never calls it again? Really?
> Argh, that's exactly what I showed.
> And the sky might be blue.


What is it that is being allocated in that call, and why does it occur
on MacOS but not, for example, Linux? Or is it a feature of valgrind on
MacOS and not of gfortran on MacOS?

$.02 -Ron Shepard
spectrum (01-19-19, 01:46 PM)
Because Steve (Kargl) kindly told us the source to look at (with additional explanations
about the codes!), maybe we can try reading it first and then ask
more specific questions (if necessary)...

And so, here is my poor attempt to understand where 136 bytes come from
(which may be wrong, of course :)

-----
Starting from this code (which is suggested above)


we can find "random_seed_i4" here


which looks like this (just some excerpt, please see the original for details)

void random_seed_i4 (GFC_INTEGER_4 *size, gfc_array_i4 *put, gfc_array_i4 *get)
{
...
/* Check that we only have one argument present. */
if ((size ? 1 : 0) + (put ? 1 : 0) + (get ? 1 : 0) > 1)
runtime_error ("RANDOM_SEED should have at most one argument present.");

// If size is requested, set the value.
if (size != NULL) ...

// Now get the internal state.
xorshift1024star_state* rs = get_rand_state(); // Line 1

if (get != NULL)
{
// code for GET
}
else
{
// code for PUT
}
...
}

We can find get_rand_state() used in Line 1 somewhere above in the code:



static xorshift1024star_state*
get_rand_state (void)
{
/* For single threaded apps. */
static xorshift1024star_state rand_state;

if (__gthread_active_p ())
{
void* p = __gthread_getspecific (rand_state_key);
if (!p)
{
p = xcalloc (1, sizeof (xorshift1024star_state)); // Line 2
__gthread_setspecific (rand_state_key, p);
}
return p;
}
else
return &rand_state;
}

and the definition of xorshift1024star_state is given just above:

typedef struct
{
bool init;
int p;
uint64_t s[16];
}
xorshift1024star_state;

So we see in Line 2

p = xcalloc (1, sizeof (xorshift1024star_state));

which seems to be allocating fresh memory for an internal state.
And an interesting part is whether the size of xorshift1024star_state agrees with
136 bytes, which seems the case if the size of bool is 4 bytes (??)

136 bytes = 8 bytes * 16 (for s[16]) + 4 bytes (for p) and 4 bytes (for init)

But I'm confused because I get 1 for sizeof(bool) (in stdbool.h) with gcc-8 on my Mac.
Possibly, 4 bytes due to so-called "padding" in the struct...?
Ev. Drikos (01-19-19, 03:55 PM)
On 19/01/2019 1:46 PM, spectrum wrote:
> Because Steve (Kargl) kindly told us the source to look at (with additional explanations
> about the codes!), maybe we can try reading it first and then ask
> more specific questions (if necessary)...


As it seems, the mac specific issue is that the memory allocated via
"xcalloc" isn't freed when the program exits.

The following patch (a quick & dirty hack) that frees one pointer makes
me feel (possibly wrong) that we aren't facing a Valgrind issue
(although Valgrind feels more comfortable in Linux rather in a Mac):

$ cat gcc7-get-rand.patch
--- libgfortran/intrinsics/random.c 2019-01-19 10:45:20.000000000 +0200
+++ libgfortran/intrinsics/random.c 2019-01-19 10:48:18.000000000 +0200
@@ -38,6 +38,8 @@
#include <fcntl.h>
#include "time_1.h"

+#include <stdlib.h>
+
#ifdef __MINGW32__
#define HAVE_GETPID 1
#include <process.h>
@@ -206,6 +208,16 @@
};

+#ifdef __APPLE__
+static void* g_p = NULL;
+static void mac_cleaner(){
+ if ( g_p ) {
+ free( g_p ) ;
+ g_p = NULL;
+ }
+}
+#endif
+
static __gthread_key_t rand_state_key;

static xorshift1024star_state*
@@ -221,6 +233,10 @@
{
p = xcalloc (1, sizeof (xorshift1024star_state));
__gthread_setspecific (rand_state_key, p);
+#ifdef __APPLE__
+ g_p = p ;
+ atexit(mac_cleaner);
+#endif
}
return p;
}

macbook:tars suser$
FortranFan (01-19-19, 06:11 PM)
On Saturday, January 19, 2019 at 6:46:44 AM UTC-5, spectrum wrote:

[..]
> 136 bytes, which seems the case if the size of bool is 4 bytes (??)
> 136 bytes = 8 bytes * 16 (for s[16]) + 4 bytes (for p) and 4 bytes (for init)
> ..


On Saturday, January 19, 2019 at 8:55:20 AM UTC-5, Ev. Drikos wrote:

> ..
> As it seems, the mac specific issue is that the memory allocated via
> "xcalloc" isn't freed when the program exits.
> ..


@spectrum, @Ev. Drikos,

Great analysis, it's so much more beneficial for readers when you start working with actual code and tools and with your thinking hats on than when discussion threads meander "in vacuo" as they tend do around here.

By the way, the issue ain't limited to MAC OS, it is seen on Windows as well - see below. It may help with someone can retry OP's case on Linux with Dr. Memory analysis tool?


--- begin console output on Windows ---

C:\temp>type p.f90
program random_test
integer :: seed_size
intrinsic :: random_seed
call random_seed( size=seed_size )
write(*,*) 'seed_size=', seed_size
end program random_test
C:\temp>gfortran p.f90 -o p.exe

C:\temp>"C:\Program Files (x86)\Dr. Memory\bin\drmemory.exe" p.exe -show_reachable
~~Dr.M~~ Dr. Memory version 1.11.0
~~Dr.M~~ (Uninitialized read checking is not yet supported for 64-bit)
~~Dr.M~~ Running "p.exe -show_reachable"
seed_size= 33
~~Dr.M~~
~~Dr.M~~ Error #1: LEAK 136 direct bytes 0x0000000003641d60-0x0000000003641de8 + 0 indirect bytes
~~Dr.M~~ # 0 replace_calloc [d:\drmemory_package\common\alloc_replace.c:2616]
~~Dr.M~~ # 1 ntdll.dll!RtlRestoreLastWin32Error +0x3f (0x00007ff9cfbf83d0 <ntdll.dll+0x83d0>)
~~Dr.M~~ # 2 _gfortrani_xcalloc [../../../gcc-9-20180805-mingw/libgfortran/runtime/memory.c:83]
~~Dr.M~~ # 3 _gfortran_random_seed_i4 [../../../gcc-9-20180805-mingw/libgfortran/intrinsics/random.c:223]
~~Dr.M~~ # 4 MAIN__
~~Dr.M~~ # 5 insert [../../../gcc-9-20180805-mingw/libgfortran/io/unit.c:206]
~~Dr.M~~ # 6 _gfortrani_fbuf_init [../../../gcc-9-20180805-mingw/libgfortran/io/fbuf.c:42]
~~Dr.M~~ # 7 msvcrt.dll!getptd_noexit
~~Dr.M~~ # 8 msvcrt.dll!signal
~~Dr.M~~ # 9 _gfortran_set_options [../../../gcc-9-20180805-mingw/libgfortran/runtime/compile_options.c:183]
~~Dr.M~~ #10 main
~~Dr.M~~
~~Dr.M~~ ERRORS FOUND:
~~Dr.M~~ 0 unique, 0 total unaddressable access(es)
~~Dr.M~~ 0 unique, 0 total invalid heap argument(s)
~~Dr.M~~ 0 unique, 0 total GDI usage error(s)
~~Dr.M~~ 0 unique, 0 total handle leak(s)
~~Dr.M~~ 0 unique, 0 total warning(s)
~~Dr.M~~ 1 unique, 1 total, 136 byte(s) of leak(s)
~~Dr.M~~ 0 unique, 0 total, 0 byte(s) of possible leak(s)
~~Dr.M~~ ERRORS IGNORED:
~~Dr.M~~ 3 potential error(s) (suspected false positives)
~~Dr.M~~ 4 unique, 5 total, 444 byte(s) of still-reachable allocation(s)
~~Dr.M~~ (re-run with "-show_reachable" for details)

C:\temp>
--- end output ---
spectrum (01-20-19, 12:00 AM)
On Sunday, January 20, 2019 at 1:11:13 AM UTC+9, FortranFan wrote:

> By the way, the issue ain't limited to MAC OS, it is seen on Windows as well - see

below. It may help with someone can retry OP's case on Linux with Dr. Memory analysis
tool?
>


I've just tried "Dr.Memory", and it seems very nice (though the Mac version
has some restrictions as compared to Linux).

Downloading Mac and Linux binaries from here


and following the explanations here,



I've tried this command on Mac (unfortunately, the Mac version seems to
allow only 32-bit compilation)

$ gfortran-8 -m32 -g -fno-inline -fno-omit-frame-pointer test.f90
$ {install-dir}/bin/drmemory -- ./a.out

Then I get an output slightly different from Windows:

~~Dr.M~~ Error #1: LEAK 136 direct bytes 0x0093a8a0-0x0093a928 + 0 indirect bytes
~~Dr.M~~ # 0 replace_calloc [/drmemory_package/common/alloc_replace.c:2616]
~~Dr.M~~ # 1 libgfortran.5.dylib!_gfortrani_xcalloc
~~Dr.M~~ # 2 libgfortran.5.dylib!get_rand_state
~~Dr.M~~ # 3 libgfortran.5.dylib!_gfortran_random_seed_i4
~~Dr.M~~ # 4 libgfortran.5.dylib!_gfortran_random_seed_i4

Though this doesn't show the file location in detail, it shows the name of
functions in a bit more detail, e.g. "# 2 ... get_rand_state".

On the other hand, everything works fine (no memory leaks at all) on Linux
(Ubuntu18.04LTS + gfortran-8.2), even for a modified code with threading (-fopenmp).
spectrum (01-20-19, 12:09 AM)
On Saturday, January 19, 2019 at 10:55:20 PM UTC+9, Ev. Drikos wrote:
> atexit(mac_cleaner);


Mac cleaner XD (a nice name... :-)
Ev. Drikos (01-20-19, 09:29 AM)
On 19/01/2019 3:55 PM, Ev. Drikos wrote:
> ...
> The following patch (a quick & dirty hack) that frees one pointer makes
> me feel (possibly wrong) that we aren't facing a Valgrind issue
> (although Valgrind feels more comfortable in Linux rather in a Mac):
> ...


Perhaps, a mac specific solution that seems to be functional and
extensible (ie to MINGW or whatever) could be something like:

$ cat /Users/suser/pc/rules/patches/gcc7/gcc7-get-rand.patch
--- libgfortran/intrinsics/random.c 2019-01-19 19:32:22.000000000 +0200
+++ libgfortran/intrinsics/random.c 2019-01-19 20:34:46.000000000 +0200
@@ -208,6 +208,10 @@

static __gthread_key_t rand_state_key;

+#ifdef __APPLE__
+static bool g_p=false;
+#endif
+
static xorshift1024star_state*
get_rand_state (void)
{
@@ -221,6 +225,9 @@
{
p = xcalloc (1, sizeof (xorshift1024star_state));
__gthread_setspecific (rand_state_key, p);
+#ifdef __APPLE__
+ g_p = true ;
+#endif
}
return p;
}
@@ -944,7 +951,20 @@
static void __attribute__((destructor))
destructor_random (void)
{
+#ifdef __APPLE__
+ if (__gthread_active_p ())
+ {
+ void* p = NULL;
+ if (g_p)
+ p = __gthread_getspecific (rand_state_key);
+ __gthread_key_delete (rand_state_key);
+ if (p)
+ free(p);
+ }
+#else
if (__gthread_active_p ())
__gthread_key_delete (rand_state_key);
+#endif
+
}
#endif
macbook:pc suser$
Ev. Drikos (03-11-19, 06:23 PM)
On 20/01/2019 12:00 AM, spectrum wrote:
> ...
> I've just tried "Dr.Memory", and it seems very nice (though the Mac version
> has some restrictions as compared to Linux).
> ...
> Though this doesn't show the file location in detail, it shows the name of
> functions in a bit more detail, e.g. "# 2 ... get_rand_state".
> On the other hand, everything works fine (no memory leaks at all) on Linux
> (Ubuntu18.04LTS + gfortran-8.2), even for a modified code with threading (-fopenmp).
> ---


Just for the record, Instruments reports some leaks like "_dyld_start*",
as reported also by Valgrind. Not sure if this is indeed a problem but I
think it isn't. I couldn't use however Instruments with code created ie
by gfortran-8.2. So, I could just compare few messages only.

> ...
> # But, my experience above may be because my OSX(10.11) is too old (compared to
> the latest version, Mojave 10.14).


In 10.13 the situation seems to be similar (maybe identical).
gah4 (03-11-19, 08:04 PM)
On Monday, March 11, 2019 at 9:23:57 AM UTC-7, Ev. Drikos wrote:

(snip)

> Just for the record, Instruments reports some leaks like "_dyld_start*",
> as reported also by Valgrind. Not sure if this is indeed a problem but I
> think it isn't. I couldn't use however Instruments with code created ie
> by gfortran-8.2. So, I could just compare few messages only.


dyld is the dynamic linker. It shouldn't be surprising that it
allocates some space, especially for actually dynamically linking,
and doesn't return it so soon.

Similar Threads