experchange > fortran

Ron Shepard (01-16-19, 08:20 AM)
In the process of tracking down some memory errors with valgrind, I seem
to have found an error with the intrinsic random_seed() in gfortran 7.3
and 8.2. This is on MacOS (several versions). Here is a sample program
that shows the problem:

program random_test
integer :: seed_size
intrinsic :: random_seed
call random_seed( size=seed_size )
write(*,*) 'seed_size=', seed_size
end program random_test

When this is compiled at any level of optimization with gfortran, and
run with “valgrind a.out”, it reports

==16832== LEAK SUMMARY:
==16832== definitely lost: 136 bytes in 1 blocks
==16832== indirectly lost: 0 bytes in 0 blocks
==16832== possibly lost: 72 bytes in 3 blocks
==16832== still reachable: 200 bytes in 6 blocks
==16832== suppressed: 18,091 bytes in 153 blocks
==16832== Rerun with --leak-check=full to see details of leaked memory

It is that first line, definitely lost, that concerns me. Further
experimentation shows that each subsequent call to random_seed() causes
an additional 40 bytes to be lost.

Is this indeed a problem with gfortran? Or with MacOS? Or is it a false
positive with valgrind? Or is there a problem with my program?

$.02 -Ron Shepard
gah4 (01-16-19, 10:58 AM)
On Tuesday, January 15, 2019 at 10:20:12 PM UTC-8, Ron Shepard wrote:
[..]
> run with “valgrind a.out”, it reports
> ==16832== LEAK SUMMARY:
> ==16832== definitely lost: 136 bytes in 1 blocks


I tried some to figure it out, but didn't find the answer.

Note that you ask for seed_size, but don't actually do anything
with a seed. It seems to me that it might allocate space for
a seed, but not actually do anything with it. Exactly why that
gives the message, I don't know.

If you call it billions of times, does it still use 40 bytes each time?
paul.richard.thomas (01-16-19, 11:41 AM)
On Wednesday, 16 January 2019 06:20:12 UTC, Ron Shepard wrote:
[..]
> Is this indeed a problem with gfortran? Or with MacOS? Or is it a false
> positive with valgrind? Or is there a problem with my program?
> $.02 -Ron Shepard


Hi Ron,

I find no memory loss with versions 6.4.1, 7.4.1, 8.2.1 and 9.0.0 on FC28/x86_64.

Cheers

Paul
spectrum (01-16-19, 12:06 PM)
On my computer (a bit old, osx10.11), valgrind reports the same amount of leak
(136 bytes) for the same program (with --leak-check=full"):

==58479== 136 bytes in 1 blocks are definitely lost in loss record 41 of 61
==58479== at 0x100008D4F: calloc (in /usr/local/Cellar/valgrind/3.14.0/lib/valgrind/vgpreload_memcheck-amd64-darwin.so)
==58479== by 0x100012902: _gfortrani_xcalloc (in /usr/local/Cellar/gcc/8.2.0/lib/gcc/8/libgfortran.5.dylib)
==58479== by 0x100000EAE: main (test.f90:6)
==58479==
==58479== LEAK SUMMARY:
==58479== definitely lost: 136 bytes in 1 blocks
==58479== indirectly lost: 0 bytes in 0 blocks
==58479== possibly lost: 0 bytes in 0 blocks
==58479== still reachable: 0 bytes in 0 blocks
==58479== suppressed: 22,148 bytes in 186 blocks

The output remains the same for this slightly modified code.

subroutine random_test
integer :: seed_size, idum, i
intrinsic :: random_seed
idum = 0
do i = 1, 10000
call random_seed( size=seed_size )
idum = max( idum, seed_size )
enddo
print *, "idum = ", idum
end

program main
call random_test
end

Maybe specific for Mac...?
gah4 (01-16-19, 04:29 PM)
On Wednesday, January 16, 2019 at 2:06:52 AM UTC-8, spectrum wrote:

(snip)

> Maybe specific for Mac...?


On some systems, random seeding does some I/O operations,
to get some randomness from the OS. That might allocate
some buffers somewhere, and would be OS specific.

Well, also C library specific in some cases.
Gary Scott (01-16-19, 05:54 PM)
On 1/16/2019 8:29 AM, gah4 wrote:
> On Wednesday, January 16, 2019 at 2:06:52 AM UTC-8, spectrum wrote:
> (snip)
> On some systems, random seeding does some I/O operations,
> to get some randomness from the OS. That might allocate
> some buffers somewhere, and would be OS specific.
> Well, also C library specific in some cases.

IO to a slow device/process would be a really horrible implementation :(
gah4 (01-16-19, 09:55 PM)
On Wednesday, January 16, 2019 at 7:54:17 AM UTC-8, Gary Scott wrote:

(snip, I wrote)

> > On some systems, random seeding does some I/O operations,
> > to get some randomness from the OS. That might allocate
> > some buffers somewhere, and would be OS specific.


> IO to a slow device/process would be a really horrible implementation :(


It is just for the seed, if you are not setting the seed,
that it uses some I/O on some systems.

OS X has /dev/random, a pseudo-device for random numbers.
No physical I/O, and I don't know the overhead.

But as above, in the libgfortran case, it is just for seeding.

The get/set seed routines might be slow, so one should not
try calling them billions of times.
Ev. Drikos (01-17-19, 01:35 PM)
On 16/01/2019 8:20 AM, Ron Shepard wrote:
[..]
> ==16832==         suppressed: 18,091 bytes in 153 blocks
> ==16832== Rerun with --leak-check=full to see details of leaked memory
> It is that first line, definitely lost, that concerns me...


As it seems this problem appears in macOS 10.13 with the most recent
GNU Fortran versions, ie:

1) With valgrind-3.13 on macOS 10.12 this line mentions a 0 bytes
leak, if the program has been compiled ie by gfortran 4.8 or 7.1.

2) With valgrind-3.14 on macOS 10.13 this line mentions a 0 bytes
leak, if the program has been compiled ie by gfortran 4.8.

3) With valgrind-3.14 on macOS 10.13 this line mentions 136 bytes
leak, if the program has been compiled ie by gfortran 7.1 or 7.3.

The archaic gfortran-4.8 (prints: seed_size=12) can be found at:


PS: I can't figure out if this problem is important or not.
Ev. Drikos (01-17-19, 02:43 PM)
On 17/01/2019 1:35 PM, Ev. Drikos wrote:
> ...
>   As it seems this problem appears in macOS 10.13 with the most recent
>   GNU Fortran versions, ie:


A restatement:

I've seen this problem in macOS 10.13 with the most recent
GNU Fortran versions, ie:
[..]
Ron Shepard (01-17-19, 05:52 PM)
On 1/17/19 6:43 AM, Ev. Drikos wrote:
> On 17/01/2019 1:35 PM, Ev. Drikos wrote:
>      A restatement:
>      I've seen this problem in macOS 10.13 with the most recent
>      GNU Fortran versions, ie:


The leak also occurs on MacOS 10.11.6 and gcc/gfortran 7.3. However, the
extra 40 byte leak for additional random_seed() calls does not apply on
this combination. I think all of my valgrind installations were with
version 3.13. I just updgraded to valgrind 3.14 on this machine, and it
still reports the 136 byte loss with no extra 40 byte increment.

I still do not know for certain that this memory is lost. It might be a
false positive with valgrind. I also do not know if it is the underlying
PRNG that is leaking the memory or if it is just the fortran interface
to it that is the problem. I'm assuming it is written in C or some other
low-level language.

I am debugging some code that has memory problems, and I located this
issue during that process. My code still has its own error somewhere,
and I am using this information mostly to try to isolate my bugs from
the gfortran bugs.

$.02 -Ron Shepard
gah4 (01-18-19, 01:12 AM)
On Thursday, January 17, 2019 at 7:52:08 AM UTC-8, Ron Shepard wrote:

(snip)

> The leak also occurs on MacOS 10.11.6 and gcc/gfortran 7.3. However, the
> extra 40 byte leak for additional random_seed() calls does not apply on
> this combination. I think all of my valgrind installations were with
> version 3.13. I just updgraded to valgrind 3.14 on this machine, and it
> still reports the 136 byte loss with no extra 40 byte increment.


> I still do not know for certain that this memory is lost. It might be a
> false positive with valgrind.


If a program loses memory in a loop, allocating some each time
through, and then overwriting all pointers to it, that is a problem.

Protected memory OS pretty much have to release a program's
memory when it finishes (otherwise it isn't doing much protection),
and in general it isn't bad to let it do that. About the only
reason to release it, is to satisfy valgrind users.

For one example, opening a file for reading and not closing it is
not normally a problem. (When writing, it is sometimes necessary to
close to be sure that the last buffer is out. I believe Fortran
guarantees to flush the buffer and close, especially from the days
before CLOSE.)

That might leave a buffer allocated, though.
David Jones (01-18-19, 02:43 AM)
gah4 wrote:

> Protected memory OS pretty much have to release a program's
> memory when it finishes (otherwise it isn't doing much protection),


It depends what one means by "finished". My experience with a memory
leak (on Windows) was that the memory was still reserved after a
Fortran "stop", with the program's process-id still appearing to be
running in the OS's lists. Using the OS to terminate the process did
release the memory.
Ron Shepard (01-18-19, 03:30 AM)
On 1/17/19 5:12 PM, gah4 wrote:
> For one example, opening a file for reading and not closing it is
> not normally a problem. (When writing, it is sometimes necessary to
> close to be sure that the last buffer is out. I believe Fortran
> guarantees to flush the buffer and close, especially from the days
> before CLOSE.)
> That might leave a buffer allocated, though.


I am not familiar enough with what valgrind detects. Would this cause a
problem with valgrind? In my test program I wrote out a value to
standard output, which presumably would have allocated a buffer somewhere.

$.02 -Ron Shepard
steve kargl (01-18-19, 03:47 AM)
Ron Shepard wrote:

> On 1/17/19 5:12 PM, gah4 wrote:
> I am not familiar enough with what valgrind detects. Would this cause a
> problem with valgrind? In my test program I wrote out a value to
> standard output, which presumably would have allocated a buffer somewhere.


I should probably ignore this thread, but ...

The source code for random_seed() is available for anyone to read.
There is a leak to the extent that if your installed gfortran can use
pthreads, then the first call to random_seed() will cause it to allocate
memory for the internal state of the PRNG. As this is the global state
of the PRNG, it is released when the program exits. If your gfortran
is installed where pthreads is not available, there is no leak as the
internal is stored in static memory.
gah4 (01-18-19, 04:53 AM)
On Thursday, January 17, 2019 at 5:47:42 PM UTC-8, steve kargl wrote:

(snip)

> I should probably ignore this thread, but ...


> The source code for random_seed() is available for anyone to read.


I did try to find it, but didn't find the actual random_seed().

I found the C routines that do the actual work, but not the one
actually called random_seed(), and specifically not the one
that returns SIZE.

I thought a google search with site:github.com/gcc-mirror
would find it.

OK, I found it now. It seems that they are called either
random_seed_i4 or random_seed_i8, and I was looking for one
actually called random_seed. (Assuming the above were
called from somewhere else.)

So, yes, even if someone understands C, it isn't so easy
to actually find in the source.

As for the actual question, one might assume that asking
for the SIZE, doesn't do anything else, such as initialize
the generator. Not that I think there is anything wrong
with doing it, but it isn't so obvious.

For anyone else interested, it is in:


Similar Threads