X-Recipient:	archive-cygwin AT delorie DOT com
Mailing-List:	contact cygwin-help AT cygwin DOT com; run by ezmlm
List-Id:	<cygwin.cygwin.com>
List-Subscribe:	<mailto:cygwin-subscribe AT cygwin DOT com>
List-Archive:	<http://sourceware.org/ml/cygwin/>
List-Post:	<mailto:cygwin AT cygwin DOT com>
List-Help:	<mailto:cygwin-help AT cygwin DOT com>, <http://sourceware.org/ml/#faqs>
Sender:	cygwin-owner AT cygwin DOT com
Mail-Followup-To:	cygwin AT cygwin DOT com
Delivered-To:	mailing list cygwin AT cygwin DOT com
Authentication-Results:	sourceware.org; auth=none
X-Spam-SWARE-Status:	No, score=-6.4 required=5.0 tests=AWL,BAYES_00,FREEMAIL_FROM,GIT_PATCH_2,HTML_MESSAGE,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.2 spammy=eat, laptop, Initial, U*corinna-cygwin
X-HELO:	mail-qt0-f179.google.com
DKIM-Signature:	v=1; a=rsa-sha256; c=relaxed/relaxed; d=bertacco-it.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:from:date:message-id:subject:to; bh=lyTdZcAE4XiZxns4kdul4B014C1jmDslO0OXFeBTse0=; b=tHilx8nG38KvrRncVGHvvaz7KasWZICyqi5DmEt2t6d3FUrCe6S9f9gWSZl+XHvb4F uJoXstU9amIwDTw+0x+GxtTXCAH75DZLdaGMd7BFbk6xXL+Brk7iUM/822yvNk66lroc p0NJjlWkfe+Dic5YYohgUuiJrVVREyjbdeDBmYEKEw2LfM2pfKC2Y1tKYV/1yyuHGe7C Vnk2147N7lwRJgN9p+aGhlTKKN1ePCKjX+VyQQ/RIRy+8mDcqwC1bWkmOxz0AtBc+xtp NhnTZMin8pUWdtA6MA+qk8ogIsik9uN79rVfvp/SJoWsjk2BUON9F6zMgLgi3uXC1Fe2 FHkw==
MIME-Version:	1.0
Sender:	livio AT bertacco DOT it
From:	Livio Bertacco <lbertacco AT tiscali DOT it>
Date:	Tue, 21 Aug 2018 16:53:56 +0200
Message-ID:	<CAL0dvti5rj=MtBjcPzrijT+YGazwWCE-utu=BD2TV9YL8Qc-5Q@mail.gmail.com>
Subject:	Re: memory reported in /proc/pid/status is wrongly scaled
To:	cygwin AT cygwin DOT com

Thanks Corinna, I tried the developer snapshot and the VmSize line from
"status" file looks good now.

What do you think about adding the VmPeak line too? (I can create the patch
myself).

>  On second thought there's more wrong than just that.
> Just dividing by page_size or allocation_granularity results in too small
values.

Yes, now that you mention it, this rounding up is actually still missing
for the vmrss and vmmaxrss values in the "stat" file (which just divides by
page_size):
1150
<https://cygwin.com/git/gitweb.cgi?p=newlib-cygwin.git;a=blob;f=winsup/cygwin/fhandler_process.cc#l1150>
vmrss = vmc.WorkingSetSize / page_size;
1151
<https://cygwin.com/git/gitweb.cgi?p=newlib-cygwin.git;a=blob;f=winsup/cygwin/fhandler_process.cc#l1151>
vmmaxrss = ql.MaximumWorkingSetSize / page_size;
and, by the way, is it correct that these two are reported in 4k pages
instead of 64k pages?
"statm" reports e.g. VmRss (that's the second value in the statm line) in
64k pages so that value doesn't match the same Rss amount (24th value) from
"stat".

Livio


> ---------- Forwarded message ----------
> From: Corinna Vinschen <corinna-cygwin AT cygwin DOT com>
> To: cygwin AT cygwin DOT com
> Cc:
> Bcc:
> Date: Fri, 17 Aug 2018 20:44:02 +0200
> Subject: Re: memory reported in /proc/pid/status is wrongly scaled
> On Aug 17 19:14, Corinna Vinschen wrote:
> > On Aug 17 16:05, Livio Bertacco wrote:
> > >  Hi,
> > > While playing with reading process memory usage in Linux and cygwin, I
> > > found that cygwin reports too large values in /proc/self/status (in
> 2.10.0
> > > and earlier).
> > > Whenever I was allocating a few kB in my test program, the VmSize line
> in
> > > /proc/self/status was growing several times faster.
> > >
> > > Small bash script to show the issue:
> > > #!/bin/bash
> > > pid=$$
> > > vmsizesplit=($(grep VmSize /proc/$pid/status))
> > > vmsize1="${vmsizesplit[1]}"
> > > echo Initial memory reported in status: $vmsize1 kB
> > > echo Allocating a 1000 kB string (bash can use more memory)
> > > eat=$(printf '%1024000s')
> > > vmsizesplit=($(grep VmSize /proc/$pid/status))
> > > vmsize2="${vmsizesplit[1]}"
> > > echo Current memory reported in status: $vmsize2 kB
> > > echo Difference is $[$vmsize2-$vmsize1] kB
> > >
> > > Running this in cygwin on my laptop I get:
> > > Initial memory reported in status: 84928 kB
> > > Allocating a 1000 kB string (bash can use more memory)
> > > Current memory reported in status: 106880 kB
> > > Difference is 21952 kB
> > >
> > > While bash may use quite more than 1000 kb in this case, 22x times
> larger
> > > doesn't seem right.
> > >
> > > Checking source file fhandler_process.cc, the
> > > function format_process_status which writes the "status" proc file
> > > retrieves memory usage via get_mem_values. Get_mem_values obtains that
> info
> > > from NtQueryInformationProcess/PagefileUsage which is in bytes, then
> it
> > > scales it to pages dividing by wincap.page_size:
> > > 1515: *vmsize = vmc.PagefileUsage / wincap.page_size ();
> > >
> > > Then format_process_status scales it back, in theory to bytes, and
> shifts
> > > it by 10 bits in order to print it out in kB:
> > > 1219:  unsigned page_size = wincap.allocation_granularity ();
> >
> > Looks like this is the bug.  get_mem_values returns all values
> > in multiple of OS page_size (4K), but format_process_status multiplies
> > with allocation_granularity (64K), leading to 16 times overallocation
> > value.  The other caller of get_mem_values, format_process_statm,
> > returns number of pages.  This must be expressed in multiples of
> > allocation_granularity since that's the virtual page_size in Cygwin.
> > But in case of format_process_status we're looking at KB values, so
> > patch 8a32c24a7bdb0, replaceing page_size with allocation_granularity,
> > was incorrect.
> >
> > Good catch!
> >
> > I'll revert patch 8a32c24a7bdb0 for 2.11.0.
>
> On second thought there's more wrong than just that.  Just dividing
> by page_size or allocation_granularity results in too small values.
> I applied patches to return more correct values and I made sure
> the values in status and statm are consistently rounded up to
> Cygwin's page size of 64K.
>
> I uploaded new developer snapshots to https://cygwin.com/snapshots
> for testing.  Please give them a try.
>
>
> Thanks,
> Corinna
>
> --
> Corinna Vinschen                  Please, send mails regarding Cygwin to
> Cygwin Maintainer                 cygwin AT cygwin DOT com
> Red Hat
>
>

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

- Raw text -