Subject: RE: Bug in gzip's stdout handling
From: "Thomas Hammer" <[email-masked]>
To: <[email-masked]>
Cc: <[email-masked]>, <[email-masked]>
Date: Mon, 17 Nov 2003 23:19:37 +0100

> -----Original Message-----
> From: Igor Pechtchanski [mailto:[email-masked]]
> Sent: 14. november 2003 17:22
> To: Thomas Hammer
> Cc: [email-masked]
> Subject: RE: Bug in gzip's stdout handling
>
<...>

Hi Igor,

This turned out to be a very long mail, so here's the summary:
- I figured it out. I had forgotten to delete the original mount table from
the registry before reinstalling cygwin. All my drives were still mounted in
textmode, and that's why cat and redirection failed.
- I don't know why mount and cygcheck -svr gives different reports regarding
mount mode.
- I have one question it would be nice if you answered. It's at the end of
the email.

And here's the rest of the email:

>
> (2) shows that you used to have text mounts.  Your cygcheck.out shows
> binary mounts, so that's probably not the problem.

I forgot to mention that I reverted to my original version of cygwin (with
DOS-style newlines) when I discovered that using UNIX-style newlines didn't
fix the problem.

I did the following:

1) Renamed c:\cygwin to c:\cygwin_old
2) Reinstalled cygwin, specifying UNIX-style newlines when asked
3) Test if "cat file.bin | gzip -c > filecopy.gz" produced a valig gz-file.
   It didn't
4) Deleted c:\cygwin and renamed c:\cygwin_old to c:\cygwin

I did not delete the registry keys or the local cygwin package directory. I
don't know is this matters or not.

I didn't know about text vs binary mounts until you mentioned it, and did
some reading up. One weird thing I came accross is that if I run mount, the
information seems to conflict with the information from cygcheck -svr.

$ mount
C:\cygwin\bin on /usr/bin type system (textmode)
C:\cygwin\lib on /usr/lib type system (textmode)
C:\cygwin on / type system (textmode)
c: on /cygdrive/c type user (textmode,noumount)
s: on /cygdrive/s type user (textmode,noumount)

The relevant lines from cygcheck -svr:

a:  fd           N/A    N/A
c:  hd  NTFS   57231Mb  47% CP CS UN PA FC
d:  cd           N/A    N/A
o:  net          N/A    N/A
s:  net NTFS   76308Mb  76% CP CS UN PA FC

C:\cygwin      /          system  binmode
C:\cygwin/bin  /usr/bin   system  binmode
C:\cygwin/lib  /usr/lib   system  binmode
.              /cygdrive  system  binmode,cygdrive

It looks to me as if mount claims all mounts are text mounts (which would
explain my problems, I guess). Whereas cygcheck -svr claims all my mounts
are binmode.

I'm beginning to wonder if I did something wrong when reinstalling cygwin
and specifying binary mode. I'm giving it another try now.

1) Renamed c:\cygwin to c:\cygwin_old
2) Renamed the HKCU\Software\Cygnus Solutions registry key
3) Renamed the local cygwin package dir.
4) Reinstalled cygwin, specifying UNIX-style newlines when asked
5) Test if "cat file.bin | gzip -c > filecopy.gz" produced a valig gz-file.

It didn't.

Running mount gives the same result as before (shows textmode on all
mounts).

I tried installing cygwin (downloaded setup.exe from www.cygwin.com) from
scratch on another computer in my office. That computer also runs WindowsXP.
$HOME wasn't defined here, but it was on my primary computer. Don't know if
that mattered.

On that computer, mount shows all mountpoints as being of type binmode, and
"cat file.bin | gzip -c > out.bin" works as it should.

I have no Idea why my primary computer insists on mounting everything in
textmode :-(.

I did try to remove my .bashrc-file on my primary computer. It didn't help -
and I couldn't find anything in there related to mounting.

I wonder where the mount table is stored. Maybe it for some reason survived
a reinstall of cygwin...

Please read on for some more discoveries.

>
> Please try the following:  "sed '' binaryfile.bin > acopy.bin" and compare
> the files.  Also "echo | sed '' > test.out; od -c test.out" and "perl -e
> 'print q{ }x2560' | gzip -c > test.gz; od -c test.gz".

thammer@blackbox /cygdrive/c/temp/temp
$ echo | sed ''  >test.out

thammer@blackbox /cygdrive/c/temp/temp
$ od -c test.out
0000000  \r  \n
0000002

thammer@blackbox /cygdrive/c/temp/temp
$ perl -e 'print q{ }x2560' | gzip -c > test.gz

thammer@blackbox /cygdrive/c/temp/temp
$ od -c test.gz
0000000 037 213  \b  \0   =   . 271   ?  \0 003 355 301 201  \0  \0  \0
0000020  \0 303     225 371   S 036 344   U 001  \0 360   d   6 271 357
0000040 252  \0  \r  \n  \0  \0
0000046

Back to the mounting thread of thought.

I tried to mount a directory as binary:

$ mount -b c:\\temp\\temp\\t /mytest
mount: warning - /mytest does not exist.

$ cd /mytest

thammer@blackbox /mytest
$ echo | sed ''  >test.out; od -c test.out
0000000  \n
0000001

Hey, I'm getting somewhere :-).

thammer@blackbox /mytest
$ ls -l
total 109
-rwx------    1 thammer  mkgroup    110755 Nov 17 21:51 bin.jpg

thammer@blackbox /mytest
$ cat bin.jpg | gzip -c > another.jpg.gz

thammer@blackbox /mytest
$ gunzip another.jpg.gz

thammer@blackbox /mytest
$ ls -l
total 218
-rw-r--r--    1 thammer  mkgroup    110755 Nov 17 22:08 another.jpg
-rwx------    1 thammer  mkgroup    110755 Nov 17 21:51 bin.jpg

thammer@blackbox /mytest
$ comp another.jpg bin.jpg
Comparing another.jpg and bin.jpg...
Files compare OK

I think there's something fishy about the mount table and where it's
located. I think it is for some reason stored _outside_ the c:\cygwin
directory. Which means it won't get updated when I do a reinstall of cygwin.

To verify this, I renamed c:\cygwin to c:\cygwin_new, and c:\cygwin_old to
c:\cygwin, and opened a bash shell.

thammer@blackbox /
$ mount
c:\temp\temp\t on /mytest type system (binmode)
C:\cygwin\bin on /usr/bin type system (textmode)
C:\cygwin\lib on /usr/lib type system (textmode)
c:\cygwin on / type system (textmode)
c: on /cygdrive/c type user (textmode,noumount)
s: on /cygdrive/s type user (textmode,noumount)

Hey! The /mytest mount is still there, as I suspected.

Where could that mount table be....

RTFM :).

When I reinstalled cygwin, I forgot to delete the "HKLM\Software\Cygnus
Solutions" registry subtree, I just deleted the "HKCU\Software\Cygnus
Solutions" subtree.

Oh well.

...

I have one final question I would be very happy if you could answer:

The problem I originally reported to you can be solved by myself by making
sure that I specifically mount the directory I'm working in in binmode. I
consider this an acceptable solution, but I don't feel good about it. I'm
going to have to put this in makefiles that will be distributed to our
customers, some of which might have text mounts, and having mount
instructions there seems plain wrong.

The question is this: Is there any way that I _from_the_commandline_ can
force all programs and the shell (for redirection) to use binary mode even
if the file is on a drive that is mounted in textmode? I'd like to do
something like this:

CYGWIN=somemagicstring bash -c cat file.bin | gzip -c > file2.bin.gz

/* Actually it would be more like this, with the gzip command
   deep within the Makefile :
  CYGWIN=somemagicstring bash -c make dist
*/

I tried s/somemagicstring/binmode/, and s/somemagicstring/tty/, but neither
did me any good.

If you have any suggestions as to if it's possible to avoid depending on a
binary mount, I'd be happy to hear it.

Thanks for the help so far :).

.Thomas

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/