Mail Archives: cygwin/2008/07/31/06:54:41
Arg.. simple repro is this Python code, with the appropriate gcc source tree:
#! /usr/bin/env python
import os
import sys
def Run(Directory, Command):
sys.stderr.flush()
sys.stdout.flush()
print("cd " + Directory + " && " + Command)
PreviousDirectory = os.getcwd()
os.chdir(Directory)
if os.name == "nt":
Command = "sh -c \"" + Command + "\""
sys.stderr.flush()
sys.stdout.flush()
ExitCode = os.system(Command)
os.chdir(PreviousDirectory)
sys.stderr.flush()
sys.stdout.flush()
if ExitCode != 0:
sys.exit(ExitCode)
if not os.path.isdir("./obj"):
Run(".", "mkdir ./obj")
if not os.path.isfile("./obj/Makefile"):
Run("./obj", "../gcc/configure")
Run(".", "rm -rf ./obj/gmp")
Run("./obj", "make configure-gmp")
Repros in Python, not outside of Python? Why? That is the question.
Other tidbits:
flex actually fails to exec m4 either way, just that one way it manages to create lex.yy.c first.
It fails to exec m4 because gmp's configure has set M4=m4-not-needed.
This is almost certainly a simple bug in gmp's configure.
A fix would be to probe for external programs (yacc, lex), earlier, before it sets M4.
Or unset M4 after it is done with it.
But there is still something amiss, maybe Cygwin specific, maybe not.
Oh, and whether or not gmp sets M4 varies somewhat on build/host/target.
Usually, configuring gmp on its own, probably it is set.
But within gcc, generally not, because gcc configures it with processor=none for reasons I don't understand..
I guess it'd be nice if flex printed what it was trying to run -- getenv("M4") and not just "m4".
I'll follow up separately, at least gcc and gmp..
Still not completely solved.
I do have a quick repro though, you can strip down the configure script a bunch and it repros in just a few seconds.
Going through the strace logs....but need to read flex also for when it creates the output vs. running m4..
- Jay
> From: jayk123 AT hotmail DOT com
> To: cygwin AT cygwin DOT com
> Subject: RE: flex: exec failed?
> Date: Sat, 26 Jul 2008 12:40:01 +0000
>
>
> anecdotes...
>
> I've seen this quite a few times now, like 10.
>
> Always this same error -- always configure unable to determine the (f)lex output, I think always configuring gmp.
> (Maybe binutils sometimes, due to the flex source directory, understood, red herring.)
>
> I've done "a lot" else successfully, configure/built gcc/binutils/gmp/mpfr, this is the one failure I've seen a few of.
> That is -- lots of stuff works, I'm not messing things up in some large way, but this fails "often".
>
> I'm not using tee any longer, but still running all of this within Python (os.system).
> I don't usually check config.log, but it always the same configure test.
>
> Peeling one layer, either cd'ing in cmd or bash to gcc toplevel and doing "make configure-gmp" or two layers, cd'ing to gcc/gmp and "whatever/configure whatever", has a much much higher success rate. And then I can run my stuff, which is incremental enough, and so I skip past this.
>
> I stopped installing Cygwin at the root, though I've seen it both work and not-work that way.
> I am usually on XP/AMD64, and maybe every repro is on that.
>
> I do notice comments about thread unsafety in Cygwin's fork/exec code.
> My Python is single threaded (I was getting intermittent errors about unable to allocate locks).
>
> I'll have to put strace in there and focus in on it, instead of continuing to workaround.
>
> ..Jay
>
>
>> From: jayk123 AT hotmail DOT com
>> To: cygwin AT cygwin DOT com
>> Subject: FW: flex: exec failed?
>> Date: Wed, 16 Jul 2008 18:55:06 +0000
>>
>>
>> It failed again without the time/tee. Time to try with -disable-bootstrap, and if it still failed, reboot, and...
>>
>>
>>
>>
>>> From: jayk123 AT hotmail DOT com
>>> To: cygwin AT cygwin DOT org
>>> Subject: flex: exec failed?
>>> Date: Wed, 16 Jul 2008 18:53:23 +0000
>>>
>>>
>>> anyone familiar with this:
>>>
>>> flex: fatal internal error, exec failed
>>> flex: error writing output file lex.yy.c
>>>
>>> It comes while configuring gmp, in a merged ("Cygnus") gcc tree.
>>> I've been able to build this several times, various configurations.
>>> (various host/target and options, mostly build=cygwin, but
>>> on some other systems too, unrelated..)
>>>
>>> What I am doing differently this time is two things.
>>>
>>> 1) I'm no longer -disable-bootstrap.
>>>
>>> I am running under a wrapper .py file.
>>> (It goes through multiple configurations.)
>>> That isn't changed, but:
>>>
>>> 2) previously:
>>> ./build.py
>>> now:
>>> time (sh -c "./build.py | tee 1.log")
>>> or
>>> time (./build.py | tee 1.log)
>>>
>>> I have tried both.
>>>
>>> config.log nearby:
>>>
>>> configure:32814: checking for flex
>>> configure:32840: result: flex
>>> configure:32853: checking for yywrap in -lfl
>>> configure:32883: gcc -o conftest.exe -g -fkeep-inline-functions -DNO_ASM conftest.c -lfl>&5
>>> configure:32889: $? = 0
>>> configure:32893: test -z
>>> || test ! -s conftest.err
>>> configure:32896: $? = 0
>>> configure:32899: test -s conftest.exe
>>> configure:32902: $? = 0
>>> configure:32915: result: yes
>>> configure:32993: checking lex output file root
>>> configure:33004: flex conftest.l
>>> flex: fatal internal error, exec failed
>>> flex: error writing output file lex.yy.c
>>> configure:33007: $? = 1
>>> configure:33014: error: cannot find output from flex; giving up
>>>
>>>
>>> shell code nearby (configure):
>>>
>>>
>>> if test "x$LEX" != "x:"; then
>>> echo "$as_me:$LINENO: checking lex output file root">&5
>>> echo $ECHO_N "checking lex output file root... $ECHO_C">&6
>>> if test "${ac_cv_prog_lex_root+set}" = set; then
>>> echo $ECHO_N "(cached) $ECHO_C">&6
>>> else
>>> # The minimal lex program is just a single line: %%. But some broken lexes
>>> # (Solaris, I think it was) want two %% lines, so accommodate them.
>>> cat>conftest.l <&5
>>> (eval $LEX conftest.l) 2>&5
>>> ac_status=$?
>>> echo "$as_me:$LINENO: \$? = $ac_status">&5
>>> (exit $ac_status); }
>>> if test -f lex.yy.c; then
>>> ac_cv_prog_lex_root=lex.yy
>>> elif test -f lexyy.c; then
>>> ac_cv_prog_lex_root=lexyy
>>> else
>>> { { echo "$as_me:$LINENO: error: cannot find output from $LEX; giving up">&5
>>> echo "$as_me: error: cannot find output from $LEX; giving up">&2;}
>>> { (exit 1); exit 1; }; }
>>> fi
>>> fi
>>>
>>>
>>> flex code nearby:
>>>
>>>
>>> /** Fork and exec entire filter chain.
>>> * @param chain The head of the chain.
>>> * @return true on success.
>>> */
>>> bool filter_apply_chain (struct filter * chain)
>>> {
>>> int pid, pipes[2];
>>>
>>> /* Tricky recursion, since we want to begin the chain
>>> * at the END. Why? Because we need all the forked processes
>>> * to be children of the main flex process.
>>> */
>>> if (chain)
>>> filter_apply_chain (chain->next);
>>> else
>>> return true;
>>>
>>> /* Now we are the right-most unprocessed link in the chain.
>>> */
>>>
>>> fflush (stdout);
>>> fflush (stderr);
>>>
>>> if (pipe (pipes) == -1)
>>> flexerror (_("pipe failed"));
>>>
>>> if ((pid = fork ()) == -1)
>>> flexerror (_("fork failed"));
>>>
>>> if (pid == 0) {
>>> /* child */
>>>
>>> /* We need stdin (the FILE* stdin) to connect to this new pipe.
>>> * There is no portable way to set stdin to a new file descriptor,
>>> * as stdin is not an lvalue on some systems (BSD).
>>> * So we dup the new pipe onto the stdin descriptor and use a no-op fseek
>>> * to sync the stream. This is a Hail Mary situation. It seems to work.
>>> */
>>> close (pipes[1]);
>>> if (dup2 (pipes[0], fileno (stdin)) == -1)
>>> flexfatal (_("dup2(pipes[0],0)"));
>>> close (pipes[0]);
>>> fseek (stdin, 0, SEEK_CUR);
>>>
>>> /* run as a filter, either internally or by exec */
>>> if (chain->filter_func) {
>>> int r;
>>>
>>> if ((r = chain->filter_func (chain)) == -1)
>>> flexfatal (_("filter_func failed"));
>>> exit (0);
>>> }
>>> else {
>>> execvp (chain->argv[0],
>>> (char **const) (chain->argv));
>>> flexfatal (_("exec failed"));
>>> }
>>>
>>> exit (1);
>>> }
>>>
>>> /* Parent */
>>> close (pipes[0]);
>>> if (dup2 (pipes[1], fileno (stdout)) == -1)
>>> flexfatal (_("dup2(pipes[1],1)"));
>>> close (pipes[1]);
>>> fseek (stdout, 0, SEEK_CUR);
>>>
>>> return true;
>>> }
>>>
>>>
>>> It seems like a gratuitous use of fork/exec in Flex but oh well, I guess
>>> that's old fashioned inefficient fairly portable multi-threading.
>>>
>>>
>>> This is on Windows XP on AMD64 ("XP" == "2003").
>>>
>>>
>>> I tried deleting flex, like maybe it is optional, like how m4/autoconf are.
>>> I think the configure script still ran it, and it failed.
>>>
>>> I have Cygwin installed to root, so that I can use all the same paths between
>>> Cygwin and Windows, without requiring symlinks. This is otherwise, *a lot*,
>>> working fine.
>>>
>>> (In particular, I used to symlink Cygwin /obj to Windows \obj and that mostly
>>> works, but symlinks seem to cause grief in various places, such as
>>> if using Cygwin Python. I know this is frowned upon, but really it is proving
>>> convenient. No need to escape backslashes, no need to ever use cygpath,
>>> as long as I use one drive, as long as Win32 code doesn't demand full paths,
>>> as long as I don't use symlinks too much, etc. very reasonable restrictions)
>>>
>>>
>>> I'll try to narrow it down and investigate more later myself.
>>> Like, currently I'm running the toplevel gcc configure.
>>> I'll have to try just gmp. Get it small enough and then strace and maybe gdb.
>>> For my current run I'll just remove the time/tee.
>>>
>>>
>>> - Jay
--
Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Problem reports: http://cygwin.com/problems.html
Documentation: http://cygwin.com/docs.html
FAQ: http://cygwin.com/faq/
- Raw text -