X-Recipient:	archive-cygwin AT delorie DOT com
DKIM-Filter:	OpenDKIM Filter v2.11.0 sourceware.org D25F13846056
DKIM-Signature:	v=1; a=rsa-sha256; c=relaxed/relaxed; d=cygwin.com;
	s=default; t=1657381093;
	bh=aKqxV+bjj8N8/a0fPStbP+hm0eHWl9VJFI17M0HOT2A=;
	h=To:Subject:Date:List-Id:List-Unsubscribe:List-Archive:List-Post:
	List-Help:List-Subscribe:From:Reply-To:From;
	b=nXaSUiEmRQC3Ekak1AJHhS8BXeAY8RnryOBDV1RiBC1wjdBPKgCDXoNeirtTH4LTZ
	L0wGmErvmByAOi82801/4Ibf7FXYaiZ0dDGoeVOVRKEDfKSjNy8XR9IR1QhT1ykNhF
	hsKf6rGfjiPoL0rsI0wMROUPe1dsrENo1Qsy0XSE=
X-Original-To:	cygwin AT cygwin DOT com
Delivered-To:	cygwin AT cygwin DOT com
DMARC-Filter:	OpenDMARC Filter v1.4.1 sourceware.org 9D10038485B8
X-SBRS-Extended:	Low
X-IronPortListener:	ces-out
X-IronPort-AV:	E=Sophos;i="5.92,258,1650945600"; d="scan'208";a="340863691"
ARC-Seal:	i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
	b=Rd3BDZ3ArQb3iLvyJR5XRmuMTta4YO10uXFTWpNKryscw6mu7JSapjiXlaym/wI4zCi53Mil4BiPGZ67fQeEm4BbO60dLcgc1KRpsowxKMUPTmm8LruGPKYbi3yb8zXyXO8bnCR0sa17oD22HKL4dcigHlPDtsdF008hGZnmkZkbGlCRajmjNQcgZ/9CKsKbY6zpSRZT45kWlDQtZ2qZ5aCV1qTB5XAQvHGy7rviRY61UoCOqGXnT75JWjWi5hkdUIQjDVmE2Z2yV6VEhNYYtRlklTa7dmFmivbZzLGPU/XL2Vc7YG2clldJIihmnk3FXzE+5MCxGQRG09vBlfA64A==
ARC-Message-Signature:	i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
	s=arcselector9901;
	h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
	bh=yGbHjyZBFNjt737/bwSctXwRwr9Pi8vuQQTM5pP+BIA=;
	b=MNm147Qj21yWg69amUDRLwgnguR+jxyd4z844mdDl0qzP1HoPaRxnG3+G32Ya6BoQiGHCFAmg9Lq3/2WH/U7z8Sdagrnjk8XuhaQravAsah0fdVfrIRrLMIXHY/ad0h3YrUrEus8AIFL4sR1jgmKxlIz6zIvkn5LakVjGfgC99YBZeGStg6Kvmyk/s9pCH3y/4SHY/b6MRP8HrKFbp68vaT8Xp4ooMORtdrj6lLBw6QIY5YI0us3cQOQZ/9ytoeDk4iJJtFIiu/Hw6KvIZoe3yOgJyPZBKJvaCDiX05inyV4jMgwvYWIeh1fZrnsfOi17BtzA3HlWBiXYrUZmP31Fg==
ARC-Authentication-Results:	i=1; mx.microsoft.com 1; spf=pass
	smtp.mailfrom=ncbi.nlm.nih.gov; dmarc=pass action=none
	header.from=ncbi.nlm.nih.gov; dkim=pass header.d=ncbi.nlm.nih.gov; arc=none
To:	"'cygwin AT cygwin DOT com'" <cygwin AT cygwin DOT com>
Subject:	Spurious / persistent "exception" condition in half-closed sockets
Thread-Topic:	Spurious / persistent "exception" condition in half-closed
	sockets
Thread-Index:	AdiTqIzsqd7Y2Y4cQ8Wpylby9/6cSQ==
Date:	Sat, 9 Jul 2022 15:37:00 +0000
Message-ID:	<DM8PR09MB7095465D639E6A96C5615E84A5859@DM8PR09MB7095.namprd09.prod.outlook.com>
Accept-Language:	en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-ms-publictraffictype:	Email
x-ms-office365-filtering-correlation-id:	11a7ec06-a45c-4716-09da-08da61c0ddb0
x-ms-traffictypediagnostic:	DM6PR09MB5989:EE_
x-ms-exchange-senderadcheck:	1
x-ms-exchange-antispam-relay:	0
x-microsoft-antispam:	BCL:0;
x-microsoft-antispam-message-info:	k+vcjE2Eh7CAWFWAPJ2Yf9yzEewK5EO4XYNwLvpM0ynHE5oJoy+gAGdoE00opeNZZkAKrBPeCUsDxc9jfdDqbmFBEipym8wuSAMOPZGkhk/8/H8KGU0Itkk+cyWCpuSU2yGOsNsQtNFSf4+Qv/wwziaRHzcsQPubSXKLXVvm5wWJcuECX1ZVpLI5Q5obxyTx7uhhPgOr1APgNkA6IqCIzps3yE6C7862FmAtoH+EFYwbp/sJe60xYWUetGzjCRowRXvtXUaG/4k7XpvFH6FEkRwFpM62trSzMwOnE78jcyuS8vNpgrtYo0qTmGsF+NmHvSBNmfDJclzggFvbLxQ2/LRVbzNt4PBhrcaCOVZJzfJO+dpW6mRldzTqbveSQxg9/5/oMd6nLAieyTajGJk0Z5np/3R2xEdBc78CmdjswlF0E4qUDXKDhEqPiwVh1yziK3QReG4gRcOcyZiTDWnvfB76XI0fLC+CUayRK0nRQB5EK1r92OEb3YY7+HanGkJpslXYRJgKN4aeX/1P8ZX/09ThR30lmob/b3MsIPkKyueKIaB2P3H69zSf40U4LD2Dd0Eeo/vRkrhVAOL2fQC2SxBmdg5wabrTG4GLG2362f+OBF5iDgC0UCcIG1GSaNS5/1Dsx9w2FZ1KDZxRNhWV7ofa47hyLdKYcbjVkCvLe4QJneAr//SfmCfrz+eCzSuRKM1OC6OlPsombg9ZF4/t0L3a6hmzf+TIgwllQB5IludqMp017vE7Mwm4/Yz7OHFkah96teYH1+105EduASagIy+KiguKHlv+pbvveL/XtrFN4SlNDqA4y7eZTIru21u+I/cpOzhGHk9mKzlONCk3DA==
x-forefront-antispam-report:	CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:;
	IPV:NLI; SFV:NSPM; H:DM8PR09MB7095.namprd09.prod.outlook.com; PTR:; CAT:NONE;
	SFS:(13230016)(4636009)(366004)(33656002)(2906002)(8936002)(5660300002)(498600001)(38100700002)(186003)(86362001)(83380400001)(52536014)(26005)(122000001)(55016003)(6506007)(76116006)(9686003)(66556008)(71200400001)(8676002)(64756008)(66476007)(66446008)(38070700005)(6916009)(66946007)(7696005)(491001);
	DIR:OUT; SFP:1102;
x-ms-exchange-antispam-messagedata-chunkcount:	1
x-ms-exchange-antispam-messagedata-0:	=?us-ascii?Q?2XDiKBG0X8JSRVYu2bhnqwWDp+5Qjc0n8UItDAADcR77def3T7sgTY08UCJR?=
	=?us-ascii?Q?uMP4p+uIU2Cm1ZQ0BJ4DReAQJqXVL8hhhFoOBr0chGBPf9qPBozSVlS49mpD?=
	=?us-ascii?Q?H1cp2/as/cI9UQ16ErpY8raR1k5gQZiszm4Gqme9iqBK6RdCEA/D97TZQRbh?=
	=?us-ascii?Q?zs5jUE84VZztPIAIZkK+aqd4r4+Hghe41daL4nPKBEMWwnpaYQ+N1j3gbh+t?=
	=?us-ascii?Q?cT9szG35AQNK0DOg/6UiD2gSUaoiJfzoG04o3T//DcR1Tzjs7cSXCHKtaBzz?=
	=?us-ascii?Q?Kbv+HAb6y/IaVQ4MMqkEZRYuz7CA3PKA/gAHuN79eMas0ddlnPsaH02LscCC?=
	=?us-ascii?Q?tP2z4tHe3BpwOEkZDor/hsdv43BsKJSvmvd0qubFxUi66waRr6QbuMe0OROV?=
	=?us-ascii?Q?JYrut8kkRe59FcAXJKD/bSZ2aE158+0XedB3CtQY4iMbbdJr/smMImyFrsyh?=
	=?us-ascii?Q?s22Gm8PVtQnlXzkv7c1BvVl77ezg2ijFNYcP5uyEP5ERS/YHjxyQh8WY3hqg?=
	=?us-ascii?Q?OOF4Ys7eCPiMKOlYwUyl+RHi7zusz7NPt4lGUmL8fHO2pXq5O0/dwBoVgj5D?=
	=?us-ascii?Q?LD4OLTJhTGUHLbEdQjslMuVzmpNWqbqwPn15pomEm6DydGakzX7JLJ4mGU7J?=
	=?us-ascii?Q?nM4JRJ3N6Iq6kds7cB7Y4Xnu65KcbSQufFjrZq5HDSZS0D0tvrou2aM7L+1w?=
	=?us-ascii?Q?c95UODLgCibu16Ufs42+TXBvjMnR9ukLHzdCowkqLdzZWxuUk4QFKbVHiHRl?=
	=?us-ascii?Q?FnY9JeIQtuaSrhmBgs5GRpNyQMxdWvmfKo0pUpkBVeTkG6m9CgF+Dq/wq5AG?=
	=?us-ascii?Q?uT9OEsd40Huz5Jt/apgBgNu5OcUhAIVF064+KA017muZIVrL/N5TqysWUr6N?=
	=?us-ascii?Q?p6h0U51cwhIa7VoRcpKUKKejzTWOHB9l4ppiGTOQqPHvnl0EFOyBYBB62BLN?=
	=?us-ascii?Q?VHWuFX1kZmuFp5i7M0r0aWnpIWMCedAI+RIX69wQfbiazy3UYLPH+Kr/cs9v?=
	=?us-ascii?Q?WtFE5aPkZBO+s3P5zKfazKl8STXOcecuBIz10Z+XZ0vekolA6i/7WcB01Dhp?=
	=?us-ascii?Q?9ncr9P/fgnfyCCT/XQFQbfvNW4cg8SC/Id2u+H9gMV+TWOrDcN1h17m9fpgj?=
	=?us-ascii?Q?LXU85RvUik7BjT9EU0VIynacy4dipPPEyYmiMJsDDTgndO1AylWcWrVBqU1v?=
	=?us-ascii?Q?FOL1YqehoaoHiQ5nbcNYZCiNgFQ1VduNWflU5ZloY7/x7EB3J1E7RG6NrbnX?=
	=?us-ascii?Q?Pw/NrNaS8RIng3ehVeye+nnjJ1vUdJGZOBYDfQ16GU2RFvFlYx7+C/HAGHe4?=
	=?us-ascii?Q?oClhgidAEyDip5uM/hJDL5W36itDNIL4VA4XF9HlWe3gkVfURjhGMf9QO5Ba?=
	=?us-ascii?Q?QModYRH2PGgHuE8D85vCu7mpyNC39IS7Pv1ZattPDPg1yVT/xM0pjt4bGpqY?=
	=?us-ascii?Q?VnyFF8U3qjYQmGmP8z7mdKc5BF1gOYzMkXq6pIylc2utiNTqe05OSkp4G31m?=
	=?us-ascii?Q?CvFj1T2TG48xutiOxpbmQfBgWPOBeZENpSbw/Bv+jyMPSr67CHgHAPzwGRGA?=
	=?us-ascii?Q?8zgdW5Jr/KQfWwpgNwM=3D?=
MIME-Version:	1.0
X-MS-Exchange-CrossTenant-AuthAs:	Internal
X-MS-Exchange-CrossTenant-AuthSource:	DM8PR09MB7095.namprd09.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id:	11a7ec06-a45c-4716-09da-08da61c0ddb0
X-MS-Exchange-CrossTenant-originalarrivaltime:	09 Jul 2022 15:37:00.6950 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader:	Hosted
X-MS-Exchange-CrossTenant-id:	14b77578-9773-42d5-8507-251ca2dc2b06
X-MS-Exchange-Transport-CrossTenantHeadersStamped:	DM6PR09MB5989
X-OriginatorOrg:	ncbi.nlm.nih.gov
X-Spam-Status:	No, score=-2.0 required=5.0 tests=BAYES_00, DKIM_SIGNED,
	DKIM_VALID, DKIM_VALID_EF, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE,
	T_SPF_HELO_PERMERROR autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version:	SpamAssassin 3.4.6 (2021-04-09) on
	server2.sourceware.org
X-BeenThere:	cygwin AT cygwin DOT com
X-Mailman-Version:	2.1.29
List-Id:	General Cygwin discussions and problem reports <cygwin.cygwin.com>
List-Archive:	<https://cygwin.com/pipermail/cygwin/>
List-Post:	<mailto:cygwin AT cygwin DOT com>
List-Help:	<mailto:cygwin-request AT cygwin DOT com?subject=help>
List-Subscribe:	<https://cygwin.com/mailman/listinfo/cygwin>,
	<mailto:cygwin-request AT cygwin DOT com?subject=subscribe>
From:	"Lavrentiev, Anton \(NIH/NLM/NCBI\) \[C\] via Cygwin" <cygwin AT cygwin DOT com>
Reply-To:	"Lavrentiev, Anton \(NIH/NLM/NCBI\) \[C\]" <lavr AT ncbi DOT nlm DOT nih DOT gov>
Sender:	"Cygwin" <cygwin-bounces+archive-cygwin=delorie DOT com AT cygwin DOT com>

Hi all,

It took me awhile to figure this one out, but I think I have a good test case to
demonstrate a (rather serious, actually) issue with Cygwin sockets and select/poll.

In short, when a reading end of a socket half closes for write (basically, signaling the
other end of no more data to expect, resulting in TCP FIN and, subsequently, EOF in the other
end's read()), if that end keeps reading the still incoming remaining data, it will face with
a lot of "exception" conditions, which are just spurious.  That will also burn CPU instead of
doing a proper wait (so a read() failed with EAGAIN, and then waited for with select()
(or poll() -- the same issue) will be attempted again immediately (as the socket would be reported
as "ready" with an "exception"), and result in another EAGAIN, etc etc...
Eventually, there will be successful reads squeezed in between, though...

Here's the "client" code ("server" code follows):

$ cat client.c
#include <errno.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <arpa/inet.h>
#include <netinet/in.h>


static void error(const char* what)
{
    fflush(stdout);
    perror(what);
    exit(1);
}


int main(int argc, const char* argv[])
{
    struct sockaddr_in sin;
    size_t total = 0;
    int c = socket(AF_INET, SOCK_STREAM, 0);

    if (c == -1)
        error("socket");

    memset(&sin, 0, sizeof(sin));
    sin.sin_family      = AF_INET;
    sin.sin_addr.s_addr = htonl(INADDR_LOOPBACK);
    sin.sin_port        = htons(atoi(argv[1]));

    if (connect(c, (struct sockaddr*) &sin, (socklen_t) sizeof(sin)) != 0)
        error("connect");
    if (fcntl(c, F_SETFL, fcntl(c, F_GETFL, 0) | O_NONBLOCK) == -1)
        error("fcntl");
#ifdef BUG
    if (shutdown(c, SHUT_WR) != 0)
        error("shutdown");
#endif

    for(;;) {
        char buf[1000];
        ssize_t n = read(c, buf, sizeof(buf));

        if (n > 0) {
            printf("%zu byte%s received from server\n", n, &"s"[n==1]);
            total += n;
            continue;
        }
        if (n == 0) {
            printf("Connection closed, %zu byte%s received\n",
                   total, &"s"[total==1]);
            break;
        }
        if (errno != EAGAIN  &&  errno != EWOULDBLOCK)
            error("read");
        fflush(stdout);
        perror("read");
        for (;;) {
            fd_set rfds, efds;
            struct timeval tv;
            int m;

            FD_ZERO(&rfds);
            FD_ZERO(&efds);
            FD_SET(c, &rfds);
            FD_SET(c, &efds);
            memset(&tv, 0, sizeof(tv));
            tv.tv_sec = 2;

            printf("Waiting...\n");
            m = select(c + 1, &rfds, 0/*wfds*/, &efds, &tv);
            if (!m)
                continue;
            if (m < 0)
                error("select");
            if (FD_ISSET(c, &efds)) {
                printf("Exception??\n");
                break;
            }
            if (FD_ISSET(c, &rfds)) {
                printf("Read-ready!\n");
                break;
            }
            error("select bug");
            abort();
        }
    }
    close(c);
    printf("Bye-bye\n");
    return 0;
}

$ cat server.c
#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <arpa/inet.h>
#include <netinet/in.h>


static void error(const char* what)
{
    fflush(stdout);
    perror(what);
    exit(1);
}


int main(int argc, const char* argv[])
{
    struct sockaddr_in sin;
    int s = socket(AF_INET, SOCK_STREAM, 0);

    if (s == -1)
        error("socket");

    memset(&sin, 0, sizeof(sin));
    sin.sin_family      = AF_INET;
    sin.sin_addr.s_addr = htonl(INADDR_LOOPBACK);
    sin.sin_port        = htons(atoi(argv[1]));

    if (bind(s, (struct sockaddr*) &sin, (socklen_t) sizeof(sin)) != 0)
        error("bind");
    if (listen(s, 1) != 0)
        error("listen");

    for(;;) {
        size_t total = 0;
        socklen_t sinlen = sizeof(sin);
        int c = accept(s, (struct sockaddr*) &sin, &sinlen);
        if (c < 0)
            error("accept");
        printf("Client accepted\n");

        for (;;) {
            char buf[1000];
            size_t len = rand() & 255;
            ssize_t n = write(c, buf, len ? len : 1);
            if (n <= 0)
                error("write");
            total += n;
            printf("%zu byte%s sent to client\n", n, &"s"[n==1]);
            if (rand() & 1)
                usleep(100);
            if (!(rand() % 11)) {
                printf("Closing connection, %zu byte%s sent\n",
                       total, &"s"[total==1]);
                break;
            }
        }
        close(c);
    }
}

$ cc -Wall -o server server.c
$ cc -Wall -o client client.c

Start the server (which just sends random garbage to the client, once it's accepted)
in a separate Cygwin terminal:

$ ./server 5555

Now run the client from another Cygwin terminal:

$ ./client 5555

You should see the client connecting and receiving (maybe sometimes waiting)
but never having a blank read (EAGAIN) after a successful select() (that was
read-ready).  Try running the client a few times to see how it works.

Now, since the client is not sending anything (or done sending, in the real case
scenario), it'd want to notify the server that it's only going to receive
(by issuing a shutdown() call).  Recompile the client with -DBUG enabled:

$ cc -Wall -DBUG -o client client.c
$ ./client 5555

When you start the client again, you'd see a ton of Exceptions, and all the waits
(select(), but poll() works exactly the same say, checked) return immediately and
the client keeps spinning around the read()s -- most of them are blank with EAGAIN.
In the end, the client does get everything sent to it, though, but with A LOT of
unnecessary CPU cycles.  Now suppose that you have thousands of such clients,
that would create a lot of unnecessary contention.  Try running the client a few
times to see how disastrous those blank reads can be in numbers!

That's not a correct behavior -- you can check that same code running it on Linux
(or BSD -- Mac).  I don't think there's any "exception" in the socket, to begin
with.  Also, it looks like the condition is simply stuck in there and is not
properly re-evaluated (as the I/O still progressing).

Thanks for looking!

Anton Lavrentiev
Contractor NIH/NLM/NCBI


-- 
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple

- Raw text -