Mailing-List: contact cygwin-developers-help AT sourceware DOT cygnus DOT com; run by ezmlm
List-Subscribe: <mailto:cygwin-developers-subscribe AT sources DOT redhat DOT com>
List-Archive: <http://sources.redhat.com/ml/cygwin-developers/>
List-Post: <mailto:cygwin-developers AT sources DOT redhat DOT com>
List-Help: <mailto:cygwin-developers-help AT sources DOT redhat DOT com>, <http://sources.redhat.com/ml/#faqs>
Sender: cygwin-developers-owner AT sources DOT redhat DOT com
Delivered-To: mailing list cygwin-developers AT sources DOT redhat DOT com
Date: 15 Nov 2001 16:21:56 -0500
Message-ID: <20011115212156.5563.qmail@lizard.curl.com>
From: Jonathan Kamens <jik AT curl DOT com>
To: cygwin-developers AT cygwin DOT com
Subject: TCP connections can occasionally fail because of a winsock bug
Mime-Version: 1.0
Content-Type: multipart/mixed; boundary="Multipart_Thu_Nov_15_16:21:56_2001"
Content-Transfer-Encoding: 7bit

--Multipart_Thu_Nov_15_16:21:56_2001
Content-Type: text/plain; charset=US-ASCII

If you run "ssh <any-host> date" over and over again with the current
openssh package from Cygwin installed, it will occasionally fail.
You're more likely to get it to happen if you run a bunch of loops,
e.g.:

  HOST=<whatever>
  while ssh $HOST date > /dev/null; do true; done &
  while ssh $HOST date > /dev/null; do true; done &
  while ssh $HOST date > /dev/null; do true; done &
  while ssh $HOST date > /dev/null; do true; done &
  while ssh $HOST date > /dev/null; do true; done &
  while ssh $HOST date > /dev/null; do true; done &
  while ssh $HOST date > /dev/null; do true; done &
  while ssh $HOST date > /dev/null; do true; done &

Eventually, all the loops will exit because they'll all encounter a
failure.  This will happen more quickly on SMP machines, but it'll
eventually happen on uniprocessor machines too (although you may need
to run more loops concurrently to make it happen).

I've dug deeply enough into this to determine that I believe the
problem is caused by a bug in winsock.  I can get the problem to
manifest itself completely independently from Cygwin.  See the full
description in the attached program, which one of my coworkers with an
MSDN subscription is going to forward to Microsoft to see what they
have to say about it.

This problem is in no way unique to ssh.  Any program which uses
connect() to connect to remote hosts could occasionally fail.  Note,
furthermore, that the problem still occurs if the client program
doesn't bind() the socket before the connect() -- the only reason I do
the bind() in the sample program below is to make it possible to print
out the local port number after a failure.

I don't think there's anything that Cygwin can do about this.  I can't
imagine the disgustingness that would ensue if we attempted to code a
workaround for this failure.  Yech.  I'm just sending this message to
cygwin-developers as a heads-up so that if anybody encounters any
mysterious TCP connection failures with Cygwin in the future, you'll
know that there's a possibility that the failures have nothing to do
with Cygwin.

The reason why this started happening in ssh when it wasn't happening
before is this change which was checked into the OpenSSH source tree
on August 6:

   - markus AT cvs DOT openbsd DOT org 2001/07/25 14:35:18
     [readconf.c ssh.1 ssh.c sshconnect.c]
     cleanup connect(); connection_attempts 4 -> 1; from 
     eivind AT freebsd DOT org

That is, before August 6, ssh would attempt to connect four times by
default before giving up; after August 6, it only attempts to connect
once.  We should probably do something in the Cygwin ssh package to
work around the problem; I'll send a separate message to cygwin-apps
about that.

  Jonathan Kamens

--Multipart_Thu_Nov_15_16:21:56_2001
Content-Type: text/plain; name="doecho.c"; charset=US-ASCII
Content-Transfer-Encoding: 7bit

/*

  This program illustrates a bug in the Windows winsock layer.  This
  bug manifests itself both on Windows NT 4.0 and Windows 2000.  The
  bug is that the winsock layer is apparently willing to assign a
  local port number to a socket which is actually already in use, such
  that when the program later tries to connect() the socket to a
  remote port, the connection fails with WSAEADDRINUSE.

  When you run the program, it will keep creating and closing socket
  connections over and over again.  You will notice in the output that
  once it gets WSAEADDRINUSE for a particular local port, it is never
  given that port again by winsock; apparently, winsock keeps track
  *after the fact* of which local ports are in use, but doesn't notice
  before the fact.  Eventually, bind will fail rather than connect,
  with WSAENOBUFS rather than WSAEADDRINUSE, and then the program will
  exit.

  If you run it again immediately after it exits, you will notice that
  it ends up using many of the ports which it was previously avoiding
  using.  So, apparently, once winsock decides that a port is in use,
  it stops using it in that process even when it becomes available
  again.  It could be argued that that's a second bug, in addition to
  the first bug that it assigns an in-use port in the first place.
*/

/*

  With Visual Studio, compile this program as "cl doecho.c
  ws2_32.lib".

  With Cygwin, compile this program with "gcc -mno-cygwin -mwindows
  doecho.c -lws2_32" if you want a native Windows version or "gcc
  doecho.c -lws32_32" for a Cygwin version.  Both the native Windows
  and Cygwin versions illustrate the bug.

  With Linux, compile the program with "gcc doecho.c".  You can then
  run it essentially forever and it will never print any errors, thus
  illustrating that there is a bug in winsock that isn't in Linux.

*/

/*

  Run the program with two arguments -- an IP address in dotted quad
  notation, and a port number.

  Separately from this program, you need to run a listener on the
  specified address and port.  All the listener should do is accept
  and close connections.  Something like this Perl script:
   
  use Socket;

  ($port = shift) || die;

  socket(ACCEPTOR, AF_INET, SOCK_STREAM, 0) || die;

  $iaddr = INADDR_ANY;
  $sockaddr = sockaddr_in($port, $iaddr);

  setsockopt(ACCEPTOR, SOL_SOCKET, SO_REUSEADDR, pack("l", 1));

  bind(ACCEPTOR, $sockaddr) || die;
  listen(ACCEPTOR, SOMAXCONN) || die;

  while (accept(CLIENT, ACCEPTOR)) {
      close(CLIENT);
  }

*/

#ifndef WIN32
#ifdef _MSC_VER
#define WIN32
#endif
#endif

#include <sys/types.h>
#include <stdio.h>
#include <string.h>
#ifdef WIN32
#include <winsock2.h>
#else
#include <sys/socket.h>
#include <netinet/in.h>
#endif

#ifndef WIN32
#define wsaperror perror
#else
void wsaperror(char *);
#endif

main(int argc, char *argv[])
{
#ifdef WIN32
  WSADATA wsaData;
#endif
  int succeeded = 0;
    struct sockaddr_in addr = {0};
    int i;
    char *endptr = NULL;

    if (argc != 3) {
      fprintf(stderr, "Must specify IP address and port arguments\n");
      exit(1);
    }

    addr.sin_family = AF_INET;
    if ((addr.sin_addr.s_addr = inet_addr(argv[1])) == INADDR_NONE) {
      fprintf(stderr, "First argument must be IP address\n");
      exit(1);
    }
    addr.sin_port = strtol(argv[2], &endptr, 10);
    if (! (*argv[2] && endptr && ! *endptr)) {
      fprintf(stderr, "Second argument must be port number\n");
      exit(1);
    }
    addr.sin_port = htons(addr.sin_port);

#ifdef WIN32
  if (WSAStartup(MAKEWORD(2, 2), &wsaData) != 0) {
    wsaperror("WSAStartup");
    exit(1);
  }
#endif
  while (1) {
    int s = socket(AF_INET, SOCK_STREAM, 0);
    struct sockaddr_in local_addr = {0};
#ifdef WIN32
    int name_len = sizeof(local_addr);
#else
    socklen_t name_len = sizeof(local_addr);
#endif

    if (s < 0) {
      wsaperror("socket");
      break;
    }

    i = 1;
    if (setsockopt(s, SOL_SOCKET, SO_REUSEADDR, (char *) &i, sizeof(i)) < 0) {
      wsaperror("setsockopt");
      break;
    }

    local_addr.sin_family = AF_INET;
    local_addr.sin_addr.s_addr = INADDR_ANY;

    if (bind(s, (struct sockaddr *) &local_addr, sizeof(local_addr)) < 0) {
      wsaperror("bind");
      break;
    }

    if (getsockname(s, (struct sockaddr *) &local_addr, &name_len) < 0) {
      wsaperror("getsockname after bind");
      break;
    }

    if (connect(s, (struct sockaddr *) &addr, sizeof(addr)) < 0) {
      fprintf(stderr, "Local port failed: %d\n", ntohs(local_addr.sin_port));
      wsaperror("connect");
      continue;
    }

    succeeded++;

    if (getsockname(s, (struct sockaddr *) &local_addr, &name_len) < 0) {
      wsaperror("getsockname");
      break;
    }

    if (send(s, "foo\n", 4, 0) < 0) {
      wsaperror("send");
      break;
    }

/*      shutdown(s, 2); */

#ifdef WIN32
    closesocket(s);
#else
    close(s);
#endif

    fprintf(stderr, "%d\n", ntohs(local_addr.sin_port));
    fflush(stdout);
  }

  fprintf(stderr, "%d connections succeeded\n", succeeded);

#ifdef WIN32
  WSACleanup();
#endif
  exit(1);
}

#ifdef WIN32
void wsaperror(char *str) 
{
  int err = WSAGetLastError();
  char *errstr;

  switch (err) {
  case WSAEADDRINUSE:
    errstr = "WSAEADDRINUSE";
    break;
  case WSAENOBUFS:
    errstr = "WSAENOBUFS";
    break;
  default:
    errstr = "Unknown error";
    break;
  }
  
  fprintf(stderr, "%s: Error %d (%s)\n", str, err, errstr);
}
#endif
--Multipart_Thu_Nov_15_16:21:56_2001--