X-Recipient: archive-cygwin AT delorie DOT com
X-SWARE-Spam-Status: No, hits=-1.9 required=5.0	tests=BAYES_00,RFC_ABUSE_POST,T_RP_MATCHES_RCVD
X-Spam-Check-By: sourceware.org
From: "Estabrook, Edward" <Edward DOT Estabrook AT Polycom DOT com>
To: "cygwin AT cygwin DOT com" <cygwin AT cygwin DOT com>
Date: Fri, 10 Dec 2010 14:37:31 -0800
Subject: Bug? UDP packet transmitted via sendto() intermitently not being transmitted to network
Message-ID: <4483E11778CA964F9644B68621E1CA4646FAD19D@CRPMBOXPRD02.polycom.com>
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
Mailing-List: contact cygwin-help AT cygwin DOT com; run by ezmlm
List-Id: <cygwin.cygwin.com>
List-Subscribe: <mailto:cygwin-subscribe AT cygwin DOT com>
List-Archive: <http://sourceware.org/ml/cygwin/>
List-Post: <mailto:cygwin AT cygwin DOT com>
List-Help: <mailto:cygwin-help AT cygwin DOT com>, <http://sourceware.org/ml/#faqs>
Sender: cygwin-owner AT cygwin DOT com
Mail-Followup-To: cygwin AT cygwin DOT com
Delivered-To: mailing list cygwin AT cygwin DOT com

Hello,
I have a fairy complicated SIPp scenario running in cygwin-1.7.7(0.230/5/3)=
 on Windows XP which is losing a single UDP packet intermittently.  At this=
 point I am fairly confident SIPp is not the culprit and that the packet is=
 being dropped 'somewhere' between the sendto() call and wireshark / being =
transmitted to the interface. The same test case running on Linux (Fedora 9=
) works consistently.=20=20

I have shown (via debugger and printf statements) that the missing packet i=
s being transmitted via the sendto() system call with the appropriate messa=
ge buffer, size and socket information and yet is not being placed on the w=
ire.  Indeed detailed analysis of SIPp execution shows that the sequence of=
 steps within SIPp is identical in the working and failing case.  As a resu=
lt I am now concerned that there may be something amiss with the UDP packet=
 processing layer in cygwin.  Within the same SIPp scenario, it is always t=
he same packet that is lost if one is lost, but the loss occurs only about =
50% of the time it is executed.  The packet is a SIP 200 OK response which =
is supposed to be transmitted after the script has received (and ignored) ~=
45 UDP REGISTER messages spread out over ~3 minutes.=20=20

I will explain the symptom in more detail a little further on in this e-mai=
l.

Generally I am seeking information regarding:
1) Where can I download and install an older version of Cygwin (say 1.5 era=
) to test if the problem is newly introduced?
2) What tools are available to measure potential packet loss within cygwin =
itself? (the Windows stack is not reporting dropped UDP packets, and the pa=
ckets are not even making it into a locally running instance of wireshark).=
  Is there a way to dump / access / sniff the inputs and outputs to the cyg=
win IP stack?  Traffic volume is low so verbosity would not pose a problem.
3) Is there additional documentation regarding the cygwin IP translation la=
yer?  I see in the release notes that various enhancements have been made r=
ecently, and I cannot help but worry that perhaps an obscure issue in this =
aggregation of packets for delivery to Windows is introducing a problem.  I=
s it possible to shut this off and simply force each packet to be delivered=
 without additional buffering?

Unfortunately the only steps to reproduce (that I am yet aware of) is a 50-=
step sequence of SIP signaling which would be difficult to detail sufficien=
tly here.  I will cover some pertinent details below.

The Windows XP system is configured with several IPs associated with an int=
erface.  Two parallel instances of SIPp are started, each associated with a=
 different IP (but using the same MAC address).  SIPp is simulating two SIP=
 proxies while a SIP device sends and receives traffic.  The traffic repres=
ented by the scenario is ~50 packets over 3 minutes (though the interface i=
s in promiscuous mode).  I have included the scenario details below.  It is=
 always the packet marked  '48  <---------- 200(11)' which is dropped, but =
dropping only happens about =BD the time.  At this point I am fairly confid=
ent SIPp is not the culprit and that the packet is being dropped 'somewhere=
' between sendto() and wireshark / the interface.

The code:
DEBUG("sendto(%d, buffer, %d, 0, %s:%hu [&=3D%p], %d)",=20
socket->ss_fd, len, inet_ntoa( ((struct sockaddr_in*)dest)->sin_addr ),=20
ntohs(((struct sockaddr_in*)dest)->sin_port),=20
dest,=20
SOCK_ADDR_SIZE(dest));
rc =3D sendto(socket->ss_fd, buffer, len, 0, (struct sockaddr *)dest, SOCK_=
ADDR_SIZE(dest));
DEBUG_OUT("return %d", rc);

Produces this debug output:
socket_write_primitive() in sipp.cpp:2484 - sendto(7, buffer, 491, 0, 172.2=
3.79.237:5060 [&=3D0x6fc8b8], 16)
socket_write_primitive() in sipp.cpp:2492 - (Leaving) - return 491
and I have verified that the buffer contents are correct.  The packet, howe=
ver, does not appear in wireshark or the network.

Milliseconds later SIPp executes the same code path to generate a SUBSCRIBE=
 message (step 49 below) and this packet is transmitted.=20=20

All suggestions will be much appreciated!
                                Messages  Retrans   Timeout   Unexpected-Msg
1  ----------> REGISTER(1 )           1         0         0         0=20=20=
=20=20=20=20=20=20
  <---------- 200(1 )                1         0=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
  <---------- SUBSCRIBE(2 )          1         0=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
  ----------> 200(2 )                1         0         0         0=20=20=
=20=20=20=20=20=20
  ----------> NOTIFY(2 )             1         0         0         0=20=20=
=20=20=20=20=20=20
  <---------- 200(2 )                1         0=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
  ----------> SUBSCRIBE(3 )          1         0         0         0=20=20=
=20=20=20=20=20=20
  <---------- 202(3 )                1         0=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
  <---------- NOTIFY(3 )             1         0=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
10  ----------> 200(3 )                1         0         0         0=20=
=20=20=20=20=20=20=20

  ----------> REGISTER(1 )           1         0         0         0=20=20=
=20=20=20=20=20=20
  <---------- 200(1 )                1         0=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
              [ NOP ]=20=20=20=20=20=20=20=20=20=20=20=20=20=20
  ----------> NOTIFY(2 )             1         0         0         0=20=20=
=20=20=20=20=20=20
  <---------- 200(2 )                1         0=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
              [ NOP ]=20=20=20=20=20=20=20=20=20=20=20=20=20=20
  ----------> INVITE(4 )             1         0         0         0=20=20=
=20=20=20=20=20=20

  <---------- 183(4 )                1         0=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
  ----------> NOTIFY(2 )             1         0         0         0=20=20=
=20=20=20=20=20=20
20  <---------- 200(2 )                1         0=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
  <---------- 200(4 )                1         0=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
  ----------> ACK(4 )                1         0         0         0=20=20=
=20=20=20=20=20=20
  ----------> NOTIFY(2 )             1         0         0         0=20=20=
=20=20=20=20=20=20
  <---------- 200(2 )                1         0=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
  [   2000ms] Pause              1                             0=20=20=20=
=20=20=20=20=20
              [ NOP ]=20=20=20=20=20=20=20=20=20=20=20=20=20=20
  ----------> INVITE(4 )             1         0         0         0=20=20=
=20=20=20=20=20=20
  ----------> INVITE(4 )             1         0         0         0=20=20=
=20=20=20=20=20=20
  ----------> INVITE(4 )             1         0         0         0=20=20=
=20=20=20=20=20=20
30  ----------> INVITE(4 )             1         0         0         0=20=
=20=20=20=20=20=20=20
  ----------> INVITE(4 )             1         0         0         0=20=20=
=20=20=20=20=20=20
  ----------> INVITE(4 )             1         0         0         0=20=20=
=20=20=20=20=20=20
  ----------> INVITE(4 )             1         0         0         0=20=20=
=20=20=20=20=20=20

  ----------> REGISTER(1 )           1         0         0         0=20=20=
=20=20=20=20=20=20
  ----------> REGISTER(1 )           1         0         0         0=20=20=
=20=20=20=20=20=20
  ----------> REGISTER(1 )           1         0         0         0=20=20=
=20=20=20=20=20=20
  ----------> REGISTER(1 )           1         0         0         0=20=20=
=20=20=20=20=20=20
  ----------> REGISTER(1 )           1         0         0         0=20=20=
=20=20=20=20=20=20
  ----------> REGISTER(1 )           1         0         0         0=20=20=
=20=20=20=20=20=20
40  ----------> REGISTER(1 )           1         0         0         0=20=
=20=20=20=20=20=20=20
41  ----------> REGISTER(1 )           1         0         0         0=20=
=20=20=20=20=20=20=20
42  ----------> REGISTER(1 )           1         0         0         0=20=
=20=20=20=20=20=20=20
43  ----------> REGISTER(1 )           1         0         0         0=20=
=20=20=20=20=20=20=20
44  ----------> REGISTER(7 )           1         0         0         0     =
   29961878-68089d97-40475bea AT 172 DOT 23 DOT 79 DOT 237
45  ----------> REGISTER(7 )           1         0         0         0=20=
=20=20=20=20=20=20=20
46  ----------> REGISTER(7 )           1         0         0         0     =
   29961878-68089d97-40475bea AT 172 DOT 23 DOT 79 DOT 237
47  ----------> REGISTER(11)           1         0         0         0     =
   f63768a9-a13cc854-432ed2c3 AT 172 DOT 23 DOT 79 DOT 237
48  <---------- 200(11)                1         0=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
49  <---------- SUBSCRIBE(12)          1         0=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
50  ----------> 200(12)                0         0         0         1=20=
=20=20=20=20=20=20=20
  ----------> NOTIFY(12)             0         0         0         0=20=20=
=20=20=20=20=20=20
  <---------- 200(12)                0         0=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
  ----------> SUBSCRIBE(13)          0         0         0         0=20=20=
=20=20=20=20=20=20
  <---------- 202(13)                0         0=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
  <---------- NOTIFY(13)             0         0=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
  ----------> 200(13)                0         0         0         0=20=20=
=20=20=20=20=20=20

  [   1000ms] Pause              0                             0=20=20=20=
=20=20=20=20=20

              [ NOP ]=20=20=20=20=20=20=20=20=20=20=20=20=20=20
  ----------> NOTIFY(12)             0         0         0         0=20=20=
=20=20=20=20=20=20
  <---------- 200(12)                0         0=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
              [ NOP ]=20=20=20=20=20=20=20=20=20=20=20=20=20=20
  ----------> INVITE(14)             0         0         0         0=20=20=
=20=20=20=20=20=20

  <---------- 183(14)                0         0=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
  ----------> NOTIFY(12)             0         0         0         0=20=20=
=20=20=20=20=20=20
  <---------- 200(12)                0         0=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
  <---------- 200(14)                0         0=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
  ----------> ACK(14)                0         0         0         0=20=20=
=20=20=20=20=20=20
  ----------> NOTIFY(12)             0         0         0         0=20=20=
=20=20=20=20=20=20
  <---------- 200(12)                0         0=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
  ----------> REGISTER(11)           0         0         0         0=20=20=
=20=20=20=20=20=20
  <---------- 200(11)                0         0=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
  ----------> REGISTER(11)           0         0         0         0=20=20=
=20=20=20=20=20=20
  <---------- 200(11)                0         0=20=20=20=20=20=20=20=20=20=
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
------- Waiting for active calls to end. Press [q] again to force exit. ---=
----

Best Regards,
Ed

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple