Bandwidth issues in the Sprint network

[Brian Raaen]

[V***@vt.edu]

[Scott Weeks]

--- ***@zcorum.com wrote

I am currently having problems get upload bandwidth on a Sprint circuit. I am
using a full OC3 circuit. I am doing fine on downloading data, but uploading
data I can only get about 5Mbps with ftp or a speedtest. I have tested
against multiple networks and this has stayed the same. Monitoring Cacti
graphs and the router I do get about 30Mbps total traffic outbound, but
individual (flows/ip?) test always seem limited. I would like to know if
anyone else sees anything similar, or where I can get help. The assistance I
have gotten from Sprint up to this point is that they find no problems. Due
to the consistency of 5Mbps I am suspecting rate limiting, but wanted to know
if I was overlooking something else
-----------------------------------

I would not use one FTP session to test bandwidth. The rate limiting may be in the FTP software or other area of the computer. Likewise, Speedtest servers (in my opinion) are more marketing tools than testing tools. Try several similarly configured (but separate boxes) FTP servers simultaneously. If you see it go up by a factor of three you've found the issue

I have had to push my four OC-12s to Sprint to the max at times and get full BW. That's in Hawaii, but I imagine it's the same as other areas

scot

[Robert D. Scott]

See if you can find a nother connector that can help with using iperf. Also
make sure any system testing systems have tuned IP stacks. That info is als
linked from the iperf web page.

http://dast.nlanr.net/Projects/Iperf

http://www.psc.edu/networking/projects/tcptune

Robert D. Scott ***@ufl.ed
Senior Network Engineer 352-273-0113 Phon
CNS - Network Services 352-392-2061 CNS Receptionis
University of Florida 352-392-9440 FA
Florida Lambda Rail 352-294-3571 FLR NO
Gainesville, FL 3261

-----Original Message----
From: owner-***@merit.edu [mailto:owner-***@merit.edu] On Behalf O
Scott Week
Sent: Monday, April 07, 2008 5:24 P
To: ***@merit.ed
Subject: Re: Bandwidth issues in the Sprint networ


--- ***@zcorum.com wrote

I am currently having problems get upload bandwidth on a Sprint circuit.
am using a full OC3 circuit. I am doing fine on downloading data, bu
uploading data I can only get about 5Mbps with ftp or a speedtest. I hav
tested against multiple networks and this has stayed the same. Monitorin
Cacti graphs and the router I do get about 30Mbps total traffic outbound
but individual (flows/ip?) test always seem limited. I would like to kno
if anyone else sees anything similar, or where I can get help. Th
assistance I have gotten from Sprint up to this point is that they find n
problems. Due to the consistency of 5Mbps I am suspecting rate limiting
but wanted to know if I was overlooking something else
-----------------------------------

I would not use one FTP session to test bandwidth. The rate limiting may b
in the FTP software or other area of the computer. Likewise, Speedtes
servers (in my opinion) are more marketing tools than testing tools. Tr
several similarly configured (but separate boxes) FTP server
simultaneously. If you see it go up by a factor of three you've found th
issue

I have had to push my four OC-12s to Sprint to the max at times and get ful
BW. That's in Hawaii, but I imagine it's the same as other areas

scot

[Scott Weeks]

------- ***@umn.edu wrote: ----------
I would like to second the recommendation and go one further. Internet2
has released a performance toolkit that is run from CD. I would like to
-----------------------------------------
Robert D. Scott wrote
> See if you can find a nother connector that can help with using iperf. Also
------------------------------------------

The thing to note about most tools like these is you need a box on both sides of the circuit using the same software. One could be 'out there' on the internet, but the further 'out there' your other box is, the less valid your test is

scot

[Martin Hannigan]

Has this circuit ever run clean(normal)

-M


On Mon, Apr 7, 2008 at 1:06 PM, Brian Raaen <***@zcorum.com> wrote
> I am currently having problems get upload bandwidth on a Sprint circuit. I a
> using a full OC3 circuit. I am doing fine on downloading data, but uploadin
> data I can only get about 5Mbps with ftp or a speedtest. I have teste
> against multiple networks and this has stayed the same. Monitoring Cact
> graphs and the router I do get about 30Mbps total traffic outbound, bu
> individual (flows/ip?) test always seem limited. I would like to know i
> anyone else sees anything similar, or where I can get help. The assistance
> have gotten from Sprint up to this point is that they find no problems. Du
> to the consistency of 5Mbps I am suspecting rate limiting, but wanted to kno
> if I was overlooking something else

> -
> Brian Raae
> Network Enginee
> ***@zcorum.co

[Brian Raaen]

[Sam Stickland]

Could be your TCP window size? A 17520 byte TCP window (Windows 2000)
will cause a single flow to top out at 5Mbps at about 50ms. What is the
latency on the link

Try some figures here and see what limit you might be hitting

http://www.wand.net.nz/~perry/max_download.php?bits_per_second=155000000&ack_size=40&no_delayed_acks=2&mss=1460&rtt=35&wsize=17520&ploss=

Sa

Brian Raaen wrote
> I am currently having problems get upload bandwidth on a Sprint circuit. I am
> using a full OC3 circuit. I am doing fine on downloading data, but uploading
> data I can only get about 5Mbps with ftp or a speedtest. I have tested
> against multiple networks and this has stayed the same. Monitoring Cacti
> graphs and the router I do get about 30Mbps total traffic outbound, but
> individual (flows/ip?) test always seem limited. I would like to know if
> anyone else sees anything similar, or where I can get help. The assistance I
> have gotten from Sprint up to this point is that they find no problems. Due
> to the consistency of 5Mbps I am suspecting rate limiting, but wanted to know
> if I was overlooking something else

>

[Brian Raaen]

[Mike Gonnason]

On Tue, Apr 8, 2008 at 9:19 AM, Brian Raaen <***@zcorum.com> wrote
> I have been using the Java based versions of the speed test. At this point
> have had some Sprint people get in contact with me so I will see what the
> find. Thank you for all your help to everyone




> -
> Brian Raae
> Network Enginee
> ***@zcorum.co



> > On Monday 07 April 2008, you wrote
> > I am currently having problems get upload bandwidth on a Sprint circuit.
> am using a full OC3 circuit. I am doing fine on downloading data, bu
> uploading data I can only get about 5Mbps with ftp or a speedtest. I hav
> tested against multiple networks and this has stayed the same. Monitorin
> Cacti graphs and the router I do get about 30Mbps total traffic outbound, bu
> individual (flows/ip?) test always seem limited. I would like to know i
> anyone else sees anything similar, or where I can get help. The assistance
> have gotten from Sprint up to this point is that they find no problems. Du
> to the consistency of 5Mbps I am suspecting rate limiting, but wanted t
> > know if I was overlooking something else
>
> > -
> > Brian Raae
> > Network Enginee
> > ***@zcorum.co
>


Most of the speed test sites on the Internet basically issue a HTT
GET request to a server and time the download. For upload they utiliz
a HTTP POST via a CGI script and time that. The main issue I have wit
these speed tests is that they only use a single TCP session for dat
transfer, which is fine if you have a large or self adjusting TC
window size and a relatively low latency link

However for high capacity links, it is unlikely (but possible) tha
you are planning to use a single TCP session and consume all th
available capacity. Realistically you will have a few doze
server/applications/users and produce hundreds/thousands of TC
sessions which will fully utilize the link

For our PtP customers that have concerns regarding capacity,
generally they suggest setup iperf at both ends and run a few test
with multiple TCP sessions so they can independently verify. Hopefull
Sprint will take your concerns to heart and assist you with testing

-Mike Gonnaso

[Michael Holstein]

> Does anyone know of bootable Linux CD with iperf on it
>

Knoppix STD (security tools distro

http://www.knoppix-std.org/tools.htm

Cheers

Michael Holstei
Cleveland State Universit

[Murphy, William]

Not 100% sure about iperf but I2 has a nice Network Performance Toolki
that runs on top of Knoppix and they have a downloadable ISO image..

Get the ISO here..
http://e2epi.internet2.edu/network-performance-toolkit.htm

Interesting doc on configuring toolkit from SLAC..
http://confluence.slac.stanford.edu/display/IEPM/Network+Performance+To
lki

Bill Murph
Senior Network Analys
University of Texas Health Science Center - Housto

-----Original Message----
From: owner-***@merit.edu [mailto:owner-***@merit.edu] On Behalf O
Michael Holstei
Sent: Wednesday, April 09, 2008 10:00 A
To: ***@iname.co
Cc: ***@merit.ed
Subject: Re: Bandwidth issues in the Sprint networ


> Does anyone know of bootable Linux CD with iperf on it
> =2

Knoppix STD (security tools distro

http://www.knoppix-std.org/tools.htm

Cheers

Michael Holstei
Cleveland State Universit

[Mike Gonnason]

A quick search comes up with Scientific Linux, but I cannot provid
any claims to suitability. I have never even heard of it before, bu
it is provided as a LiveCD

http://linux.web.psi.ch/livecd/software.htm

-Mike Gonnaso

On Wed, Apr 9, 2008 at 6:28 AM, Frank Bulk <***@iname.com> wrote

> Does anyone know of bootable Linux CD with iperf on it

> Fran


> -----Original Message----
> From: owner-***@merit.edu [mailto:owner-***@merit.edu] On Behalf Of Mik
> Gonnaso
> Sent: Wednesday, April 09, 2008 9:05 A
> To: ***@merit.ed
> Subject: Re: Bandwidth issues in the Sprint networ




> On Tue, Apr 8, 2008 at 9:19 AM, Brian Raaen <***@zcorum.com> wrote
> > I have been using the Java based versions of the speed test. At thi
> point
> > have had some Sprint people get in contact with me so I will see wha
> the
> > find. Thank you for all your help to everyone
>
> > -
> > Brian Raae
> > Network Enginee
> > ***@zcorum.co
>
> > > On Monday 07 April 2008, you wrote
> > > I am currently having problems get upload bandwidth on a Sprin
> circuit.
> > am using a full OC3 circuit. I am doing fine on downloading data, bu
> > uploading data I can only get about 5Mbps with ftp or a speedtest.
> hav
> > tested against multiple networks and this has stayed the same
> Monitorin
> > Cacti graphs and the router I do get about 30Mbps total traffic outbound
> bu
> > individual (flows/ip?) test always seem limited. I would like to know i
> > anyone else sees anything similar, or where I can get help. Th
> assistance
> > have gotten from Sprint up to this point is that they find no problems
> Du
> > to the consistency of 5Mbps I am suspecting rate limiting, but wanted t
> > > know if I was overlooking something else
> >
> > > -
> > > Brian Raae
> > > Network Enginee
> > > ***@zcorum.co
> >
>

> Most of the speed test sites on the Internet basically issue a HTT
> GET request to a server and time the download. For upload they utiliz
> a HTTP POST via a CGI script and time that. The main issue I have wit
> these speed tests is that they only use a single TCP session for dat
> transfer, which is fine if you have a large or self adjusting TC
> window size and a relatively low latency link

> However for high capacity links, it is unlikely (but possible) tha
> you are planning to use a single TCP session and consume all th
> available capacity. Realistically you will have a few doze
> server/applications/users and produce hundreds/thousands of TC
> sessions which will fully utilize the link

> For our PtP customers that have concerns regarding capacity,
> generally they suggest setup iperf at both ends and run a few test
> with multiple TCP sessions so they can independently verify. Hopefull
> Sprint will take your concerns to heart and assist you with testing

> -Mike Gonnaso

[r***@rochester.rr.com]

You can also use ttcp from the command line, useful if its cisco on both ends of the circuit

sr01#ttc
transmit or receive [receive]: transmi
Target IP address: 1.1.1.
calculate checksum during buffer write [y]:
perform tcp half close [n]:
send buflen [32768]:
send nbuf [2048]:
bufalign [16384]:
bufoffset [0]:
port [5001]:
sinkmode [y]:
buffering on writes [y]:
show tcp information at end [n]:

ttcp-t: buflen=32768, nbuf=2048, align=16384/0, port=5001 tcp -> 1.1.1.

sr02#ttcp
transmit or receive [receive]:
receive packets asynchronously [n]:
perform tcp half close [n]:
receive buflen [32768]:
bufalign [16384]:
bufoffset [0]:
port [5001]:
sinkmode [y]:
rcvwndsize [32768]:
ack frequency [0]:
delayed ACK [y]:
show tcp information at end [n]:

ttcp-r: buflen=32768, align=16384/0, port=500
rcvwndsize=32768, delayedack=yes tc

---- Michael Holstein <***@csuohio.edu> wrote:
>
>
> > Does anyone know of bootable Linux CD with iperf on it
> >
>
> Knoppix STD (security tools distro
>
> http://www.knoppix-std.org/tools.htm
>
> Cheers
>
> Michael Holstei
> Cleveland State Universit

[Brian Raaen]

[Lincoln Dale]

even with tuned TCP window sizes, make sure you don't have TCP syncookie
enabled on either endpoint

many syncookie implementations have implications on supporting RFC1323 options

cheers

lincoln


> -----Original Message----
> From: owner-***@merit.edu [mailto:owner-***@merit.edu] On Behalf Of Bria
> Raae
> Sent: Friday, 18 April 2008 7:00 A
> To: ***@merit.ed
> Subject: Re: Bandwidth issues in the Sprint networ
>
> Some people wanted to know what I found the problem to be. I hav
> discovered
> the problem for a fact is the TCP window size on uploads. I have a Linux bo
> that I changed the Window sizes to match and I still get 32k on a uploa
> window and 64k on a download window. With a ping time of 50ms I have a ma
> theoretical throughput of 5.2Mbps Which is about what I was getting. Th
> formula to calculate this is the following
>
> (((Ts/Tw)*Rtd)/1000)+((Ts*8)/(Lr*1000))
>
> Where the following ar
>
> Ts = Transfer size in Byte
> Tw = Tcp Window size in Byte
> Rtd = Round trip Delay in millisecond
> Lr = Line rate in bp
>
> At this point I am still trying to locate the offending device that i
> changing the window size. After I determine for sure whether the problem i
> with my router, the sprint network, or another upstream system I will le
> everybody know what I find
>
> -
> Brian Raae
> Network Enginee
> ***@zcorum.co
>
>
> On Monday 07 April 2008, Brian Raaen wrote
> > I am currently having problems get upload bandwidth on a Sprint circuit.
> a
> > using a full OC3 circuit. I am doing fine on downloading data, bu
> uploadin
> > data I can only get about 5Mbps with ftp or a speedtest. I have teste
> > against multiple networks and this has stayed the same. Monitoring Cact
> > graphs and the router I do get about 30Mbps total traffic outbound, bu
> > individual (flows/ip?) test always seem limited. I would like to know i
> > anyone else sees anything similar, or where I can get help. The assistanc
>
> > have gotten from Sprint up to this point is that they find no problems
> Du
> > to the consistency of 5Mbps I am suspecting rate limiting, but wanted t
> kno
> > if I was overlooking something else

[Chris Adams]

Once upon a time, Lincoln Dale <***@interlink.com.au> said
> even with tuned TCP window sizes, make sure you don't have TCP syncookie
> enabled on either endpoint

IIRC Linux (at least) syncookies only come into play when you are bein
syn-flooded (i.e. when the kernel has to start dropping syns). Havin
them enabled at other times has no impact, so there's rarely (if ever)
reason to disable them

--
Chris Adams <***@hiwaay.net
Systems and Network Administrator - HiWAAY Internet Service
I don't speak for anybody but myself - that's enough trouble

[Mike Gonnason]

On Thu, Apr 17, 2008 at 1:00 PM, Brian Raaen <***@zcorum.com> wrote
> Some people wanted to know what I found the problem to be. I have discovered
> the problem for a fact is the TCP window size on uploads. I have a Linux bo
> that I changed the Window sizes to match and I still get 32k on a uploa
> window and 64k on a download window. With a ping time of 50ms I have a ma
> theoretical throughput of 5.2Mbps Which is about what I was getting. Th
> formula to calculate this is the following

> (((Ts/Tw)*Rtd)/1000)+((Ts*8)/(Lr*1000))

> Where the following ar

> Ts = Transfer size in Byte
> Tw = Tcp Window size in Byte
> Rtd = Round trip Delay in millisecond
> Lr = Line rate in bp

> At this point I am still trying to locate the offending device that i
> changing the window size. After I determine for sure whether the problem i
> with my router, the sprint network, or another upstream system I will le
> everybody know what I find


> -
> Brian Raae
> Network Enginee
> ***@zcorum.co




> On Monday 07 April 2008, Brian Raaen wrote
> > I am currently having problems get upload bandwidth on a Sprint circuit.
> a
> > using a full OC3 circuit. I am doing fine on downloading data, bu
> uploadin
> > data I can only get about 5Mbps with ftp or a speedtest. I have teste
> > against multiple networks and this has stayed the same. Monitoring Cact
> > graphs and the router I do get about 30Mbps total traffic outbound, bu
> > individual (flows/ip?) test always seem limited. I would like to know i
> > anyone else sees anything similar, or where I can get help. The assistanc
>
> > have gotten from Sprint up to this point is that they find no problems. Du
> > to the consistency of 5Mbps I am suspecting rate limiting, but wanted t
> kno
> > if I was overlooking something else
>
> > -
> > Brian Raae
> > Network Enginee
> > ***@zcorum.co
>


Thanks for reporting back to curious minds

Mike Gonnaso