Daily Archives: Tuesday, February 25, 2014

Feb 26, 2014 Update:

Verizon appears to have solved the problem on their end
and an MTU of 1500 will work again. The original problem lasted from at least the beginning
of Feb 24, 2014 until mid-day Feb 26, 2014.

Verizon’s escalation team later called me back and provided the following information:

New uplinks were installed in Lewisville and Plano which were improperly configured and the configuration has been corrected.

The information below remains for historical information on the problem and a workaround.

Background on the Problem

I live in the Dallas market for Verizon FiOS which is where it seems the problems are happening.
The issue manifests primarily for me as an inability to play content reliably on YouTube,
but I had many other issues on other CDNs. Trying to view other sites, such as MitchRibar.com, would never succeed for some other FiOS subscribers and me.

The error displayed in Chrome was ERR_CONNECTION_RESET.

List of sites affected

The following sites were reported affected by friends, others on the Internet, and me:

• YouTube
• stackoverflow.com
• icloud.com
• appleid.apple.com
• packagist.org
• 500px.com
• blog.iso50.com
• mitchribar.com
• youtrack.jetbrains.com
• upi.com
• soundcloud.com
• azlyrics.com
• EA Games
• Battlefield 4 (Online Gaming including Xbox Live)
• Seemingly random problems on other sites usually when loading CDN resources

Diagnosis

I called Verizon multiple times and they were no help of course. Since a ping worked and since a traceroute exited their network they said that the problem was either on my side or YouTube’s side and didn’t care that it affected multiple sites. I tried to explain the difference between ICMP and a TCP session but they aren’t that smart, of course. They wouldn’t even talk to me until I plugged in their router which is a terrible piece of equipment I haven’t used for years. I obliged and that’s when I got the above from them. They would not let me talk to a higher tier of support.

However, after finding out some friends had the same issue, I was tipped off to this forum post by started by another Verizon customer. You’ll see most if not all in the thread are from the Dallas market.

I did some ping tests to validate that the problem is the MTU setting. Somewhere in Verizon’s network, close to the DFW side of the route, someone has messed up the MTU and reduced it from the default of 1500 to 1496. Keep in mind that there is a 28 byte header so the successful (non-fragmented) ping size + 28 = MTU.

In the below paste, -f prevents fragmentation of the packet and -l 1472/1468 sets the ping packet length. Keep in mind that the IP header adds 28 bytes and also ping parameters are different on different platforms. This example is from Windows, but check the parameters for your platform to set these options.

>ping mitchribar.com -f -l 1472

Pinging mitchribar.com [205.134.224.227] with 1472 bytes of data:
Request timed out.

Ping statistics for 205.134.224.227:
    Packets: Sent = 1, Received = 0, Lost = 1 (100% loss),
Control-C
^C

>ping mitchribar.com -f -l 1468

Pinging mitchribar.com [205.134.224.227] with 1468 bytes of data:
Reply from 205.134.224.227: bytes=1468 time=42ms TTL=55

As you can see, 1472 (which equates to a 1500 MTU [1472+28=1500]) did not work. Lowering it until I got to 1468 worked, which equates to an MTU of 1496, so you can see, because of Verizon’s now-broken network, we must lower the MTU from the default of 1500 to 1496 to ensure the packets traverse the network correctly.

I don’t use Verizon’s router (I use DD-WRT and changing the MTU is easy: Setup -> MTU) but Jake Smith provided these screenshots to me that I edited to show the steps. They come from a regular Verizon FiOS router.

Steps to Change the MTU

Connect to your router using a web browser

There are many resources available to find out what your router’s LAN IP address is. Connect to this address in your router. Your FiOS router has the default password printed on it if you have not changed it.

My router’s IP is 192.168.0.1, but yours may (and probably will) be different. With my example, I would navigate to http://192.168.0.1 in my web browser.