The Great Tele2.fr Network Outage of 2008

Although this isn't an Open Source blog entry it does show how a lack of Openness and Customer Care will cause your business to lose customers.

I am publishing it here so that others will discover the Customer Care capability of perhaps the worst Internet Service Provider in France - www.tele2.fr.

Please check back for updates and perhaps even a conclusion to the problem.

Day 1 - 20080130 - Wednesday

The Tele2.fr network failed for us sometime in the morning of 30th January 2008 , I noticed it by the red light on the US Robotics 9105 router , after a restart the router came back up but I was unable to browse the internet , sites like www.google.com and jaiku.com would never display.

Strangely though , my bittorrent client was still running albeit more slowly than usual.

I could ping addresses on my LAN so I knew my DNS server was running but bizarrely I could also ping the non-displaying sites above :-

ping google.com
PING google.com (72.14.207.99) 56(84) bytes of data.
64 bytes from eh-in-f99.google.com (72.14.207.99): icmp_seq=1 ttl=244 time=174 ms
64 bytes from eh-in-f99.google.com (72.14.207.99): icmp_seq=2 ttl=244 time=156 ms
64 bytes from eh-in-f99.google.com (72.14.207.99): icmp_seq=3 ttl=244 time=152 ms
64 bytes from eh-in-f99.google.com (72.14.207.99): icmp_seq=4 ttl=244 time=341 ms

--- google.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3000ms
rtt min/avg/max/mdev = 152.981/206.537/341.791/78.531 ms

Next I tried our password protected websites , these would bring up the Apache username and password requester box and allow you to click OK but no site would ever return.

This shows that traffic was leaving our network / router , some was getting back in and the rest being dropped or filtered in some way.

I thought I might be able to get web access through the tor onion routing system as this is able to breach the great firewall of China , but alas this also showed the same packet loss as direct browsing and didn't work.

Inexplicably a few sites still worked directly ; www.ibm.com , www.theregister.co.uk and www.tele2.fr site.

Access to the Skype network was available at all times during the outage on both my Ubuntu workstation and via my Nokia N800 , but I'm not able to access the www.skype.com website.

Gizmoproject didn't fair so well probably due to their non-P2P nature. The VOIP service was marginal sometimes our phone and ATA could register and would work and sometimes not.
Access to the GizmoProject Jabber / XMPP server was also completely blocked to both the GizmoProject client and other Jabber clients.
I tried accessing the freenode.net IRC network with Pidgin both directly and through tor but this always failed to connect.

Secure IMAP access to the google email servers was also patchy , it would take hours for Thunderbird to get enough access to the server to download each message , some with attachments didn't download at all , yet I could still send small text emails without a problem.

I sent and email to support@tele2.fr asking for an explanation and an expected fix date.

Day 2 - 20080131 - Thursday

In the evening my french speaking wife (Marie-Laure) called them and got to speak to Technical Support at tele2.fr , after being asked what router we had , what colour its LEDs were and what our routers internal TCP/IP address was he eventually admitted that its an "Internal Network Fault" at tele2.fr and would be fixed within 48 hours.

Day 3 - 20080201 - Friday

Received a call from Graham Moore who also uses tele2.fr for his ISP , he was also having the same browsing problems. A pain for him but a relief for me as I could now definitely rule out any network or hardware issue here and place the blame firmly at tele2.fr's door.

Received a reply to the email I sent on 20080130 asking me to either call their 3.4 cents per minute hotline or visit their tele2.fr webpage.

Day 4 - 20080202 - Saturday

Two days after sending my email I got a standard reply asking me to ring a Technical Support number , so that night my (french) wife called them and was told it was an "Internal Network Error" at tele2.fr and would be fixed within 48 hours.

In the morning of day four my wife called again to see why we still couldn't browse the internet , this time we were told to change the PPP MTU size in the router from the default (and correct) 1500 down to 1400.
How this packet size change could ever hope to fix a web filtering problem I cannot understand , feel free to enlighten me in the comments.

Day 5 - 20080203 - Sunday

Day 6 - 20080204 - Monday

Depression setting in.

Sent an email to support.fr@tele2.com and Nadege.Lagree@tele2.com asking "What is going on" and "How long is the Network Outage going to take to fix ?"

Day 7 - 20080205 - Tuesday

After a week I could see it was getting nowhere with tele2.fr support and started looking for ways to get around the problem myself.
I first asked my friend Dave for some help with a proxy or ssh connection but got only got a negative response later in the day.
Luckily another friend appeared on Skype and was able to provide me with a working public proxy server and a link to tools.rosinstrument.com which contains a list of more servers - big thanks to you Tom.
This allowed us to get back on the net and connected with the world again. yeehaa!

Got a reply back from the email I sent on 20080204 saying :-
"We acknowledge receipt of your request concerning your internet connection and inform you that an internal network error is still running."

Sent a second email to support@tele2.fr with a link to this blog entry.

Day 8 - 20080206 - Wednesday

After posting to my Jaiku micro-blogging page I am now the number one results on Google for the phrase "tele2.fr outage" , cool.

Day 9 - 20080207 - Thursday

Still no direct Internet access , luckily the proxy servers I am using are still working.

After 3 emails , two phone calls , one web based message and now nine days offline our problem isn't even showing up on the Tele2.fr Network Status page.

Tried downloading the latest linuxoutlaws.com podcast with wget :-

mark@core5:~/Desktop$ wget -c --no-proxy http://cdn2.libsyn.com/linuxoutlaws/Linux_Outlaws_21_-_Beer_from_a_Mug.mp3
--15:04:22--  http://cdn2.libsyn.com/linuxoutlaws/Linux_Outlaws_21_-_Beer_from_a_Mug.mp3
           => `Linux_Outlaws_21_-_Beer_from_a_Mug.mp3'
Resolving cdn2.libsyn.com... 198.78.215.122, 204.160.98.122
Connecting to cdn2.libsyn.com|198.78.215.122|:80... connected.
HTTP request sent, awaiting response... 

and it just sits there forever....

Marie-Laure called again tonight to find out if there has been any progress , this time she was told that the "Internal Network Error" still exists.

The technical support person didn't know how long it was going to take to be fixed and has even stopped giving people fix dates.

Scott stepped up and volunteered the use of his server to run a proxy we could access , unfortunately we haven't configured it fully and its so safe we can't access it yet.
The HTTPS proxy we were using have disappeared so we are unable to access email via the web interface again.

Day 10 - 20080208 - Friday

Still no access to the Web or EMail.

Got a reply from the email I sent on 20080205 saying :- "Someone who speaks english will answer you."

Used Tele2.fr's Contact Page to send them a link to this blog entry.

Day 11 - 20080209 - Saturday

Dave allowed me to install Squid for Windows onto his home server and set it up to give us access , we can now access the World Wide Web again at a blazingly fast 8Mbit!

Thanks Dave.

Day 12 - 20080210 - Sunday

Still no access to the web or Google's IMAPS mail system.

Sent an email to support.fr@tele2.com with a link to this blog entry.

Day 13 - 20080211 - Monday

Tested www , mail and ftp access this morning , still all unavailable.
Almost two weeks of ADSL downtime and still no explanation or apology from tele2.fr support.

I have replied to PetsRus in the poitoucharentes.angloinfo.com forums who is having similar problems as us with tele2.fr's incompetence , feel free to leave a reply there or here below.

Day 14 - 20080212 - Tuesday

Today marks two weeks of downtime for tele2.fr. Just to recap for all those who are joing the party late.

* Services / Protocols blocked by tele2.fr :- http , https , imaps email , tor , ftp , jabber.
* Services / Protocols still available but slow :- ntp , bittorrent , emule , skype , gizmoproject sip.
* Services / Protocols available through a proxy :- http , https.

To date there have been :- no announcements , no explaination , no apology , no official workaround.

Sent another email to support@tele2.fr with a link to this blog entry.

After fourteen days of apathy by tele2.fr and blogging by me , the phrase "tele2 outage" is now the number one result on Google's Search Engine , to celebrate it I have posted a screenshot as my image of the day on photoblog.weloveit.info.

Got a reply back from the email we sent via their web based message system saying :-

We have given all our attention to your request regarding the network problem stopping our customer from surfing the Net.
Please accept our excuses for the delay in replying to it.
We are informing you that the problem still exists and that we cannot give you a fixing date.
However, once the problem solved, our customers will be entitled to ask for a refund on their invoice.

Day 15 - 20080213 - Wednesday

.............Time Passes.

Day 16 - 20080214 - Thursday

Received another reply from tele2.fr support asking us to resend our complaint so that they can provide a more appropriate solution , I replied with a link to this blog entry.

Day 17 - 20080215 - Friday

Direct tele2.fr web browsing and email still unavailable.

I've setup a second proxy on a a friends Ubuntu powered laptop in France so should have a more reliable uptime for browsing.

Day 18 - 20080216 - Saturday

The third weekend without a proper Internet connection begins , how can any modern company be so incompetent ?
This sort of downtime must be intentional , I suspect it is some sort of filtering technology being implemented.
Why don't tele2.fr setup a transparent proxy and route our connections through it ?
Are other ISP's having problems or is it just tele2.fr ? , Anyone using AliceADSL , Free.fr , Orange , Club-Internet care to comment ?
Thanks.

Day 19 - 20080217 - Sunday

Dave has moved his squid proxy server to a PC that is on 24/7 giving us a much more reliable connection , Thanks again dood.

Of course Tele2.fr is still down.

Day 20 - 20080218 - Monday

Tele2.fr still down...

I posted this article to my StumbleUpon page this morning , you can it too by clicking the icon , hopefully this will raise awareness of this issue and perhaps stir tele.fr into some action. Thanks.

We also got a reply today to the email I sent a little over a week ago on 20080207 , it reads :-

After having checked our technicians dossiers, we can see that you have contacted them on Feb 07.
They have informed you about the incident occurring on the network and they have invited you to be patient.
If however, this problem persists, we invite you to call our hotline.

How rude , it sounds like we are disturbing them while they work - like the guys actually fixing the problem are the ones that have to respond to their upset customers emails.
Why should I be patient ? I have paid for ADSL and it hasn't worked now for almost three weeks!

Jaiku is also down today with a hard disk problem , lets hope that the tele2.fr technicians are not working on that too!

Day 21 - 20080219 - Tuesday

Today we celebrate three weeks of downtime for tele2.fr.

Jaiku returned after about 20 hours of downtime , whereas tele2.fr have now passed 500 hours of downtime - appalling service.

Day 22 - 20080220 - Wednesday

More time passes........

Day 23 - 20080221 - Thursday

I wonder if there is a French equivalent to the UK Trading Standards Office ?

Day 23 - 20080222 - Friday

Another day , another dead tele2.fr connection.
Une autre jour , Une autre mort connexion avec tele2.fr.

It only took 23 days for the Police to track down the DC Sniper in the movie D.C. Sniper: 23 Days of Fear , unlike tele2.fr who are incapable of fixing an "Internal network Error in 23 days.

Sent an email to subs@french-news.com from the www.french-news.com website asking if anyone else is suffering a major outage with other ISPs in France.

Could this outage be due to the Cable Cutting Disruption in the Middle East ?

Day 24 - 20080223 - Saturday

The fourth weekend without direct Internet access.

I have posted to the DSL Reports forums here , hopefully this will generate some objective responses.

Day 25 - 20080224 - Sunday

The incompetence of Tele2 has started me looking for alternatives , so far Budget-Telecom's ADSL Offering looks the best - approximately the same price but with the router included in the monthly costs.

MLW has written a letter asking for a new January bill with an updated invoice for the month where they supplied ADSL for only 29 days instead of the billed 31 days ;)

Day 26 - 20080225 - Monday

Posted a reply on a Wall Street Journal article where Amazon's S3 service was down for 2-3 hours , certainly not as bad as tele2.fr's 26 day outage.

In the evening I got the chance to idle in Civray for an hour , I took my Nokia N800 of course and managed to find an open wireless access point and connect.
It was so nice to be able to post to my twitxr account without a proxy server , collect my mail in Modest , connect to my GizmoProject account and chat to Dad via Skype.

Day 27 - 20080226 - Tuesday

Day 28 - 20080227 - Wednesday

Four weeks of downtime with tele2 today , still no web browsing , email , cvs or ftp access.
I have given up sending email to tele2 and I am certainly not wasting money phoning them to find out why their network is still broken.
How can any company be so lame - stupid tele2

Posted this story to swik.net.

Without irony tele2.fr sent us the bill for our Telephone and ADSL today :-

Here they have the cheek to charge us for ADSL after 28 days of no service.

They even have the gall to charge us for calls to their helpdesk enquiring about the problem and when it is likely to be fixed.
You can see we have wasted almost an hour of our lives as well as 4.467 Euros.

Day 29 - 20080228 - Thursday

Day 30 - 20080229 - Friday

Congratulations tele2.fr , your uptime has now sunk below the 50% level for the year so far , compare this to companies like Google who boast a 99.9% uptime. How pathetic.
If you are looking for the worst ISP in France , then give tele2.fr a try.

After a month of tele2.fr lameness and no fix in sight I was staring to get desperate

A Google search for other tele2.fr users revealed a fellow Open Source user who I emailed asking if they were having similar connection problems. Within two hours they had replied and offered suggestions on how to get the problem fixed and a link to some forums where this tele2.fr outage was being discussed.

As per Aurore's information I logged into my USRobotics router via telnet and changed the PPPMTU size :-

ifconfig ppp41 mtu 1462

and then added an extra rule to the iptables firewall :-

iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu

to make these changes permenant through a reboot I did a save :-

save

Later when I looked at the man pages for details of the rules I found this :-

TCPMSS
This target allows to alter the MSS value of TCP SYN packets, to control the maximum size for that connection (usually limiting it to your outgoing interface’s MTU
minus 40). Of course, it can only be used in conjunction with -p tcp. It is only valid in the mangle table.
This target is used to overcome criminally braindead ISPs or servers which block ICMP Fragmentation Needed packets. The symptoms of this problem are that everything
works fine from your Linux firewall/router, but machines behind it can never exchange large packets:
1) Web browsers connect, then hang with no data received.
2) Small mail works fine, but large emails hang.
3) ssh works fine, but scp hangs after initial handshaking.

This man page was written almost five years ago on Mar 09, 2002 , yet almost biblically they have predicted tele2.fr's braindeadness. Amazing!

As expected IMAPS email now works via Thunderbird and Modest , I am also able to connect to the GizmoProject Jabber server via Gajim and my Nokia N800.

Day 31 - 20080301 - Saturday

Day 32 - 20080302 - Sunday

Day 33 - 20080303 - Monday

Day 34 - 20080304 - Tuesday

Day 35 - 20080305 - Wednesday

Day 36 - 20080306 - Thursday

Day 37 - 20080307 - Friday

The connection today seems even less stable if that were possible , although I am connected and my bittorrent client is running access to the web seems slow and incosistant , sometimes it is possible to load a webpage and others the browser just waits while it fetches data. odd
It took me several attempts to upload a the WP-SpamFree plugin to our PhotoBlog.

Day 38 - 20080308 - Saturday

Noticed this reply on the poitoucharentes.angloinfo forums mentioning that the problems have occured becuase of an unannounced change from IPv4 to IPv6.
Myrgen reports that using PPPoE instead of PPPoA, and using the LLC-based multiplex protocol, will make things run smoothly.
I have changed the configuration of my US Robotic 9105 to match and its working.
Thanks Myrgen!

Day 39 - 20080309 - Sunday

Day 40 - 20080310 - Monday

Still no news from tele2.fr , our connection is working with the workaround at the moment although the speed seems slower than normal.

Day 41 - 20080311 - Tuesday

Day 42 - 20080312 - Wednesday

Day 43 - 20080313 - Thursday

Day 44 - 20080314 - Friday

Day 45 - 20080315 - Saturday

Day 46 - 20080316 - Sunday

Graham Moore's www.fastmail.fm account is now threatened with closure due to inactivity because he cannot login due to not Internet access. I have sent fastmail.fm support an email asking for the closure deadline to be put back a month. Fingers crossed.

Day 47 - 20080317 - Monday
...
Day 65 - 20080404 - Saturday

Well we have still had no information from tele2.fr apologising , telling us what happened , how to configure our router or what the hell actually happened two months ago.

The internet connection is currently working and by downloading a well seeded torrent so it looks like its working at full speed.

Big thanks go out to Dave Jenkins for supplying us with a way to surf during the outage.

Day 70 - 20080409 - Wednesday

Tele2.fr are now threatening to cut our ADSL connection as we haven't paid the February bill , nice considering the fact that they didn't supply us with working ADSL during that month.

Day ?? - 20080506 - Tuesday

Marie-Laure hasn't been able to get to www.plantes-et-jardins for a while yet when I asked my Dad in the UK he could access it without any problems.

I checked the proxy server and blocklist and they didn't seem to be the problem.

As a test I changed the PPP MTU from 1462 down to 1400 and suddenly the site was available. odd.