When Live Streams go bad: ITV and the Euros

Having lived in Europe for the last 6 years, my family have come to love watching football tournaments. We got hooked while in Croatia in 2016, watching the games in outside pubs overlooking the sea, but the tradition held through the 2018 World Cup (camping on the wet Ireland coast, football was a great excuse to head to a pub and get a hot meal and a pint or two.).This year has not been an exception, as we’ve been following all our favourite teams through the 2020 Euros.

Monday June 28

Monday was a bit of a roller coaster for our family. We are die hard Croatia fans, and down 3-1 with minutes left, Croatia had two quick scores and TIED THE GAME to go to extra time!

The First Live Hiccup

I was doing some work, with iTV streaming on my second monitor. The kids were watching iTV on the TV downstairs. They all started cheering – I looked over, and 15 seconds later, the goal was scored.

Syncing live video is next to impossible. If you are streaming video on two sources in the same room, it is unlikely that they will be perfectly synced. But it appeared at that moment that the TV was about 15-20 seconds ahead of the browser based app that I was using.

I went downstairs to watch the rest of the match, and we sadly watched Croatia just melt apart in extra time, losing the game to Spain.

The Second Hiccup

The 2nd game Monday night was France vs. Switzerland. Many of my colleagues at api.video are French, so I was excited to watch this game and root with them over Slack.

3:23 into the game, ITV stopped responding. Fans across the UK saw the iTV buffering icons, and then it all failed and refused to play any more football.

Now, we’d been having Wi-Fi issues at home, so we bounced the router. No dice. TV still failed. We pulled out a laptop. Same issue. I tried on my work computer – exactly the same failure, even several minutes later into the game.

I then pulled out my Android phone – and the video worked!!! I watched a bit of the game, but then we started to play a game as a family, so I muted my phone, and went on to other things.

This morning, I did a quick Google search, and I found that the UK was fuming because ITV’s apps failed during a BIG match.

https://www.thesun.co.uk/sport/football/15424260/france-switzerland-itv-hub-broken-corner/

Hold on a second: The Android app DID work, while the ITV browser/smart TV app did not.

There’s probably some interesting video work going on here, and I want to figure out what the difference is!

Let’s do some Packet sniffing

To be 100% clear, the data collected for this analysis was not done during a football game, but was just observation of ITV live streams the next day.

I used Fiddler to capture the data, using my Mac as a proxy and routing all the network traffic through Fiddler.

Web Packets

Watching ITV live with Chrome has this pattern of file requests:

The first file is ~ 2KB and is a MPD file. This is a manifest file that describes MPEG-DASH streaming. This text file links to all of the video and audio segments that are used for playback.

The next two files: 73 KB. and 2 MB files are .dash files – are segments of audio and video. These are requested by the player, synced up, and played on the screen.

What have we learned

1. The ITV web app is streaming MPEG-DASH video.
2. The host domain for the Manifest files is: csm-e-ceitvaeuw1live208-0b7cfe9d9db25a210.play.dar.itv.com
3. The host for the MPEG-DASH video files is: dar-itv1simadotcom.itv.com

What I find very interesting about this is most video streams I have seen have the mpd and dash files on the same server. Since the MPD files point to the dash files – both need to be available for the video to successfully play back, so one server for all the content generally seems, well, easier, an 2 just ups the chance of a failure.

Android Packets

Ok, let’s watch the same live program in the Android app:

1. We have new domains. The mobile content is being served through Akamai, and not directly from ITV. Akamai is a content distribution network (CDN) and is built to help deliver content quickly through localised caches across the internet. Note: this does not mean that the ITV domains seen in the Chrome capture are not hosted on a CDN, but it appears that at the very least, the CDNs are DIFFERENT for mobile vs. the web.
2. The mobile streams are HLS – not DASH. the ~300 byte files are m3u8 manifest files, followed by the 1.5 mB .ts files (that contain the video)

So, what caused the issue?

I honestly don’t know. I didn’t go investigate when the video wasn’t streaming – I went to do other things :). In hindsight, I wish I had opened devTools to see which request was failing in Chrome.

What we Do know:

1. The DASH streams were failing, while HLS streams worked. Could ITV have rolled the players over to HLS? I don’t know.
2. The HLS streams use a different CDN setup (with “akamaized” urls). The DASH streams have some Akamai headers i the DASH files, but *not* in the MPD manifest files.
3. The DASH manifest files are on a different server than the video files.

What is interesting is that there are several significant differences in how ITV streams video from the web vs. to mobile. If you favourite live stream DOES go down – perhaps trying on a different device format makes sense.