You may have read that Microsoft recently had their Youtube API key revoked from Google, leaving their visually impressive app more or less dead in the water (Source). This story followed a previous cease and desist from Google on the grounds that an earlier incarnation of the application did not properly display advertisements for certain content (Source). What may not be clear is that these are but a couple of well reported responses in what has actually been a series of ongoing changes that Youtube has been employing behind the scenes. Changes that have long served to thwart developers from creating a stable Youtube client for Windows Phone.
Let’s start by taking a look at what it takes to playback a video on Windows Phone. Initially, Microsoft provided a Youtube application on Windows Phone that was little more than a wrapper around m.youtube.com. This mobile version of Youtube is a stripped down version of the official youtube.com site that serves up videos that are encoded for html 5 playback. Not only does the mobile site give less interesting search results than its counterpart, it serves up results that look relevant upon glance but users are unfortunately met with the harrowing “This video is not available on mobile” message upon clicking on many of these results. To try this experience for yourself on desktop, simply install User-Agent Switcher to your browser and select Windows Phone.
I used the Chrome extension available here to obtain the screenshot below. This is the result of entering a standard youtube URL (http://www.youtube.com/watch?v=bnVUHWCynig) which is then redirected to (http://m.youtube.com/watch?v=bnVUHWCynig&desktop_uri=%2Fwatch%3Fv%3DbnVUHWCynig)
Developers took notice of the shortcomings of this Youtube wrapper and began creating their own applications which provided enhanced features like in-app video playback without the need to invoke the wrapper application. At this point in time, the two heavy-weights contenders in this space were MetroTube and SuperTube. The concept for achieving playback was relatively the same across most apps at the time. A web request would be sent to a youtube url typically with the parameter nomobile=1 appended to ensure the desktop page was retrieved (i.e. http://www.youtube.com/watch?v=bnVUHWCynig&nomobile=1). Then the page was simply parsed with a regex to pull out the mp4 link from the page source (http:[^;,]*?itag=18[^;,]*type=video\/mp4.*quality=medium). This would then be fed into a media control capable of mp4 playback. Early on this was usually done with the Silverlight Media Framework (now MMP). It should be noted that this link is tied to ip, double url-encoded, and is extremely long.
This worked for awhile, but eventually, you saw apps breaking across the Windows Phone ecosystem. MetroTube was even removed from the Windows Store. Server-side, Youtube made slight adjustments requiring a change to the regex that was currently baked-in to most apps. Specifically a variable named ‘signature’ was renamed ‘sig’. This fact meant that regex wouldn’t be enough since a string replace had to be performed. The solution was essentially to carry on as usual but replace “sig=” with “signature=” and you had a working mp4 link. As a result many applications had to recertify with this change applied in order to work properly. Youtube then got wise to this and began randomly ordering the variables required to construct the mp4 url. Many developers employed a solution which re-ordered the values into key value pairs allowing for easy modification and reconstruction of the mp4 url.
Thus, taking into account finding the mp4 link, random ordering of variables, and need to perform string replace, solutions started to get complex. For specifics, I suggest referring to Download YouTube Videos as MP4 over on userscripts.org. The discussion there also give a nice rundown of the breaking changes introduced by Youtube over the last couple years and are very much up to date.
At this point, many developers realized they can’t bake code into apps anymore and needed to be able to inject turing complete code that could enact on the payload. A proof of concept titled BulletProof In-App Youtube Video Playback for WP7 using MMP : POC appeared on codeplex.com that used a hidden browser control to run remote hosted javascript allowing for server side changes to be deployed en masse to Windows Phone applications in the event that Youtube were to make breaking changes. Youtube did make further changes, changing the variable “sig” to simply the letter “s” and most apps were able to keep up with this change rather easily.
Then came encrypted signatures. Typically, when a video was encountered that had the “s” signature you would perform a string replace to “signature” but it later became evident that these types of videos had an encrypted signature in the youtube link. A decryption function that was contained in the associated player was required to assemble a proper signature. For a while, these decryption functions were relatively static and could be added into a remote parser. However, Youtube eventually began rolling these functions from weekly, to daily, and currently hourly. Apps like MetroTube and SuperTube currently do not allow for playback of these types of content. You can test this by attempting to playback any Vevo published music videos as they all contain encrypted signatures. What is odd here is that Windows Phone has an official Vevo application, so why is this particular content singled out?
In order to obtain the decryption function for these videos requires getting the url of the associated player which is contained in the video page and parsing out that function and then applying it to your ip-assigned signature. The player is of course hosted on a domain seperate from youtube.com (s.ytimg). This means, that parsers using remotely hosted javascript would need to have a remote web request shim baked into the client or use a jsonp proxy to break out of same origin policy in order to make the request to the player, obtain the decryption function, and then apply it in the parser.
Assuming you have all of that figured out, Youtube applied another change yesterday that likely intentionally singles out Webrequests from Windows Phone 8. Applications which obtain the video page using a WebRequest on WP8 (NOT WP7) are given a 303 redirect to http://www.youtube.com/das_captcha?next=/. Specifically, if a request is sent in the form:
GET /watch?v=AWpsOqh8q0M&feature=youtube_gdata HTTP/1.1
Accept: */*
Accept-Encoding: identity
User-Agent: NativeHost
Host: www.youtube.com
Connection: Keep-Alive
Playing back the above request, you can see that Youtube is now redirecting requests to videopage urls made via WebRequest on WP8.
This is because, the default behavior of WebRequest on WP8 does not supply a referrer value. This is an anomaly that does not exist on WP7. The behavior was first reported a year ago on Tomahawk’s blog. Thus, Youtube can in theory assume that Web Requests with a User-Agent value of “NativeHost” that are missing a Referer value are coming from a WP8 device and as such are given the following payload which renders as a “large volume of requests” warning when viewed in a browser:
To confirm this behavior I proxied a WP8 dev device through Fiddler and captured the responses, even swapping to three different networks which had not made any requests to Youtube. I would like to point out that this redirection was not occurring 24 hours ago. A partner publisher of mine confirmed that his WP8 apps stopped playing videos suddenly last night but was confused because his WP7 variants were unaffected. I would also like to stress that there was no difference in how video playback was achieved in both versions of the app. Upon capturing responses from both apps, only the WP8 app was redirected. The key difference being that the WP7 app supplies a Referer value in its header whereas the WP8 app lacks one. If any random value is given to an appended Referer value when playing back the original request, the proper video page payload is returned. This may be a coincidence, but this could only have been enabled due to a Server Side change on Youtube’s end.
For curiosity, I also watched the requests being sent by other Youtube clients to see what was happening. It was interesting to note the fallback methods certain applications employed when encountering playback failure. Fortunately, MetroTube spoofs the UserAgent value and is currently immune to this issue.
Before closing, I would like to express some opinion on the matter. It is definitely not 100% morally sound for developers to have to resort to reverse engineering Youtube’s protocols, and I do not condone the practice if it harms content producers. However, Youtube has essentially left WP developers no other choice. On alternative platforms, official video players exist to allow a sensible playback experience. Microsoft recently went through the trouble of reverse engineering Youtube’s own ad serving protocol to attempt to solve the content provider problem for good after being previously ousted and was told again that their app can not be allowed. That’s what happened to Microsoft, imagine what the indie developers are feeling. This raises the question, does Youtube enforce cryptography to protect content providers, to thwart competition, or both? No one wants to write rolling regexes and untestable injected code. We just want a sensible Youtube experience on Windows Phone. At this time, only the old dogs who have been fighting the battle for the last 2+ years really know what to do when it comes to Youtube client development on Windows Phone and they are war torn and unsung. It is my hope that the guys out there carrying on the good fight do so diligently and that their spirit doesn’t break in light of Youtube’s changes. Ultimately, it is my hope that we can move forward with the proposed partnership between Google and Microsoft and finally work together on a first class Youtube app for Windows. Until then, we must make do and march forward. If you are a supporter or a developer let your voice be heard and let’s get some official support for Windows Phone!