Cloudflare Blocking XMLRPC Download

Opensubtitles API
1

A user reported subtitle download is not working in IINA (XMLRPC API). I debugged this and it is because instead of returning subtitle file contents the response to the GET request contains HTML, a web page with the title Just a moment... and text that says Enable JavaScript and cookies to continue.

This seems to be a part of the Cloudflare challenge system attempting to confirm the request is coming from an actual human.

This could be what is causing FileBot to fail as reported in the post FileBot - 403 Forbidden error in the opensubtitles.org forum.

1 Like
2

Unfortunately it is not only IINA, subtitles fail to download in Kodi addons and Stremio addons as well.

Does it makes sense to also post this on the opensubtitles.org forum? What do you think?

3

Pretty sure the topic FileBot - 403 Forbidden error is about the same root cause.

I tried replying to that post, but as that reply was my first post in that forum it is currently awaiting moderator approval.

4

With respects to KODI, see the topic New OpenSubtitles.com KODI Subtitles addon.

After reproducing the problem with the XMLRPC API I tested a development version of IINA that uses the REST API and did not encounter the Cloudflare problem. Either I got lucky and Cloudflare didn’t feel like putting up a challenge or the REST API does not have this problem.

Are you using that latest KODI addon version?

5

This affects all apps using the XML-RPC API, including my Android app.

A web browser can handle the Javascript returned by Cloudflare and ultimately handle the download. I wonder if a WebView would not be able to handle it. Seems an heavy workaround though.

6

I can confirm that this workaround works (tested on Android with WebView):

If a subtitle download request fails with a 403:

  • spawn a WebView with the download URL and let Cloudflare do the check
  • once the check is done, retrieve the cookie for URL from the WebView Cookie Store
  • now in you app make the sub download request with your favorite http client with these headers:

“Host: hostname of the download URL”
“User-Agent: the exact user-agent of the WebView”
“Cookie: the cookie returned by Cloudflare”

After you have successfully retrieved the Cookie, you can re-use it until it is eventually invalid (not sure if it ever expires). If that’s the case, retrieve a new Cookie with the same method.

7

I could be wrong, but from a quick read of Cloudflare documentation the workaround may not work in all cases. The idea behind the Cloudflare system is to reduce the number of times visitors are required to prove humanness by solving a CAPTCHA. From the blog post The end of the road for Cloudflare CAPTCHAs:

As a result, we can fine-tune the difficulty of the challenge itself and avoid showing visual puzzles to more than 90% of human requests, while at the same time presenting harder challenges to visitors that exhibit non-human behaviors.

This system is using heuristics to reduce not eliminate the need for a visitor to solve a CAPTCHA.

Cloudflare presented me with one of these puzzles when I was browsing the Open Subtitles web site using Firefox.

If Cloudflare decides for whatever reason to present an interactive challenge, how will the workaround react?

8

In the topic Download requests result in http error 403 “Forbidden” Open Subtitles responded indicating the problem has been corrected. I just finished testing and downloaded worked for me.

DO NOT forget about the end of 2023 deadline to convert to the REST API as discussed in OpenSubtitles.org API deprecation and end of life.