HTTP/HTTPS redirect too deep when server uses cookies to authenticate connection

[pacbard]

Hi!

I am running in a weird hedge case when connecting to a feed that uses a redirect+cookie to authenticate the connection.

Here is a script inspired by #33 that replicates the issue:

require 'fetcher'

url = 'https://link.springer.com/search.rss?facet-content-type=Article&facet-journal-id=40753&channel-name=International+Journal+of+Research+in+Undergraduate+Mathematics+Education'

worker = Fetcher::Worker.new

response = worker.get( url )

puts response.code
puts response.message
puts response.content_type
puts response.body[0..100]

which results in

❯ ruby debug.rb
[debug] fetch - get(_response) src: https://link.springer.com/search.rss?facet-content-type=Article&facet-journal-id=40753&channel-name=International+Journal+of+Research+in+Undergraduate+Mathematics+Education
[debug] using direct net http access; no proxy configured
[debug] GET /search.rss?facet-content-type=Article&facet-journal-id=40753&channel-name=International+Journal+of+Research+in+Undergraduate+Mathematics+Education uri=https://link.springer.com/search.rss?facet-content-type=Article&facet-journal-id=40753&channel-name=International+Journal+of+Research+in+Undergraduate+Mathematics+Education, redirect_limit=5
[debug] 302 Found location=https://idp.springer.com/authorize?redirect_uri=https://link.springer.com/search.rss?facet-content-type=Article&facet-journal-id=40753&channel-name=International+Journal+of+Research+in+Undergraduate+Mathematics+Education&client_id=springerlink&response_type=cookie
[debug] GET /authorize?redirect_uri=https://link.springer.com/search.rss?facet-content-type=Article&facet-journal-id=40753&channel-name=International+Journal+of+Research+in+Undergraduate+Mathematics+Education&client_id=springerlink&response_type=cookie uri=https://idp.springer.com/authorize?redirect_uri=https://link.springer.com/search.rss?facet-content-type=Article&facet-journal-id=40753&channel-name=International+Journal+of+Research+in+Undergraduate+Mathematics+Education&client_id=springerlink&response_type=cookie, redirect_limit=4
[debug] 302 Found location=https://idp.springer.com/transit?redirect_uri=https%3A%2F%2Flink.springer.com%2Fsearch.rss%3Ffacet-content-type%3DArticle&code=95d6f221-7e47-47eb-b2ff-eafd9a6bf63d
[debug] GET /transit?redirect_uri=https%3A%2F%2Flink.springer.com%2Fsearch.rss%3Ffacet-content-type%3DArticle&code=95d6f221-7e47-47eb-b2ff-eafd9a6bf63d uri=https://idp.springer.com/transit?redirect_uri=https%3A%2F%2Flink.springer.com%2Fsearch.rss%3Ffacet-content-type%3DArticle&code=95d6f221-7e47-47eb-b2ff-eafd9a6bf63d, redirect_limit=3
[debug] 302 Found location=https://link.springer.com/search.rss?facet-content-type=Article&error=cookies_not_supported&code=95d6f221-7e47-47eb-b2ff-eafd9a6bf63d
[debug] GET /search.rss?facet-content-type=Article&error=cookies_not_supported&code=95d6f221-7e47-47eb-b2ff-eafd9a6bf63d uri=https://link.springer.com/search.rss?facet-content-type=Article&error=cookies_not_supported&code=95d6f221-7e47-47eb-b2ff-eafd9a6bf63d, redirect_limit=2
[debug] 302 Found location=null?idp_error=cookies_not_supported
[debug] url relative; try to make it absolute
[debug] GET /null?idp_error=cookies_not_supported uri=https://link.springer.com/null?idp_error=cookies_not_supported, redirect_limit=1
[debug] 302 Found location=https://idp.springer.com/authorize?redirect_uri=https://link.springer.com/null?idp_error=cookies_not_supported&client_id=springerlink&response_type=cookie
[debug] GET /authorize?redirect_uri=https://link.springer.com/null?idp_error=cookies_not_supported&client_id=springerlink&response_type=cookie uri=https://idp.springer.com/authorize?redirect_uri=https://link.springer.com/null?idp_error=cookies_not_supported&client_id=springerlink&response_type=cookie, redirect_limit=0
[debug] 302 Found location=https://idp.springer.com/transit?redirect_uri=https%3A%2F%2Flink.springer.com%2Fnull%3Fidp_error%3Dcookies_not_supported&code=c2d1ee0e-a1a9-48a4-acd5-e6ff6cbce67c
Traceback (most recent call last):
	2: from debug.rb:7:in `<main>'
	1: from /Library/Ruby/Gems/2.6.0/gems/fetcher-0.4.5/lib/fetcher/worker.rb:56:in `get'
/Library/Ruby/Gems/2.6.0/gems/fetcher-0.4.5/lib/fetcher/worker.rb:166:in `get_response': HTTP redirect too deep (ArgumentError)

I can also replicate the infinite redirect using curl -L 'https://link.springer.com/search.rss?facet-content-type=Article&facet-journal-id=40753&channel-name=International+Journal+of+Research+in+Undergraduate+Mathematics+Education' on my machine.

At least in curl, explicitly setting the cookies for the connection with -b cookies.txt breaks the infinite redirect.

Is there a way to set the connection cookies in fetcher to allow for this remote authentication case?

Thank you!

[geraldb]

See issue #39 reported some hours ago today @ #39 - a fix for adding cookie support is planned.

PS: Tip for debugging requests you can use the fetch command line tool (incl. in fetchter and thus pluto on local installs):

 $ fetch -v "https://link.springer.com/search.rss?facet-content-type=Article&facet-journal-id=40753&channel-name=International+Journal+of+Research+in+Undergraduate+Mathematics+Education"

[pacbard]

It's exactly the same issue. Sorry for the duplicate report.