Help with libsoup and http://data.phishtank.com

[gnumdk (Cédric Bellegarde)]

Hello, Eolie is using phishtank to get it phishing database. Since some time, getting data from URI below is not working anymore.

I do not understand what is happening, when executing code below, I can see in returned data: Please enable cookies.

Getting this URI with “curl -L” and wget works.

from gi.repository import Soup

uri="http://data.phishtank.com/data/2eba68d50e3b974d6831c25e45d34f691056ef410a357c186d5cb94b7bf4048b/online-valid.json"
session = Soup.Session.new()
session.set_property("accept-language-auto", True)
jar = Soup.CookieJarText.new("/tmp/jar.txt", False)
session.add_feature(jar)
request = session.request(uri)
st=request.send()
bytes = st.read_bytes(1024, None)
data = bytes.get_data()
while data:
    print(data)
    bytes = st.read_bytes(1024, None)
    data = bytes.get_data()

Thanks for help!

[ntd (Nicola Fontana)]

Not sure this answer is going to help but the your URI does a double redirection using the Location HTTP header, the first one to the https version of the same URI.

My wild guesses:

1. they added https support and check the cookies there, while your /tmp/jar.txt file is still http based (cookies for http or https protocols can be different);
2. they check cookies on the last redirect, in which case you should generate the cookies dynamically or you must hardcode the target URI in both your code and your cookies text file;
3. you must not send cookies (e.g. curl works without sending any cookie);
4. PhishTank changed their API;
5. …