HttpClient not exposing redirect response codes

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

HttpClient not exposing redirect response codes

T Lee Davidson
I have run into a couple of sites that are responding with 301 (Moved Permanently) and 302 (Moved Temporarily) codes. However,
HttpClient.Code yields "0" for both of those sites/codes (and HttpClient.Reason is empty). 200 & 404 codes are reported
successfully.

Have I missed something in the properties that is preventing proper disclosure of redirect response codes?

Or is this a bug?

(Project attached.)

--
Lee

[System]
Gambas=3.9.2
OperatingSystem=Linux
Kernel=4.4.57-18.3-default
Architecture=x86_64
Distribution=SuSE NAME="openSUSE Leap"
VERSION="42.2"
ID=opensuse
ID_LIKE="suse"
VERSION_ID="42.2"
PRETTY_NAME="openSUSE Leap 42.2"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:opensuse:leap:42.2"
BUG_REPORT_URL="https://bugs.opensuse.org"
HOME_URL="https://www.opensuse.org/"
Desktop=KDE5
Theme=QtCurve
Language=en_US.UTF-8
Memory=3951M

[Libraries]
DBus=libdbus-1.so.3.8.14
OpenGL=libGL.so.1.2.0



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Gambas-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gambas-user

httpclienttest-0.0.1.tar.gz (17K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: HttpClient not exposing redirect response codes

Tobias Boege-2
On Tue, 25 Apr 2017, T Lee Davidson wrote:

> I have run into a couple of sites that are responding with 301 (Moved
> Permanently) and 302 (Moved Temporarily) codes. However, HttpClient.Code
> yields "0" for both of those sites/codes (and HttpClient.Reason is empty).
> 200 & 404 codes are reported successfully.
>
> Have I missed something in the properties that is preventing proper disclosure of redirect response codes?
>
> Or is this a bug?
>
> (Project attached.)
>

I stumbled upon this too one time. Apparently my solution was to scan
HttpClient.Headers for the presence of a "Location: " header.

So what I'm saying is, I only have a workaround, no explanation. I think
it's a bug.

Regards,
Tobi

--
"There's an old saying: Don't change anything... ever!" -- Mr. Monk

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Gambas-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gambas-user
Reply | Threaded
Open this post in threaded view
|

Re: HttpClient not exposing redirect response codes

PICCORO McKAY Lenz
its a bug..and also not:  due the redirection.. in low level must handle by
oursleft..  i mean "moved" where?

Lenz McKAY Gerardo (PICCORO)
http://qgqlochekone.blogspot.com

2017-04-25 13:26 GMT-04:00 Tobias Boege <[hidden email]>:

> On Tue, 25 Apr 2017, T Lee Davidson wrote:
> > I have run into a couple of sites that are responding with 301 (Moved
> > Permanently) and 302 (Moved Temporarily) codes. However, HttpClient.Code
> > yields "0" for both of those sites/codes (and HttpClient.Reason is
> empty).
> > 200 & 404 codes are reported successfully.
> >
> > Have I missed something in the properties that is preventing proper
> disclosure of redirect response codes?
> >
> > Or is this a bug?
> >
> > (Project attached.)
> >
>
> I stumbled upon this too one time. Apparently my solution was to scan
> HttpClient.Headers for the presence of a "Location: " header.
>
> So what I'm saying is, I only have a workaround, no explanation. I think
> it's a bug.
>
> Regards,
> Tobi
>
> --
> "There's an old saying: Don't change anything... ever!" -- Mr. Monk
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Gambas-user mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gambas-user
>
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Gambas-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gambas-user
Reply | Threaded
Open this post in threaded view
|

Re: HttpClient not exposing redirect response codes

PICCORO McKAY Lenz
i forgot to mention, you have received a X-Frame-Options: DENY , that the
other reason of the problem..

Lenz McKAY Gerardo (PICCORO)
http://qgqlochekone.blogspot.com

2017-04-25 13:32 GMT-04:00 PICCORO McKAY Lenz <[hidden email]>:

> its a bug..and also not:  due the redirection.. in low level must handle
> by oursleft..  i mean "moved" where?
>
> Lenz McKAY Gerardo (PICCORO)
> http://qgqlochekone.blogspot.com
>
> 2017-04-25 13:26 GMT-04:00 Tobias Boege <[hidden email]>:
>
>> On Tue, 25 Apr 2017, T Lee Davidson wrote:
>> > I have run into a couple of sites that are responding with 301 (Moved
>> > Permanently) and 302 (Moved Temporarily) codes. However, HttpClient.Code
>> > yields "0" for both of those sites/codes (and HttpClient.Reason is
>> empty).
>> > 200 & 404 codes are reported successfully.
>> >
>> > Have I missed something in the properties that is preventing proper
>> disclosure of redirect response codes?
>> >
>> > Or is this a bug?
>> >
>> > (Project attached.)
>> >
>>
>> I stumbled upon this too one time. Apparently my solution was to scan
>> HttpClient.Headers for the presence of a "Location: " header.
>>
>> So what I'm saying is, I only have a workaround, no explanation. I think
>> it's a bug.
>>
>> Regards,
>> Tobi
>>
>> --
>> "There's an old saying: Don't change anything... ever!" -- Mr. Monk
>>
>> ------------------------------------------------------------
>> ------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>> _______________________________________________
>> Gambas-user mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/gambas-user
>>
>
>
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Gambas-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gambas-user
Reply | Threaded
Open this post in threaded view
|

Re: HttpClient not exposing redirect response codes

Benoît Minisini
In reply to this post by T Lee Davidson
Le 25/04/2017 à 19:12, T Lee Davidson a écrit :

> I have run into a couple of sites that are responding with 301 (Moved
> Permanently) and 302 (Moved Temporarily) codes. However, HttpClient.Code
> yields "0" for both of those sites/codes (and HttpClient.Reason is
> empty). 200 & 404 codes are reported successfully.
>
> Have I missed something in the properties that is preventing proper
> disclosure of redirect response codes?
>
> Or is this a bug?
>
> (Project attached.)
>

HttpClient is "just" libcurl, so it may be a libcurl thing (that library
has gazillons of options). I will check...

--
Benoît Minisini

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Gambas-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gambas-user
Reply | Threaded
Open this post in threaded view
|

Re: HttpClient not exposing redirect response codes

T Lee Davidson
On 04/25/2017 01:36 PM, Benoît Minisini wrote:

> Le 25/04/2017 à 19:12, T Lee Davidson a écrit :
>> I have run into a couple of sites that are responding with 301 (Moved
>> Permanently) and 302 (Moved Temporarily) codes. However, HttpClient.Code
>> yields "0" for both of those sites/codes (and HttpClient.Reason is
>> empty). 200 & 404 codes are reported successfully.
>>
>> Have I missed something in the properties that is preventing proper
>> disclosure of redirect response codes?
>>
>> Or is this a bug?
>>
>> (Project attached.)
>>
>
> HttpClient is "just" libcurl, so it may be a libcurl thing (that library
> has gazillons of options). I will check...
>

Thank you for taking the time to check into that, Benoît.

In the meantime, for anyone else needing to loop through redirects (currently scanning for a "Location" header), here's a short
example subroutine:

Private Sub TestPage(sUrl As String)

   Dim hClient As New HttpClient
   Dim iMaxRedirects As Integer = 5 'Prevent infinite loop bot trap
   Dim iCurrentRedirects As Integer = 0
   Dim bFound As Boolean

   hClient.UserAgent = "AppleWebKit"
   hClient.Async = False
   hClient.URL = sUrl
   hClient.Timeout = 15
   Inc Application.Busy
   Do
     bFound = True 'Reset redirect flag
     hClient.Get
     If hClient.Headers.Find("Location:*", gb.Like) > -1 Then
       bFound = False 'Redirection
       If iCurrentRedirects = iMaxRedirects Then Break
       Inc iCurrentRedirects
       hClient.URL = Replace(hClient.Headers[hClient.Headers.Find("Location:*", gb.Like)], "Location: ", "")
       Print hClient.URL
       hClient.Headers.Clear 'Must manually clear headers array or "Location" header will persist causing redundant looping
     Endif
   Loop Until bFound = True
   Dec Application.Busy

   If hClient.Status < 0 Then
     Print "ERROR"
   End If

   Print "Success? -> " & hClient.Code

End


---
Lee


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Gambas-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gambas-user
Reply | Threaded
Open this post in threaded view
|

Re: HttpClient not exposing redirect response codes

PICCORO McKAY Lenz
hi lee, in the example, when i debug , the header does not return the
required contents to made the location! i mean i donot test u'r code, but
before send that mail i tested and does not work.. with the specific url..
due that the problem relies on libcurl implementatio of httpclient module
in gambas..

maybe a solution (ugly but proper) its to parse options directly to the
curl implementation

i found some others misctakes/bugs making gambas http/web projects but for
now i must finish my works at job

Lenz McKAY Gerardo (PICCORO)
http://qgqlochekone.blogspot.com

2017-04-25 16:29 GMT-04:00 T Lee Davidson <[hidden email]>:

> On 04/25/2017 01:36 PM, Benoît Minisini wrote:
> > Le 25/04/2017 à 19:12, T Lee Davidson a écrit :
> >> I have run into a couple of sites that are responding with 301 (Moved
> >> Permanently) and 302 (Moved Temporarily) codes. However, HttpClient.Code
> >> yields "0" for both of those sites/codes (and HttpClient.Reason is
> >> empty). 200 & 404 codes are reported successfully.
> >>
> >> Have I missed something in the properties that is preventing proper
> >> disclosure of redirect response codes?
> >>
> >> Or is this a bug?
> >>
> >> (Project attached.)
> >>
> >
> > HttpClient is "just" libcurl, so it may be a libcurl thing (that library
> > has gazillons of options). I will check...
> >
>
> Thank you for taking the time to check into that, Benoît.
>
> In the meantime, for anyone else needing to loop through redirects
> (currently scanning for a "Location" header), here's a short
> example subroutine:
>
> Private Sub TestPage(sUrl As String)
>
>    Dim hClient As New HttpClient
>    Dim iMaxRedirects As Integer = 5 'Prevent infinite loop bot trap
>    Dim iCurrentRedirects As Integer = 0
>    Dim bFound As Boolean
>
>    hClient.UserAgent = "AppleWebKit"
>    hClient.Async = False
>    hClient.URL = sUrl
>    hClient.Timeout = 15
>    Inc Application.Busy
>    Do
>      bFound = True 'Reset redirect flag
>      hClient.Get
>      If hClient.Headers.Find("Location:*", gb.Like) > -1 Then
>        bFound = False 'Redirection
>        If iCurrentRedirects = iMaxRedirects Then Break
>        Inc iCurrentRedirects
>        hClient.URL = Replace(hClient.Headers[hClient.Headers.Find("Location:*",
> gb.Like)], "Location: ", "")
>        Print hClient.URL
>        hClient.Headers.Clear 'Must manually clear headers array or
> "Location" header will persist causing redundant looping
>      Endif
>    Loop Until bFound = True
>    Dec Application.Busy
>
>    If hClient.Status < 0 Then
>      Print "ERROR"
>    End If
>
>    Print "Success? -> " & hClient.Code
>
> End
>
>
> ---
> Lee
>
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Gambas-user mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gambas-user
>
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Gambas-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gambas-user
Reply | Threaded
Open this post in threaded view
|

Re: HttpClient not exposing redirect response codes

Benoît Minisini
In reply to this post by T Lee Davidson
Le 25/04/2017 à 19:12, T Lee Davidson a écrit :

> I have run into a couple of sites that are responding with 301 (Moved
> Permanently) and 302 (Moved Temporarily) codes. However, HttpClient.Code
> yields "0" for both of those sites/codes (and HttpClient.Reason is
> empty). 200 & 404 codes are reported successfully.
>
> Have I missed something in the properties that is preventing proper
> disclosure of redirect response codes?
>
> Or is this a bug?
>
> (Project attached.)
>

I have just tested your project, and I get all the expected answers
(301, 302, 404 & 200).

libcurl (the library used by gb.net.curl) has a flag to automatically
follow http redirections, but it is disabled by default.

So I don't understand the result you have. Which version of libcurl do
you use on your system?

--
Benoît Minisini

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Gambas-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gambas-user
Reply | Threaded
Open this post in threaded view
|

Re: HttpClient not exposing redirect response codes

T Lee Davidson
On 04/27/2017 03:01 AM, Benoît Minisini wrote:

> Le 25/04/2017 à 19:12, T Lee Davidson a écrit :
>> I have run into a couple of sites that are responding with 301 (Moved
>> Permanently) and 302 (Moved Temporarily) codes. However, HttpClient.Code
>> yields "0" for both of those sites/codes (and HttpClient.Reason is
>> empty). 200 & 404 codes are reported successfully.
>>
>> Have I missed something in the properties that is preventing proper
>> disclosure of redirect response codes?
>>
>> Or is this a bug?
>>
>> (Project attached.)
>>
>
> I have just tested your project, and I get all the expected answers
> (301, 302, 404 & 200).
>
> libcurl (the library used by gb.net.curl) has a flag to automatically
> follow http redirections, but it is disabled by default.

I thought so. But, I looked and looked and did not find any way to access that flag either in HttpClient or in Curl (gb.net.curl).


> So I don't understand the result you have. Which version of libcurl do
> you use on your system?
>

The version of libcurl on my system is libcurl4, Version 7.37.0-16.3.1 (libcurl.so.4.3.0).

BTW, I just tried again to see if it was just a transient issue, but, no, I still get neither 301 nor 302.


---
Lee



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Gambas-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gambas-user
Reply | Threaded
Open this post in threaded view
|

Re: HttpClient not exposing redirect response codes

T Lee Davidson
In reply to this post by Benoît Minisini
On 04/27/2017 03:01 AM, Benoît Minisini wrote:

> Le 25/04/2017 à 19:12, T Lee Davidson a écrit :
>> I have run into a couple of sites that are responding with 301 (Moved
>> Permanently) and 302 (Moved Temporarily) codes. However, HttpClient.Code
>> yields "0" for both of those sites/codes (and HttpClient.Reason is
>> empty). 200 & 404 codes are reported successfully.
>>
>> Have I missed something in the properties that is preventing proper
>> disclosure of redirect response codes?
>>
>> Or is this a bug?
>>
>> (Project attached.)
>>
>
> I have just tested your project, and I get all the expected answers
> (301, 302, 404 & 200).
>
> libcurl (the library used by gb.net.curl) has a flag to automatically
> follow http redirections, but it is disabled by default.
>
> So I don't understand the result you have. Which version of libcurl do
> you use on your system?
>

Actually, I don't think it is a libcurl issue since the first response header does contain the appropriate text and code.

If I run the example project with slightly different output:

   rMatch = New RegExp(hClient.Headers[0], ".*? (\\d+) (.*)")
   TextArea1.Text = hClient.Headers[0] & "\n"
   TextArea1.Text &= "Parsed header code: " & IIf(rMatch[1], rMatch[1].Text, "No code found.") & "\n"
   TextArea1.Text &= "Parsed header reason: " & IIf(rMatch[2], rMatch[2].Text, "No reason found.") & "\n"
   TextArea1.Text &= "HttpClient.Code: " & hClient.Code & "\n"
   TextArea1.Text &= "HttpClient.Reason: " & hClient.Reason & "\n\n"

For the two problem sites, I get:

HTTP/1.1 301 Moved Permanently
Parsed header code: 301
Parsed header reason: Moved Permanently
HttpClient.Code: 0
HttpClient.Reason:

and

HTTP/1.1 302 Moved Temporarily
Parsed header code: 302
Parsed header reason: Moved Temporarily
HttpClient.Code: 0
HttpClient.Reason:


For the 200 & 404 codes, it works as expected.

Thinking it may be a Charset issue, I made sure to send a "Accept-Charset: utf-8" header - no difference.

I've taken a quick look at CHttpClient.c and found http_parse_header, but I don't understand what "ret = (ret * 10) + (*p -
'0')", among others(!), does.


---
Lee


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Gambas-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gambas-user