Jump to content

Sky Slate Blueberry Blackcurrant Watermelon Strawberry Orange Banana Apple Emerald Chocolate
Photo

[Func] HTTPRequest: for Web APIs [AHK-B + AHK-L+Unicode+x64]


  • Please log in to reply
380 replies to this topic
twhyman
  • Members
  • 348 posts
  • Last active: Sep 17 2014 01:55 AM
  • Joined: 07 Dec 2005

twhyman:
maybe you want to exklude your username and password from that sample.


Thanks, did not noticed its was posted in two place in the code, I only changed one place :D

np, i recommend to change password also..

Done! :D
(\__/) This is Bunny.
(='.'=) Cut, copy, and paste bunny onto your sig.
(")_(") Help Bunny gain World Domination.

VxE
  • Moderators
  • 3622 posts
  • Last active: Dec 24 2015 02:21 AM
  • Joined: 07 Oct 2006
@ twhyman, I found the bug. Once again, the problem is in one of the regex needles (particularly, the one that looks for the codepage in the header is not case-insensitive).

Bugfix (10-09-2011): The regex to detect the 'codepage' attribute of the 'Content-Type' header has been fixed. Please download version (10-9-2011) from the OP.

I guess I should mention that [b]if[/i] the request headers have at least one 'Cookie' header, the INTERNET_FLAG_NO_COOKIES flag is added automatically (along with the standard flags INTERNET_FLAG_KEEP_CONNECTION, INTERNET_FLAG_RELOAD, and INTERNET_FLAG_NO_CACHE_WRITE )

twhyman
  • Members
  • 348 posts
  • Last active: Sep 17 2014 01:55 AM
  • Joined: 07 Dec 2005

@ twhyman, I found the bug. Once again, the problem is in one of the regex needles (particularly, the one that looks for the codepage in the header is not case-insensitive).

Bugfix (10-09-2011): The regex to detect the 'codepage' attribute of the 'Content-Type' header has been fixed. Please download version (10-9-2011) from the OP.

I guess I should mention that [b]if[/i] the request headers have at least one 'Cookie' header, the INTERNET_FLAG_NO_COOKIES flag is added automatically (along with the standard flags INTERNET_FLAG_KEEP_CONNECTION, INTERNET_FLAG_RELOAD, and INTERNET_FLAG_NO_CACHE_WRITE )


Thanks for the updated version, But it's still not working, When using a sniffer the Hebrew packet looks just the same as before like in a picture i posted before where every letter is encoded to one "number" not two.

Please help :(
(\__/) This is Bunny.
(='.'=) Cut, copy, and paste bunny onto your sig.
(")_(") Help Bunny gain World Domination.

VxE
  • Moderators
  • 3622 posts
  • Last active: Dec 24 2015 02:21 AM
  • Joined: 07 Oct 2006
OK, one more time. I think I've squished the bug at last.

And for those who are interested, the syntax for uploading a file directly is
HTTPRequest( url, response_var, headers_var, "upload: c:\somefile.png" )
The file's contents are read as they are uploaded, which limits the amount of memory used.

twhyman
  • Members
  • 348 posts
  • Last active: Sep 17 2014 01:55 AM
  • Joined: 07 Dec 2005

@ twhyman, I found the bug. Once again, the problem is in one of the regex needles (particularly, the one that looks for the codepage in the header is not case-insensitive).

Bugfix (10-09-2011): The regex to detect the 'codepage' attribute of the 'Content-Type' header has been fixed. Please download version (10-9-2011) from the OP.

I guess I should mention that if[/i] the request headers have at least one 'Cookie' header, the INTERNET_FLAG_NO_COOKIES flag is added automatically (along with the standard flags INTERNET_FLAG_KEEP_CONNECTION, INTERNET_FLAG_RELOAD, and INTERNET_FLAG_NO_CACHE_WRITE )


Hi,

It seems that the encoding part works! the Hebrew encoding is just like the PHP packet :D, BUT i have found a another bug :D the Content-Length is buggy, I noticed that for every Hebrew letter I write the Content-Length is not calculated correctly for example here is a packet when sending only the letter A in english:

**IMAGE REMOVED**

Here is a Hebrew packet:
[b]**IMAGE REMOVED**


Take a look at the Hebrew packet, For every Hebrew letter I write it takes a char from the end, as you can see the ">" is missing and its killing the XML.
(\__/) This is Bunny.
(='.'=) Cut, copy, and paste bunny onto your sig.
(")_(") Help Bunny gain World Domination.

nimda
  • Members
  • 4368 posts
  • Last active: Aug 09 2015 02:36 AM
  • Joined: 26 Dec 2010
How's OAuth coming?

VxE
  • Moderators
  • 3622 posts
  • Last active: Dec 24 2015 02:21 AM
  • Joined: 07 Oct 2006
@ twhyman: I fixed that bug... it turns out I had some variables mixed up :oops: It should handle codepage conversions correctly now.

Bugfix (10-10-2011): codepage conversion of POST data should now be fully functional.


@ nimda: oauth has been on the back burner as I take on other endeavors. Was there a particular API you have in mind?

nimda
  • Members
  • 4368 posts
  • Last active: Aug 09 2015 02:36 AM
  • Joined: 26 Dec 2010
imgur was what I was thinking of.
How much does OAuth vary from site-to-site?

VxE
  • Moderators
  • 3622 posts
  • Last active: Dec 24 2015 02:21 AM
  • Joined: 07 Oct 2006
In theory, OAuth 1.0a is a standard, so the authorization algorithm should be compatible with any API that uses it.

I only disclaimed a compatibility guarantee over there because I don't want to spend a lot of time debugging everyone's API wrappers.

twhyman
  • Members
  • 348 posts
  • Last active: Sep 17 2014 01:55 AM
  • Joined: 07 Dec 2005

@ twhyman: I fixed that bug... it turns out I had some variables mixed up :oops: It should handle codepage conversions correctly now.

Bugfix (10-10-2011): codepage conversion of POST data should now be fully functional.
...

@VxE all seems to be working great now :D thank you very much you rock!

Hope I dont find any more bugs :p

thank you again for your help!
(\__/) This is Bunny.
(='.'=) Cut, copy, and paste bunny onto your sig.
(")_(") Help Bunny gain World Domination.

DerBen
  • Members
  • 7 posts
  • Last active: Oct 16 2011 01:20 AM
  • Joined: 27 Jun 2010
correct me if i'm wrong (please, lol)
but I assume to use a referer to download a binary file, I'd do this:

	 headers:= "Referer: " URL1 "`n"
	 options:= "n> file.torrent`nBinary`n"
	 httpRequest(URL2,BinData,headers,options)


However.. it doesn't seem to work for me.

VxE
  • Moderators
  • 3622 posts
  • Last active: Dec 24 2015 02:21 AM
  • Joined: 07 Oct 2006
I have no way of knowing whether a 'referrer' is required to download a torrent from whatever site you're dealing with. If you can copy the link to the torrent, paste it into your browser address bar, hit 'enter' then download the file, then you can probably use UrlDownloadToFile.

The 'binary' option is a bit of a misnomer, what it does is bypasses the codepage conversion on downloaded text (which only happens if the response headers indicate that the downloaded data is some kind of text). Furthermore, the 'output-file' option writes the downloaded data byte-for-byte to the disk, completely untouched by codepage conversion.

If present, the output file path must be a folder that exists (the working dir is used if only the file name is supplied). Otherwise the option is ignored.

HTTPRequest is fairly verbose, so if it's not doing what you expect, it should give you enough information to know what it is doing. What does it return in the output data (or file) and output headers?

DerBen
  • Members
  • 7 posts
  • Last active: Oct 16 2011 01:20 AM
  • Joined: 27 Jun 2010
Thanks for the response!

http://www.monova.org/ <-- any torrent there will not download without a referrer. I tested this in firefox, I turned off cookies and referrers (different times and together) only time it didn't want to give me the torrent file was when referrers were off.

urldownloadtofile, doesn't work there, for the referrer reason. It's why I tried your script :)

My code above works fine for other sites I tested, that don't require the referrer (they work with urldownload as well). With your script, I get the file, headers and it's all good for other sites.

But on monova, pick any link, then click 'download via torrent'.. you will get redirected to the previous URL (the page that has the torrent download link) if the referrer is not working. Thus it downloads HTML and gives me the headers for the html page.

So I assume something isn't quite right in my or your code :p or I'd be getting the torrent.

VxE
  • Moderators
  • 3622 posts
  • Last active: Dec 24 2015 02:21 AM
  • Joined: 07 Oct 2006
h := "referer: http://www.monova.org/torrent/4806552/Kung_Fu_Panda_2_DVDRip_XviD-TARGET.html"
u := "http://www.monova.org/download/torrent/4806552/2011-10-13/f9bc5d627c015988213fc5668c6270a49fa872ec/Kung_Fu_Panda_2_DVDRip_XviD-TARGET.torrent"
httprequest( u, d := "", h, "n> c:\test.torrent" )
msgbox % h
The above code worked for me (of course I copied the links from the site manually).

I'm shooting fish in the dark here, since you don't give much info. Are you parsing HTML to get the URLs? If so, are you making sure that they don't contain HTML entities (or other formatting problems)?

DerBen
  • Members
  • 7 posts
  • Last active: Oct 16 2011 01:20 AM
  • Joined: 27 Jun 2010
Thought I was providing enough data :p

Well here goes, I already made a program that uses urldownloadtofile, but it lacks control. It searches websites and scans pages for the right URLs. The program works quite well so far and uses the right URLs.

<!-- m -->http://sourceforge.n...cts/dbtv/files/<!-- m --> is the program..

I've tried parsing the URLs to remove the %hx tags (if any) and leaving them alone. Among other things..

On that note: I just tried your example, as something simple to narrow down the problem..

#include HTTPRequest.ahk

h := "referer: http://www.monova.org/torrent/4806552/Kung_Fu_Panda_2_DVDRip_XviD-TARGET.html"
u := "http://www.monova.org/download/torrent/4806552/2011-10-13/f9bc5d627c015988213fc5668c6270a49fa872ec/Kung_Fu_Panda_2_DVDRip_XviD-TARGET.torrent"
httprequest( u, d := "", h, "n> test.torrent" )
msgbox % h

and it downloads an HTML page too.. So I assume it's my AHK install, autohotkey_L Version v1.1.00.00 or vista..