Jump to content

Sky Slate Blueberry Blackcurrant Watermelon Strawberry Orange Banana Apple Emerald Chocolate
Photo

[Func] HTTPRequest: for Web APIs [AHK-B + AHK-L+Unicode+x64]


  • Please log in to reply
380 replies to this topic
VxE
  • Moderators
  • 3622 posts
  • Last active: Dec 24 2015 02:21 AM
  • Joined: 07 Oct 2006
 
HTTPRequest - A function for submitting http(s) requests to the web and handling the responses.

Version: 2.49 - Tested compatible with AHK v1.0.48.05 and AHK v1.1.00.00 (Unicode); Presumed compatible with later versions, ANSI, and x64.

              Download HTTPRequest.ahk      or      Download HTTPRequest.zip

Overview: I wrote HTTPRequest specifically to service client-authenticated web APIs (like Amazon and eBay), which it does to my satisfaction. It can also upload data and download resources (e.g: HTML, images, etc...) from the internet, optionally saving them as a file on disk. The core of HTTPRequest is WinINet.dll, which comes with Internet Explorer.

The Basics: HTTPRequest has 4 parameters, the second and third are BYREF and used for both input and output. HTTPRequest returns either the number of bytes downloaded OR the length of the text downloaded, depending on how it treats the downloaded data.
; This is the order of HTTPRequest's parameters
BytesDownloaded := HTTPRequest( URL, In_POST__Out_Data, In_Out_HEADERS, Options )
  [*:1h5kyocc]1). URL (required) - The URL of the internet resource you wish to access.

  [*:1h5kyocc]2). In_POST__Out_Data (optional) - A variable. Before the function is called, this variable should hold any data you wish to upload. If you don't wish to upload data, make this variable blank (empty). A simple way to do this is to assign an empty string to it. After the function completes successfully, this variable holds the response data.
HTTPRequest does not convert uploaded text to utf-8 automatically. If you use unicode AHK, see the "charset" option below.

  [*:1h5kyocc]3). In_Out_HEADERS (recommended) - A variable. Prior to calling the function, put a (`n) newline delimited list of custom request headers into this variable (or make it blank, if you don't need to supply any extra headers). After the function completes, this variable holds the HTTP response headers and/or messages about any errors the function encountered. Do not ignore this parameter, it often contains very useful debugging info.

  [*:1h5kyocc]4). Options (optional) - A newline delimited list of keywords that modify how HTTPRequest works.
Headers: are a collection of information about the client/server, the request/response, and about the data being transferred. Even if parameter 3 is empty, HTTPRequest still submits request headers. There are a few request headers you should take special note of:  [*:1h5kyocc]Content-Length: 12345 - HTTPRequest will add the "Content-Length" header to the request headers if it detects data to upload. If HTTPRequest can't determine the content length automatically (e.g: you are uploading non-text data), you must provide the "Content-Length" header with the total number of bytes you wish to upload. NOTE for UNICODE AHK: if you are uploading text, the number of bytes in text is twice the length of the text.

  [*:1h5kyocc]Content-Type: application/x-www-form-urlencoded - If the "Content-Length" is greater than zero, HTTPRequest will add the "Content-Type" header unless you already provided it. HTTPRequest's default values for this header are "application/octet-stream" when uploading from a file and the file's extension is neither "txt" nor "xml"; "text/xml" when the string "<?xml" appears near the beginning of the data; or "application/x-www-form-urlencoded" in all other cases.

  [*:1h5kyocc]Content-MD5: - Some web APIs require this header because they need to verify the integrity of the uploaded data. HTTPRequest can fill in the value automatically if you leave it blank (i.e: supply the header name and colon without the value like "Content-MD5:"). Also, if this appears in the response headers, HTTPRequest will add a "Computed-MD5" pseudo-header for your convenience.

  [*:1h5kyocc]User-Agent: MyScriptName/1.0 (Language=AutoHotkey/1.0.48.05; Platform=WIN_XP) - HTTPRequest will add the "User-Agent" header to every request. By default, your script name, AHK version, and OS are used. If this is unacceptable to you, provide your own "User-Agent" header.
  Options: modify how HTTPRequest behaves. Text in blue is only a syntax example.  [*:1h5kyocc]AutoProxy - This option tells HTTPRequest to use Internet Exlporer's proxy configuration. If your Internet Explorer is not configured to use a proxy, this option has no effect.

  [*:1h5kyocc]Binary - Normally, after downloading text, HTTPRequest will try to convert its codepage to the script's codepage, which is essential if you're using unicode AHK. The "binary" option tells HTTPRequest NOT to do the conversion (i.e: HTTPRequest will treat the downloaded data as raw binary data).

  [*:1h5kyocc]Callback: Function_Name [, Third Parameter Value] - Tells HTTPRequest to call "Function_Name()" at regular intervals during the data transfer. "Function_Name()" must NOT require more than 3 parameters. When HTTPRequest calls the function, the first parameter's value ranges from -1 to +1, where values less than 0 indicate an upload in progress and values greater than 0 indicate a download. The second parameter's value is the number of bytes anticipated in the transfer (i.e: the value of the "Content-Length" header). The third parameter's value is whatever you put as "Third Parameter Value". The callback function may return the string "CANCEL" to tell HTTPRequest to cancel the transaction.

  [*:1h5kyocc]Charset: utf-8 - Tells HTTPRequest to convert the text in "In_POST__Out_Data" from the script's codepage to the character set specified. HTTPRequest contains a lookup table with the supported charset names, which was copied from http://msdn.microsof...6(v=vs.85).aspx.

  [*:1h5kyocc]Codepage: 65001 - The more 'technical' counterpart to the "charset" option, where the number on the right indicates the desired codepage number.

  [*:1h5kyocc]+NO_COOKIES - Tells WinINet to NOT automatically handle cookies (by default, WinINet uses Internet Explorer's cookie pool). This is required if you want your script to manage cookies with HTTPRequest. If your request headers contain any "Cookie" headers, this option is applied automatically.

  [*:1h5kyocc]+NO_AUTO_REDIRECT - Internet flags may be added or removed from a partiular request by putting a plus (+) or minus (-) sign in front of the flag's name, respectively (with or without the preceding "INTERNET_FLAG_"). Some flags are added automatically, depending on whether the connection uses SSL.

  [*:1h5kyocc]+IGNORE_UNKNOWN_CA - Security flags may be added or removed from a partiular request by putting a plus (+) or minus (-) sign in front of the flag's name, respectively (use the full name with "SECURITY_FLAG_" in front when the flag name would otherwise be ambiguous).

  [*:1h5kyocc]Method: POST - Tells HTTPRequest to use the indicated verb for the request. Typically, HTTPRequest uses "POST" if it is uploading data or "GET" otherwise. Use this option for web API functions that require a different verb. Supported verbs are GET, HEAD, POST, PUT, DELETE, OPTIONS, TRACE.

  [*:1h5kyocc]Proxy: <!-- m -->http://www.exampleproxy.com:1234<!-- m --> - Tells HTTPRequest to use a proxy for the request.
        To use a debugging proxy (like Fiddler), add this option to HTTPRequest: Proxy: localhost:8888 (be sure to supply the correct port number).

  [*:1h5kyocc]ProxyBypass: <!-- m -->http://www.autohotkey.com<!-- m --> - Tells HTTPRequest NOT to use a proxy to access [www.autohotkey.com]. This has no effect unless HTTPRequest has been told to use a proxy via the "proxy" option.

  [*:1h5kyocc]Resume: C:\My Partially Downloaded File.jpg - Tells HTTPRequest to continue downloading the specified file. It is up to you to make sure that the URL points to the same resource as the one used to download the file in the first place. When the download succeeds, "In_POST__Out_Data" is made blank.

  [*:1h5kyocc]SaveAs: C:\Some File.htm - Tells HTTPRequest to save the data downloaded to your hard disk. This option simulates UrlDownloadToFile. When the download succeeds, "In_POST__Out_Data" is made blank. You may use FileRead to read the data into memory later.

  [*:1h5kyocc]Upload: C:\My Awesome Pic.png - Tells HTTPRequest to use "C:\My Awesome Pic.png" as the source of data to upload. Please consult your webservice's documentation for any other conditions for uploading data. When using this option, I highly recommend supplying the appropriate "Content-Type" header.
Finally, this is a work in progress and features may be added/removed. Old versions will not remain available and I don't keep a change log. If you have questions, comments, bug reports, or requests for features, post them in this thread.

Special thanks to everyone who helped spot bugs and suggest new features. Extra special thanks to derRaphael, whose work on httpQuery inspired me to start this project.

VxE
  • Moderators
  • 3622 posts
  • Last active: Dec 24 2015 02:21 AM
  • Joined: 07 Oct 2006
Example: binary POST data. Upload Images to Imgur.com
This example no longer works since Imgur has apparently deprecated the anonymous API.


This script requires HTTPRequest (7-1-2011) and your Imgur Anonymous API key. Running this script will prompt you to select an image file, upload the file to Imgur.com, and return the URL of the image.
The response XML is saved to a file in the same folder as the chosen image, so you can delete the image if you wish.
; Select an image and upload it to Imgur.com via the anonymous API.Anonymous_API_Key := "&lt; Your Imgur Anonymous API Key&gt;"FileSelectFile, image_file, 2, ::{450d8fba-ad25-11d0-98a8-0800361b1103}, Upload Image To Imgur, Image Files (*.jpg; *.jpeg; *.gif; *.png; *.bmp)If !( ErrorLevel ){	image_url := Imgur_Upload( image_file, Anonymous_API_Key, xml )	FileAppend, % XML_MakePretty( xml ), % info_file := RegexReplace( image_file, ".*\K\..*", "-Imgur Info.txt" )	If ( image_url )		MsgBox, 0, Upload Successful!, % ( Clipboard := image_url ) "`n`nHas been put on the clipboard."	Else		MsgBox, 0, Upload Failed!, Something went wrong. For details, please see %info_file%}ExitappImgur_Upload( image_file, Anonymous_API_Key, byref output_XML="" ) { ; -----------------------------; Uploads one image file to Imgur via the anonymous API and returns the URL to the image.; To acquire an anonymous API key, please register at http://imgur.com/register/api_anon.; This function was written by [VxE] and relies on the HTTPRequest function, also by [VxE].; HTTPRequest can be found at http://www.autohotkey.com/forum/viewtopic.php?t=73040	Static Imgur_Upload_Endpoint := "http://api.imgur.com/2/upload.xml"	FileGetSize, size, % image_file	FileRead, output_XML, % "*c " image_file	If HTTPRequest( Imgur_Upload_Endpoint "?key=" Anonymous_API_Key, output_XML		, Response_Headers := "Content-Type: application/octet-stream`nContent-Length: " size		, "Callback: CustomProgress" )	&& ( pos := InStr( output_XML, "&lt;original&gt;" ) )		Return SubStr( output_XML, pos + 10, Instr( output_XML, "&lt;/original&gt;", 0, pos ) - pos - 10 )	Else Return "" ; error: see response} ; Imgur_Upload( image_path, Anonymous_API_Key, byref output_XML="" ) -----------------------------CustomProgress( pct, total ) {If ( pct = "" )	TooltipElse If ( pct &lt; 0 )	Tooltip, % "Uploading " Round( 100 * ( pct + 1 ), 1 ) "%. "	. Round( ( pct + 1 ) * total, 0 ) " of " total " bytes."Else If ( 0 &lt;= pct )	Tooltip, % "Downloading " Round( 100 * pct, 1 ) "%. "	. Round( pct * total, 0 ) " of " total " bytes."}XML_MakePretty( XML, Tab="`t" ) { ; ----------------------------------------------------------------; Function by [VxE]. Adds newlines and tabs between XML tags to give human-friendly arrangement to; an XML stream. 'Tab' contains the string to use as an indentation unit (it may be more readable to; use 2 or 3 spaces instead of a full tab... so it's up to you!).	oel := ErrorLevel, PrevCloseTag := 0, tabs := "", tablen := StrLen( tab )	StringLen, pos, XML	Loop, Parse, XML, &lt;, % "`t`r`n "		If ( A_Index = 1 )			VarSetCapacity( XML, pos, 0 )		Else		{			StringGetPos, pos, A_LoopField, &gt;			StringMid, b, A_LoopField, pos, 1			StringLeft, a, A_LoopField, 1			If !( OpenTag := a != "/" ) * ( CloseTag := a = "/" || a = "!" || a = "?" || b = "/" )				StringTrimRight, tabs, tabs, tablen			XML .= ( OpenTag || PrevCloseTag ? tabs : "" ) "&lt;" A_LoopField			If !( PrevCloseTag := CloseTag ) * OpenTag				tabs := ( tabs = "" ? "`n" : tabs ) tab		}	Return XML, ErrorLevel := oel} ; XML_MakePretty( XML, Tab="`t" ) ----------------------------------------------------------------
                                                                        yI8ly.png

sumon
  • Moderators
  • 1317 posts
  • Last active: Dec 05 2016 10:14 PM
  • Joined: 18 May 2010
Let me be the first to be impressed. It seems well structured and functioning, and I can't wait to try it out. I also like your way of handling input -> output, as for me I'll make it easy and name the variables "POST" in the case of a post.

I have two requests/questions:

1) An example of transferring raw binary data, since that's something that I know I will find useful, and probably others too. For example for [img] imgur's API (see "upload"), or later, to upload apps to Appifyer.com.

2) The main differences between this and httpQuery. Their main functionality is the same, if I am not majorly mistaken?

VxE
  • Moderators
  • 3622 posts
  • Last active: Dec 24 2015 02:21 AM
  • Joined: 07 Oct 2006

2) The main differences between this and httpQuery. Their main functionality is the same, if I am not majorly mistaken?

Indeed, though my function dares to differ in several ways. I don't want to enumerate the points (out of respect for derRaphael), but I wrote HTTPRequest because httpQuery did not satisfy some of my requirements.


The Imgur anonymous API looks very useful (and it looks like a good demo for OAuth). I'll look into it in the next few weeks.

sumon
  • Moderators
  • 1317 posts
  • Last active: Dec 05 2016 10:14 PM
  • Joined: 18 May 2010

2) The main differences between this and httpQuery. Their main functionality is the same, if I am not majorly mistaken?

Indeed, though my function dares to differ in several ways. I don't want to enumerate the points (out of respect for derRaphael), but I wrote HTTPRequest because httpQuery did not satisfy some of my requirements.


The Imgur anonymous API looks very useful (and it looks like a good demo for OAuth). I'll look into it in the next few weeks.


Oh, indeed, but I was referring to the anonymous/open API (where you just need an API key and a binary image file - getting an API key is easy), in general a guide purely on how to post raw binary data, since it's (like you said) a bit more advanced.

On the topic of oAuth, I think you would be a hero if you managed to (using your slightly more "convenient" HTTPRequest function) provide an example for authenticating with oAuth, since that authentication is becoming more and more common. Read more about it, and Octal's start, here.

derRaphael
  • Members
  • 872 posts
  • Last active: Mar 19 2013 04:42 PM
  • Joined: 23 Nov 2007
nice one ;)

good to know that httpQuery has now a worthy successor.

when talking of OAuth, it might be worth an idea to have build in cookie support (ideally with keeping session cookie data while following request relocations). some services (OAuth2, OpenSocial) have tendencies to move tokens and auhtorization information into cookie parts - so it will ease using these services (like facebook)

8)
dR

All scripts, unless otherwise noted, are hereby released under CC-BY

guest3456
  • Members
  • 1704 posts
  • Last active: Nov 19 2015 11:58 AM
  • Joined: 10 Mar 2011
whats the difference between this (and httpquery) compared to InternetFileRead?
<!-- m -->http://www.autohotke...pic.php?t=45718<!-- m -->

tank
  • Administrators
  • 4345 posts
  • AutoHotkey Foundation
  • Last active: Oct 13 2016 01:04 AM
  • Joined: 21 Dec 2007
support for AHK_l and unicode for one
Never lose.
WIN or LEARN.

VxE
  • Moderators
  • 3622 posts
  • Last active: Dec 24 2015 02:21 AM
  • Joined: 07 Oct 2006
@guest3456: I hadn't looked at InternetFileRead before, but from a cursory inspection, it seems that the main difference is in the function premises. InternetFileRead is meant to download files from the internet (with special attention to large files) whereas HTTPRequest is meant to handle requests to, and responses from, web APIs. I think httpQuery's premise is closest to 'mimic a browser's ability to submit POST data and retrieve results from those transactions'.

I don't think it's constructive for me to enumerate which functions have the shiniest bells and whistles.... Every user should select the appropriate tool for their needs (or write their own).


@dR: Thank you for your blessing on this project, I truly appreciate it.

Cookie support was part of some of the earlier iterations of this project. I didn't include it in this release because i hadn't thought of a graceful and intuitive way to integrate the feature.

As it stands, cookies should appear in the response as 'Set-Cookie' headers, and the user can submit cookies using the 'Cookie' header (see RFC 2965). It's possible for the user to manage cookies if needed.

I took a similar approach to redirects and response codes other than 2xx, leaving it up to the user to decide what to do based on the response headers.

SKAN
  • Administrators
  • 9115 posts
  • Last active:
  • Joined: 26 Dec 2005

The Imgur anonymous API looks very useful (and it looks like a good demo for OAuth). I'll look into it in the next few weeks.


It will be very helpful [VxE].
I currently use <!-- m -->http://imguruploader...ases/view/40965<!-- m --> but it is .net and requires installation.

Uberi
  • Moderators
  • 1119 posts
  • Last active: May 02 2015 06:05 PM
  • Joined: 23 Aug 2010
Nice work! I'm glad that there's finally a valid alternative to my HTTPQuery version, as I don't have to maintain it any more :D. Also, HTTPS support is awesome, couldn't get it to work with any other methods.

VxE
  • Moderators
  • 3622 posts
  • Last active: Dec 24 2015 02:21 AM
  • Joined: 07 Oct 2006
Thanks, SKAN and Uberi.

BUGFIX: 6-20-2011. If you have an older version, please re-download from the link in the OP. The bug was in the call to HttpOpenRequest with the verb parameter for 'POST' operations. Also, there were typos in some of the auto-headers.

I'll keep a running commentary on bugfixes as I find/fix them. The function date (version?) can be found on the first comment line within the function.

[Edit] Well, it turns out that Imgur's API has two different modes and the anonymous API does not require OAuth. So, instead of demonstrating OAuth, I posted a demo of uploading binary data via POST (the demo script is just below the top post).

TWEAK: 6-29-2011: I expanded the automatic recognition of text-based 'Content-Type' responses. This version is required for the Imgur upload example to work correctly. If you don't trust the auto text recognition, and you know your API responses will be some sort of text, then by all means use VarSetCapacity( output_var, -1 ) after calling HTTPRequest.

sumon
  • Moderators
  • 1317 posts
  • Last active: Dec 05 2016 10:14 PM
  • Joined: 18 May 2010
Yeah, imgur has two modes, but oAuth is something that in general is a widely used method, so that's why I requested that. imgur is just a decent example of oAuth.

Anyway, thanks for sharing the imgur upload example. Basically you just supply the binary data as POST data I take it from the FileRead as I understand it. For some reason, I get this error message all the time:

<?xml version="1.0" encoding="utf-8"?><error>
<message>Image format not supported, or image is corrupt.</message>
<request>/2/upload.xml</request>
<method>post</method>
<format>xml</format>
<parameters></parameters>
</error>


AHK_L x32 Unicode, my API key is not the problem afaik.

VxE
  • Moderators
  • 3622 posts
  • Last active: Dec 24 2015 02:21 AM
  • Joined: 07 Oct 2006

Yeah, imgur has two modes, but oAuth is something that in general is a widely used method, so that's why I requested that. imgur is just a decent example of oAuth.

Lol, I'm still working on OAuth.. just the different Imgur modes had me a bit confused at first. It will take a little while longer though. (OAuth is for accessing an account through the Imgur API... but I had started coding for the anonymous mode...).

Anyways, that error happens because I didn't test the Imgur example with AHK-L and forgot to add the '*c' option to FileRead. I have edited the example (after testing it with AHK-L U 32b) and it should work now.

sumon
  • Moderators
  • 1317 posts
  • Last active: Dec 05 2016 10:14 PM
  • Joined: 18 May 2010
Oh you are, nice to hear. Want contact with another who is? Contact octalmage. Sorry about the confusion!

Also, thanks alot for the bug fix, will be great. Gonna improve Zizorz alot soon...