Jump to content

Sky Slate Blueberry Blackcurrant Watermelon Strawberry Orange Banana Apple Emerald Chocolate
Photo

[Func] HTTPRequest: for Web APIs [AHK-B + AHK-L+Unicode+x64]


  • Please log in to reply
380 replies to this topic
SKAN
  • Administrators
  • 9115 posts
  • Last active:
  • Joined: 26 Dec 2005

a demo of uploading binary data via POST


Thank you very much. :)

BTW, provide .BMP in FileSelect. ImgUR automatically converts it into PNG.

fragman
  • Members
  • 1591 posts
  • Last active: Nov 12 2012 08:51 PM
  • Joined: 13 Oct 2009
ImgUr upload is very much appreciated. I'll check this out later...is there a way to monitor upload progress?

sumon
  • Moderators
  • 1317 posts
  • Last active: Dec 05 2016 10:14 PM
  • Joined: 18 May 2010

ImgUr upload is very much appreciated. I'll check this out later...is there a way to monitor upload progress?


Just guessing here, but since the upload is synchronous, I think not. Might be way wrong though.

By the way I am pretty sure it is spelled imgur, and actually pronounced 'imager'. I bet Appifyer with -er wasn't so wrong after all...

Edit: Actually, they are not themselves very consistent in their capitalization, in their logo and site title it's imgur, but sometimes in the site it's refered to as "Imgur".

nfl
  • Guests
  • Last active:
  • Joined: --
Thank you, this function is really awesome.
It would be nice to have an additional parameter for calling a function dynamically just before reading the content of the url, in every iteration of the loop and after the loop, so you can create a nice download progress.
If you don't want to add this feature, I will try to implement it for my personal use :)

VxE
  • Moderators
  • 3622 posts
  • Last active: Dec 24 2015 02:21 AM
  • Joined: 07 Oct 2006

Thank you very much. :)

BTW, provide .BMP in FileSelect. ImgUR automatically converts it into PNG.

I forgot bmp :oops: [fixed]


It would be nice to have an additional parameter for calling a function dynamically just before reading the content of the url, in every iteration of the loop and after the loop, so you can create a nice download progress.
If you don't want to add this feature, I will try to implement it for my personal use :)

I had considered going this route, but I got stuck trying to decide the best way to accept the callback function name from the user. Actually implementing this feature looks relatively simple, just replace HttpSendRequest with HttpSendRequestEx and loop InternetWriteFile at [1|2|4]K increments, with a callback on each loop.

I'm open to suggestions on how the callback should be specified. In my brainstorming, I though of 2 things
Either:
1. put the callback function name in the input headers. (isn't it too ugly to use the headers parameter for non-header options?)
OR
2. use a (predefined) function to store the progress, and let the user put their 'callback' routine on a timer.

nfl
  • Guests
  • Last active:
  • Joined: --
I have created a little example. That's how I would implement it, I have commented the relevant parts:

Modified HTTPRequest:
; ##################################################################################################
; ###                                       HTTPRequest                                          ###
; ##################################################################################################

HTTPRequest( url, byref in_POST_out_DATA="", byref inout_HEADERS="", proxy_info="", progressFunc="" ) { ; -----------
; Function by [VxE] (6-29-2011). Special thanks to derRaphael for inspiring this function.
; Submits one request to the specified URL and returns the number of bytes in the response.
; 'in_POST_out_DATA' must be a variable, which may contain data to be send as POST data. If the
; request completes successfully, 'in_POST_out_DATA' receives the response data. 'inout_HEADERS'
; must be a variable, and may contain headers to use for the request. If the request completes
; successfully, 'inout_HEADERS' receives the response headers; otherwise it receives an error summary.
; NOTE: proxy_info should be blank if you don't wish to use a proxy, otherwise it should be the
; address of the proxy you wish to use. Anything to the right of the first newline character is
; considered to be part of the proxy bypass list. E.g: proxy_info := "www.proxy.com`nwww.google.com"
; would tell HTTPRequest to use the proxy 'www.proxy.com', but not for urls with 'www.google.com'.
; NOTE: If the function encounters an error, an error message will be put into 'inout_HEADERS' and
; the function will return '0'. Since it is possible for a successful request to elicit a response
; with zero bytes, you should consult the response headers to determine if an error occured. On the
; other hand, if you are requesting data, a zero-byte response would indicate an error anyways.
; IMPORTANT: each header in 'inout_HEADERS' must conform to the following format:
; "<header name>: <header text>", and multiple headers MUST be separated by a linefeed.
; THE FOLLOWING HEADERS ARE HANDLED SPECIALLY:
; Content-Length  -> the header is added automatically IF AND ONLY IF the post data is not empty.
;                    Use the Content-Length header to override data length  auto-detection.
; Content-MD5     -> the value is computed automatically IF AND ONLY IF the VALUE is left blank.
; User-Agent      -> the header is set automatically if it isn't specified or if the value is blank.
;                    NOTE: The automatic user-agent contains the script file's name and OS version.
;                    If this is not desirable, please specify your own user-agent.
; Referrer        -> is uncorrected to 'Referer' because that's the actual official header name.
; +Flag           -> the value must either be an exact power of 2 or the NAME of one of the internet
;                    flags specified below. Use this to set custom flags for a request.
;                    E.g: "+Flag: INTERNET_FLAG_FORMS_SUBMIT", OR "+Flag: 0x40"
; >>              -> indicates the beginning of a file path to which to write the downloaded data.
;                    After the data has been written to the indicated file, the file is read into
;                    'in_POST_out_DATA' (this function still returns the number of bytes downloaded,
;                    which may not be the actual length of the data). This will overwrite the file
;                    if it already exists. This is functionally similar to URLDownloadToFile.
; IMPORTANT: for users of unicode versions of AHK: if you want to submit text as POST data (either
; a query string or XML feed or other text) AND your target url does not accept UTF-16 (wide-char)
; text, HTTPRequest can automatically convert the POST text to UTF-8, but ONLY IF you supply a
; Content-Type header with the attribute 'charset=UTF-8'. E.g: Content-Type: text/xml charset=UTF-8

     Static URL_Components, WorA := "", ModuleName := "WinINet.dll"
		, Scheme, Host, User, Pass, UrlPath, ExtraInfo, URL_Components
		, INTERNET_OPEN_TYPE_DIRECT := 1, INTERNET_OPEN_TYPE_PROXY := 3, hModule := 0
		, INTERNET_FLAG_DONT_CACHE                     := 0x04000000
		, INTERNET_FLAG_NO_CACHE_WRITE                 := 0x20000000
		, INTERNET_FLAG_FORMS_SUBMIT                   := 0x00000040
		, INTERNET_FLAG_FROM_CACHE                     := 0x01000000
		, INTERNET_FLAG_FWD_BACK                       := 0x00000020
		, INTERNET_FLAG_HYPERLINK                      := 0x00000400
		, INTERNET_FLAG_IGNORE_CERT_CN_INVALID         := 0x00001000
		, INTERNET_FLAG_IGNORE_CERT_DATE_INVALID       := 0x00002000
		, INTERNET_FLAG_IGNORE_REDIRECT_TO_HTTP        := 0x00008000
		, INTERNET_FLAG_IGNORE_REDIRECT_TO_HTTPS       := 0x00004000
		, INTERNET_FLAG_KEEP_CONNECTION                := 0x00400000
		, INTERNET_FLAG_MAKE_PERSISTENT                := 0x02000000
		, INTERNET_FLAG_MUST_CACHE_REQUEST             := 0x00000010
		, INTERNET_FLAG_NEED_FILE                      := 0x00000010
		, INTERNET_FLAG_NO_AUTH                        := 0x00040000
		, INTERNET_FLAG_NO_AUTO_REDIRECT               := 0x00200000
		, INTERNET_FLAG_NO_CACHE_WRITE                 := 0x04000000
		, INTERNET_FLAG_NO_COOKIES                     := 0x00080000
		, INTERNET_FLAG_NO_UI                          := 0x00000200
		, INTERNET_FLAG_OFFLINE                        := 0x01000000
		, INTERNET_FLAG_FROM_CACHE                     := 0x08000000
		, INTERNET_FLAG_PRAGMA_NOCACHE                 := 0x00000100
		, INTERNET_FLAG_RAW_DATA                       := 0x40000000
		, INTERNET_FLAG_READ_PREFETCH                  := 0x00100000
		, INTERNET_FLAG_RELOAD                         := 0x80000000
		, INTERNET_FLAG_RESTRICTED_ZONE                := 0x00020000
		, INTERNET_FLAG_RESYNCHRONIZE                  := 0x00000800
		, INTERNET_FLAG_SECURE                         := 0x00800000
		, iFlagList := "
		( LTRIM JOIN
			,INTERNET_FLAG_DONT_CACHE,INTERNET_FLAG_NO_CACHE_WRITE,INTERNET_FLAG_FORMS_SUBMIT
			,INTERNET_FLAG_FROM_CACHE,INTERNET_FLAG_FWD_BACK,INTERNET_FLAG_HYPERLINK
			,INTERNET_FLAG_IGNORE_CERT_CN_INVALID,INTERNET_FLAG_IGNORE_CERT_DATE_INVALID
			,INTERNET_FLAG_IGNORE_REDIRECT_TO_HTTP,INTERNET_FLAG_IGNORE_REDIRECT_TO_HTTPS
			,INTERNET_FLAG_KEEP_CONNECTION,INTERNET_FLAG_MUST_CACHE_REQUEST
			,INTERNET_FLAG_NEED_FILE,INTERNET_FLAG_NO_AUTH,INTERNET_FLAG_NO_AUTO_REDIRECT
			,INTERNET_FLAG_NO_CACHE_WRITE,INTERNET_FLAG_NO_COOKIES,INTERNET_FLAG_NO_UI
			,INTERNET_FLAG_OFFLINE,INTERNET_FLAG_FROM_CACHE,INTERNET_FLAG_PRAGMA_NOCACHE
			,INTERNET_FLAG_RAW_DATA,INTERNET_FLAG_READ_PREFETCH,INTERNET_FLAG_RELOAD
			,INTERNET_FLAG_RESTRICTED_ZONE,INTERNET_FLAG_RESYNCHRONIZE,INTERNET_FLAG_SECURE,
		)"

	If ( WorA = "" ) ; Initialize Static Varaibles
	{
		WorA := A_IsUnicode ? "W" : "A" ; Either 'A' (ansi) or 'W' (wide-chars).
	; Filling the URL_Components structure with the addresses of static variables only needs to
	; happen once per script instance. For unicode, the string capacities are doubled.
	; URL_COMPONENTS structure > http://msdn.microsoft.com/en-us/library/aa385420%28v=VS.85%29.aspx
		VarSetCapacity( URL_Components, 60, 0 )
		NumPut( 60, URL_Components, 0, "Int" )
		VarSetCapacity( Scheme, 16 << !! A_IsUnicode, 0 )
		NumPut( &Scheme, URL_Components, 4, "UInt" )
		VarSetCapacity( Host, 2048 << !! A_IsUnicode, 0 )
		NumPut( &Host, URL_Components, 16, "UInt" )
		VarSetCapacity( User, 2048 << !! A_IsUnicode, 0 )
		NumPut( &User, URL_Components, 28, "UInt" )
		VarSetCapacity( Pass, 2048 << !! A_IsUnicode, 0 )
		NumPut( &Pass, URL_Components, 36, "UInt" )
		VarSetCapacity( UrlPath, 4096 << !! A_IsUnicode, 0 )
		NumPut( &UrlPath, URL_Components, 44, "UInt" )
		VarSetCapacity( ExtraInfo, 4096 << !! A_IsUnicode, 0 )
		NumPut( &ExtraInfo, URL_Components, 52, "UInt" )
	}

	inout_HEADERS := "`r`n" inout_HEADERS "`r`n" ; Padding... yes it's important.

	; Determine the length of the POST data and auto-add the content-type if needed
	If RegexMatch( inout_HEADERS, "i)\v\h*Content-Length:\h*\K(?:0x[\da-f]+|\d+)", Content_Length )
	{
		Content_Length := RegexReplace( Content_Length + 0.0, ".*\K\..*" ) ; coerce to decimal
		; Give a default content-type header if there IS POST data but no content-type header.
		If !RegexMatch( inout_HEADERS, "i)\v\h*Content-Type:\h*\K\w+", Content_Type )
		{
			StringGetPos, pos, in_POST_out_DATA, <?xml
			If !( ErrorLevel ) && ( pos < 5 )
				Content_Type := "text/xml"
			Else Content_Type := "application/x-www-form-urlencoded"
			inout_HEADERS .= "Content-Type: " Content_Type "`r`n"
		}
	}
	Else ; the POST is either blank or contains text so we can determine the length automatically.
	{
		StringLen, Content_Length, in_POST_out_DATA
		If ( 0 < ( Content_Length := RegexReplace( Content_Length + 0.0, ".*\K\..*" ) ) )
		{
			inout_HEADERS .= "Content-Length: " Content_Length "`r`n"
			; Give a default content-type header if there IS POST data but no content-type header.
			If !RegexMatch( inout_HEADERS, "i)\v\h*Content-Type:\h*\K\V+", Content_Type )
			{
				StringGetPos, pos, in_POST_out_DATA, <?xml
				If !( ErrorLevel ) && ( pos < 5 )
					Content_Type := "text/xml"
				Else Content_Type := "application/x-www-form-urlencoded"
				inout_HEADERS .= "Content-Type: " Content_Type "`r`n"
			}
		}
	}

	; If the user wants to POST text in UTF-8, but they are using a unicode-version of AHK,
	; convert the POST data to UTF-8 and recalculate the length
	If ( A_IsUnicode && InStr( Content_Type, "charset=UTF-8" ) )
	{
		buffers := in_POST_out_DATA
		; WideCharToMultiByte > http://msdn.microsoft.com/en-us/library/dd374130%28v=vs.85%29.aspx
		VarSetCapacity( in_POST_out_DATA, size := DllCall( "WideCharToMultiByte"
			, "UInt", 65001, "UInt", 0, "UInt", &buffers, "UInt", Content_Length
			, "UInt", 0, "UInt", 0, "UInt", 0, "UInt", 0 ), 0 )
		DllCall( "WideCharToMultiByte"
			, "UInt", 65001, "UInt", 0, "UInt", &buffers, "UInt", Content_Length
			, "UInt", &in_POST_out_DATA, "UInt", size, "UInt", 0, "UInt", 0 )
		size := RegexReplace( size + 0.0, ".*\K\..*" )
		StringReplace, inout_HEADERS, inout_HEADERS, % "Content-Length: " Content_Length "`r`n", % "Content-Length: " size "`r`n"
		Content_Length := size
	}

	; Determine the accept type
	If !RegexMatch( inout_HEADERS, "i)\v\h*Accept:\h*\K\V+", Accept_Types )
		Accept_Types := "text/xml, text/* q=0.2, */* q=0.1"

	; Get the agent, if it's specified. Otherwise, add an auto-generated agent that has enough
	; info to satisfy any API that requires an informative agent.
	If !RegexMatch( inout_HEADERS, "i)\v\h*User-Agent:\h*\K\V+", Agent )
		inout_HEADERS .= "User-Agent: " ( Agent := RegexReplace( A_ScriptName, ".*\K\..*" )
				. "/1.0 (Language=AutoHotkey/" A_AhkVersion "; Platform=" A_OSVersion ")" ) "`r`n"

	; See if the user wants to output to a file.
	If RegexMatch( inout_HEADERS, "i)\v\h*>>?\h*\K(?:\w:)?[^`t`r`n:*?""<>|]+", output_file )
	{
		SplitPath, output_file,, folder
		IfNotExist, % folder
			output_file := ""
	}

	; Check the referer url
	RegexMatch( inout_HEADERS, "i)\v\h*Referr?er:\h+\K\V+", Referer_URL )

	; Check the content-MD5 header
	If RegexMatch( inout_HEADERS, "i)\v\h*Content-MD5:\h*\K\w*", pos ) && 40 != StrLen( pos )
		inout_HEADERS := RegexReplace( inout_HEADERS, "i)\v\h*Content-MD5:\K\h+\V*"
					, " " HTTPRequest_MD5( in_POST_out_DATA, Content_Length ) )

	; Typical flags for normal HTTP requests.
	Flags := 0
	Flags |= INTERNET_FLAG_KEEP_CONNECTION
	Flags |= INTERNET_FLAG_RELOAD
	Flags |= INTERNET_FLAG_NO_CACHE_WRITE

	; Properly format the headers. For each line in the headers, check to make sure it's formatted
	; like a header (Name: Value) and if it is, then append it to the the actual headers followed
	; by CRLF. Also, check the headers for additional flags the user may want to use.
	Loop, Parse, inout_HEADERS, `n, % "`t`r "
		If ( A_Index = 1 )
			inout_HEADERS := ""
		Else If RegexMatch( A_LoopField, "^(?<name>[^\h:]+):\h*\K.+", header )
				If ( headername != "+Flag" && headername != "-Flag" )
					inout_HEADERS .= headername ": " header "`r`n"
			Else IfInString, iFlagList, % "," header ","
					Flags := Asc( A_LoopField ) = 45 ? ~%header% & Flags : %header% | Flags
				Else If ( header = 1 << Ln( header ) / Ln( 2 ) )
					Flags := Asc( A_LoopField ) = 45 ? ~header & Flags : header | Flags

	; Load WinINet.dll. Because the 'hModule' is static, we can tell if the function interrupted
	; itself. If the function is interrupting itself, it shouldn't unload WinINet before ending.
	If !( interrupted := 0 != hModule )
	&& !( hModule := DllCall( "LoadLibrary" WorA, "UInt", &ModuleName ) )
	{
		inout_HEADERS := "There was a problem loading WinINet.dll. ErrorLevel = " ErrorLevel ", A_LastError = " A_LastError
		Return 0
	}

	; Put the sizes into the URL_Components structure (the sizes are the same for unicode and ansi
	; because the sizes are actually a character count, not a byte count).
	NumPut( 16, URL_Components, 8, "Int" )
	NumPut( 2048, URL_Components, 20, "Int" )
	NumPut( 2048, URL_Components, 32, "Int" )
	NumPut( 2048, URL_Components, 40, "Int" )
	NumPut( 4096, URL_Components, 48, "Int" )
	NumPut( 4096, URL_Components, 56, "Int" )

	; InternetCrackUrl > http://msdn.microsoft.com/en-us/library/aa384376%28VS.85%29.aspx
	If !DllCall( "WinINet\InternetCrackUrl" WorA, "UInt", &URL, "Int", StrLen( URL ), "UInt", 0, "UInt", &URL_Components )
	{
		inout_HEADERS := "There was a problem with the provided URL (InternetCrackUrl). ErrorLevel = " ErrorLevel ", A_LastError = " A_LastError
		If !( interrupted )
			hModule := 0 & DllCall( "FreeLibrary", "UInt", hModule )
		Return 0
	}
	; The port should always be 80... but if it's zero, then something went terribly wrong
	If !( Port := NumGet( URL_Components, 24, "UShort" ) )
	{
		inout_HEADERS := "There was a problem with the provided URL. The connection port could not be determined."
		If !( interrupted )
			hModule := 0 & DllCall( "FreeLibrary", "UInt", hModule )
		Return 0
	}

	; Update the internal lengths of the strings that were just cracked
	VarSetCapacity( Scheme, -1 )
	VarSetCapacity( Host, -1 )
	VarSetCapacity( User, -1 )
	VarSetCapacity( Pass, -1 )
	VarSetCapacity( UrlPath, -1 )
	VarSetCapacity( ExtraInfo, -1 )
	Query := UrlPath ExtraInfo

	If ( Scheme = "https" ) ; Apply these flags to HTTPS requests
		Flags |= INTERNET_FLAG_IGNORE_CERT_CN_INVALID | INTERNET_FLAG_SECURE | INTERNET_FLAG_IGNORE_CERT_DATE_INVALID
	Else If ( Scheme != "http" )
	{
	; Schemes other than HTTP and HTTPS are not supported by this function.
		inout_HEADERS := "HTTPRequest does not support '" Scheme "' type connections."
		If !( interrupted )
			hModule := 0 & DllCall( "FreeLibrary", "UInt", hModule )
		Return 0
	}

	; Handle the proxy specifications (if any)
	Loop, Parse, proxy_info, `n, % "`t`r "
		If ( A_Index = 1 )
			proxy_url := A_LoopField, proxy_info := ""
		Else proxy_info .= A_LoopField "`r`n"
	StringTrimRight, proxy_info, proxy_info, 2
	bUseProxy := 0 != InStr( proxy, "." ) ; don't bother verifying the proxy url
		
	; Tweak the accept type string to look like a list
	Loop, Parse, Accept_Types, `,
		Loop, Parse, A_LoopField, % Chr( 59 + !( pos := A_Index ) ), % "`t`n`r "
			If ( A_Index = 1 )
				If ( pos = 1 )
					Accept_Types := A_LoopField
				Else Accept_Types .= "`n" A_LoopField
	VarSetCapacity( int_array, pos + 1 << 2, 0 )

	; Build an array of pointers to the valid accept type strings and insert nulls into the
	; accept types string to make it look like a collection of null-terminated strings.
	pos := 0
	Loop, Parse, Accept_Types, `n
	{
		NumPut( &Accept_Types + pos, int_array, A_Index - 1 << 2, "UInt" )
		pos += StrLen( A_LoopField ) << !!A_IsUnicode
		NumPut( 0, Accept_Types, pos, A_IsUnicode ? "UShort" : "UChar" )
		pos += 1 << !!A_IsUnicode
	}

	; Get an internet handle. InternetOpen > http://msdn.microsoft.com/en-us/library/aa385096(v=VS.85).aspx
	hInternet := DllCall( "WinINet\InternetOpen" WorA
		, "UInt", &Agent
		, "UInt", bUseProxy ? INTERNET_OPEN_TYPE_PROXY : INTERNET_OPEN_TYPE_DIRECT
		, "UInt", bUseProxy ? &proxy_url : 0
		, "UInt", bUseProxy && proxy_info = "" ? 0 : &proxy_info
		, "UInt", 0 )

	If !( hInternet )
	{
		inout_HEADERS := "There was a problem opening an internet handle. ErrorLevel = " ErrorLevel ", A_LastError = " A_LastError
		If !( interrupted )
			hModule := 0 & DllCall( "FreeLibrary", "UInt", hModule )
		Return 0
	}

	; Open a connection. InternetConnect > http://msdn.microsoft.com/en-us/library/aa384363%28v=VS.85%29.aspx
	hConnection := DllCall( "WinINet\InternetConnect" WorA, "UInt", hInternet
		, "UInt", &Host
		, "UInt", Port
		, "UInt", &User
		, "UInt", &Pass
		, "UInt", 3 ; INTERNET_SERVICE_HTTP = 3
		, "UInt", Flags
		, "UInt", 0 )

	If !( hConnection )
	{
		inout_HEADERS := "There was a problem opening a connection to the host. ErrorLevel = " ErrorLevel ", A_LastError = " A_LastError
		hInternet := 0 & DllCall( "WinINet\InternetCloseHandle", "UInt", hInternet )
		If !( interrupted )
			hModule := 0 & DllCall( "FreeLibrary", "UInt", hModule )
		Return 0
	}

	; Open a request. HttpOpenRequest > http://msdn.microsoft.com/en-us/library/aa384233%28v=VS.85%29.aspx
	hRequest := DllCall( "WinINet\HttpOpenRequest" WorA, "UInt", hConnection
		, "Str", Content_Length = 0 ? "GET" : "POST"
		, "UInt", &Query
		, "Str", "HTTP/1.1"
		, "UInt", &Referer_URL
		, "UInt", &int_array
		, "UInt", Flags )

	If !( hRequest )
	{
		inout_HEADERS := "There was a problem opening the request. ErrorLevel = " ErrorLevel ", A_LastError = " A_LastError
		hInternet := 0 & DllCall( "WinINet\InternetCloseHandle", "UInt", hConnection )
		hInternet := 0 & DllCall( "WinINet\InternetCloseHandle", "UInt", hInternet )
		If !( interrupted )
			hModule := 0 & DllCall( "FreeLibrary", "UInt", hModule )
		Return 0
	}

	; apply the headers to the request ( to allow header errors to be detected and reported )
	pos := DllCall( "WinINet\HttpAddRequestHeaders" WorA, "UInt", hRequest
		, "Str", inout_HEADERS
		, "UInt", StrLen( inout_HEADERS )
		, "UInt", 0x20000000 ) ; HTTP_ADDREQ_FLAG_ADD = 0x20000000
	If !( pos )
	{
		inout_HEADERS := "There was a applying one or more headers to the request. ErrorLevel = " ErrorLevel ", A_LastError = " A_LastError "`nHeaders:`n" inout_HEADERS
		StringReplace, inout_HEADERS, inout_HEADERS, `r`n, `n, A
		hInternet := 0 & DllCall( "WinINet\InternetCloseHandle", "UInt", hRequest )
		hInternet := 0 & DllCall( "WinINet\InternetCloseHandle", "UInt", hConnection )
		hInternet := 0 & DllCall( "WinINet\InternetCloseHandle", "UInt", hInternet )
		If !( interrupted )
			hModule := 0 & DllCall( "FreeLibrary", "UInt", hModule )
		Return 0
	}

	; Send the request. HttpSendRequest > http://msdn.microsoft.com/en-us/library/aa384247%28v=VS.85%29.aspx
	pos := DllCall( "WinINet\HttpSendRequest" WorA, "UInt", hRequest
		, "UInt", 0
		, "UInt", 0
		, "UInt", &in_POST_out_DATA
		, "UInt", Content_Length )

	If !( pos )
	{
		inout_HEADERS := "There was a problem sending the request. ErrorLevel = " ErrorLevel ", A_LastError = " A_LastError
		hInternet := 0 & DllCall( "WinINet\InternetCloseHandle", "UInt", hRequest )
		hInternet := 0 & DllCall( "WinINet\InternetCloseHandle", "UInt", hConnection )
		hInternet := 0 & DllCall( "WinINet\InternetCloseHandle", "UInt", hInternet )
		If !( interrupted )
			hModule := 0 & DllCall( "FreeLibrary", "UInt", hModule )
		Return 0
	}

	VarSetCapacity( int_array, 4, 0 ) ; recycle this variable (use it as an INT)
	; Query the request for ready data. Actualy, it waits for data to become ready.
	; InternetQueryDataAvailable > http://msdn.microsoft.com/en-us/library/aa385100%28v=VS.85%29.aspx
	DllCall( "WinINet\InternetQueryDataAvailable", "UInt", hRequest, "UInt", &int_array, "UInt", 0, "UInt", 0 )

	VarSetCapacity( inout_HEADERS, 4096, 0 ) ; use 4K as first-try for response header length.
	NumPut( 4096, int_array )
	Loop 2 ; Get the response headers separated by CRLF. The first line has the HTTP response code
	{
		; HttpQueryInfo > http://msdn.microsoft.com/en-us/library/aa384238%28v=VS.85%29.aspx
		If ( pos := DllCall( "WinINet\HttpQueryInfo" WorA, "UInt", hRequest
			, "UInt", 22 ; HTTP_QUERY_RAW_HEADERS_CRLF = 22
			, "UInt", &inout_HEADERS
			, "UInt", &int_array
			, "UInt", 0 ) )
				Break

		If ( A_LastError = 122 ) ; ERROR_INSUFFICIENT_BUFFER = 122
			VarSetCapacity( inout_HEADERS, NumGet( int_array ) + 2, 0 )
	}

	If !( pos )
	{
		inout_HEADERS := "There was a problem reading the response headers. ErrorLevel = " ErrorLevel ", A_LastError = " A_LastError
		hInternet := 0 & DllCall( "WinINet\InternetCloseHandle", "UInt", hRequest )
		hInternet := 0 & DllCall( "WinINet\InternetCloseHandle", "UInt", hConnection )
		hInternet := 0 & DllCall( "WinINet\InternetCloseHandle", "UInt", hInternet )
		If !( interrupted )
			hModule := 0 & DllCall( "FreeLibrary", "UInt", hModule )
		Return 0
	}

	; Update response header outputvar length and remove carriage returns
	VarSetCapacity( inout_HEADERS, -1 )
	StringReplace, inout_HEADERS, inout_HEADERS, `r`n, `n, A
	; Get the content type (well use it to see if we should treat the response as a string)
	StringGetPos, pos, inout_HEADERS, % "`nContent-Type: "
	pos += 16
	StringMid, Accept_Types, inout_HEADERS, pos, InStr( inout_HEADERS "`n", "`n", 0, pos ) - pos
	If ( bTextdata := InStr( Accept_Types, "text/" ) = 1
		|| InStr( Accept_Types, "/xml" )
		|| InStr( Accept_Types, "/atom" )
		|| InStr( Accept_Types, "/json" )
		|| InStr( Accept_Types, "/x-www-form-urlencoded" )
		|| InStr( Accept_Types, "/xhtml" )
		|| InStr( Accept_Types, "/html" )
		|| InStr( Accept_Types, "/soap" ) )
		Codepage := InStr( Accept_Types, "charset=ISO-8859-1" ) ? 28591 : 65001

	; Download the response data
	Size := 0
	;--------- BEGIN Custom Download Progress ------------
	RegExMatch(inout_Headers,"Content-Length:\s+?(?P<Size>\d+)",full)
	SplitPath, URL, FileName,,,, DN
	FileName:=(FileName ? FileName : DN)
	%progressFunc%(true,fullSize,Filename)
	;--------- END Custom Download Progress ------------
	If ( output_file != "" )
	{
		; The user wants to save the info as a file, so delete the file if it exists.
		IfExist, % output_file
			FileDelete, % output_file

		; Use binary-mode file write. CreateFile > http://msdn.microsoft.com/en-us/library/aa363858%28v=vs.85%29.aspx
		If !( hFile := DllCall( "CreateFile" WorA, "Str", output_file
					, "Uint", 0x40000000 ; GENERIC_WRITE = 0x40000000
      				, "Uint", 0, "UInt", 0, "UInt", 4 ; OPEN_ALWAYS = 4
					, "Uint", 0, "UInt", 0 ) )
			inout_HEADERS .= "`nHTTPRequest Error: Could not create/open the file for writing. ErrorLevel = " ErrorLevel ", A_LastError = " A_LastError
		Else
		{
			; Read from the internet response and write to the file
			Loop
			{
				VarSetCapacity( buffers, 4096, 0 ) ; Use a 4K buffer
				; InternetReadFile > http://msdn.microsoft.com/en-us/library/aa385103%28v=VS.85%29.aspx
				pos := DllCall( "WinINet\InternetReadFile", "UInt", hRequest
					, "UInt", &buffers
					, "UInt", 4096
					, "UInt", &int_array )
				If !( pos && NumGet( int_array ) )
				{
					; CloseHandle > http://msdn.microsoft.com/en-us/library/ms724211%28v=vs.85%29.aspx
					DllCall( "CloseHandle", "Uint", hFile )
					If !( pos )
						inout_HEADERS .= "`nHTTPRequest Warning: InternetReadFile Failed. ErrorLevel = " ErrorLevel ", A_LastError = " A_LastError
					Else FileRead, in_POST_out_DATA, % output_file
					Break
				}
				Sleep -1
				Size += buffersize := NumGet( int_array )
				; WriteFile > http://msdn.microsoft.com/en-us/library/aa365747%28v=vs.85%29.aspx
				If !DllCall( "WriteFile", "UInt", hFile, "UInt", &buffers, "Int", buffersize, "UInt", &int_array, "UInt", 0 )
				{
					DllCall( "CloseHandle", "Uint", hFile )
					FileDelete, % output_file
					inout_HEADERS .= "`nHTTPRequest Error: There was a problem writing to the file. ErrorLevel = " ErrorLevel ", A_LastError = " A_LastError
					Break
				}
				;--------- BEGIN Custom Download Progress ------------
				%progressFunc%(Size,fullSize,Filename)
				;--------- END Custom Download Progress ------------
				Sleep -1
			}
			buffers := ""
		}
	}
	Else
	{
		; Read the response and store it into a pseudo-array of buffers
		Loop
		{
			buffers := A_Index
			VarSetCapacity( HTTPRequest_Buffer_%A_Index% := "", 4096, 0 )
			; InternetReadFile > http://msdn.microsoft.com/en-us/library/aa385103%28v=VS.85%29.aspx
			pos := DllCall( "WinINet\InternetReadFile", "UInt", hRequest
				, "UInt", &HTTPRequest_Buffer_%A_Index%
				, "UInt", 4096
				, "UInt", &int_array )
			If !( pos && NumGet( int_array ) )
				Break
			Size += HTTPRequest_BufferSize_%A_Index% := NumGet( int_array )
			;--------- BEGIN Custom Download Progress ------------
			%progressFunc%(Size,fullSize,Filename)
			;--------- END Custom Download Progress ------------
			Sleep -1
		}
		If !( pos )
			inout_HEADERS .= "`nHTTPRequest Warning: InternetReadFile Failed. ErrorLevel = " ErrorLevel ", A_LastError = " A_LastError
		VarSetCapacity( in_POST_out_DATA, Size + 1 << !!A_IsUnicode, 0 ) ; always put an ending null, even for non-text data
		Size := 0
		Loop % buffers - 1 ; Then copy the buffered data into the output parameter.
		{
			If ( A_IsUnicode ) && bTextdata ; convert ANSI or UTF-8 into Wide-Char (UTF-16)
				; MultiByteToWideChar > http://msdn.microsoft.com/en-us/library/dd319072%28v=vs.85%29.aspx
				Size += DllCall( "MultiByteToWideChar"
						, "UInt", CodePage, "UInt", 0
						, "UInt", &HTTPRequest_Buffer_%A_Index%
						, "UInt", HTTPRequest_BufferSize_%A_Index%
						, "UInt", &in_POST_out_DATA + Size
						, "UInt", HTTPRequest_BufferSize_%A_Index% << 1 ) << 1
			Else ; the script isn't unicode, so just copy byte for byte
			{
				; MoveMemory > http://msdn.microsoft.com/en-us/library/aa366788%28v=vs.85%29.aspx
				DllCall( "RtlMoveMemory"
						, "UInt", &in_POST_out_DATA + Size
						, "UInt", &HTTPRequest_Buffer_%A_Index%
						, "UInt", HTTPRequest_BufferSize_%A_Index% )
				Size += HTTPRequest_BufferSize_%A_Index%
			}
			HTTPRequest_Buffer_%A_Index% := ""
		}
	
		; If the content-type is text, update the output data length
		If ( bTextdata )
			VarSetCapacity( in_POST_out_DATA, -1 )
		; Scrapped idea: if content-type = text/xml, insert newlines and tabs to make it pretty
	} ; End Else

	;--------- BEGIN Custom Download Progress ------------
	%progressFunc%(false,fullSize,Filename)
	;--------- END Custom Download Progress ------------
	; InternetCloseHandle > http://msdn.microsoft.com/en-us/library/aa384350%28v=VS.85%29.aspx
	hRequest := 0 & DllCall( "WinINet\InternetCloseHandle", "UInt", hRequest )
	hConnection := 0 & DllCall( "WinINet\InternetCloseHandle", "UInt", hConnection )
	hInternet := 0 & DllCall( "WinINet\InternetCloseHandle", "UInt", hInternet )
	If !( interrupted )
		hModule := 0 & DllCall( "FreeLibrary", "UInt", hModule )
	Return Size
} ; HTTPRequest( url, byref in_POST_out_DATA="", byref inout_HEADERS="", proxy_info="" ) -----------

HTTPRequest_MD5( byref data, length=-1 ) { ; -------------------------------------------------------
; Computes the MD5 hash of a data blob of length 'length'. If 'length' is less than zero, this
; function assumes that 'data' is a null-terminated string and determines the length automatically.
	; static variables and constants r[0~63], encoded here as bytes with an offset of 64
	; ( that means the real value is the byte value minus 64, e.g: r[0] = 7, so 7 + 64 = 71 = 'G' )
	Static S, k, p:=0, r := "GLQVGLQVGLQVGLQVEINTEINTEINTEINTDKPWDKPWDKPWDKPWFJOUFJOUFJOUFJOU"

	VarSetCapacity( S, 64, 0 ) ; Initialize the block buffer S and constants p and k[0~63]
	IfEqual, p, 0, Loop % VarSetCapacity( k, 256 + !( p := &S ) ) >> 2 & 64
		NumPut( Floor(Abs(Sin(A_Index)) * 2**32 ), k, A_Index - 1 << 2, "UInt" )

	; autodetect message length if it's not specified (or is not positive)
	IfLess, length, 1, StringLen, length, data

	; initialize running accumulators and terminator (the terminator is appended to the message)
	ha := 0x67452301, hb := 0xEFCDAB89, hc := 0x98BADCFE, hd := 0x10325476

	; Begin rolling the message. This loop does 1 iteration for each 64 byte block such that the
	; last block has fewer than 55 bytes in it ( to leave room for the terminator and data length )
	Loop % length + 72 >> 6
	{
		If ( f := length - 64 > ( e := A_Index - 1 << 6 ) ? 64 : length > e ? length - e : 0 )
			DllCall( "RtlMoveMemory", "UInt", p, "UInt", &data + e, "Int", f ) ; copy the block
		If ( f != 64 && e <= length ) ; append the terminator to the message
			NumPut( 128, S, f, "UChar" )
		IfLess, f, 56, Loop 8 ; if this is the real last block, insert the data length in BITS
			NumPut( ( length << 3 >> ( A_Index - 1 << 3 ) ) & 255, S, 55 + A_Index, "UChar" )
		
		a := ha, b := hb, c := hc, d := hd ; copy running accumulators to intermediate variables

		Loop 64 ; begin rolling the block. These operations have been condensed and obfuscated.
		{ ; For i from 0 to 63 {
			e := NumGet( r, ( 2 - !A_IsUnicode ) * i := A_Index - 1, "UChar" ) & 31
			f := 0 = ( j := i >> 4 ) ? (b&c)|(~b&d) : j=1 ? (d&b)|(~d&c) : j=2 ? b^c^d : c^(~d|b)
			g := (( i * ( 3817 >> j * 3 & 7 ) + ( 328 >> j * 3 & 7 ) & 15 ) << 2 ) + p
			w := (*(g+3) << 24 | *(g+2) << 16 | *(g+1) << 8 | *g) + a + f + NumGet(k,i<<2,"UInt")
			a := d, d := c, c := b, b += w << e | (( w & 0xFFFFFFFF ) >> ( 32 - e ))
		}
		; add the intermediate variables to the running accumlators (making sure to mod by 2**32)
		ha := ha+a&0xFFFFFFFF, hb := hb+b&0xFFFFFFFF, hc := hc+c&0xFFFFFFFF, hd := hd+d&0xFFFFFFFF
		VarSetCapacity( S, 64, 0 ) ; Clear the block ( set bits to zero )
	}
	Loop 32 ; convert the running accumulators into 32 hex digits
		i := Chr( 96 + ( A_Index + 7 >> 3 ) ), S .= SubStr( "123456789abcdef0"
		, h%i% >> ( ( A_Index - 1 + ( A_Index & 1 ) - !( A_Index & 1 ) & 7 ) << 2 ) & 15, 1 )
	Return S ; return the hex digits
} ; HTTPRequest_MD5( byref data, length=-1 ) -------------------------------------------------------

Example code (derived from InternetFileRead):
file := "http://www.autohotkey.net/~Lexikos/AutoHotkey_L/AutoHotkey_L_Install.exe"
size := HTTPRequest( file, data, "", "", "ShowProgress" )
MsgBox %size%
write_bin(data,A_ScriptDir . "\AutoHotkey_L_Install.exe",size)
ExitApp

#Include %A_ScriptDir%\res
#Include httpRequest_custom.ahk


ShowProgress( WP=0, LP=0, Msg="" ) {
 If ( WP=1 ) {
	 SysGet, m, MonitorWorkArea, 1
	 y:=(mBottom-46-2), x:=(mRight-370-2), VarSetCapacity( Size,16*2,1 )
	 DllCall( "shlwapi.dll\StrFormatByteSize64A", Int64,LP, AStr,Size, UInt,16 )
	 Size := ( Size="0 bytes" ) ? N : "«" Size "»"
	 Progress, CWE6E3E4 CT000020 CBF73D00 x%x% y%y% w370 h46 B1 FS8 WM700 WS400 FM8 ZH8 ZY3
			 ,, %Msg%  %Size%, InternetFileRead(), Tahoma
	 WinSet, Transparent, 210, InternetFileRead()
 }
 Progress,% (P:=Round(WP/LP*100)),% "Memory Download: " wp " / " lp " [ " P "`% ]"
 IfEqual,wP,0, Progress, Off
}

write_bin(byref bin,filename,size){
   h := DllCall("CreateFile","str",filename,"Uint",0x40000000
            ,"Uint",0,"UInt",0,"UInt",4,"Uint",0,"UInt",0)
   IfEqual h,-1, SetEnv, ErrorLevel, -1
   IfNotEqual ErrorLevel,0,ExitApp ; couldn't create the file
   r := DllCall("SetFilePointerEx","Uint",h,"Int64",0,"UInt *",p,"Int",0)
   IfEqual r,0, SetEnv, ErrorLevel, -3
   IfNotEqual ErrorLevel,0, {
      t = %ErrorLevel%              ; save ErrorLevel to be returned
      DllCall("CloseHandle", "Uint", h)
      ErrorLevel = %t%              ; return seek error
   }
   result := DllCall("WriteFile","UInt",h,"Str",bin,"UInt"
               ,size,"UInt *",Written,"UInt",0)
   h := DllCall("CloseHandle", "Uint", h)
   return, 1
}


fragman
  • Members
  • 1591 posts
  • Last active: Nov 12 2012 08:51 PM
  • Joined: 13 Oct 2009
Looks nice! Is there any way to do an asynchronous request that works on a separate thread? I have a main loop running constantly in my program which should continue executing while an upload is in progress.

VxE
  • Moderators
  • 3622 posts
  • Last active: Dec 24 2015 02:21 AM
  • Joined: 07 Oct 2006

Looks nice! Is there any way to do an asynchronous request that works on a separate thread? I have a main loop running constantly in my program which should continue executing while an upload is in progress.

The asynchronous HTTP flow is quite different from the synchronous flow, so I probably won't pursue that path. You could always use a separate instance of AHK to do uploads/downloads (poor-man's multithreadding).


@nfl: That's a good start. I have added a callback option to HTTPRequest that works for both upload and download.

UPDATE: 7-1-2011. I made several significant changes to HTTPRequest, so please re-download. I separated the flags and output file options from the headers parameter and joined them with the new callback option and proxy options in the fourth parameter, which is now called 'options'.

The new callback option allows you to specify a function that gets called after each 'chunk' of data that gets transferred. The first parameter receives a decimal that represents the percentage completion of the operation and the second parameter receives the total number of bytes in the operation.

Because the chunk size is 4K, please don't expect the progress to be completely smooth, especially for smaller transfers. Also, in my tests, there was often a delay between finishing writing the data and receiving a response... I don't think that has anything to do with the HTTPRequest code itself though.

fragman
  • Members
  • 1591 posts
  • Last active: Nov 12 2012 08:51 PM
  • Joined: 13 Oct 2009

Looks nice! Is there any way to do an asynchronous request that works on a separate thread? I have a main loop running constantly in my program which should continue executing while an upload is in progress.

The asynchronous HTTP flow is quite different from the synchronous flow, so I probably won't pursue that path. You could always use a separate instance of AHK to do uploads/downloads (poor-man's multithreadding).


Then I'll have to see if I can modify your function so I can call it multiple times for each chunk or something like that. Running a separate process might be an option but it would require IPC or some command line parameters and sending progress messages to the main instance of the program.

Edit: I have successfully integrated ImgUr upload into 7plus. I now use the second process method and have the program call itself and send window messages for the progress notification. I've also seen the delay near the end of the file upload.

bichlepa
  • Members
  • 62 posts
  • Last active: Dec 17 2016 08:45 PM
  • Joined: 04 Jul 2011
Hello, forgive me please for my bad English.:oops:
I am trying to send cookies to the webpage <!-- m -->http://deru.contribute.dict.cc/<!-- m -->.
I am a very newbie in this field.

I got so far:
#noenv
URL      := "http://deru.contribute.dict.cc/"
Headers=
(
Cookie: cmbba2=history; cmbba1=edit; options_str=sres-pop--limveri-n--nres-50--linkw-y--ttslink-y--hlrows-y--linkp-y--cmenu-y--fict-y--asug-y--nickname-bichlepa--co_nres-50--home_rcol-w--co_rcol-src; 
)


HTTPRequest( url, buffer, Headers, "") ;connect

;Save recieved Webpage to file and show it:
filedelete,Seite.html 
fileappend,%buffer%,Seite.html,utf-8
run,Seite.html
exitapp
#include HTTPRequest.ahk
For any reason the cookie does not change anything. For example it should effect that the page will show 50 word pairs instead of 10.
I use ahk_L Unicode 32bit.
Please help!

VxE
  • Moderators
  • 3622 posts
  • Last active: Dec 24 2015 02:21 AM
  • Joined: 07 Oct 2006

For any reason the cookie does not change anything. For example it should effect that the page will show 50 word pairs instead of 10.
I use ahk_L Unicode 32bit.
Please help!


I'm very sorry about that. I did some more research on cookies with WinINet and it looks like you have to use "+Flag: INTERNET_FLAG_NO_COOKIES" (in parameter 4) for custom cookies to work. E.g:
URL := "http://deru.contribute.dict.cc/"
Headers=
(
Cookie: cmbba2=history; cmbba1=edit; options_str=sres-pop--limveri-n--nres-50--linkw-y--ttslink-y--hlrows-y--linkp-y--cmenu-y--fict-y--asug-y--nickname-bichlepa--co_nres-50--home_rcol-w--co_rcol-src; 
)
httprequest( url, data := "", headers, "+Flag: INTERNET_FLAG_NO_COOKIES `n >Seite.html" )

BUGFIX: (7-4-2011). Fixed a minor bug in parsing the output file option when the file path had a colon in it.

bichlepa
  • Members
  • 62 posts
  • Last active: Dec 17 2016 08:45 PM
  • Joined: 04 Jul 2011
Yay! It works! :lol:
Thank you very much. Your HTTPRequest is great!

bichlepa
  • Members
  • 62 posts
  • Last active: Dec 17 2016 08:45 PM
  • Joined: 04 Jul 2011
Hello again.
I have an other problem.
I want my script to be able to log in dict.cc. But it doesn't work with ahk_L unicode 32bit. But everything works with basic ahk.
This ist my code:
URL:= "http://users.dict.cc/urc_logn.php"
PostData=
(
hinz=bichlepa&kunz=I_dont_say&rmotc=on
)

Headers=
(
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Host: users.dict.cc
)

httprequest( url, PostData, headers, "+Flag: INTERNET_FLAG_NO_COOKIES" )
With ahk_L I get the answer that I have not entered any username. The answer should be that the username or password is invalid. (The password in this code is wrong, you understand :) )
I need it to work with ahl_L because of the unicode charset.

VxE
  • Moderators
  • 3622 posts
  • Last active: Dec 24 2015 02:21 AM
  • Joined: 07 Oct 2006
I'm sorry HTTPRequest is giving you such a headache, but I truly appreciate your bringing these bugs to my attention.

It turns out that the bug comes from flawed regex.

...when the newline character is at its default of CRLF (`r`n), two dots are required to match it (not one).

This meant that the regex '\v' would not match '`r`n', but would match either if they appeared alone. I have fixed the regex needles to use '\v+'.

The way I tested the UTF-16 -> UTF-8 POST conversion was by comparing the MD5 hash of the POST data between AHK-basic and AHK-L(u), in case anyone was wondering.

BUGFIX: (7-7-2011). Fixed flawed regex needles for detecting certain headers. Please re-download.

Also, please note that the source code is now sporting a license and limitation of liability based on the 3-clause BSD license.

bichlepa
  • Members
  • 62 posts
  • Last active: Dec 17 2016 08:45 PM
  • Joined: 04 Jul 2011
Thank you! It works! :D

I'm sorry HTTPRequest is giving you such a headache

I would have much more headache if your function would not exist and if you wouldn't help that quickly. :wink: