Jump to content

Sky Slate Blueberry Blackcurrant Watermelon Strawberry Orange Banana Apple Emerald Chocolate
Photo

[function] httpQuery GET and POST requests - update 0.3.6

Started by derRaphael , Jul 08 2008 10:58 AM

  • Please log in to reply
96 replies to this topic

Poll: Should a more generic function be released which will include the already available web functions, such as header queries, uri encoding, base encoding, etc ? (158 member(s) have cast votes)

Should a more generic function be released which will include the already available web functions, such as header queries, uri encoding, base encoding, etc ?

  1. yes, i'd like to have one function to get all neccessary http functionalities in one instead of collecting each for my own (174 votes [96.67%])

    Percentage of vote: 96.67%

  2. no, i prefer collecting the functions i need (5 votes [2.78%])

    Percentage of vote: 2.78%

  3. neither nor ... explained in post (1 votes [0.56%])

    Percentage of vote: 0.56%

Vote Guests cannot vote
derRaphael
  • Members
  • 872 posts
  • Last active: Mar 19 2013 04:42 PM
  • Joined: 23 Nov 2007
Introduction
hi all
This is a small function for handling POST and GET queries natively in AHK so external utilities to handle it are not needed any more. It can be used where no additional utilities, such as cURL or wget are wanted.
The latest version handles following features:

[*:4qpwruga] ports in URL
[*:4qpwruga] Username:Password@domain style URLs (see this answer to avoid common problems)
[*:4qpwruga] SSL aka https
[*:4qpwruga] HeaderInformations / Dumping / Storing
[*:4qpwruga] DownloadSizes / CurrentSize (for download progressbars)
[*:4qpwruga] Flags for InternetConnection handling (auto follow feature etc)
[*:4qpwruga] Referrer for Queries
[*:4qpwruga] Accepttypes for Queries
[*:4qpwruga] Proxy handling
[*:4qpwruga] timeout handling
[*:4qpwruga] custom UserAgent[/list]Usage

Usage is pretty simple:
Introducing with verion 0.3.6 the function supports an alternate syntax. which eases the function use. see example VI for details on this. To cut it short:
When the 1st parameter is not empty and contains a var holding an url, the httpQuery will try to return the data directly - which eliminates the need for that additional varsetcapacity call. However the old syntax is still available and working, so scripts using this function dont need to be changed.
Keep in mind, that when dealing with binary data - as for downloads of zip files, executables, or pictures, it is neccessary to assign the 1st parameter as an empty var and the 2nd containing the url.
Using the new syntax:

html := httpQUERY(URL:="http://url") will result the html of the standard HTML which is usually retrieved. The length will be stored as errorlevel. The query method is GET as the length of Postparams is zero.
html := httpQUERY(URL:="http://url",POSTDATA) will work as the GET above just with POSTDATA being transmitted. The query method is POST as the length of Postparams is non zero.
Using the old syntax:

You need to define a Variable which will receive the returned databuffer.
When only Strings are expected as returnvalues, autohotkey needs to learn the new size, so a VarSetCapacity(buffer,-1) is needed. This has been modified to support binary downloads aswell.
httpQUERY(buffer:="",URL) will result the length of the standard HTML which is usually retrieved. The html itself is stored in the variable buffer. The query method is GET as the length of Postparams is zero.
httpQUERY(buffer:="",URL,POSTDATA) will work as the GET above just with POSTDATA being transmitted. The query method is POST as the length of Postparams is non zero.


The URL now supports following scheme:
<!-- m -->http://username:pass... ... s#fragment<!-- m -->

Since httpQuery has been updated to use InternetCrackURL from winINet, all essential parts will be recognized. so there is no need to set up any additional parameters. Attention: When u need to authetificate in the Website the username / password attempt will not work. u have to submit those parameters via POST or GET method.

Additional Parameters:
To see a dump of the received httpHeaders, there is buildIn support for a global Variable named httpQueryOps. It may consist of one or more Verbs. For now "showHeader", "storeHeader", and "updateSize" verbs are supported. If You use storeHeader the complete Header will be saved in a variable named HttpQueryHeader which will be made global at runtime. The verb updateSize will make two variables globally Available: httpQueryFullSize and httpQueryCurrentSize. An usage example to show a download status indicator is included

Following Variables are evaluated globally:
httpAgent: can hold an individual UserAgent Code. The default is AutoHotkeyScript
httpProxy: the name of a proxy server to use. default = 0
httpProxyByPass: list of domains which will not be used with the proxy. default = 0
httpQueryReferer: an URL which is recognized by QueriedServer, as the location where the Request was generated mind though its "Referrer" the name of the variable is httpQueryReferer [sic]
httpQueryAcceptType: this variable lets u specify your accepted stream formats for the results of the query
httpQueryDwFlags: if in need for any special flags for the current connection this is the variable to set (example V shows an useCase for this feat)

I think these are pretty self explaining.
Examples

Example I: Showing the raw HTML result of a search for httpQuery in AutoHotkey's forum in an edit control 
; exmpl.searchAHKforum.httpQuery.ahk
; Searches the forum for a given Phrase: in this case httpQuery
#noenv
html     := ""
URL      := "http://www.autohotkey.com/forum/search.php?mode=results"
POSTData := "search_keywords=httpQuery&search_terms=all&search_forum=-1&"
          . "search_time=0&search_fields=all&show_results=topics&return_chars=500"
          . "&sort_by=0&sort_dir=DESC"

length := httpQuery(html,URL,POSTdata)
varSetCapacity(html,-1)

Gui,Add,Edit,w600 +Wrap r25,% html
Gui,Show
Return

GuiClose:
GuiEscape:
   ExitApp
   
#include httpQuery.ahk
Remarks: To make a function work with POST parameters, it is essential to know which parameters are actually sent. These can be obtained by studying the HTML source of the target URL or by using analyzing tools such as the HttpFox Addon for Firefox. You might also want to use the following example to learn the nature of the form in a HTML document
Example II: Showing a dump of a HTML form to help making valid POST parameters
; exmpl.formdump.httpquery.ahk
; Form Dumper v0.1b (w) 9th July 2008 by derRaphael
#NoEnv
InputBox
 ,URL                                                                  ; OutputVariable
 ,Enter URL to analyze                                                 ; Title of box
 ,Please enter the complete URL starting with http:// to be analysed   ; Descriptive text 
 ,,,,,,,,http://www.autohotkey.com/forum/search.php                    ; default value

	html := ""
	htmlLength := httpQuery(html,URL)
	VarSetCapacity(html,-1)

; The Complete Form Node from given URL's HTML
	RegExMatch(html,"i)<form.+?</form>",formNode)
; Just the formtag for further analyzing
	RegExMatch(formNode,"i)<form[^>]+?>",formTag)
; The name of the form
	RegExMatch(formTag,"i)NAME=""?(?P<Name>.+?)""?>",form)
; The method used to process Data
	RegExMatch(formTag,"i)METHOD=""?(?P<Method>.+?)""?\s",form)
; The complete address used to send data to
	RegExMatch(formTag,"i)ACTION=""?(?P<Action>.+?)""?\s",form)
; just the url
	RegExMatch(formAction,"i)(?P<URL>[^\?]+)",formA_)
; any existing GET parameters
	RegExMatch(formAction,"i)\?(?P<GET>.*)",formA_)
	; Fix & to & as delimiter
	StringReplace,formA_GET,formA_GET,&`;,&,All

	startPosI := startPosS := startPosT := 0
	inpCount := selCount := txtCount := 0

	Loop,
	{
		If (startPosI:=RegExMatch(formNode,"i)<input[^>]+>",inputTag,startPosI+1)) {
			inpCount++
			formInput%InpCount% := inputTag
		} else {
			noMoreInput := 1
		}
		If (startPosS:=RegExMatch(formNode,"i)<select.*?</select>",selectNode,startPosS+1)) {
			selCount++
			formSelect%selCount% := selectNode
		} else {
			noMoreSelectOptions := 1
		}
		If (startPosT:=RegExMatch(formNode,"i)<textarea.*?</textarea>",textareaNode,startPosT+1)) {
			txtCount++
			formTextarea%txtCount% := textareaNode
		} else {
			noMoreTextArea := 1
		}
		if (NoMoreInput) && (NoMoreTextarea) && (NoMoreSelectOptions) {
			break
		}
	}
	Loop,% inpCount
		Inputs .= formInput%A_Index% "`n"
	StringReplace,inputs,inputs,<,%A_Tab%<,All
	Loop,% selCount
		Selects .= formSelect%A_Index% "`n"
	StringReplace,Selects,Selects,</option>,</option>`n,All
	StringReplace,Selects,Selects,<option,%A_Tab%<option,All
	Selects := RegExReplace(selects,"i)<select[^>]+>","$0`n")
		
	Loop,% txtCount
		txtAreas .= formTextArea%A_Index% "`n"
	StringReplace,txtAreas,txtAreas,</textarea>,</textarea>`n,All
	StringReplace,txtAreas,txtAreas,<textarea,%A_Tab%<textarea,All

; dump results to Gui
Gui,Add,Tab2, w800 h20,Analyzed Content|found Form|raw HTML
Gui,Tab,1
Gui,Add,Edit,w800 h600 yp+20 xp , % ""
		. "formTag:`n" formTag "`n`n"
        . "formMethod:`t" formMethod "`n"
        . "formAction:`t" formAction "`n"
        . "formActionURL:`t" formA_URL "`n"
        . "formActionGET:`t" formA_GET "`n`n"
		. "Total Inputs:`t" inpCount "`n"
		. inputs
		. "`nTotal Selects:`t" selCount "`n"
		. selects "`n"
		. "`nTotal TextAreas:`t" txtCount "`n"
		. txtAreas "`n"
Gui,Tab,2
Gui,Add,Edit,w800 h600 yp xp , % formNode
Gui,Tab,3
Gui,Add,Edit,w800 h600 yp xp , % html
Gui,Show,,Statistics for Form: "%formName%" (%URL%)
return

GuiClose:
GuiEscape:
	ExitApp
	
#include httpQuery.ahk
Remarks: Using this function provides informations of which elements are used in a form which will be submitted. It neither catches modded JavaScript content nor more than one form. It has been tested with google.com, autohotkey.com's searchform and autohotkey.net's pastebin
Example III: Downloading binary data and saving it to a local file.
; exmpl.downloadBinary.httpQuery.ahk
; This example downloads the latest AHK environment and stores
; the received binary data to a file.
#noenv
data     := ""
URL      := "https://ahknet.autohotkey.com/programs/AutoHotkey104706.zip"
httpQueryOps := "updateSize"
SetTimer,showSize,10
length   := httpQuery(data,URL)
Tooltip
if (write_bin(data,"ahk.exe",length)!=1)
	MsgBox "There was an Error!"
else
	MsgBox AHK Source downloaded and saved as "ahk.zip"!
Return

showSize:
   Tooltip,% HttpQueryCurrentSize "/" HttpQueryFullSize
return

GuiClose:
GuiEscape:
   ExitApp

write_bin(byref bin,filename,size){
   h := DllCall("CreateFile","str",filename,"Uint",0x40000000
            ,"Uint",0,"UInt",0,"UInt",4,"Uint",0,"UInt",0)
   IfEqual h,-1, SetEnv, ErrorLevel, -1
   IfNotEqual ErrorLevel,0,ExitApp ; couldn't create the file
   r := DllCall("SetFilePointerEx","Uint",h,"Int64",0,"UInt *",p,"Int",0)
   IfEqual r,0, SetEnv, ErrorLevel, -3
   IfNotEqual ErrorLevel,0, {
      t = %ErrorLevel%              ; save ErrorLevel to be returned
      DllCall("CloseHandle", "Uint", h)
      ErrorLevel = %t%              ; return seek error
   }
   result := DllCall("WriteFile","UInt",h,"Str",bin,"UInt"
               ,size,"UInt *",Written,"UInt",0)
   h := DllCall("CloseHandle", "Uint", h)
   return, 1
}

#include httpQuery.ahk
Remarks: This example downloads the current Autohotkey Source and saves it to local disk as ahk.zip in the same path as the script. While it does so, it shows the current progress of the download

Example IV: Uploading an image to Imageshack using the official (free) API
; exmpl.imageshack.httpQuery.ahk
; This example uploads an image and constructs a multipart/form-data Type
; for fileuploading and returns the XML which is returned to show the stored Imagepath
	FileSelectFile,image
	FileGetSize,size,%image%
	SplitPath,image,OFN
	FileRead,img,%image%
	VarSetCapacity(placeholder,size,32)
	boundary := makeProperBoundary()
	post:="--" boundary "`ncontent-disposition: form-data; name=""MAX_FILE_SIZE""`n`n"
		. "1048576`n--" boundary "`ncontent-disposition: form-data; name=""xml""`n`nyes`n--"
		. boundary "`ncontent-disposition: form-data; name=""fileupload""; filename="""
		. ofn """`nContent-type: " MimeType(img) "`nContent-Transfer-Encoding: binary`n`n" 
		. placeholder "`n--" boundary "--"
	headers:="Content-type: multipart/form-data, boundary=" boundary "`nContent-Length: " strlen(post)
	DllCall("RtlMoveMemory","uInt",(offset:=&post+strlen(post)-strlen(Boundary)-size-5)
			,"uInt",&img,"uInt",size)
	size := httpQuery(result:="","http://www.imageshack.us/index.php",post,headers)
	VarSetCapacity(result,-1)
	Gui,Add,Edit,w800 h600, % result
	Gui,Show
return

GuiClose:
GuiEscape:
	ExitApp

makeProperBoundary(){
	Loop,26
		n .= chr(64+a_index)
	n .= "0123456789"
	Loop,% StrLen(A_Now) {
		Random,rnd,1,% StrLen(n)
		Random,UL,0,1
		b .= RegExReplace(SubStr(n,rnd,1),".$","$" (round(UL)? "U":"L") "0")
	}
	Return b
}

MimeType(ByRef Binary) {
	MimeTypes:="424d image/bmp|4749463 image/gif|ffd8ffe image/jpeg|89504e4 image/png|4657530"
			 . " application/x-shockwave-flash|49492a0 image/tiff"
	@:="0123456789abcdef"
	Loop,8
		hex .= substr(@,(*(a:=&Binary-1+a_index)>>4)+1,1) substr(@,((*a)&15)+1,1)
	Loop,Parse,MimeTypes,|
		if ((substr(hex,1,strlen(n:=RegExReplace(A_Loopfield,"\s.*"))))=n) 
			Mime := RegExReplace(A_LoopField,".*?\s")
	Return (Mime!="") ? Mime : "application/octet-stream"
}

#include httpQuery.ahk
Remarks: Uploading binary data where an encodingtype "multipart/form-data" was needed, didnt work with classic POST usage. this example shows how to build the complete POST request in a different manner and to include raw binary data with no encoding such as base64 or ascii85

Example V: Modifying dwFlags for special useCases where a 302 is the returncode such as with dynamic created content urls
#NoEnv
	Gui,add,edit,w800 h400 vPasteBin, this is a testtext!
	Gui,Add,Button,gPaste wp, Upload this text to http://pastebin.com
	Gui,Show
return

Paste:
	Gui,Submit,Nohide
	URL := "http://pastebin.com/pastebin.php" 
	POSTDATA := "parent_pid=&format=text&code2=" uriEncode(PasteBin) "&"
			  . "poster=&paste=Send&expiry=d&email=&"
	httpQueryOps := "showHeader storeHeader"
	httpQueryDwFlags := (INTERNET_FLAG_NO_AUTO_REDIRECT:= 0x00200000)
	length := httpQuery(HTML:="",URL,POSTDATA)
	VarSetCapacity(HTML,-1)
	gui,destroy
	Gui,Add,Edit,w800 h600,% HttpQueryHeader "`n`n" html
	Gui,Show
Return

GuiClose:
GuiEscape:
   ExitApp

uriEncode(str)
{ ; v 0.3 / (w) 24.06.2008 by derRaphael / zLib-Style release
	b_Format := A_FormatInteger
	data := ""
	SetFormat,Integer,H
	Loop,Parse,str
		if ((Asc(A_LoopField)>0x7f) || (Asc(A_LoopField)<0x30) || (asc(A_LoopField)=0x3d))
			data .= "%" . ((StrLen(c:=SubStr(ASC(A_LoopField),3))<2) ? "0" . c : c)
		Else
			data .= A_LoopField
	SetFormat,Integer,%b_format%
	return data
}

#include httpQUERY.ahk
Remarks: Some WebServices such as PasteBin.com generate Dynamic Location Urls in order to show processed content. Using httpQuery in a default manner, results in a 404 Not found error when the script tries to access the location to fast. Specifying the NO_FOLLOW dwFlag solves this problem but needs a 2nd request to grab the fresh received Location Header.

Example VI: Introducing the new syntax coming with version 0.3.6
; Syntax as it's supported from v 0.3.6 
; resulting directly in a textual content eliminating the need for varsetcapacity
; Please note, that the url when passed as 1st parameter has to be
; assigned to a var - such as below
html := httpQuery( url:="http://www.autohotkey.com/download/index.htm" )
MsgBox,0, Length: %ErrorLevel%, % html
; to not break scripts, the known syntax (pre 0.3.6) is still supported
; this is neccessary for binary downloads
; (dont varsetcapacity - it'll break the binary data)
length := httpQuery( data:="", "http://www.autohotkey.com/download/index.htm" )
VarSetCapacity( data, -1 )
MsgBox,0, Length: %length%, % data

Remarks: The only thing which is quite unsatisfying, is the fact that the url has to assigned as a var and when downloading binaries, the data must be empty.
httpQuery 0.3.6
; httpQuery-0-3-6.ahk
httpQuery(byref p1 = "", p2 = "", p3="", p4="")
{   ; v0.3.6 (w) Oct, 26 2010 by derRaphael / zLib-Style release
   ; currently the verbs showHeader, storeHeader, and updateSize are supported in httpQueryOps
   ; in case u need a different UserAgent, Proxy, ProxyByPass, Referrer, and AcceptType just
   ; specify them as global variables - mind the varname for referrer is httpQueryReferer [sic].
   ; Also if any special dwFlags are needed such as INTERNET_FLAG_NO_AUTO_REDIRECT or cache
   ; handling this might be set using the httpQueryDwFlags variable as global
   global httpQueryOps, httpAgent, httpProxy, httpProxyByPass, httpQueryReferer, httpQueryAcceptType
       , httpQueryDwFlags
   ; Get any missing default Values
   
   ;v0.3.6
   ; check for syntax
   if ( VarSetCapacity(p1) != 0 )
      dReturn:=true,  result := "", lpszUrl := p1, POSTDATA := p2, HEADERS  := p3
   else
      result := p1, lpszUrl := p2, POSTDATA := p3, HEADERS  := p4
   
   defaultOps =
   (LTrim Join|
      httpAgent=AutoHotkeyScript|httpProxy=0|httpProxyByPass=0|INTERNET_FLAG_SECURE=0x00800000
      SECURITY_FLAG_IGNORE_UNKNOWN_CA=0x00000100|SECURITY_FLAG_IGNORE_CERT_CN_INVALID=0x00001000
      SECURITY_FLAG_IGNORE_CERT_DATE_INVALID=0x00002000|SECURITY_FLAG_IGNORE_CERT_WRONG_USAGE=0x00000200
      INTERNET_OPEN_TYPE_PROXY=3|INTERNET_OPEN_TYPE_DIRECT=1|INTERNET_SERVICE_HTTP=3
   )
   Loop,Parse,defaultOps,|
   {
      RegExMatch(A_LoopField,"(?P<Option>[^=]+)=(?P<Default>.*)",http)
      if StrLen(%httpOption%)=0
         %httpOption% := httpDefault
   }

   ; Load Library
   hModule := DllCall("LoadLibrary", "Str", "WinINet.Dll")

   ; SetUpStructures for URL_COMPONENTS / needed for InternetCrackURL
   ; http://msdn.microsoft.com/en-us/library/aa385420(VS.85).aspx
   offset_name_length:= "4-lpszScheme-255|16-lpszHostName-1024|28-lpszUserName-1024|"
                  . "36-lpszPassword-1024|44-lpszUrlPath-1024|52-lpszExtrainfo-1024"
   VarSetCapacity(URL_COMPONENTS,60,0)
   ; Struc Size               ; Scheme Size                  ; Max Port Number
   NumPut(60,URL_COMPONENTS,0), NumPut(255,URL_COMPONENTS,12), NumPut(0xffff,URL_COMPONENTS,24)
   
   Loop,Parse,offset_name_length,|
   {
      RegExMatch(A_LoopField,"(?P<Offset>\d+)-(?P<Name>[a-zA-Z]+)-(?P<Size>\d+)",iCU_)
      VarSetCapacity(%iCU_Name%,iCU_Size,0)
      NumPut(&%iCU_Name%,URL_COMPONENTS,iCU_Offset)
      NumPut(iCU_Size,URL_COMPONENTS,iCU_Offset+4)
   }

   ; Split the given URL; extract scheme, user, pass, authotity (host), port, path, and query (extrainfo)
   ; http://msdn.microsoft.com/en-us/library/aa384376(VS.85).aspx
   DllCall("WinINet\InternetCrackUrlA","Str",lpszUrl,"uInt",StrLen(lpszUrl),"uInt",0,"uInt",&URL_COMPONENTS)

   ; Update variables to retrieve results
   Loop,Parse,offset_name_length,|
   {
      RegExMatch(A_LoopField,"-(?P<Name>[a-zA-Z]+)-",iCU_)
      VarSetCapacity(%iCU_Name%,-1)
   }
   nPort:=NumGet(URL_COMPONENTS,24,"uInt")
   
   ; Import any set dwFlags
   dwFlags := httpQueryDwFlags
   ; For some reasons using a selfsigned https certificates doesnt work
   ; such as an own webmin service - even though every security is turned off
   ; https with valid certificates works when
   if (lpszScheme = "https")
      dwFlags |= (INTERNET_FLAG_SECURE|SECURITY_FLAG_IGNORE_CERT_CN_INVALID
               |SECURITY_FLAG_IGNORE_CERT_WRONG_USAGE)

   ; Check for Header and drop exception if unknown or invalid URL
   if (lpszScheme="unknown") {
      Result := "ERR: No Valid URL supplied."
      Return StrLen(Result)
   }

   ; Initialise httpQuery's use of the WinINet functions.
   ; http://msdn.microsoft.com/en-us/library/aa385096(VS.85).aspx
   hInternet := DllCall("WinINet\InternetOpenA"
                  ,"Str",httpAgent,"UInt"
                  ,(httpProxy != 0 ?  INTERNET_OPEN_TYPE_PROXY : INTERNET_OPEN_TYPE_DIRECT )
                  ,"Str",httpProxy,"Str",httpProxyBypass,"Uint",0)

   ; Open HTTP session for the given URL
   ; http://msdn.microsoft.com/en-us/library/aa384363(VS.85).aspx
   hConnect := DllCall("WinINet\InternetConnectA"
                  ,"uInt",hInternet,"Str",lpszHostname, "Int",nPort
                  ,"Str",lpszUserName, "Str",lpszPassword,"uInt",INTERNET_SERVICE_HTTP
                  ,"uInt",0,"uInt*",0)

   ; Do we POST? If so, check for header handling and set default
   if (Strlen(POSTDATA)>0) {
      HTTPVerb:="POST"
      if StrLen(Headers)=0
         Headers:="Content-Type: application/x-www-form-urlencoded"
   } else ; otherwise mode must be GET - no header defaults needed
      HTTPVerb:="GET"   

   ; Form the request with proper HTTP protocol version and create the request handle
   ; http://msdn.microsoft.com/en-us/library/aa384233(VS.85).aspx
   hRequest := DllCall("WinINet\HttpOpenRequestA"
                  ,"uInt",hConnect,"Str",HTTPVerb,"Str",lpszUrlPath . lpszExtrainfo
                  ,"Str",ProVer := "HTTP/1.1", "Str",httpQueryReferer,"Str",httpQueryAcceptTypes
                  ,"uInt",dwFlags,"uInt",Context:=0 )

   ; Send the specified request to the server
   ; http://msdn.microsoft.com/en-us/library/aa384247(VS.85).aspx
   sRequest := DllCall("WinINet\HttpSendRequestA"
                  , "uInt",hRequest,"Str",Headers, "uInt",Strlen(Headers)
                  , "Str",POSTData,"uInt",Strlen(POSTData))

   VarSetCapacity(header, 2048, 0)  ; max 2K header data for httpResponseHeader
   VarSetCapacity(header_len, 4, 0)
   
   ; Check for returned server response-header (works only _after_ request been sent)
   ; http://msdn.microsoft.com/en-us/library/aa384238.aspx
   Loop, 5
     if ((headerRequest:=DllCall("WinINet\HttpQueryInfoA","uint",hRequest
      ,"uint",21,"uint",&header,"uint",&header_len,"uint",0))=1)
      break

   If (headerRequest=1) {
      VarSetCapacity(res,headerLength:=NumGet(header_len),32)
      DllCall("RtlMoveMemory","uInt",&res,"uInt",&header,"uInt",headerLength)
      Loop,% headerLength
         if (*(&res-1+a_index)=0) ; Change binary zero to linefeed
            NumPut(Asc("`n"),res,a_index-1,"uChar")
      VarSetCapacity(res,-1)
   } else
      res := "timeout"

   ; Get 1st Line of Full Response
   Loop,Parse,res,`n,`r
   {
      RetValue := A_LoopField
      break
   }
   
   ; No Connection established - drop exception
   If (RetValue="timeout") {
      html := "Error: timeout"
      return -1
   }
   ; Strip protocol version from return value
   RetValue := RegExReplace(RetValue,"HTTP/1\.[01]\s+")
   
    ; List taken from http://en.wikipedia.org/wiki/List_of_HTTP_status_codes
   HttpRetCodes := "100=Continue|101=Switching Protocols|102=Processing (WebDAV) (RFC 2518)|"
              . "200=OK|201=Created|202=Accepted|203=Non-Authoritative Information|204=No"
              . " Content|205=Reset Content|206=Partial Content|207=Multi-Status (WebDAV)"
              . "|300=Multiple Choices|301=Moved Permanently|302=Found|303=See Other|304="
              . "Not Modified|305=Use Proxy|306=Switch Proxy|307=Temporary Redirect|400=B"
              . "ad Request|401=Unauthorized|402=Payment Required|403=Forbidden|404=Not F"
              . "ound|405=Method Not Allowed|406=Not Acceptable|407=Proxy Authentication "
              . "Required|408=Request Timeout|409=Conflict|410=Gone|411=Length Required|4"
              . "12=Precondition Failed|413=Request Entity Too Large|414=Request-URI Too "
              . "Long|415=Unsupported Media Type|416=Requested Range Not Satisfiable|417="
              . "Expectation Failed|418=I'm a teapot (RFC 2324)|422=Unprocessable Entity "
              . "(WebDAV) (RFC 4918)|423=Locked (WebDAV) (RFC 4918)|424=Failed Dependency"
              . " (WebDAV) (RFC 4918)|425=Unordered Collection (RFC 3648)|426=Upgrade Req"
              . "uired (RFC 2817)|449=Retry With|500=Internal Server Error|501=Not Implem"
              . "ented|502=Bad Gateway|503=Service Unavailable|504=Gateway Timeout|505=HT"
              . "TP Version Not Supported|506=Variant Also Negotiates (RFC 2295)|507=Insu"
              . "fficient Storage (WebDAV) (RFC 4918)|509=Bandwidth Limit Exceeded|510=No"
              . "t Extended (RFC 2774)"
   
   ; Gather numeric response value
   RetValue := SubStr(RetValue,1,3)
   
   ; Parse through return codes and set according informations
   Loop,Parse,HttpRetCodes,|
   {
      HttpReturnCode := SubStr(A_LoopField,1,3)    ; Numeric return value see above
      HttpReturnMsg  := SubStr(A_LoopField,5)      ; link for additional information
      if (RetValue=HttpReturnCode) {
         RetMsg := HttpReturnMsg
         break
      }
   }

   ; Global HttpQueryOps handling
   if strlen(HTTPQueryOps)>0 {
      ; Show full Header response (usefull for debugging)
      if (instr(HTTPQueryOps,"showHeader"))
         MsgBox % res
      ; Save the full Header response in a global Variable
      if (instr(HTTPQueryOps,"storeHeader"))
         global HttpQueryHeader := res
      ; Check for size updates to export to a global Var
      if (instr(HTTPQueryOps,"updateSize")) {
         Loop,Parse,res,`n
            If RegExMatch(A_LoopField,"Content-Length:\s+?(?P<Size>\d+)",full) {
               global HttpQueryFullSize := fullSize
               break
            }
         if (fullSize+0=0)
            HttpQueryFullSize := "size unavailable"
      }
   }

   ; Check for valid codes and drop exception if suspicious
   if !(InStr("100 200 201 202 302",RetValue)) {
      Result := RetValue " " RetMsg
      return StrLen(Result)
   }

   VarSetCapacity(BytesRead,4,0)
   fsize := 0
   Loop            ; the receiver loop - rewritten in the need to enable
   {               ; support for larger file downloads
      bc := A_Index
      VarSetCapacity(buffer%bc%,1024,0) ; setup new chunk for this receive round
      ReadFile := DllCall("wininet\InternetReadFile"
                  ,"uInt",hRequest,"uInt",&buffer%bc%,"uInt",1024,"uInt",&BytesRead)
      ReadBytes := NumGet(BytesRead)    ; how many bytes were received?
      If ((ReadFile!=0)&&(!ReadBytes))  ; we have had no error yet and received no more bytes
         break                         ; we must be done! so lets break the receiver loop
      Else {
         fsize += ReadBytes            ; sum up all chunk sizes for correct return size
         sizeArray .= ReadBytes "|"
      }
      if (instr(HTTPQueryOps,"updateSize"))
         Global HttpQueryCurrentSize := fsize
   }
   sizeArray := SubStr(sizeArray,1,-1)   ; trim last PipeChar
   
   VarSetCapacity( ( dReturn == true ) ? result : p1 ,fSize+1,0)      ; reconstruct the result from above generated chunkblocks
   Dest := ( dReturn == true ) ? &result : &p1                 ; to a our ByRef result variable
   Loop,Parse,SizeArray,|
      DllCall("RtlMoveMemory","uInt",Dest,"uInt",&buffer%A_Index%,"uInt",A_LoopField)
      , Dest += A_LoopField
      
   DllCall("WinINet\InternetCloseHandle", "uInt", hRequest)   ; close all opened
   DllCall("WinINet\InternetCloseHandle", "uInt", hInternet)
   DllCall("WinINet\InternetCloseHandle", "uInt", hConnect)
   DllCall("FreeLibrary", "UInt", hModule)                    ; unload the library
   
   if ( dReturn == true ) {
      VarSetCapacity( result, -1 )
      ErrorLevel := fSize
      return Result
   } else 
      return fSize                      ; return the size - strings need update via VarSetCapacity(res,-1)
}

Additional Informations / Needs to be done

[*:4qpwruga] For now the SSL functionality (aka https or secured communication) works but just when using trusted SSL servers. Using servers with self-signed certificates such as WebMin wont work and needs further research.Version History

v0.1 - Initial Version (Tue Jul 08, 2008)
v0.1b - Fixed Typo, thx (Thu Jul 10, 2008)
v0.3.1 - added Features such as InternetCrackURL, SSL, header Return Values, changed function call.
v0.3.2 - added new httpQueryOps Verb to update full and current download Size
v0.3.3 - added new global Vars to handle 302 answers properly by using custom dwFlags (thx skrommel for pointing this out).
v0.3.4 - fixed a minor bug which might return wrong received data constructed from memory data
v0.3.5 - fixed the regEx searching for http proto version - Thx temp01
v0.3.6 - introducing new syntax, which eliminates the varsetcapacity for textual data such as pure html

Remarks: Thanks to Heresy who helped a big deal in collecting all needed winINet calls to make httpPOST happen, and to ahklerner, who collected the neccessary SSL parameters in his version of this script.
Also a big thanks to all who took their time to review this lil function :)

greets
DerRaphael
#1 - Posted 08 July 2008 - 10:58 AM

All scripts, unless otherwise noted, are hereby released under CC-BY
  • Guests
  • Last active:
  • Joined: --
Thanks god we have native http function now
#2 - Posted 08 July 2008 - 11:06 AM
Joy2DWorld
  • Members
  • 562 posts
  • Last active: Jun 30 2014 07:48 PM
  • Joined: 04 Dec 2006
clean!


the IF structure 
if ((InStr(Server,"http://")) or (InStr(Server,"ftp://"))) {
      Protocoll := (RegExMatch(Server,"(^http://|^ftp://)",pr)) ? pr1 : "unknown"
   }
seems redundant...


(although have not tested time to do two INSTR vs. one regexmatch)


and would suggested 
if !instr(url,":")
	url = http://%url%
or such.


cf. <!-- m -->http://www.autohotke...topic10393.html<!-- m -->

and <!-- m -->http://www.autohotke... ... 6&start=30<!-- m --> (which Sean, using COM, shows how to have url request work as 'callback' so you don't have to wait for page//script continues even if download does not!)
#3 - Posted 08 July 2008 - 02:41 PM
Joyce Jamce
engunneer
  • Moderators
  • 9162 posts
  • Last active: Sep 12 2014 10:36 PM
  • Joined: 30 Aug 2005
the regex is needed due to odd urls such as: http://go.fark.com/c....com/ci_9797611
#4 - Posted 08 July 2008 - 04:52 PM
Common Answers - Tutorials Forum - Humongous FAQ - CALC (Common AHK Library Collection) - Expressions Explained (deleyd) - phpBB Search - Think AHK_L is too complicated?
Joy2DWorld
  • Members
  • 562 posts
  • Last active: Jun 30 2014 07:48 PM
  • Joined: 04 Dec 2006
and the "IF" structure is needed for ?
#5 - Posted 08 July 2008 - 05:51 PM
Joyce Jamce
neXt
  • Members
  • 549 posts
  • Last active: May 20 2015 02:38 AM
  • Joined: 18 Mar 2007
so this function can be used in auto login scripts? Can someone post an example if yes please.

P.S. Nice work DerRaphael!
#6 - Posted 08 July 2008 - 11:29 PM
heresy
  • Members
  • 291 posts
  • Last active: Sep 26 2008 10:47 PM
  • Joined: 11 Mar 2008
random xkcd comics getter :)

LButton : Drag GUI
LButton Double Click : change image
RButton : Exitapp

;WinXP+ & needs HttpQuery()
GetPic()
OnMessage(0x203, "GetPic")
OnMessage(0x200, "ShowAlt")
Return

GuiContextMenu:
Exitapp

Move:
PostMessage,0xA1,2,,,A
Return

GetPic(){
    WinGetPos, X, Y,,, RandXKCD
    Gui, Destroy
    HttpQuery(raw, "http://dynamic.xkcd.com/comic/random/")
    VarSetCapacity(raw, -1)
    RegExMatch(raw,"Sm)<img src=""(?P<pic>http://imgs.xkcd.com/comics.+?)"" title=""(?P<title>.+?)"" alt=(?P<alt>"".+?)"" /><br/>", _)
    Passer(_Alt,_Title)
    SplitPath, _pic, PicFile
    picloc := A_Temp "\" PicFile
    If !FileExist(picloc)
        URLDownloadToFile, %_pic%, %picloc%
    Gui, Margin, 0,0
    Gui, +ToolWindow -Caption
    Gui, Add, Picture, gMove h-1, %picloc%
    Gui, Show, NA AutoSize x%x% y%y%, RandXKCD

}

ShowAlt(){
    Global a, b
    If A_Gui=1
        ToolTip, %a% "`n" %b%
    Else
        ToolTip
}

Passer(Byref _Alt, Byref _Title){
    Global a:=_Alt, b:=_Title
}

#7 - Posted 09 July 2008 - 09:28 AM
Easy WinAPI - Dive into Windows API World
Benchmark your AutoHotkey skills at PlayAHK.com
derRaphael
  • Members
  • 872 posts
  • Last active: Mar 19 2013 04:42 PM
  • Joined: 23 Nov 2007
updated example section for a form dumper

derRaphael
#8 - Posted 09 July 2008 - 09:50 AM
n-l-i-d
  • Guests
  • Last active:
  • Joined: --
Nice! :)

But isn't there a typo in your code?
httpQUERY(URL, POSTDATA="", AGENT="AutohotkeyScript", HTTP_PO[color=red]S[/color]T=80)
, "uInt",hInternet,"Str",ServerName, "Int",HTTP_PO[color=red]R[/color]T

#9 - Posted 09 July 2008 - 11:02 AM
heresy
  • Members
  • 291 posts
  • Last active: Sep 26 2008 10:47 PM
  • Joined: 11 Mar 2008
Getting first hit from google search.
looping the regex part will get the whole result

MsgBox % Google("AutoHotkey")

Google(Query){
    Fp=1
    StringReplace, Query, Query, ", `%22, All
    StringReplace, Query, Query,%A_Space%,+, All
    URL := "http://www.google.com/search?q=" . Query
    httpQUERY(raw, URL)
    VarSetCapacity(raw, -1)   
        StringReplace, Raw, Raw,<b>,, All
        StringReplace, Raw, Raw,</b>,, All
    Fp := RegExMatch(Raw,"Sm)<div class=g><a href=""(?P<Link>.+?)""", _, Fp)
    Fp := RegExMatch(Raw,"Sm)class=l>(?P<Title>.+?)</a>", _, Fp)
    Fp := RegExMatch(Raw,"Sm)<div class=std>(?P<description>.+?)<br><span", _, Fp)
    Return _Link "`n" _Title "`n" _Description
}

#10 - Posted 09 July 2008 - 03:21 PM
Easy WinAPI - Dive into Windows API World
Benchmark your AutoHotkey skills at PlayAHK.com
tank
  • Administrators
  • 4345 posts
  • AutoHotkey Foundation
  • Last active: May 02 2019 09:16 PM
  • Joined: 21 Dec 2007
Very interesting DerRaphael
very very interesting
#11 - Posted 09 July 2008 - 08:33 PM
Never lose.
WIN or LEARN.
tank
  • Administrators
  • 4345 posts
  • AutoHotkey Foundation
  • Last active: May 02 2019 09:16 PM
  • Joined: 21 Dec 2007
wow been toying a bit here might end up canning my own library this is impressive
#12 - Posted 09 July 2008 - 08:43 PM
Never lose.
WIN or LEARN.
derRaphael
  • Members
  • 872 posts
  • Last active: Mar 19 2013 04:42 PM
  • Joined: 23 Nov 2007
fixed typo, thx n-l-i-d
#13 - Posted 10 July 2008 - 02:25 AM

All scripts, unless otherwise noted, are hereby released under CC-BY
ahklerner
  • Members
  • 1386 posts
  • Last active: Oct 08 2014 10:29 AM
  • Joined: 26 Jun 2006
I wrapped the WinInet.dll functions to be StdLib compatible. It can be found here:
<!-- m -->http://www.autohotke...pic.php?t=33561<!-- m -->

The WININET lib includes the functionality above as well as the FTP functions that were spread across the forum. This allows a single include that can be utilized for HTTP and FTP communication.
#14 - Posted 10 July 2008 - 04:46 PM
Posted Image
ʞɔпɟ əɥʇ ʇɐɥʍ
bigsalgo911
  • Members
  • 29 posts
  • Last active: Jul 12 2008 04:34 AM
  • Joined: 02 Jul 2008
So this works great...however. I am having a problem getting this to work with any variables in the POSTdata field.

Here is the code I'm using:

URL := "http://ecase.xxx.yyy.net/login.asp"
POSTdata := "logonid=%login%&password=%pass%"
httpQUERY(URL,POSTDATA)

the variables are defined earlier. I have tried many combinations of using and not using quotes and I cannot figure it out. It works perfectly when I enter my login information, but it can't seem to read the variable. What am I doing wrong?
#15 - Posted 12 July 2008 - 01:37 AM
Back to Scripts and Functions
  1. AutoHotkey Community
  2. → AutoHotkey
  3. → Scripts and Functions