Jump to content

Sky Slate Blueberry Blackcurrant Watermelon Strawberry Orange Banana Apple Emerald Chocolate
Photo

UrlDownloadToVar


  • Please log in to reply
100 replies to this topic
tank
  • Administrators
  • 4345 posts
  • AutoHotkey Foundation
  • Last active: May 02 2019 09:16 PM
  • Joined: 21 Dec 2007

where are these unresolved issues documented? is there a post in this thread that i'm missing?

um seriosuly did you actually read the thread?
<!-- m -->http://www.autohotke... ... 911#125911<!-- m -->

where can i read about these differences?

microsoft.com these are both microsoft API's i just googled WinHTTP or WinInet and got plenty of good materials
its best not to play with fire if you dont know weather to use gas or wood to fuel it
Never lose.
WIN or LEARN.

guest3456
  • Guests
  • Last active:
  • Joined: --
thank you sir

steadycookie
  • Members
  • 24 posts
  • Last active: Oct 13 2011 08:13 PM
  • Joined: 27 Mar 2009
Since the CoHelper.ahk is retired, in fact, I did not find it in forum. Then I revised the codes below and wrap it to a function. I also remove the parts to get header and status, only keeping the part to get webpage content.

Hope useful. Thanks!

;downloaded from http://www.autohotkey.com/forum/viewtopic.php?t=10466&postdays=0&postorder=asc&start=15
;Modified by steadycookie 06/14/2010
;****usage: UrlDownloadToVar(URL, sResponseText) 

; Requires Windows Vista, Windows XP SP1, or Windows 2000 Professional SP3 and later.
; Requires WinHTTP 5.0 and Internet Explorer 5.01 or later on Windows XP, Windows 2000, and Windows NT 4.0.
; http://msdn2.microsoft.com/en-us/library/aa384106.aspx

;#NoEnv
;Need COM library: http://www.autohotkey.com/forum/viewtopic.php?t=22923

UrlDownloadToVar(URL, byRef sResponseText) 
{	
	COM_CoInitialize()
	pWHR := COM_ActiveXObject("WinHttp.WinHttpRequest.5.1") ; CreateObject
	
	SetTimeouts()
	
	; Use "HEAD" method, if it is not neccesary to retrieve ResponseText
	sMethod := "GET", COM_Ansi2Unicode(sMethod, wMethod)
	sUrl := URL, COM_Ansi2Unicode(sUrl, wUrl)
		;msgbox % sUrl
	DllCall(COM_VTable(pWHR, 9), "Uint", pWHR, "Str", wMethod, "Str", wUrl) ; Open
	
	; Set UserAgent
	sUserAgent := "A WinHttpRequest Example Program"
	;sUserAgent := "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)"
	pUserAgent := COM_SysAllocString(sUserAgent)
	DllCall(COM_VTable(pWHR, 20), "Uint", pWHR, "UInt", WinHttpRequestOption_UserAgentString := 0 , "int64", 8, "int64", pUserAgent) ; put_Option
	
	DllCall(COM_VTable(pWHR, 13), "Uint", pWHR) ; Send

	;get URL webpage		
	If (sMethod != "HEAD") {
	DllCall(COM_VTable(pWHR, 16), "Uint", pWHR, "UIntP", pResponseText) ; ResponseText
	COM_Unicode2Ansi(pResponseText, sResponseText)
	COM_SysFreeString(pResponseText)
	;MsgBox % sResponseText
	}
	
	COM_Release(pWHR)
	COM_CoUninitialize()
}

SetTimeouts(Resolve=0, Connect=60000, Send=30000, Receive=60000) {
  ; http://msdn2.microsoft.com/en-us/library/aa384061.aspx
  global pWHR
  DllCall(COM_VTable(pWHR, 23), "Uint", pWHR, "Int", Resolve, "Int", Connect, "Int", Send, "Int", Receive)
}


Murp-e
  • Members
  • 531 posts
  • Last active: Sep 27 2011 11:44 AM
  • Joined: 12 Jan 2007
steadycookie: Thanks for sharing. I'm going to wait and see what yoda has to say about this, but if it's better than the default URLDownloadToVar, I again recommend editing the first post in this thread to link to this script.

tank
  • Administrators
  • 4345 posts
  • AutoHotkey Foundation
  • Last active: May 02 2019 09:16 PM
  • Joined: 21 Dec 2007
I know the answer already but do you?
i guess im Yoda?
looks like an un tested function i wonder how the average user retreives sResponseText
Never lose.
WIN or LEARN.

Murp-e
  • Members
  • 531 posts
  • Last active: Sep 27 2011 11:44 AM
  • Joined: 12 Jan 2007

i wonder how the average user retreives sResponseText

sResponseText seems to be the return value of the function i.e. the source code of the webpage.

This seems to work just fine, should it supersede the original function?

Sean
  • Members
  • 2462 posts
  • Last active: Feb 07 2012 04:00 AM
  • Joined: 12 Feb 2007
The interface used in the top post was abandoned long-time ago. The equivalent to that for COM.ahk is as follows.
sURL := "http://www.autohotkey.com/"

COM_Init()
pwhr := COM_CreateObject("WinHttp.WinHttpRequest.5.1")
COM_Invoke(pwhr, "Open", "GET", sURL)
COM_Invoke(pwhr, "Send")
sResponseText := COM_Invoke(pwhr, "ResponseText")
COM_Release(pwhr)
COM_Term()

MsgBox, % sResponseText


tanknlid
  • Guests
  • Last active:
  • Joined: --
yet there is no return statement

Murp-e
  • Members
  • 531 posts
  • Last active: Sep 27 2011 11:44 AM
  • Joined: 12 Jan 2007
tanknlid: Wtf? sResponseText is a byref parameter so you would use it like this:

#include com.ahk
UrlDownloadToVar("http://www.google.com", googleSource) 
 msgbox % googleSource
Return


tank
  • Administrators
  • 4345 posts
  • AutoHotkey Foundation
  • Last active: May 02 2019 09:16 PM
  • Joined: 21 Dec 2007
well isnt i special i guess the frack up fairy visited my house this morning i didnt even look at the parameters
Never lose.
WIN or LEARN.

keyboardfreak
  • Members
  • 217 posts
  • Last active: Sep 27 2010 07:21 PM
  • Joined: 09 Oct 2004
I tried the script in post

<!-- m -->http://www.autohotke... ... 470#166470<!-- m -->

and it works fine. Only one little useful thing is missing. Is there a way to a give an id to the async request, so if I start several requests at the same time then I know in the callback which request the current response belongs to?


Edit: Seems like the script cannot handle multiple simultaneous url retrievals at the same time, because only the last one is retrieved if I start more than one at the same time. Would it be hard to add the ability to handle multiple requests?

sbc
  • Members
  • 321 posts
  • Last active: Jun 07 2011 10:24 AM
  • Joined: 25 Aug 2009

Some users may be perplexed how to save the binary data then. I used File.ahk Standard Library for that, however, it's a common practice to use ADODB.Stream object in scripting like:

sURL :=	"http://www.autohotkey.com/download/AutoHotkeyInstall.exe"
sFile:=	A_Temp "\AutoHotkeyInstall.exe"
bOverWrite := False

COM_Init()
pwhr :=	COM_CreateObject("WinHttp.WinHttpRequest.5.1")
COM_Invoke(pwhr, "Open", "GET", sURL)
COM_Invoke(pwhr, "Send")

If	psfa :=	COM_Invoke(pwhr, "ResponseBody")
{
;	pData:=	NumGet(psfa+12)
;	nSize:=	NumGet(psfa+16)
	pstm :=	COM_CreateObject("ADODB.Stream")
	COM_Invoke(pstm, "Type", 1)	; adTypeBinary:=1, adTypeText:=2
	COM_Invoke(pstm, "Open")
	COM_Invoke_(pstm, "Write", [color=red]0x2011[/color], psfa) ; VT_ARRAY | VT_UI1
	COM_Invoke(pstm, "SaveToFile", sFile, bOverWrite ? 2:1)
	COM_Invoke(pstm, "Close")
	COM_Release(pstm)
	COM_SafeArrayDestroy(psfa)
}

COM_Release(pwhr)
COM_Term()

What would be the best way to save downloaded data into a file directly on the fly? With this method, the script's memory size corresponds to increase the size of the downloading file. Therefore, if a huge file is specified to download, the script memory size becomes huge.

I've tried your UrlDownloadToFile Progress but it uses URLDownloadToFile API and it does not support asynchronous mode so it freezes the script's GUI windows while downloading.

A quick google search gave me a reference to URLOpenPullStream to use asyncronous mode. But I'm not skilled enough to implement it.

To me this COM method is the most preferable since it can also set a timeout. So I'm wondering if it is possible to download data into a file without increasing(keep minimum increase) the script memory size.

Sean
  • Members
  • 2462 posts
  • Last active: Feb 07 2012 04:00 AM
  • Joined: 12 Feb 2007
You want to download large files with these? These solutions are not suitable to download large files, it will be slow. Anyway, use the asynchronous version of WinHttpRequest posted in this thread, by implementing currently commented out OnResponseDataAvailable (A_EventInfo = 4). This is similar to, but simpler than, URLOpenStream which is less cumbersome (but less controllable) than URLOpenPullStream.

sbc
  • Members
  • 321 posts
  • Last active: Jun 07 2011 10:24 AM
  • Joined: 25 Aug 2009

You want to download large files with these? These solutions are not suitable to download large files, it will be slow. Anyway, use the asynchronous version of WinHttpRequest posted in this thread, by implementing currently commented out OnResponseDataAvailable (A_EventInfo = 4). This is similar to, but simpler than, URLOpenStream which is less cumbersome (but less controllable) than URLOpenPullStream.

Thanks, I tried it but I don't seem to be able to modify it for the Unicode and 64bit build. :?

Edit: I found something relevant: ResponseStream

RaptorX
  • Members
  • 751 posts
  • Last active: Feb 19 2015 02:47 AM
  • Joined: 19 Feb 2010
here is the code for Unicode and Ansi builds of AHK_L

UrlDownloadToVar(URL, ByRef Result, UserAgent = "", Proxy = "", ProxyBypass = "") {
    ; Requires Windows Vista, Windows XP, Windows 2000 Professional, Windows NT Workstation 4.0,
    ; Windows Me, Windows 98, or Windows 95.
    ; Requires Internet Explorer 3.0 or later.
    pFix:=a_isunicode ? "W" : "A"
    hModule := DllCall("LoadLibrary", "Str", "wininet.dll") 

    AccessType := Proxy != "" ? 3 : 1
    ;INTERNET_OPEN_TYPE_PRECONFIG                    0   // use registry configuration 
    ;INTERNET_OPEN_TYPE_DIRECT                       1   // direct to net 
    ;INTERNET_OPEN_TYPE_PROXY                        3   // via named proxy 
    ;INTERNET_OPEN_TYPE_PRECONFIG_WITH_NO_AUTOPROXY  4   // prevent using java/script/INS 

    io := DllCall("wininet\InternetOpen" . pFix
    , "Str", UserAgent ;lpszAgent 
    , "UInt", AccessType 
    , "Str", Proxy 
    , "Str", ProxyBypass 
    , "UInt", 0) ;dwFlags 

    iou := DllCall("wininet\InternetOpenUrl" . pFix
    , "UInt", io 
    , "Str", url 
    , "Str", "" ;lpszHeaders 
    , "UInt", 0 ;dwHeadersLength 
    , "UInt", 0x80000000 ;dwFlags: INTERNET_FLAG_RELOAD = 0x80000000 // retrieve the original item 
    , "UInt", 0) ;dwContext 

    If (ErrorLevel != 0 or iou = 0) { 
        DllCall("FreeLibrary", "UInt", hModule) 
        return 0 
    } 

    VarSetCapacity(buffer, 10240, 0)
    VarSetCapacity(BytesRead, 4, 0)

    Loop 
    { 
        ;http://msdn.microsoft.com/library/en-us/wininet/wininet/internetreadfile.asp
        irf := DllCall("wininet\InternetReadFile", "UInt", iou, "UInt", &buffer, "UInt", 10240, "UInt", &BytesRead) 
        VarSetCapacity(buffer, -1) ;to update the variable's internally-stored length

        BytesRead_ = 0 ; reset
        Loop, 4  ; Build the integer by adding up its bytes. (From ExtractInteger-function)
            BytesRead_ += *(&BytesRead + A_Index-1) << 8*(A_Index-1) ;Bytes read in this very DllCall

        ; To ensure all data is retrieved, an application must continue to call the
        ; InternetReadFile function until the function returns TRUE and the lpdwNumberOfBytesRead parameter equals zero.
        If (irf = 1 and BytesRead_ = 0)
            break
        Else ; append the buffer's contents
        {
            a_isunicode ? buffer:=StrGet(&buffer, "CP0")
            Result .= SubStr(buffer, 1, BytesRead_ * (a_isunicode ? 2 : 1))
        }

        /* optional: retrieve only a part of the file
        BytesReadTotal += BytesRead_
        If (BytesReadTotal >= 30000) ; only read the first x bytes
        break                      ; (will be a multiple of the buffer size, if the file is not smaller; trim if neccessary)
        */
    }

    DllCall("wininet\InternetCloseHandle",  "UInt", iou) 
    DllCall("wininet\InternetCloseHandle",  "UInt", io) 
    DllCall("FreeLibrary", "UInt", hModule)
}

Tested on unicode but not on ansi.
AutoHotkey Toolkit [Main Project]
Scintilla Wrapper
LexAHKL