Jump to content

Sky Slate Blueberry Blackcurrant Watermelon Strawberry Orange Banana Apple Emerald Chocolate
Photo

UrlDownloadToVar


  • Please log in to reply
100 replies to this topic
olfen
  • Members
  • 115 posts
  • Last active: Dec 25 2012 09:48 AM
  • Joined: 04 Jun 2005
I recommend using the WinHTTP COM alternative from page 2 of this thread. It doesn't suffer from some of the unresolved issues in the WinInet script.

Supplementary to UrlDownloadToFile:
Not yet fully functional!
msgbox % UrlDownloadToVar("http://www.autohotkey.com/download/CurrentVersion.txt")

UrlDownloadToVar(URL, Proxy="", ProxyBypass="") {
AutoTrim, Off
hModule := DllCall("LoadLibrary", "str", "wininet.dll") 

If (Proxy != "")
AccessType=3
Else
AccessType=1
;INTERNET_OPEN_TYPE_PRECONFIG                    0   // use registry configuration
;INTERNET_OPEN_TYPE_DIRECT                       1   // direct to net
;INTERNET_OPEN_TYPE_PROXY                        3   // via named proxy
;INTERNET_OPEN_TYPE_PRECONFIG_WITH_NO_AUTOPROXY  4   // prevent using java/script/INS

io_hInternet := DllCall("wininet\InternetOpenA"
, "str", "" ;lpszAgent
, "uint", AccessType
, "str", Proxy
, "str", ProxyBypass
, "uint", 0) ;dwFlags

iou := DllCall("wininet\InternetOpenUrlA"
, "uint", io_hInternet
, "str", url
, "str", "" ;lpszHeaders
, "uint", 0 ;dwHeadersLength
, "uint", 0x80000000 ;dwFlags: INTERNET_FLAG_RELOAD = 0x80000000 // retrieve the original item
, "uint", 0) ;dwContext

If (ErrorLevel != 0 or iou = 0) {
DllCall("FreeLibrary", "uint", hModule)
return 0
}

VarSetCapacity(buffer, 512, 0)
VarSetCapacity(NumberOfBytesRead, 4, 0)
Loop
{
  irf := DllCall("wininet\InternetReadFile", "uint", iou, "uint", &buffer, "uint", 512, "uint", &NumberOfBytesRead)
  NOBR = 0
  Loop 4  ; Build the integer by adding up its bytes. - ExtractInteger
    NOBR += *(&NumberOfBytesRead + A_Index-1) << 8*(A_Index-1)
  IfEqual, NOBR, 0, break
  ;BytesReadTotal += NOBR
  DllCall("lstrcpy", "str", buffer, "uint", &buffer)
  res = %res%%buffer%
}
StringTrimRight, res, res, 2

DllCall("wininet\InternetCloseHandle",  "uint", iou)
DllCall("wininet\InternetCloseHandle",  "uint", io_hInternet)
DllCall("FreeLibrary", "uint", hModule)
AutoTrim, on
return, res
}

Edit:
Replaced "res := res . buffer" with "res = %res%%buffer%" for better performance, as Chris recommended.
Also removed A_ScriptName as lpszAgent
Edit: Added AutoTrim, off for correct concatenation (thanks PhiLho)

PhiLho
  • Moderators
  • 6850 posts
  • Last active: Jan 02 2012 10:09 PM
  • Joined: 27 Dec 2005
Nice, it has been asked some times!
Posted Image vPhiLho := RegExReplace("Philippe Lhoste", "^(\w{3})\w*\s+\b(\w{3})\w*$", "$1$2")

Chris
  • Administrators
  • 10727 posts
  • Last active:
  • Joined: 02 Mar 2004
I think I may use this for server uptime monitoring. Thanks.

Chris
  • Administrators
  • 10727 posts
  • Last active:
  • Joined: 02 Mar 2004
One minor improvement to this that might help performance when downloading huge webpages:

Replace: res := res . buffer
With: res = %res%%buffer%

The latter performs better due to an optimization.

To get even more performance out of it (once again, for large web pages or files), you could call VarSetCapacity(res, ExpectedSize) beforehand. But this requires that you have some idea of how large the file will be.

olfen
  • Members
  • 115 posts
  • Last active: Dec 25 2012 09:48 AM
  • Joined: 04 Jun 2005

One minor improvement to this that might help performance when downloading huge webpages:

Replace: res := res . buffer
With: res = %res%%buffer%

The latter performs better due to an optimization.

Thanks, I changed the script accordingly.

To get even more performance out of it (once again, for large web pages or files), you could call VarSetCapacity(res, ExpectedSize) beforehand. But this requires that you have some idea of how large the file will be.

I will look into adding a call to fetch HTTP_QUERY_CONTENT_LENGTH via HttpQueryInfo as demonstrated here.
This would also allow to give feedback on download progress. Many thanks for the idea.

Also, do you think increasing the buffer size would improve performance?

PhiLho
  • Moderators
  • 6850 posts
  • Last active: Jan 02 2012 10:09 PM
  • Joined: 27 Dec 2005

Replace: res := res . buffer
With: res = %res%%buffer%

Maybe he should switch off AutoTrim, in this case, no?
Posted Image vPhiLho := RegExReplace("Philippe Lhoste", "^(\w{3})\w*\s+\b(\w{3})\w*$", "$1$2")

Chris
  • Administrators
  • 10727 posts
  • Last active:
  • Joined: 02 Mar 2004
It might help, but it would have to be tested to find out if it's worth it.

do you think increasing the buffer size would improve performance?

That's a good question. I know it tends to do so with FTP, but these API Internet functions are probably pretty high above the network layer, so it might not make much of a difference. However, a smaller buffer size would probably make the script more responsive/granular since it would be likely to spend less time inside the call (especially when the download is slow, such as dial-up).

PhiLho
  • Moderators
  • 6850 posts
  • Last active: Jan 02 2012 10:09 PM
  • Joined: 27 Dec 2005

It might help, but it would have to be tested to find out if it's worth it.

I didn't meant it as speed improvement (althought skipping trimming might be beneficial) but, if I understood correctly the code portion, he fills the buffer with 512 bytes of data from the server, likely to be HTML, so it is no unlikely to have spaces or tabs at the end of buffer.
res := res . buffer
will preserves these spaces while
res = %res%%buffer%
will remove them, thus alter the final result, if AutoTrim is On (default).

BTW, is it usable for binary data (image)? Althought currently we can't do much use of such image in memory.
Posted Image vPhiLho := RegExReplace("Philippe Lhoste", "^(\w{3})\w*\s+\b(\w{3})\w*$", "$1$2")

olfen
  • Members
  • 115 posts
  • Last active: Dec 25 2012 09:48 AM
  • Joined: 04 Jun 2005
Chris, thanks for your considerations regarding the buffer size.
I will do some automated testing to see if it makes a difference.
But first, I will have to fix this buggy thing: Files that are larger than the buffer size are getting a queer appendage...
a := UrlDownloadToVar("http://jm.greatnow.com/ahk_test_20060618140900_517b.txt")
b := UrlDownloadToVar("http://jm.greatnow.com/ahk_spaces_20060618140900_224b.txt")
fileappend, %a%, a.txt
fileappend, %b%, b.txt
@PhiLho
AutoTrim, Off really is neccessary to prevent loss of whitespace. The second URL in the code above demonstates it. Thanks.

I'll check if it can be used to fetch binary data, as soon as the other problems are solved...

jared
  • Members
  • 7 posts
  • Last active: Jul 28 2006 08:14 AM
  • Joined: 02 Jul 2006
Autotrim should not be set to "on" at the end of the function if it wasn't befor.

Is there any way to make the function binary save to downlaod images or zips and save them to file? I'd like to use the proxy login of this function for downlaods, which UrlDownloadToFile can't do.

olfen
  • Members
  • 115 posts
  • Last active: Dec 25 2012 09:48 AM
  • Joined: 04 Jun 2005
Here's an update of the function:
UrlDownloadToVar(URL, ByRef Result, UserAgent = "", Proxy = "", ProxyBypass = "") {
  ; Requires Windows Vista, Windows XP, Windows 2000 Professional, Windows NT Workstation 4.0,
  ; Windows Me, Windows 98, or Windows 95.
  ; Requires Internet Explorer 3.0 or later.
  
  hModule := DllCall("LoadLibrary", "Str", "wininet.dll") 

  AccessType := Proxy != "" ? 3 : 1
  ;INTERNET_OPEN_TYPE_PRECONFIG                    0   // use registry configuration 
  ;INTERNET_OPEN_TYPE_DIRECT                       1   // direct to net 
  ;INTERNET_OPEN_TYPE_PROXY                        3   // via named proxy 
  ;INTERNET_OPEN_TYPE_PRECONFIG_WITH_NO_AUTOPROXY  4   // prevent using java/script/INS 

	io := DllCall("wininet\InternetOpenA" 
	, "Str", UserAgent ;lpszAgent 
	, "UInt", AccessType 
	, "Str", Proxy 
	, "Str", ProxyBypass 
	, "UInt", 0) ;dwFlags 
	
	iou := DllCall("wininet\InternetOpenUrlA" 
	, "UInt", io 
	, "Str", url 
	, "Str", "" ;lpszHeaders 
	, "UInt", 0 ;dwHeadersLength 
	, "UInt", 0x80000000 ;dwFlags: INTERNET_FLAG_RELOAD = 0x80000000 // retrieve the original item 
	, "UInt", 0) ;dwContext 
	
	If (ErrorLevel != 0 or iou = 0) { 
	  DllCall("FreeLibrary", "UInt", hModule) 
	  return 0 
	} 
	
	VarSetCapacity(buffer, 10240, 0)
	VarSetCapacity(BytesRead, 4, 0)
	
	Loop 
	{ 
	  ;http://msdn.microsoft.com/library/en-us/wininet/wininet/internetreadfile.asp
	  irf := DllCall("wininet\InternetReadFile", "UInt", iou, "UInt", &buffer, "UInt", 10240, "UInt", &BytesRead) 
	  VarSetCapacity(buffer, -1) ;to update the variable's internally-stored length
	  
    BytesRead_ = 0 ; reset
	  Loop, 4  ; Build the integer by adding up its bytes. (From ExtractInteger-function)
      BytesRead_ += *(&BytesRead + A_Index-1) << 8*(A_Index-1) ;Bytes read in this very DllCall
    
    ; To ensure all data is retrieved, an application must continue to call the
    ; InternetReadFile function until the function returns TRUE and the lpdwNumberOfBytesRead parameter equals zero.
    If (irf = 1 and BytesRead_ = 0)
	    break
	  Else ; append the buffer's contents
	    Result .= SubStr(buffer, 1, BytesRead_) 
    
    /* optional: retrieve only a part of the file
    BytesReadTotal += BytesRead_
    If (BytesReadTotal >= 30000) ; only read the first x bytes
	    break                      ; (will be a multiple of the buffer size, if the file is not smaller; trim if neccessary)
	  */
	}
	
	DllCall("wininet\InternetCloseHandle",  "UInt", iou) 
	DllCall("wininet\InternetCloseHandle",  "UInt", io) 
	DllCall("FreeLibrary", "UInt", hModule)
}

@jared: I'll try if it can be modified to handle binary data, but I can't promise anything. It won't be soon either.

skwire
  • Moderators
  • 279 posts
  • Last active: Aug 12 2014 05:16 PM
  • Joined: 18 Jan 2006
Would you mind providing an example call using the update? I can't seem to make it work at all. From my tests, it's spinning its wheels within the function's loop. Thanks.

rogerr (as guest)
  • Guests
  • Last active:
  • Joined: --
skwire: worked fine for me with this code:
#SingleInstance, force

UrlDownloadToVar("http://www.autohotkey.com/download/CurrentVersion.txt", urldata)
filedelete, urldata.txt
fileappend, %urldata%,urldata.txt
msgbox, urldata:`n%urldata%
exitapp

@olfen any time you get around to handling binary data, it will be appreciated :D

BoBo
  • Guests
  • Last active:
  • Joined: --
Noobish/BoBoish question: (how) would it be possible to process an already loaded page (the repeat of a previously POSTed form) that way?
(I guess it will be a 'localhost' kinda thing ...:?)

skwire
  • Moderators
  • 279 posts
  • Last active: Aug 12 2014 05:16 PM
  • Joined: 18 Jan 2006
Gah...my fault. I was trying to test it like this:

clipboard := UrlDownloadToVar("http://www.autohotkey.com/download/CurrentVersion.txt", urldata)

But since the function is using a ByRef return variable, that will never work. I should have been using it like this:

UrlDownloadToVar("http://www.autohotkey.com/download/CurrentVersion.txt", urldata)
clipboard := urldata