Jump to content

Sky Slate Blueberry Blackcurrant Watermelon Strawberry Orange Banana Apple Emerald Chocolate
Photo

UrlDownloadToVar


  • Please log in to reply
100 replies to this topic
Chris
  • Administrators
  • 10727 posts
  • Last active:
  • Joined: 02 Mar 2004
This should have been done earlier, but I've finally linked to this topic from the UrlDownloadToFile page.

By the way, at the top of this topic, it says "Not yet fully functional!". It might be good to revise that because it gives the impression that even basic operation is lacking.

Thanks for sharing your work.

majkinetor
  • Moderators
  • 4512 posts
  • Last active: Jul 29 2016 12:40 AM
  • Joined: 24 May 2006
It can screw up caller script as it will not return AutoTrim correctly

There should be use of A_AutoTrim in the func.
Posted Image

corrupt
  • Members
  • 2558 posts
  • Last active: Nov 01 2014 03:23 PM
  • Joined: 29 Dec 2004

This should have been done earlier, but I've finally linked to this topic from the UrlDownloadToFile page.

I had hoped that you would build in the functionality instead of linking to it.

Chris
  • Administrators
  • 10727 posts
  • Last active:
  • Joined: 02 Mar 2004
Unfortunately, I haven't had the time to develop this or much of anything else lately. However, it's still planned to have it built-in or at least part of the standard library.

corrupt
  • Members
  • 2558 posts
  • Last active: Nov 01 2014 03:23 PM
  • Joined: 29 Dec 2004

Unfortunately, I haven't had the time to develop this or much of anything else lately.

I didn't mean today :) . Sorry if my response sounded like that. No rush :) . The main reason I was thinking that this functionality might get built-in instead of being added to a Standard Library is that it seems like an option could be used to either output the data to a file or to a var since the programming for retrieving the data already exists. In general, output to a var seems more useful in most scripts and avoids the use of temporary files.

HuBa
  • Members
  • 175 posts
  • Last active: Feb 13 2012 09:51 AM
  • Joined: 24 Feb 2007
I have a serious problem with this script!

I tried to download a file that contained only 2 bytes.

And this function returned with empty string.

Why is this line so necessary?
StringTrimRight, res, res, 2
It deletes 2 bytes from the end of the file!!!

Another thing:
I think it would be useful to allow the lpszAgent to be supplied with an optional parameter.

olfen
  • Members
  • 115 posts
  • Last active: Dec 25 2012 09:48 AM
  • Joined: 04 Jun 2005
I'm too lazy, to delve into this again.

You may want to try the COM alternative instead.

; Requires Windows Vista, Windows XP SP1, or Windows 2000 Professional SP3 and later.
; Requires WinHTTP 5.0 and Internet Explorer 5.01 or later on Windows XP, Windows 2000, and Windows NT 4.0.
; http://msdn2.microsoft.com/en-us/library/aa384106.aspx

#NoEnv
#include CoHelper.ahk ; http://www.autohotkey.com/forum/topic16631.html
CoInitialize()
pWHR := ActiveXObject("WinHttp.WinHttpRequest.5.1") ; CreateObject

SetTimeouts()
SetTimeouts(Resolve=0, Connect=60000, Send=30000, Receive=30000) {
  ; http://msdn2.microsoft.com/en-us/library/aa384061.aspx
  global pWHR
  DllCall(VTable(pWHR, 23), "Uint", pWHR, "Int", Resolve, "Int", Connect, "Int", Send, "Int", Receive)
}

; Use "HEAD" method, if it is not neccesary to retrieve ResponseText
sMethod := "GET", Ansi2Unicode(sMethod, wMethod) 
sUrl := "http://microsoft.com", Ansi2Unicode(sUrl, wUrl)
DllCall(VTable(pWHR, 9), "Uint", pWHR, "Str", wMethod, "Str", wUrl) ; Open

; Set UserAgent
sUserAgent := "A WinHttpRequest Example Program"
;sUserAgent := "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)"
pUserAgent := SysAllocString(sUserAgent)
DllCall(VTable(pWHR, 20), "Uint", pWHR, "UInt", WinHttpRequestOption_UserAgentString := 0 , "int64", 8, "int64", pUserAgent) ; put_Option

DllCall(VTable(pWHR, 13), "Uint", pWHR) ; Send

DllCall(VTable(pWHR, 12), "Uint", pWHR, "UIntP", pAllResponseHeaders) ; GetAllResponseHeaders
Unicode2Ansi(pAllResponseHeaders, sAllResponseHeaders)
SysFreeString(pAllResponseHeaders)
MsgBox % sAllResponseHeaders

sHeader := "content-length", Ansi2Unicode(sHeader, wHeader)
DllCall(VTable(pWHR, 11), "Uint", pWHR, "Str", wHeader, "UIntP", pValue) ; GetResponseHeader
Unicode2Ansi(pValue, sValue)
SysFreeString(pValue)
MsgBox % sValue

VarSetCapacity(StatusCode, 4, 0)
DllCall(VTable(pWHR, 14), "Uint", pWHR, "UInt", &StatusCode) ; Status
MsgBox % NumGet(StatusCode)

DllCall(VTable(pWHR, 15), "Uint", pWHR, "UIntP", pStatusText) ; StatusText
Unicode2Ansi(pStatusText, sStatusText)
SysFreeString(pStatusText)
MsgBox % sStatusText

If (sMethod != "HEAD") {
DllCall(VTable(pWHR, 16), "Uint", pWHR, "UIntP", pResponseText) ; ResponseText
Unicode2Ansi(pResponseText, sResponseText)
SysFreeString(pResponseText)
MsgBox % sResponseText
}

Release(pWHR)
CoUninitialize()


/* Settings for put_Option method: http://msdn2.microsoft.com/EN-US/library/aa383998.aspx

WinHttpRequestOption_UserAgentString := 0
WinHttpRequestOption_URL := 1
WinHttpRequestOption_URLCodePage := 2
WinHttpRequestOption_EscapePercentInURL := 3
WinHttpRequestOption_SslErrorIgnoreFlags := 4
WinHttpRequestOption_SelectCertificate := 5
WinHttpRequestOption_EnableRedirects := 6
WinHttpRequestOption_UrlEscapeDisable := 7
WinHttpRequestOption_UrlEscapeDisableQuery := 8
WinHttpRequestOption_SecureProtocols := 9
WinHttpRequestOption_EnableTracing := 10
WinHttpRequestOption_RevertImpersonationOverSsl := 11
WinHttpRequestOption_EnableHttpsToHttpRedirects := 12
WinHttpRequestOption_EnablePassportAuthentication := 13
WinHttpRequestOption_MaxAutomaticRedirects := 14
WinHttpRequestOption_MaxResponseHeaderSize := 15
WinHttpRequestOption_MaxResponseDrainSize := 16
WinHttpRequestOption_EnableHttp1_1 := 17
WinHttpRequestOption_EnableCertificateRevocationCheck := 18

Edit: Added GetAllResponseHeaders, GetResponseHeader, Get Status, Get StatusText
Edit: Added method to set the UserAgent (and other options)
Edit: Added SetTimeouts method

HuBa
  • Members
  • 175 posts
  • Last active: Feb 13 2012 09:51 AM
  • Joined: 24 Feb 2007
Wonderfull script, thanx!

But it's just a workaround. I still don't know why is it necessary to trim the result value by 2 bytes in the original function.

olfen
  • Members
  • 115 posts
  • Last active: Dec 25 2012 09:48 AM
  • Joined: 04 Jun 2005
Sorry, I currently don't want to spend my time on it.

HuBa
  • Members
  • 175 posts
  • Last active: Feb 13 2012 09:51 AM
  • Joined: 24 Feb 2007

Sorry, I currently don't want to spend my time on it.

Ok

Tekl
  • Members
  • 814 posts
  • Last active: May 03 2009 03:28 PM
  • Joined: 24 Sep 2004
Does this solution also block the whole script like UrlDownloadToFile does? It's a pain when network is unreachable that UrlDownloadToFile will block everything even if it runs in a thread.
Tekl

olfen
  • Members
  • 115 posts
  • Last active: Dec 25 2012 09:48 AM
  • Joined: 04 Jun 2005

Does this solution also block the whole script like UrlDownloadToFile does?

Yes, same problem here.

It's a pain when network is unreachable that UrlDownloadToFile will block everything even if it runs in a thread.

I added SetTimeouts(), so you can at least define how long it blocks the script. See MSDN url for details...

z3lig
  • Members
  • 11 posts
  • Last active: Aug 20 2007 12:41 AM
  • Joined: 12 Aug 2007
Greetings,

I'm new to programming as well as AHK. I'm finding myself growing more and more obsessed with it as I come up with new ideas that AHK can make real. At any rate, if its not too much trouble, would someone more seasoned please post a functional version of this script? It would be helpful for my understanding if you could make it do the following:

1. open box with edit field to input URL. button to submit and trigger download.

Ultimately, I'm looking to parse web pages for specific content (if you can provide me a quick example of how to do this say with <!-- w -->www.google.com<!-- w --> it would be appreciated, otherwise, with time I'm sure I'll figure it out)
and then write an XML file with the content for an RSS reader to interpret and display.


Thanks in advance. Z

engunneer
  • Moderators
  • 9162 posts
  • Last active: Sep 12 2014 10:36 PM
  • Joined: 30 Aug 2005
while I understand why you posted in this thread, you may want to post in the ask for help section in the future, and include a link to this topic to show you did the research.

In any case, I would suggest starting with this thread: <!-- m -->http://www.autohotke...pic.php?t=20397<!-- m -->

and adding a Gui or an InputBox or two.

Sean
  • Members
  • 2462 posts
  • Last active: Feb 07 2012 04:00 AM
  • Joined: 12 Feb 2007

You may want to try the COM alternative instead.

Here is a demonstration of Asynchronous, i.e., Event/Callback, alternative.

NEED the latest COM Standard Library and DebugView to display the response text.

#Persistent
OnExit, CleanUp

COM_Init()
pwhr    := COM_CreateObject("WinHttp.WinHttpRequest.5.1")
psink   := ConnectIWinHttpRequestEvents(pwhr)

COM_Invoke(pwhr, "Open", "GET", "http://www.autohotkey.com/", "True")
COM_Invoke(pwhr, "Send")
Sleep, 3000
COM_Invoke(pwhr, "Open", "GET", "http://www.autohotkey.com/forum/", "True")
COM_Invoke(pwhr, "Send")
Return

CleanUp:
COM_Unadvise(NumGet(psink+8), NumGet(psink+12))
COM_Release(NumGet(psink+8))
COM_Release(pwhr)
COM_Term()
ExitApp

IWinHttpRequestEvents(this, nStatus = "", pType = "")
{
	Critical
	If	A_EventInfo = 0
		NumPut(this,pType+0)
;	Else If	A_EventInfo = 1
;	Else If	A_EventInfo = 2
	Else If	A_EventInfo = 3
		OutputDebug, % "[START]`t" . nStatus . ": " . COM_Ansi4Unicode(pType) . "`n`n"
;	Else If	A_EventInfo = 4
	Else If	A_EventInfo = 5
		OutputDebug, % COM_Invoke(NumGet(this+4), "ResponseText") . "`n[END]`n"	
	Else If	A_EventInfo = 6
		OutputDebug, % "[ERROR]`t" . nStatus . ": " . COM_Ansi4Unicode(pType) . "`n`n"
	Return	0
}

ConnectIWinHttpRequestEvents(pwhr)
{
	Static	IWinHttpRequestEvents
	If Not	VarSetCapacity(IWinHttpRequestEvents)
	{
		VarSetCapacity(IWinHttpRequestEvents,28,0), nParams=3113213
		Loop,	Parse,	nParams
		NumPut(RegisterCallback("IWinHttpRequestEvents","",A_LoopField,A_Index-1),IWinHttpRequestEvents,4*(A_Index-1))
	}
	pconn:=COM_FindConnectionPoint(pwhr,IID_IWinHttpRequestEvents:="{F97F4E15-B787-4212-80D1-D380CBBF982E}")
	psink:=COM_CoTaskMemAlloc(16), NumPut(pwhr,NumPut(&IWinHttpRequestEvents,psink+0))
	NumPut(COM_Advise(pconn,psink),NumPut(pconn,psink+8))
	Return	psink
}