Jump to content

Sky Slate Blueberry Blackcurrant Watermelon Strawberry Orange Banana Apple Emerald Chocolate
Photo

UrlDownloadToVar


  • Please log in to reply
100 replies to this topic
  • Guests
  • Last active:
  • Joined: --
Could i get the url behind a link with this? if not, is there such a script in the forums?

ie link diplayed as LINKTEXT (target="actual URL") ...then again that might need to follow javascript code and aspx playlist too..

im testing alot of ff extensions and greasemonkey scripts to find this but i need ahk anyway for the purpose of streaming with external app so if there is one in ahk that would be awesome

bmcclure
  • Members
  • 774 posts
  • Last active: Jan 04 2014 10:44 PM
  • Joined: 24 Nov 2007

Here is a demonstration of Asynchronous, i.e., Event/Callback, alternative.


Hey Sean, this looks nice, but I'm a little fuzzy on one thing: Where in here are the downloaded pages being stored to a variable? I'm trying to adapt this for my needs, and I essentially need to download a webpage into a variable so that I can run commands on it. It seems suited for this task, but I'm just having trouble grasping the COM calls I guess.

Sean
  • Members
  • 2462 posts
  • Last active: Feb 07 2012 04:00 AM
  • Joined: 12 Feb 2007

Where in here are the downloaded pages being stored to a variable?

It didn't save the page to the variable, just output the content into DebugViewer. It was a demonstration. I modified it a little to save the content to a (global) variable _ResponseText_. One thing to note: as it's asynchronous, have to set up some mechanism to notify the new content of _ResponseText_, and I used here SetTimer and MsgBox for that.

#Persistent
sURL :=	"http://www.autohotkey.com/"

OnExit, CleanUp
COM_Init()
pwhr :=	COM_CreateObject("WinHttp.WinHttpRequest.5.1")
psink:=	ConnectIWinHttpRequestEvents(pwhr)
COM_Invoke(pwhr, "Open", "GET", sURL, "True")
COM_Invoke(pwhr, "Send")
Return
CleanUp:
COM_Unadvise(NumGet(psink+8), NumGet(psink+12))
COM_Release(NumGet(psink+8))
COM_Release(pwhr)
COM_Term()
ExitApp
ResponseText:
MsgBox, % _ResponseText_
Return

IWinHttpRequestEvents(this, nStatus = "", pType = "")
{
	Critical
	If	A_EventInfo = 0
		NumPut(this,pType+0)
;	Else If	A_EventInfo = 1
;	Else If	A_EventInfo = 2
	Else If	A_EventInfo = 3
		OutputDebug, % "[START]`t" . nStatus . ": " . COM_Ansi4Unicode(pType) . "`n`n"
;	Else If	A_EventInfo = 4
	Else If	A_EventInfo = 5
	{
		Global	_ResponseText_ := COM_Invoke(NumGet(this+4), "ResponseText")
		SetTimer, ResponseText, -1
	}
	Else If	A_EventInfo = 6
		OutputDebug, % "[ERROR]`t" . nStatus . ": " . COM_Ansi4Unicode(pType) . "`n`n"
	Return	0
}

ConnectIWinHttpRequestEvents(pwhr)
{
	Static	IWinHttpRequestEvents
	If Not	VarSetCapacity(IWinHttpRequestEvents)
	{
		VarSetCapacity(IWinHttpRequestEvents,28,0), nParams=3113213
		Loop,	Parse,	nParams
		NumPut(RegisterCallback("IWinHttpRequestEvents","",A_LoopField,A_Index-1),IWinHttpRequestEvents,4*(A_Index-1))
	}
	pconn:=COM_FindConnectionPoint(pwhr,IID_IWinHttpRequestEvents:="{F97F4E15-B787-4212-80D1-D380CBBF982E}")
	psink:=COM_CoTaskMemAlloc(16), NumPut(pwhr,NumPut(&IWinHttpRequestEvents,psink+0))
	NumPut(COM_Advise(pconn,psink),NumPut(pconn,psink+8))
	Return	psink
}


Lexikos
  • Administrators
  • 9844 posts
  • AutoHotkey Foundation
  • Last active:
  • Joined: 17 Oct 2006

Could i get the url behind a link with this?

I recently wrote a demonstration of how to do the opposite - get the text of a link, given a partial URL. Below is a slightly updated version (the original used embedded C# rather than WinHttpRequest):
#NoEnv
COM_Init()
; Init now since it can be reused for multiple requests.
req := COM_CreateObject("WinHttp.WinHttpRequest.5.1")

LoginPage = http://www.autohotkey.com/forum/login.php
Redirect = privmsg.php?folder=inbox
Username = 
Password =

Gui, Add, Edit, vLoginPage W300 R1, %LoginPage%
Gui, Add, Text,, Username:
Gui, Add, Edit, vUsername XM+100 YP-2 W200, %Username%
Gui, Add, Text, XM, Password:
Gui, Add, Edit, vPassword XM+100 YP-2 W200 Password, %Password%
Gui, Add, Button, gCheckPrivmsg Default, Check Private Messages

Gui, Show,, privmsg test
return


CheckPrivmsg:

Gui, Submit, NoHide
postData := "username=" uriEncode(UserName)
         . "&password=" uriEncode(Password)
         . "&autologin=0&redirect=" uriEncode(Redirect)
         . "&login=Log+in"

; POST form data to the login page.
COM_Invoke(req, "Open", "POST", LoginPage)
COM_Invoke(req, "SetRequestHeader", "Content-Type", "application/x-www-form-urlencoded")
COM_Invoke(req, "Send", postData)
; Retrieve HTML of the response page (%Redirect%).
html := COM_Invoke(req, "ResponseText")

; Create a HTMLDocument to parse the HTML.
doc := COM_CreateObject("{25336920-03F9-11CF-8FD0-00AA00686F13}")
COM_Invoke(doc, "write", html)
COM_Invoke(doc, "close")

; "Retrieves a collection of all 'a' objects that specify the HREF property and
;  all 'area' objects in the document."
doc_links := COM_Invoke(doc, "links")

Loop, % COM_Invoke(doc_links, "length")
{
    link := COM_Invoke(doc_links, "item", A_Index-1)

    if InStr(COM_Invoke(link, "href"), "privmsg.php?folder=inbox")
    {
        text := COM_Invoke(link, "innerText")
        COM_Release(link)
        break
    }

    COM_Release(link)
}

COM_Release(doc_links)
COM_Release(doc)

; <DEBUG>
; FileDelete, check privmsg debug.html
; FileAppend, %html%, check privmsg debug.html
; Run, check privmsg debug.html
; </DEBUG>

MsgBox %text%

return


GuiClose:
ExitApp


; Authors: Titan, Ageless
;   http://www.autohotkey.com/forum/topic18876.html
uriEncode(str) {
   f = %A_FormatInteger%
   SetFormat, Integer, Hex
   If RegExMatch(str, "^\w+:/{0,2}", pr)
      StringTrimLeft, str, str, StrLen(pr)
   StringReplace, str, str, `%, `%25, All
   Loop
      If RegExMatch(str, "i)[^\w\.~%/:]", char)
         StringReplace, str, str, %char%, % "%" . SubStr(Asc(char),3), All
      Else Break
   SetFormat, Integer, %f%
   Return, pr . str
}
Given the username and password of a registered autohotkey.com/forum user, it (simultaneously) logs into the forum and retrieves your PM Inbox (page). It then parses the HTML (via COM + MSHTML) to find the text of the PM link - i.e. "You have N new messages."

If you haven't already, you'll have to register before you can test the script. ;) Alternatively, you could try it on some other phpBB forum.

I used synchronous code (for brevity ;)), so be patient if the GUI appears to stop responding.

  • Guests
  • Last active:
  • Joined: --
to Lexikos: is there a way to use the WinHttpRequest method like curl and make it click on a button ?

Eg:
curl -Ld "BUTTON_INPUT=Restart%20Cable%20Modem" http://192.168.100.1/configdata.html


Lexikos
  • Administrators
  • 9844 posts
  • AutoHotkey Foundation
  • Last active:
  • Joined: 17 Oct 2006
You can't "click on a button" without a web browser and button to click on. You can submit a form, exactly how my previous post demonstrates... It would be slightly different if the form's method is "GET", in which case I think the form data would become part of the URL.

RobOtter
  • Members
  • 133 posts
  • Last active: Jul 24 2014 11:29 AM
  • Joined: 30 Jan 2005
I´m not quite happy with this ugly kind of workaround to get URL contents into a variable...
Although I´m a programmer myself, I´m not familiar with C++ but I suspect it couldn´t be too hard to implement a function "UrlDownloadToVar" in a very similar way to "UrlDownloadToFile". Imho copying the URL content into a variable instead of a file should be a very easy task for Chris (or some other C++ guru who can maintain the source code) - et voila, we´d have a very quick and tidy solution.

Another workaround (which I haven´t tested yet) would be to use a IE control which loads the URL and then get the contents from that control. I guess this would avoid messing around with certain DLLs and so, keeping cross-platform compatibility (x-pf in the sense of different Windows OS, of course :) )
Has anyone done that yet and can confirm its feasibility?

Regards,
Rob

Yure
  • Guests
  • Last active:
  • Joined: --
I guess this line should not be here, I can't see why should we destroy the last two characters. Once I had removed this line the function worked just great for me.

StringTrimRight, res, res, 2


tank
  • Administrators
  • 4345 posts
  • AutoHotkey Foundation
  • Last active: Oct 13 2016 01:04 AM
  • Joined: 21 Dec 2007

doc := COM_CreateObject("{25336920-03F9-11CF-8FD0-00AA00686F13}")

bloody brilliant i never thought of it
Never lose.
WIN or LEARN.

LiquidGravity
  • Members
  • 156 posts
  • Last active: Oct 11 2014 04:11 PM
  • Joined: 26 Jan 2009
This thread is all very confusing and old but is linked to in the help file ( for v1.0.47.6). Could someone help clear up how to download a web page to a variable. Wrapping it up in a simple function like UrlDownloadToVar(url) would be best.

tank
  • Administrators
  • 4345 posts
  • AutoHotkey Foundation
  • Last active: Oct 13 2016 01:04 AM
  • Joined: 21 Dec 2007
whats unclear about the function on the first page
UrlDownloadToVar(URL, Proxy="", ProxyBypass="")

Never lose.
WIN or LEARN.

LiquidGravity
  • Members
  • 156 posts
  • Last active: Oct 11 2014 04:11 PM
  • Joined: 26 Jan 2009
This part is unclear about the function on the first page:

I recommend using the WinHTTP COM alternative from page 2 of this thread. It doesn't suffer from some of the unresolved issues in the WinInet script.

and this:

Not yet fully functional!


I tried the example in the first post and it seems to work but I'm trying avoid unforeseen consequences.

The alternatives posted may work as well or better but I'd rather not have a bunch of includes and more calls that I don't know.

tank
  • Administrators
  • 4345 posts
  • AutoHotkey Foundation
  • Last active: Oct 13 2016 01:04 AM
  • Joined: 21 Dec 2007

Provides balances of moneys owed by Merchants

then your choices are as follows
make up something new and snazzy to replace it impress us with your skill :D
place the includes in the standard library folder and dont use includes
prepend your script and the example to the file you need to include

abandon the rediculous concept of downloading to var and use http request by derRaphael
Never lose.
WIN or LEARN.

LiquidGravity
  • Members
  • 156 posts
  • Last active: Oct 11 2014 04:11 PM
  • Joined: 26 Jan 2009
I will check into http request. Maybe I just didn't know the right thing to search for. Thank you.

Joy2DWorld
  • Members
  • 562 posts
  • Last active: Jun 30 2014 07:48 PM
  • Joined: 04 Dec 2006
[Moderator's note: The rest of this discussion has been moved to Ask for Help: Download an executable to memory to run?.]

You can do it in other languages. Why not ahk?


Ask the author of ahk. Furthermore, UrlDownLoadToVar will not correctly handle binary data (i.e. executables). Accept it and use UrlDownloadToFile as I said.



easily fixed, btw.,

replace the building of result string .= xxx

with binary join.

BJoin(byref A, Asize, byref B, Bsize = "") {

	;if ! Asize
	;	Asize := varsetcapacity(A)
	if ! Bsize
		Bsize := varsetcapacity(B)
	;msgbox % a "-" asize "  " bsize
		
  if !(Asize + BSize)
	return 0
	
	if !BSize
		return ASize
		
	
 ;Puthere := VarResetCapacity(A,Asize,Asize + Bsize ,255) + Asize
 ;if second != try 
 {
	varsetcapacity(AA,Asize)
	DllCall("RtlMoveMemory","uInt",&AA,"uInt",&A,"uInt",Asize)
	
	;A=
	varsetcapacity(A,ASize + BSize,255)
	
	DllCall("RtlMoveMemory","uInt",&A,"uInt",&AA,"uInt",Asize)
	
	;AA =
	puthere := &A + Asize
}	
	DllCall("RtlMoveMemory","uInt", Puthere ,"uInt",&B,"uInt",BSize)
;msgbox % C "is c"	
	
	return Asize + Bsize
	
}


and make it a byref in function call.


will work fine then with bins.
Joyce Jamce