Jump to content

Sky Slate Blueberry Blackcurrant Watermelon Strawberry Orange Banana Apple Emerald Chocolate
Photo

UrlDownloadToVar


  • Please log in to reply
100 replies to this topic
  • Guests
  • Last active:
  • Joined: --
I know nothing about html, so I can not look at your results to see where the menutable id is.

But I do know the msgbox is cutting off the bottom half of your html text.

Try this.

;http://www.autohotkey.com/community/viewtopic.php?f=2&t=10466&start=75
; ComObjError(false)
FileDelete,test.txt
URL := "http://www.ereaderiq.com/free/"
FileAppend, % UrlDownloadToVar( URL ),test.txt
run notepad.exe test.txt
;MsgBox, % UrlDownloadToVar( URL ) ; ResponseText
return

UrlDownloadToVar(URL) {
 WebRequest := ComObjCreate("WinHttp.WinHttpRequest.5.1")
 WebRequest.Open("GET", URL)
 WebRequest.Send()
 Return WebRequest.ResponseText
}


Richard
  • Members
  • 19 posts
  • Last active: Mar 01 2013 11:20 PM
  • Joined: 21 Jul 2012

Try this.


Thanks for the code, it now retrieves all of the html.

pratpor
  • Members
  • 12 posts
  • Last active: Aug 31 2013 06:16 AM
  • Joined: 04 Mar 2013

I am a newbie. The function you provided didn't work for me. Dont know why. Please help!
This is how I implemented it. 

#SingleInstance, force
;UrlDownloadToVar("http://axsiom.co.in/logfile.html", urldata)
;Msgbox %urldata%
;clipboard := urldata

UrlDownloadToVar(URL, ByRef Result, UserAgent = "", Proxy = "", ProxyBypass = "") {
  ; Requires Windows Vista, Windows XP, Windows 2000 Professional, Windows NT Workstation 4.0,
  ; Windows Me, Windows 98, or Windows 95.
  ; Requires Internet Explorer 3.0 or later.
  
  hModule := DllCall("LoadLibrary", "Str", "wininet.dll") 

  AccessType := Proxy != "" ? 3 : 1
  ;INTERNET_OPEN_TYPE_PRECONFIG                    0   // use registry configuration 
  ;INTERNET_OPEN_TYPE_DIRECT                       1   // direct to net 
  ;INTERNET_OPEN_TYPE_PROXY                        3   // via named proxy 
  ;INTERNET_OPEN_TYPE_PRECONFIG_WITH_NO_AUTOPROXY  4   // prevent using java/script/INS 

	io := DllCall("wininet\InternetOpenA" 
	, "Str", UserAgent ;lpszAgent 
	, "UInt", AccessType 
	, "Str", Proxy 
	, "Str", ProxyBypass 
	, "UInt", 0) ;dwFlags 
	
	iou := DllCall("wininet\InternetOpenUrlA" 
	, "UInt", io 
	, "Str", url 
	, "Str", "" ;lpszHeaders 
	, "UInt", 0 ;dwHeadersLength 
	, "UInt", 0x80000000 ;dwFlags: INTERNET_FLAG_RELOAD = 0x80000000 // retrieve the original item 
	, "UInt", 0) ;dwContext 
	
	If (ErrorLevel != 0 or iou = 0) { 
	  DllCall("FreeLibrary", "UInt", hModule) 
	  return 0 
	} 
	
	VarSetCapacity(buffer, 10240, 0)
	VarSetCapacity(BytesRead, 4, 0)
	
	Loop 
	{ 
	  ;http://msdn.microsoft.com/library/en-us/wininet/wininet/internetreadfile.asp
	  irf := DllCall("wininet\InternetReadFile", "UInt", iou, "UInt", &buffer, "UInt", 10240, "UInt", &BytesRead) 
	  VarSetCapacity(buffer, -1) ;to update the variable's internally-stored length
	  
    BytesRead_ = 0 ; reset
	  Loop, 4  ; Build the integer by adding up its bytes. (From ExtractInteger-function)
      BytesRead_ += *(&BytesRead + A_Index-1) << 8*(A_Index-1) ;Bytes read in this very DllCall
    
    ; To ensure all data is retrieved, an application must continue to call the
    ; InternetReadFile function until the function returns TRUE and the lpdwNumberOfBytesRead parameter equals zero.
    If (irf = 1 and BytesRead_ = 0)
	    break
	  Else ; append the buffer's contents
	    Result .= SubStr(buffer, 1, BytesRead_) 
    
    /* optional: retrieve only a part of the file
    BytesReadTotal += BytesRead_
    If (BytesReadTotal >= 30000) ; only read the first x bytes
	    break                      ; (will be a multiple of the buffer size, if the file is not smaller; trim if neccessary)
	  */
	}
	
	DllCall("wininet\InternetCloseHandle",  "UInt", iou) 
	DllCall("wininet\InternetCloseHandle",  "UInt", io) 
	DllCall("FreeLibrary", "UInt", hModule)
}

UrlDownloadToVar("http://www.autohotkey.com/download/CurrentVersion.txt", urldata)
msgbox, %urldata%
exitapp

 

It returns an empty string.

Am I doing something wrong?



Cragaha
  • Members
  • 265 posts
  • Last active: Jan 04 2016 02:24 AM
  • Joined: 19 Nov 2010

@pratpor

Your function is Ansi.

Here is a simple Unicode one.

 

URL_ToVar(URL) {
    ComObjError(0)
    WebRequest := ComObjCreate("WinHttp.WinHttpRequest.5.1")
    WebRequest.Open("GET", URL)
    WebRequest.Send()
    Return WebRequest.ResponseText()
        , ComObjError(0)
    }
 


gummby8
  • Members
  • 120 posts
  • Last active: Nov 01 2017 05:53 PM
  • Joined: 05 Jul 2009
UrlDownloadToVar(URL) {
 WebRequest := ComObjCreate("WinHttp.WinHttpRequest.5.1")
 WebRequest.Open("GET", URL)
 WebRequest.Send()
 Return WebRequest.ResponseText
}

I have a slight problem with this. If the server isn't available it locks up the whole script. Is there anyway to give it a timeout?



DataLife
  • Members
  • 1022 posts
  • Last active: Nov 27 2015 01:09 AM
  • Joined: 27 Apr 2008
UrlDownloadToVar(URL) {
 WebRequest := ComObjCreate("WinHttp.WinHttpRequest.5.1")
 WebRequest.Open("GET", URL)
 WebRequest.Send()
 Return WebRequest.ResponseText
}

I have a slight problem with this. If the server isn't available it locks up the whole script. Is there anyway to give it a timeout?

 

It times out in less than 3 seconds for me.

 

Do you mean it locks up your script permanently?


Check out my scripts.  (MyIpChanger) (XPSnap) (SavePictureAs) All my scripts are tested on Windows 7, AutoHotkey 32 bit Ansi unless otherwise stated.

gummby8
  • Members
  • 120 posts
  • Last active: Nov 01 2017 05:53 PM
  • Joined: 05 Jul 2009

It times out in less than 3 seconds for me.

 

Do you mean it locks up your script permanently?

 

Maybe not permanently, but definitely longer than 3 seconds.



ageek
  • Members
  • 67 posts
  • Last active: Feb 17 2015 12:35 AM
  • Joined: 08 Feb 2009

Maybe not permanently, but definitely longer than 3 seconds.

 

Old post, I know, but this does it for me:

 

WebRequest.SetTimeOuts(3000,3000,3000,3000)


Preactive
  • Members
  • 30 posts
  • Last active: Dec 29 2015 09:18 PM
  • Joined: 17 Jul 2013

http://www.autohotke...nableredirects/

 

TUZI put up an good version of this!  And it actually works with Timeouts for Perl requests that continually update a page until finished.

 

Here is the stripped down code without logogram comments.

UrlDownloadToVar(URL,Charset="",URLCodePage="",Proxy="",ProxyBypassList="",Cookie="",Referer="",UserAgent="",EnableRedirects="",Timeout=-1)
  {
    ComObjError(0)  
    WebRequest := ComObjCreate("WinHttp.WinHttpRequest.5.1")
    If (URLCodePage<>"")    
        WebRequest.Option(2):=URLCodePage
    If (EnableRedirects<>"")
        WebRequest.Option(6):=EnableRedirects
    If (Proxy<>"")  
        WebRequest.SetProxy(2,Proxy,ProxyBypassList)
    WebRequest.Open("GET", URL, true)
    If (Cookie<>"")
      {
        WebRequest.SetRequestHeader("Cookie","tuzi")   
        WebRequest.SetRequestHeader("Cookie",Cookie)
      }
    If (Referer<>"") 
        WebRequest.SetRequestHeader("Referer",Referer)
    If (UserAgent<>"")  
        WebRequest.SetRequestHeader("User-Agent",UserAgent)
    WebRequest.Send()
    WebRequest.WaitForResponse(Timeout) 
    If (Charset="")
        Return,WebRequest.ResponseText()
    Else
      {
        ADO:=ComObjCreate("adodb.stream")  
        ADO.Type:=1 
        ADO.Mode:=3
        ADO.Open() 
        ADO.Write(WebRequest.ResponseBody())  
        ADO.Position:=0
        ADO.Type:=2 
        ADO.Charset:=Charset    
        Return,ADO.ReadText()   
      }
  }


PurpleRocket
  • Members
  • 1 posts
  • Last active: Dec 24 2015 04:25 PM
  • Joined: 24 Dec 2015
If you use ComObjError(false), you can't know about errors, you can't know even work it or not.
There is my decision:
UrlDownloadToVar(URL) {	WebRequest := ComObjCreate("WinHttp.WinHttpRequest.5.1")	try WebRequest.Open("GET", URL, true)	catch	Error {		ErrorLevel := "Wrong URL"		return false		}	WebRequest.Send()	try WebRequest.WaitForResponse()	catch	Error {		ErrorLevel := "No access to the Internet"		return false		}	HTTPStatusCode := WebRequest.status	if (SubStr(HTTPStatusCode, 1, 1) ~= "4|5") { ; 4xx — Client Error, 5xx — Server Error. wikipedia.org/wiki/List_of_HTTP_status_codes		ErrorLevel := "HTTPStatusCode: " HTTPStatusCode		return false		}	return WebRequest.ResponseText}


stealzy
  • Members
  • 4 posts
  • Last active: Apr 30 2017 12:22 PM
  • Joined: 01 Nov 2015
ComObjError(0) - Not good solution. You will never know about error in working.
URL := "http://ajax.googleapis.com/ajax/libs/jquery/1.11.0/jquery.min.js"
If answer:=UrlDownloadToVar(URL)
	MsgBox,, Success, % answer
else
	MsgBox,, Error, % ErrorLevel
Return

UrlDownloadToVar(URL) {
	; if something wrong return false, put explanation in ErrorLevel
	WebRequest := ComObjCreate("WinHttp.WinHttpRequest.5.1")
	try WebRequest.Open("GET", URL, true)
	catch	Error {
		ErrorLevel := "Wrong URL format"
		return false
		}
	WebRequest.Send()
	try WebRequest.WaitForResponse()
	catch	Error {
		ErrorLevel := "No internet access / No existing domain"
		return false
		}
	HTTPStatusCode := WebRequest.status
	if (SubStr(HTTPStatusCode, 1, 1) ~= "4|5") { ; 4xx — Client Error, 5xx — Server Error. wikipedia.org/wiki/List_of_HTTP_status_codes
		ErrorLevel := "Error HTTP Status Code: " HTTPStatusCode
		return false
		}
	return WebRequest.ResponseText
}