Jump to content

Sky Slate Blueberry Blackcurrant Watermelon Strawberry Orange Banana Apple Emerald Chocolate
Photo

Function : InternetFileRead() aka UrlDownLoadToVar()


  • Please log in to reply
161 replies to this topic
SKAN
  • Administrators
  • 9115 posts
  • Last active:
  • Joined: 26 Dec 2005

InternetFileRead()

As the name suggests, this function reads a file from internet (passed to it as URL).
It is similar to AHK FileRead command in following ways:
1) Reads the content into Memory Variable
2) Can read the specified number of bytes from the leading part of the file.
3) Data retrieved will be in binary ( but to process it, we are on our own )
Parameters :

[*:2dhzuykb]V : Variable name passed as ByRef
[*:2dhzuykb]URL : A valid existing URL from which File will be read from
[*:2dhzuykb]RB : Requested Bytes. Pass 0 to read the whole file or request the number of bytes to read.
[*:2dhzuykb]bSz : Buffer Size. Default value is 1024 ( bytes ). File will be read in small chunks as specified with this parameter. You can increase it for faster downloads but should result in less responsive GUI.
[*:2dhzuykb]DLP : The dynamic function name of the function that will handle the "Download Progress". I have included one .. You can add more in the same script and call them as per your criteria. A simple one would look like:
DLPCE( WP=0, LP=0, MSG="" ) {
 Tooltip, Downloading : %WP% / %LP%
}
[*:2dhzuykb]F : Flags. Default is INTERNET_FLAG_RELOAD|INTERNET_FLAG_DONT_CACHE which makes sure file is not read from/written into cache.


Return Value & ErrorLevel:

If there are no errors from API functions, the function will return the Bytes Read, which will be exactly equal to the Variable's capacity.
If Bytes Read is lesser than the File/Content Length or If Bytes Read is equal to Requested Bytes, AHK ErrorLevel will be set to 1
On an API error, the function will return a negative number which can interpreted as follows:

-1 Internet Connection Problem
-2 Internet Connection Problem
-4 Length of file could not be ascertained
-5 The above and one of the Internet handles could not closed
-6 The above one more of the Internet handles could not closed
-7 One of the Internet handles could not closed
-8 Both of the the Internet handles could not closed
The included dynamic function DLP() will display a progress-bar
in the right-bottom for the screen:

Posted Image
[/list]

/*
[color=#808080]       ___       _                       _   _____ _ _      ____                _ [/color]
[color=#808080]      |_ _|_ __ | |_ ___ _ __ _ __   ___| |_|  ___(_) | ___|  _ \ ___  __ _  __| |[/color]
[color=#111111]       | || '_ \| __/ _ \ '__| '_ \ / _ \ __| |_  | | |/ _ \ |_) / _ \/ _` |/ _` |[/color]
[color=#111111]       | || | | | ||  __/ |  | | | |  __/ |_|  _| | | |  __/  _ <  __/ (_| | (_| |[/color]
[color=#808080]      |___|_| |_|\__\___|_|  |_| |_|\___|\__|_|   |_|_|\___|_| \_\___|\__,_|\__,_|[/color]

                              by [color=#9F009B]SKAN[/color] ( [color=#9F009B]Suresh Kumar A N[/color], arian.suresh@gmail.com )
                                      [color=black]Created: 24-Jun-2009 | LastEdit: 15-Dec-2009[/color]
                       Forum Topic: www.autohotkey.com/forum/viewtopic.php?t=45718
           Included : [color=#F73D00]InternetFileRead()[/color], [color=#F73D00]DLP()[/color] Progress Bar, [color=#F73D00]VarZ_Save()[/color],Examples
      ____________________________________________________________________________
      Credit:

      [color=#9F009B]Olfen[/color] for his topics:

                          [color=#111111]DllCall: HttpQueryInfo - Get HTTP headers[/color]
                          - www.autohotkey.com/forum/topic10510.html
                          [color=#111111]UrlDownloadToVar()[/color]
                          - www.autohotkey.com/forum/topic10466.html
      [color=#9F009B]Lexikos[/color] for:
                          [color=#111111]For supporting this project with valuable info,
                          especially:[/color]

                          [color=#111111]code/method to support FTP read[/color]
                          - www.autohotkey.com/forum/viewtopic.php?p=277646#277646

                          [color=#111111]clarifying alternative parameter for proxy issues[/color]
                          - www.autohotkey.com/forum/viewtopic.php?p=279205#279205
                          - www.autohotkey.com/forum/viewtopic.php?p=279210#279210

      [color=#9F009B]jballi[/color] for fixing an important bug
                          - www.autohotkey.com/forum/viewtopic.php?p=496321#496321

      [color=#111111]Thanks to all the replies in Forum Topic which motivates me to perfect this.[/color]
*/

#SingleInstance Force

; Example 1: Download the leading 100 bytes of default HTML and extract a part of text.
URL := "[color=#2255FF]http://www.formyip.com/[/color]"
If ( [color=#F73D00]InternetFileRead([/color] IP, URL, 100, 100, "No-Progress" [color=#F73D00])[/color] > 0 )
  MsgBox, 64, Your External IP Address
        , % IP := SubStr( IP,SP:=InStr(IP,"My ip address ")+17,InStr(IP," ",0,SP+1)-SP )


; Example 2, Download a binary file ( AHK Script Decompiler ) and save it.
URL := "[color=#2255FF]http://www.autohotkey.com/download/Exe2Ahk.exe[/color]"
If ( [color=#F73D00]InternetFileRead([/color] binData, URL, False, 10240, "No-Progress" [color=#F73D00])[/color] > 0 && !ErrorLevel )
  If [color=#F73D00]VarZ_Save([/color] binData, A_ScriptDir "\Exe2Ahk.exe" [color=#F73D00])[/color]
     MsgBox, 64, AHK Script Compiler Downloaded and Saved, % A_ScriptDir "\Exe2Ahk.exe"


; Example 3, Download a FTP file: EditPlus 3.11 Evaluation Version (1 MB) and save it.
URL := "[color=#2255FF]ftp://ftp.editplus.com/epp311_en.exe[/color]"
If ( [color=#F73D00]InternetFileRead([/color] binData, URL [color=#F73D00])[/color] > 0 && !ErrorLevel )
    If [color=#F73D00]VarZ_Save([/color] binData, A_Temp "\epp311_en.exe" [color=#F73D00])[/color] {
         Sleep 500
         DLP( False ) ; or use Progress, off
         Run %A_Temp%\epp311_en.exe
       }

; AHK will automatically unload libraries on exit. If you are particular, here is a method
; to unload Wininet library without a handle.
DllCall( "[color=#B92300]FreeLibrary[/color]", UInt,DllCall( "[color=#B92300]GetModuleHandle[/color]", Str,"wininet.dll") )
Return ;                                                 [color=red]// end of auto-execute section //[/color]


[color=#F73D00]InternetFileRead([/color] ByRef V, URL="", RB=0, bSz=1024, DLP="DLP", F=0x84000000 [color=#F73D00])[/color] {
 Static LIB="WININET\", CL="00000000000000", N=""
 QRL := 16
 If ! DllCall( "[color=#B92300]GetModuleHandle[/color]", Str,"wininet.dll" )
      DllCall( "[color=#B92300]LoadLibrary[/color]", Str,"wininet.dll" )
 If ! hIO:=DllCall( LIB "[color=#B92300]InternetOpenA[/color]", Str,N, UInt,4, Str,N, Str,N, UInt,0 )
   Return -1
 If ! (( hIU:=DllCall( LIB "[color=#B92300]InternetOpenUrlA[/color]", UInt,hIO, Str,URL, Str,N, Int,0, UInt,F
                                                            , UInt,0 ) ) || ErrorLevel )
   Return 0 - ( !DllCall( LIB "[color=#B92300]InternetCloseHandle[/color]", UInt,hIO ) ) - 2
 If ! ( RB  )
 If ( SubStr(URL,1,4) = "ftp:" )
    CL := DllCall( LIB "[color=#B92300]FtpGetFileSize[/color]", UInt,hIU, UIntP,0 )
 Else If ! DllCall( LIB "[color=#B92300]HttpQueryInfoA[/color]", UInt,hIU, Int,5, Str,CL, UIntP,QRL, UInt,0 )
   Return 0 - ( !DllCall( LIB "[color=#B92300]InternetCloseHandle[/color]", UInt,hIU ) )
            - ( !DllCall( LIB "[color=#B92300]InternetCloseHandle[/color]", UInt,hIO ) ) - 4
 VarSetCapacity( V,64 ), VarSetCapacity( V,0 )
 SplitPath, URL, FN,,,, DN
 FN:=(FN ? FN : DN), CL:=(RB ? RB : CL), VarSetCapacity( V,CL,32 ), P:=&V,
 B:=(bSz>CL ? CL : bSz), TtlB:=0, LP := RB ? "Unknown" : CL,  %DLP%( True,CL,FN )
 Loop {
       If ( DllCall( LIB "[color=#B92300]InternetReadFile[/color]", UInt,hIU, UInt,P, UInt,B, UIntP,R ) && !R )
       Break
       P:=(P+R), TtlB:=(TtlB+R), RemB:=(CL-TtlB), B:=(RemB<B ? RemB : B), %DLP%( TtlB,LP )
       Sleep -1
 } TtlB<>CL ? VarSetCapacity( T,TtlB ) DllCall( "[color=#B92300]RtlMoveMemory[/color]", Str,T, Str,V, UInt,TtlB )
  . VarSetCapacity( V,0 ) . VarSetCapacity( V,TtlB,32 ) . DllCall( "[color=#B92300]RtlMoveMemory[/color]", Str,V
  , Str,T, UInt,TtlB ) . %DLP%( TtlB, TtlB ) : N
 If ( !DllCall( LIB "[color=#B92300]InternetCloseHandle[/color]", UInt,hIU ) )
  + ( !DllCall( LIB "[color=#B92300]InternetCloseHandle[/color]", UInt,hIO ) )
   Return -6
Return, VarSetCapacity(V)+((ErrorLevel:=(RB>0 && TtlB<RB)||(RB=0 && TtlB=CL) ? 0 : 1)<<64)
}
;  - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
; The following function is an add-on to provide "Download Progress" to InternetFileRead()
; InternetFileRead() calls DLP() dynamically, i.e., will not error-out if DLP() is missing
;  - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

[color=#F73D00]DLP([/color] WP=0, LP=0, Msg="" [color=#F73D00])[/color] {
 If ( WP=1 ) {
 SysGet, m, MonitorWorkArea, 1
 y:=(mBottom-46-2), x:=(mRight-370-2), VarSetCapacity( Size,16,0 )
 DllCall( "[color=#B92300]shlwapi.dll\StrFormatByteSize64A[/color]", Int64,LP, Str,Size, UInt,16 )
 Size := ( Size="0 bytes" ) ? N : "«" Size "»"
 Progress, CWE6E3E4 CT000020 CBF73D00 x%x% y%y% w370 h46 B1 FS8 WM700 WS400 FM8 ZH8 ZY3
         ,, %Msg%  %Size%, InternetFileRead(), Tahoma
 WinSet, Transparent, 210, InternetFileRead()
 } Progress,% (P:=Round(WP/LP*100)),% "Memory Download: " wp " / " lp " [ " P "`% ]"
 IfEqual,wP,0, Progress, Off
}
;  - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
; The following function is a part of: [color=#F73D00]VarZ 46L - Native Data Compression[/color]
; View topic : www.autohotkey.com/forum/viewtopic.php?t=45559
;  - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

[color=#F73D00]VarZ_Save([/color] byRef V, File="" [color=#F73D00])[/color] { ;   www.autohotkey.net/~Skan/wrapper/FileIO16/FileIO16.ahk
Return ( ( hFile :=  DllCall( "[color=#B92300]_lcreat[/color]", Str,File, UInt,0 ) ) > 0 )
                 ?   DllCall( "[color=#B92300]_lwrite[/color]", UInt,hFile, Str,V, UInt,VarSetCapacity(V) )
                 + ( DllCall( "[color=#B92300]_lclose[/color]", UInt,hFile ) << 64 ) : 0
}


HotKeyIt
  • Moderators
  • 7439 posts
  • Last active: Jun 22 2016 09:14 PM
  • Joined: 18 Jun 2008
Thank you, looks cool 8) and a nice progress bar :)
Can you give more info for parameters?

SKAN
  • Administrators
  • 9115 posts
  • Last active:
  • Joined: 26 Dec 2005

Thank you, looks cool 8) and a nice progress bar :)


Thanks for trying my code. :)

Can you give more info for parameters?


Sure.. I have updated the title post.

Lexikos
  • Administrators
  • 9844 posts
  • AutoHotkey Foundation
  • Last active:
  • Joined: 17 Oct 2006
Nice work. I especially like the progress window.

I was a little surprised that this part works:
VarSetCapacity(CL,[color=red]14,32[/color])
 If ! ( RB  )
 If ! DllCall( LIB "HttpQueryInfoA", UInt,hIU, Int,5, [color=red]UInt,&CL[/color], UIntP,QRL, UInt,0 )
   Return 0 - ( !DllCall( LIB "InternetCloseHandle", UInt,hIU ) )
            - ( !DllCall( LIB "InternetCloseHandle", UInt,hIO ) ) - 4
StrLen(CL) shows 14 regardless of the apparent length of the string. The code still works because AutoHotkey doesn't fully support binary zero/null-terminator in strings - HttpQueryInfoA null-terminates the string and although the internal length field of CL remains incorrect, most uses of it in an expression would stop at the null-terminator.

Anyway, Str,CL would properly update the variable's length.

It's simple to support FTP in addition to HTTP; for instance, something like this should work:
If SubStr(URL,1,4) = "ftp:"
    CL := DllCall( LIB "FtpGetFileSize", UInt,hIU, UIntP,0 )
 Else [color=#404040]If ! DllCall( LIB "HttpQueryInfoA", UInt,hIU, Int,5, [/color]Str,CL[color=#404040], UIntP,QRL, UInt,0 )
   Return 0 - ( !DllCall( LIB "InternetCloseHandle", UInt,hIU ) )
            - ( !DllCall( LIB "InternetCloseHandle", UInt,hIO ) ) - 4
[/color]
URLDownloadToFile uses InternetReadFileEx, which apparently doesn't support FTP.

I tested using a URL like ftp://user:password@host/path.

SKAN
  • Administrators
  • 9115 posts
  • Last active:
  • Joined: 26 Dec 2005

Nice work. I especially like the progress window.


I am glad you liked it. :D

I was a little surprised that this part works:
...
StrLen(CL) shows 14 regardless of the apparent length of the string.


That was the intended effect... VarSetCapacity() was inserted at the last minute of posting!
Original code used Static CL="00000000000000" and first example reset CL StrLen to 3 bytes ( Requested Bytes 100 ), and so, for the seconds call ( autohotkeyinstall.exe ), the size was 204!

Anyway, Str,CL would properly update the variable's length.


Yes.. Thank you! That was the mistake. I have reverted back to Static CL="00000000000000" and changed the Type,Arg accordingly.

It's simple to support FTP in addition to HTTP


Oh! Thank you very much Lexikos. I had tested the function for FTP with Requested Bytes and it worked... I had the wrong idea that it would require a FtpOpenFile() handle to call FtpFileGetSize() :oops:

BTW, do you know a simple way to ascertain the size of default html ( like <!-- m -->http://www.autohotkey.com/<!-- m --> ) when HTTP_QUERY_CONTENT_LENGTH fails. I am looking at ahklerner's UrlGetContents(), but that looks pretty tough for me..

Thanks again for all the help. :)

Lexikos
  • Administrators
  • 9844 posts
  • AutoHotkey Foundation
  • Last active:
  • Joined: 17 Oct 2006

BTW, do you know a simple way to ascertain the size of default html ( like <!-- m -->http://www.autohotkey.com/<!-- m --> ) when HTTP_QUERY_CONTENT_LENGTH fails.

No, but UrlDownloadToFile works without knowing the size in advance - it reads into a fixed-size buffer, writes out to file, and repeats until there's no more to read. If you know the file is text, you can simply append to a variable. Otherwise you can use GlobalAlloc to allocate memory and GlobalReAlloc to resize it, preserving data... or use some other method.

SKAN
  • Administrators
  • 9115 posts
  • Last active:
  • Joined: 26 Dec 2005

No, but UrlDownloadToFile works without knowing the size in advance - it reads into a fixed-size buffer, writes out to file, and repeats until there's no more to read. If you know the file is text, you can simply append to a variable.


I had thought on similar lines.. but I cannot generate a progress bar!

Otherwise you can use GlobalAlloc to allocate memory and GlobalReAlloc to resize it, preserving data...


Cool!.. I never knew this!

or use some other method.


I would have used VarSetCapacityEx(), but would be very slow, I guess

Lexikos
  • Administrators
  • 9844 posts
  • AutoHotkey Foundation
  • Last active:
  • Joined: 17 Oct 2006

I would have used VarSetCapacityEx(), but would be very slow, I guess

Yes, it basically has to do twice the work, and in a semi-interpreted scripting language.

Another advantage of Global(Re)Alloc is that because you reference the data by pointer, it is easily passed to or returned from a function, referenced in another structure, etc. On the other hand, string operations are more difficult and costly.

Note that there are numerous (similar) alternatives to the Global memory management functions such as the Heap, Virtual, Local or C run-time functions. See Memory Management Functions and malloc/free.

Sean
  • Members
  • 2462 posts
  • Last active: Feb 07 2012 04:00 AM
  • Joined: 12 Feb 2007

See Memory Management Functions and malloc/free.

Those omit another important function which is actually my recommendation: CoTaskMem.... Although it's not mentioned explicitly in the page, I think it's thread-safe which is not that important with current AHK though.
<!-- m -->http://msdn.microsof...y/aa366533.aspx<!-- m -->

Drugwash
  • Members
  • 1078 posts
  • Last active: May 24 2016 04:20 PM
  • Joined: 07 Sep 2008
All 3 examples work perfectly on my 98SE system. 8)

If I may: first thought when I saw the function name was: why so long? I'd make it WebFileRead - easy to use, compact.

Nice job SKAN, thanks! :wink:

tank
  • Administrators
  • 4345 posts
  • AutoHotkey Foundation
  • Last active: Oct 13 2016 01:04 AM
  • Joined: 21 Dec 2007
damnit this is brutifule
this is the coolest f$%^& thing done in ahk since COM
Never lose.
WIN or LEARN.

DeWild1
  • Members
  • 369 posts
  • Last active: Feb 28 2014 08:15 PM
  • Joined: 30 Apr 2006
holly fricken cow, you just cut out so many lines of my old code, in so many programs with this.
8) 8) 8)
Thank you!

Drugwash
  • Members
  • 1078 posts
  • Last active: May 24 2016 04:20 PM
  • Joined: 07 Sep 2008
I've already stumbled into an issue with this function: can't download anything that does not end the URL in an explicit file type. :(

Testing was performed using URL := "http://www.pcidatabase.com/reports.php?type=csv" . This should download a file called reports.php whose format depends on the trailing parameter ?type=csv.

Any idea on how this can be fixed?

Sean
  • Members
  • 2462 posts
  • Last active: Feb 07 2012 04:00 AM
  • Joined: 12 Feb 2007

Any idea on how this can be fixed?

In analogy with ReadFile, it appears to me similar as you're asking ReadFile to format the data in the file to Unicode. Does it make a sense? I highly doubt InternetReadFile knows anything about MIME type.

PS. BTW, you can use this in that case, although it's no longer maintained.
<!-- m -->http://www.autohotke...topic19475.html<!-- m -->

PS2. I just tested InternetReadFile with the following script, and it worked! So, I reckon it's a problem of HttpQueryInfo failing to retrieve the size.
<!-- m -->http://www.autohotke...topic19608.html<!-- m -->

SKAN
  • Administrators
  • 9115 posts
  • Last active:
  • Joined: 26 Dec 2005

I've already stumbled into an issue with this function: can't download anything that does not end the URL in an explicit file type.


As Sean said, this happens because HttpQueryInfo fails to retrieve the file size... and that was what I have been discussing with Lexikos a few posts above.

Any idea on how this can be fixed?


You may force download the file if you can guesstimate the file size:

URL := "http://www.pcidatabase.com/reports.php?type=csv"
If ( InternetFileRead( CSV, URL, [color=red]2**19 [/color] ) > 0 ) && !ErrorLevel ; 2**19 is 512K
  If VarZ_Save( CSV, "reports.csv" )
     Run Notepad Reports.csv

Note: AHK ErrorLevel will be 0 if complete file was downloaded else it will be 1


Refer the first example on title post... http://www.formyip.com/ would also fail if I had not specified 100 bytes..