Jump to content

Sky Slate Blueberry Blackcurrant Watermelon Strawberry Orange Banana Apple Emerald Chocolate
Photo

Function : InternetFileRead() aka UrlDownLoadToVar()


  • Please log in to reply
161 replies to this topic
TOV
  • Guests
  • Last active:
  • Joined: --
Sorted doing:
URL = ftp://%ip%/file.txt
and
global FileName
Then using the value

tov
  • Guests
  • Last active:
  • Joined: --
really stupid questions time :D

Anyway this could simply be adapted to allow.......uploading ?

I would love to have the progress bar available for uploading files !

Any chance ?? go on you know you want to 8)

Jamie
  • Members
  • 129 posts
  • Last active: Dec 02 2012 04:59 AM
  • Joined: 26 Mar 2010
I love this function but the code is ugly, no offense.

I wanted to make a version that presumes the size is unknown ahead of time (HttpQueryInfoA notwithstanding) and returns the number of bytes downloaded.

In the process of decrypting the code, I came across two things I thought might be errors. The first appears to be a typo:
If ! (( hIU:=DllCall( LIB "InternetOpenUrlA", UInt,hIO, Str,URL, Str,N, Int,0, UInt,F
                                                            , UInt,0 ) ) || ErrorLevel )    
    ; return error
Which is equivalent to
hIU:=DllCall( LIB "InternetOpenUrlA", UInt,hIO, Str,URL, Str,N, Int,0, UInt,F, UInt,0 )
  if !((hIU) || ErrorLevel)
    ; return error
If the dllcall fails, then hIU will be blank and ErrorLevel will be nonzero, which will cause it to not return error at that point. It will continue along with a bad handle.

The second issue regards this:
Loop {
    If ( DllCall( LIB "InternetReadFile", UInt,hIU, UInt,P, UInt,B, UIntP,R ) && !R )
      Break
If InternetReadFile were to fail for some reason, it returns false and sets R to zero, meaning the function will get stuck in an infinite loop, as there is no way out of the loop other than successfully reaching the end of the file.

These observations were from looking at and reformatting the code, not from actually running it and encountering these cases. I might be mistaken, but those two areas look suspicious.

Drugwash
  • Members
  • 1078 posts
  • Last active: May 24 2016 04:20 PM
  • Joined: 07 Sep 2008
Please keep at this. Downloading a file of unknown size is exactly what I need for one of my scripts and as mentioned in first posts, the script fails at doing that. Multiple subsequent tries might not be the best choice, since certain servers may limit download attempts or consider repeated attempts as flood attacks or whatever.

SKAN
  • Administrators
  • 9115 posts
  • Last active:
  • Joined: 26 Dec 2005

Please keep at this. Downloading a file of unknown size is exactly what I need


Sure friend. Allow me a couple of days. :)

Jamie
  • Members
  • 129 posts
  • Last active: Dec 02 2012 04:59 AM
  • Joined: 26 Mar 2010
Here's what I ended up with. I don't have time to document it right now but I've been using it for a month solid with no issues.

Returns the number of bytes fetched, or negative number in case of failure. Data is placed in variable V.

; 0x84000000 is INTERNET_FLAG_NO_CACHE_WRITE (0x04000000) and INTERNET_FLAG_RELOAD (0x80000000)
InternetFileRead2( ByRef V, URL, bSz=1024, F=0x84000000 ) {
  Static LIB="WININET\", N=""
  If (!DllCall( "GetModuleHandle", Str,"wininet.dll" )) {
    DllCall( "LoadLibrary", Str,"wininet.dll" ) ; load wininet.dll if it's not already loaded
  }
  hIO:=DllCall( LIB "InternetOpenA", Str,N, UInt,4, Str,N, Str,N, UInt,0 )
  If (!hIO) {
    Return -1 ; couldn't open internet IO
  }
  
  hIU:=DllCall( LIB "InternetOpenUrlA", UInt,hIO, Str,URL, Str,N, Int,0, UInt, F, UInt,0)
  If (!hIU) { ; failed to open handle to url
    DllCall( LIB "InternetCloseHandle", UInt,hIO ) ; clean up the open internet handle
    Return -2
  }
  
  ; make sure query returned 200 ok  
  VarSetCapacity(QStatus, 100) ; 100 characters for status
  HTTP_QUERY_STATUS_CODE := 19
  if (DllCall( LIB "HttpQueryInfoA", UInt, hIU, Int, HTTP_QUERY_STATUS_CODE, Str, QStatus, UIntP, 100, UInt, 0)) {
    if (QStatus != "200") {
      return -3
    }
  }
  else {
    return -4 ; couldn't get status code, treat this as a fatal error
  }
  
  TtlB := 0
  outspace := 1024
  VarSetCapacity(Q, outspace)
  VarSetCapacity(recvbuf, bSz) ; set receive buffer size (will be reused for each read)
  Loop {
    if (!DllCall( LIB "InternetReadFile", UInt, hIU, UInt, &recvbuf, UInt, bSz, UIntP, R )) {
      ; read failed for some reason
      if (A_LastError == 122) { ; 122 is ERROR_INSUFFICIENT_BUFFER
        MsgBox, InternetReadFile failed for having too small a buffer
        if (bSz > 5000000) { ; set a hard limit of 10MB to prevent it from blowing up
          return -10
        }
        bSz *= 2
        VarSetCapacity(recvbuf, bSz)
        continue ; try again with larger buffer
      }
      else {
        MsgBox, InternetReadFile failed for some unknown reason
        return -11
      }
    }
    if (R == 0) {
      break ; successfully read and number of bytes returned was zero
    }
    while (TtlB + R > outspace) {
      VarSetCapacity(T, TtlB)
      DllCall("RtlMoveMemory", Str, T, Str, Q, UInt, TtlB) ; copy to temp space
      VarSetCapacity(Q, outspace*2, 0) ; allocate more space for Q
      DllCall("RtlMoveMemory", Str, Q, Str, T, UInt, TtlB) ; copy back
      VarSetCapacity(T, 0) ; free temp space
      outspace := outspace * 2
    }
    DllCall("RtlMoveMemory", UInt, &Q+TtlB, UInt, &recvbuf, UInt, R)
    TtlB += R
  }
  
  VarSetCapacity(V, 64) 
  VarSetCapacity(V, 0) ; shrink V to the actual bytes received
  VarSetCapacity(V, TtlB) ; allocate exact space for V
  DllCall("RtlMoveMemory", Str, V, Str, Q, UInt, TtlB) ; copy data from accum buffer
  VarSetCapacity(Q, 0) ; free accumulation buffer
  ErrorLevel := 0
  return TtlB
}


aaron_cz
  • Guests
  • Last active:
  • Joined: --
I know this has been asked before, but could someone ( who knows what they are doing - so thats me out) add the progress bar to an ftp upload ?

I really like the look and feel of this, but have no idea where to start adding this for an upload.

please help ?! :)

Drugwash
  • Members
  • 1078 posts
  • Last active: May 24 2016 04:20 PM
  • Joined: 07 Sep 2008
Jamie, after a couple of tests your version works fine. I've only tested with one address - the one I needed - but hopefully it would work with others as well. Thank you very much for this birthday gift! ;) I'll soon post the script that uses the function.

SKAN
  • Administrators
  • 9115 posts
  • Last active:
  • Joined: 26 Dec 2005

Thank you very much for this birthday gift! ;)


:O Happy Birthday!! :D

Drugwash
  • Members
  • 1078 posts
  • Last active: May 24 2016 04:20 PM
  • Joined: 07 Sep 2008
Thank you! It's on the 19th, actually, but it only depends on timezone. ;)

SKAN
  • Administrators
  • 9115 posts
  • Last active:
  • Joined: 26 Dec 2005

UrlDownloadToFile works without knowing the size in advance - it reads into a fixed-size buffer, writes out to file, and repeats until there's no more to read. If you know the file is text, you can simply append to a variable. Otherwise you can use GlobalAlloc to allocate memory and GlobalReAlloc to resize it, preserving data... or use some other method.

...

advantage of Global(Re)Alloc is that because you reference the data by pointer, it is easily passed to or returned from a function, referenced in another structure, etc. On the other hand, string operations are more difficult and costly.

Note that there are numerous (similar) alternatives to the Global memory management functions such as the Heap, Virtual, Local or C run-time functions. See Memory Management Functions and malloc/free.


I opted for malloc() and found realloc() alone is enough for allocate/reallocate/free.

Many thanks for guiding me, Lexikos.

@Drugwash:

Here is a beta-version of your requirement. Please test it and let me know the status:

#SingleInstance Force
URL := "http://www.autohotkey.com/download/AutoHotkeyInstall.exe"
MsgBox, % "Bytes Read:`t" NetGet( BIN, URL ) "`nErrorLevel:`t" ErrorLevel
Return


NetGet( ByRef V="", URL="", Override="" ) {  ; NetGet v0.50 a.k.a InternetFileRead by SKAN
; www.autohotkey.com/forum/viewtopic.php?p=370318#370318     CD:13-01-2010 | LM:19-07-2010
 Static CD="CDecl", QL=16, CL="00000000000000", PRX="", POR="", Agent="AutoHotkey"
 Bytes := 0, Buffer := 1024, Page := 16384, Mode := 4, UI := "DLP",  iFlags := 0x80000000

 Loop, Parse, Override,`,=, %A_Space%               ; Override routine for local variables
   A_Index & 1  ? ( Var:= A_LoopField ) : ( %Var% := A_LoopField )
   
 If Not DllCall( "kernel32\GetModuleHandle", Str,"wininet.dll" )  ; If WININET not loaded,
        DllCall( "kernel32\LoadLibrary", Str,"wininet.dll" )      ; load it now.
        
 ; * Open Internet/URL & obtain handles *  On Error: Return 0 and set an AHK ErrorLevel
 If hIO := DllCall( "wininet\InternetOpenA", Str,Agent, UInt,Mode, Str,PRX,Str,POR,Int,0 )
   If ! ( hIU := DllCall( "wininet\InternetOpenUrlA", UInt,hIO, Str,URL, Int,0,Int,0, UInt
                                                     , iFlags ,UInt,0 )) || ErrorLevel
    Return ( ErrorLevel := 1 + !DllCall( "wininet\InternetCloseHandle", UInt,hIO ) ) << 64
    
 CL := Bytes ? Bytes ; * Ascertain filesize * If not Bytes, fetch & use HTTP/FTP Filesize
 : ((SubStr(URL,1,4)="ftp:") ? DllCall( "wininet\FtpGetFileSize",UInt, hIU, Int,0 )
 : ((DllCall( "wininet\HttpQueryInfoA", UInt,hIU,Int,5,Str,CL,UIntP,QL,Int,0 )<<64) + CL))

 SplitPath, URL, FN,,,, DN   ; * Ascertain Progress Display name & Buffer Size/Page Size *
 BSZ := ( (Bytes && Bytes<1024) ? Bytes : Buffer ),  FN := (FN ? FN:DN)
 pSZ := Bytes ? Bytes : (CL ? CL:(1024*Ceil(Page/1024))), PSZ := PSZ<BSZ ? BSZ:PSZ
 LP := Bytes ? Bytes : ( CL ? CL : "unknown" ), %UI%( True,LP,FN )

 ; * Allocate Memory * CoP is backup "Copy of Pointer" as realloc() might fail and reset P
 mErr := !( P := DllCall( "msvcrt\realloc", UInt,0,UInt,PSz, CD )), CoP:=P,  TtlB:=0, R:=0
 While !( mErr||rErr ){ ; Loop while no memory allocation error OR until read is completed
  TtlB:=TtlB+R,  ReqMem:=TtlB+BSz,  RemB:=(CL-TtlB),  BSz:=(RemB>0 && RemB<BSz ? RemB:BSz)
  If ( DllCall( "msvcrt\_msize", UInt,P ) < ReqMem ) ; Check Memory requirement & increase
  mErr := !( P := DllCall( "msvcrt\realloc",UInt,P, UInt,TtlB+PSz, CD ) ), CoP:=P
  rErr := (( DllCall( "wininet\InternetReadFile", UInt,hIU, UInt,P+TtlB,UInt,BSz,UIntP,R )
           && !R ) || ( Bytes && (TtlB >= Bytes) ))
  %UI%( TtlB,LP )
  Sleep -1
 }
 ichErr := ( ! DllCall( "wininet\InternetCloseHandle", UInt,hIU)) ; Close internet handles
         | ( ! DllCall("wininet\InternetCloseHandle", UInt,hIO)) ;  and save error as flag
        
 VarSetCapacity(V,64), VarSetCapacity(V,0), VarC := VarSetCapacity(V,TtlB,32)  ; Reset Var
 DllCall( "RtlMoveMemory", UInt,&V,UInt,CoP, UInt,TtlB ) ; Transfer from Allocated Memory.
 DllCall( "msvcrt\realloc", UInt,CoP, UInt,0, CD )       ; Free Allocated Memory
Return TtlB
}

DLP( WP=0, LP=0, Msg="" ) {
 If ( WP=1 ) {
 SysGet, m, MonitorWorkArea, 1
 y:=(mBottom-46-2), x:=(mRight-370-2), VarSetCapacity( Size,16,0 )
 DllCall( "shlwapi.dll\StrFormatByteSize64A", Int64,LP, Str,Size, UInt,16 )
 Size := ( Size="0 bytes" ) ? N : "«" Size "»"
 Progress, CWE6E3E4 CT000020 CBF73D00 x%x% y%y% w370 h46 B1 FS8 WM700 WS400 FM8 ZH8 ZY3
         ,, %Msg%  %Size%, InternetFileRead(), Tahoma
 WinSet, Transparent, 210, InternetFileRead()
 Return
 }
 Progress,% (P:=Round(WP/LP*100)),% "Memory: " wp " / " lp " [ " P "`% ]"
 IfGreater,WP,0,IfEqual,WP,%LP%, Progress, Off
}


aaron_cz
  • Guests
  • Last active:
  • Joined: --
I know this is sort of off topic.
But can anyone help me ?

aaron_cz
  • Guests
  • Last active:
  • Joined: --
would offering to pay (beer or paypal) get this working with upload and downloading ?

Drugwash
  • Members
  • 1078 posts
  • Last active: May 24 2016 04:20 PM
  • Joined: 07 Sep 2008
Thank you very much, SKAN! The beta version did work correctly with the URL I needed ( <!-- m -->http://www.pcidataba...ts.php?type=csv<!-- m --> ). I'll wait for the stable version before including it in my script, just to make sure.

SKAN
  • Administrators
  • 9115 posts
  • Last active:
  • Joined: 26 Dec 2005

I'll wait for the stable version before including it in my script, just to make sure.


I am waiting for you to tell me since I do not have access to W98.
I am already using it in my scripts without any problems ( XP-SP3 ).

:)