download urls to vars, partially/fully, via WinHttpRequest Topic is solved

Get help with using AutoHotkey and its commands and hotkeys
User avatar
jeeswg
Posts: 2198
Joined: 19 Dec 2016, 01:58
Location: UK

download urls to vars, partially/fully, via WinHttpRequest

05 Jan 2017, 14:18

tl;dr
- download file (binary) to variable (via object)
- download file (binary) to variable (via object) (initial n bytes only)

I know that UrlDownloadToFile uses
InternetReadFile to download webpages,
and that with this you can download
files (binary data) and webpages to variables, and if desired,
only the start of the webpage.

One of the reasons to download only
the start of webpage, is to retrieve
the webpage titles for a list of urls.

Is it possible to download all or only the start
of a file (binary data) or webpage via WinHttpRequest,
to a variable?

Note: WinHttpRequest is preferable to
UrlDownloadToFile/InternetReadFile
because its faster.

[EDIT: 100th post! That happened quite quickly.]
Last edited by jeeswg on 01 Sep 2017, 20:10, edited 3 times in total.
garry
Posts: 940
Joined: 22 Dec 2013, 12:50

Re: download urls to vars, partially/fullly, via WinHttpRequest

05 Jan 2017, 16:55

you mean , you search for url's ?

Code: [Select all] [Expand] [Download] GeSHi © Codebox Plus

User avatar
jeeswg
Posts: 2198
Joined: 19 Dec 2016, 01:58
Location: UK

Re: download urls to vars, partially/fullly, via WinHttpRequest

05 Jan 2017, 17:25

My two common scenarios are:
- The url is a webpage (e.g. http://ahkscript.org/),
I want to download the html as UrlDownloadToFile does,
but only the start of it, in order to retrieve the webpage title.
- The url is an image (e.g. http://ahkscript.org/static/ahk_logo.png),
I want to download the binary data for the image to a variable,
but not save it as a file. (I compare the data against an existing file,
and only if they differ do I then write the binary to a file.
E.g. YouTube thumbnails, same url, but image data can change.)

E.g. this downloads a file and saves it as a file.
Can it be amended to store it in a variable instead.
Also can it be set to download only the first n bytes.

Code: [Select all] [Expand] [Download] GeSHi © Codebox Plus

User avatar
jeeswg
Posts: 2198
Joined: 19 Dec 2016, 01:58
Location: UK

Re: download urls to vars, partially/fullly, via WinHttpRequest

22 Feb 2017, 02:30

Please if anyone can help,
I've been going round in circles trying to use
ComObjCreate("WinHttp.WinHttpRequest.5.1")
and ComObjCreate("ADODB.Stream")
to download a file e.g. an image,
and put the data from ResponseBody
into a variable.

(Btw I'm trying to get binary data from ResponseBody,
and not text from ResponseText.)
(I show an example for downloading binary to a file,
above, but I want to download it to a variable.)
Thank you.

Some relevant links I found:
Stream Object (ADO) | Microsoft Docs
https://docs.microsoft.com/en-us/sql/ado/reference/ado-api/stream-object-ado
Read Method | Microsoft Docs
https://docs.microsoft.com/en-us/sql/ado/reference/ado-api/read-method
Convert Byte Array to String - AutoHotkey Community
https://autohotkey.com/boards/viewtopic.php?t=9807
Help with HTTPRequest and BinRead please - AutoHotkey Community
https://autohotkey.com/boards/viewtopic.php?t=5716
User avatar
jeeswg
Posts: 2198
Joined: 19 Dec 2016, 01:58
Location: UK

Re: download urls to vars, partially/fullly, via WinHttpRequest

22 Feb 2017, 02:38

Thank you jNizM, excellent functions btw.
I have functions already that use InternetReadFile,
but I'm trying to investigate this object method which appears to download much faster
(and to improve my knowledge of objects for doing AHK v1/v2 script conversions,
and possibly write a tutorial on objects).

However, I don't know if the object method allows a partial download of a file
which is something that the InternetReadFile method can do.
Appreciated.
User avatar
jeeswg
Posts: 2198
Joined: 19 Dec 2016, 01:58
Location: UK

Re: download urls to vars, partially/fullly, via WinHttpRequest

22 Feb 2017, 02:45

Thanks, that's the frustrating thing, the method is fairly clear, from various sources,
it's actually the mechanics of getting binary data out of ResponseBody,
into a variable, possibly via an object, that is the problem.
To know the specifics of how to move the data within AutoHotkey.

To work out the lines I need to modify in JEE_UrlDownloadToFile above.
I've read things like VARIANT and binary array etc.
User avatar
jeeswg
Posts: 2198
Joined: 19 Dec 2016, 01:58
Location: UK

Re: download urls to vars, partially/fullly, via WinHttpRequest

22 Feb 2017, 02:49

Waitasec it might be here I'll just check.

[solved] UrlDownloadToFile slows down GUI response - AutoHotkey Community
https://autohotkey.com/boards/viewtopic.php?t=9338

Btw is there a search engine that would let you search for:
AutoHotkey ADODB Read(

Specifically, to handle the bracket.

[EDIT:]
I found a good result when I searched for:
AutoHotkey "ADODB.Stream" ResponseBody
Last edited by jeeswg on 22 Feb 2017, 03:37, edited 1 time in total.
User avatar
jeeswg
Posts: 2198
Joined: 19 Dec 2016, 01:58
Location: UK

Re: download urls to vars, partially/fullly, via WinHttpRequest  Topic is solved

22 Feb 2017, 03:36

This looks like it works:

Code: [Select all] [Expand] [Download] GeSHi © Codebox Plus



- Btw is memcpy the best dll function for memory move?
- Also, it's a C function, can it handle Ptr? It appeared to work on 64-bit when I tested it.
- Also can 'Ptr,&vIndex' be replaced with 'PtrP,0' or something similar?

[EDIT:]
- I don't know if there is a way to specify to only download the first n bytes of a file.
- I've had problems when trying to use ResponseText from webpages, to get a webpage's title,
so I can now do it by using the binary ResponseBody and converting that to text,
like I did with InternetReadFile.

[EDIT 2:]
Btw I've seen a fair number of AHK libraries with functions for WinHttpRequest,
so feel free to copy this function, changing it as much or as little as you want, to match your other functions, if you would like to add this in to your library.
Last edited by jeeswg on 22 Feb 2017, 08:07, edited 1 time in total.
tmplinshi
Posts: 1087
Joined: 01 Oct 2013, 14:57

Re: download urls to vars, partially/fullly, via WinHttpRequest

22 Feb 2017, 05:28

jeeswg wrote: I don't know if there is a way to specify to only download the first n bytes of a file.

You can use Range header. However not all sites support this.

Example:

Code: [Select all] [Download] GeSHi © Codebox Plus

whr := ComObjCreate("WinHttp.WinHttpRequest.5.1")
whr.Open("GET", "http://www.nirsoft.net/utils/nircmd.zip", true)
whr.SetRequestHeader("Range", "bytes=0-10")
whr.Send()
whr.WaitForResponse()
MsgBox % whr.ResponseBody.MaxIndex()
MsgBox % whr.Status " " whr.StatusText "`n`n" whr.GetAllResponseHeaders()
User avatar
jeeswg
Posts: 2198
Joined: 19 Dec 2016, 01:58
Location: UK

Re: download urls to vars, partially/fullly, via WinHttpRequest

22 Feb 2017, 05:42

Thanks so much, that's a great response.
It's not the end of the world if a site doesn't support it,
but it's a great help. One thing I like to do is
download the start to get the webpage title.

[EDIT:] Haha good call with the util url btw.
User avatar
jeeswg
Posts: 2198
Joined: 19 Dec 2016, 01:58
Location: UK

Re: download urls to vars, partially/fully, via WinHttpRequest

01 Sep 2017, 22:40

- download htm to variable
- download file to variable e.g. image/icon
- download htm to variable (partial) - get webpage title

Code: [Select all] [Expand] [Download] GeSHi © Codebox Plus


Return to “Ask For Help”

Who is online

Users browsing this forum: Bing [Bot], Drake, euras, Terka, Unknovvn, Yahoo [Bot] and 45 guests