REST API basic authentication via COM WinHttp.WinHttpRequest.5.1

Get help with using AutoHotkey and its commands and hotkeys
JoeWinograd
Posts: 1073
Joined: 10 Feb 2014, 20:00

REST API basic authentication via COM WinHttp.WinHttpRequest.5.1

23 May 2018, 09:41

Hi Folks,
I'm trying to write an AHK script to call this REST API:

https://dev.docparser.com

This is my first attempt at calling a REST API and I can't get past the API authentication via HTTP Basic Auth, which says this at their documentation:
This authentication method is the preferred way of authenticating your requests to Docparser. When using HTTP Basic Auth, use your secret API key as the "username" and leave the "password" blank.
I do have my secret API key.

I create the object with this AHK code:

Code: Select all

oHTTP:=ComObjCreate("WinHttp.WinHttpRequest.5.1")
If !IsObject(oHTTP)
{
  MsgBox,4112,Fatal Error,Unable to create HTTP object
  ExitApp
}
That works fine. But I can't get beyond that. Everything that I've tried comes back with this response:

{"error":"please use HTTP Basic authentication"}

If it's helpful, here's the curl code that is in their HTTP Basic Auth section:

Code: Select all

curl https://api.docparser.com/v1/ping \
   -u <secret_api_key>:
As mentioned above, I do have the <secret_api_key>. Can anyone help me with the AHK code that will achieve HTTP Basic Auth? Thanks much, Joe
User avatar
Nextron
Posts: 1274
Joined: 01 Oct 2013, 08:23
Location: Netherlands OS: Win7 x64 AHK: Unicode x32

Re: REST API basic authentication via COM WinHttp.WinHttpRequest.5.1

23 May 2018, 10:21

Before the oHTTP.send(), add these:

Code: Select all

oHTTP.SetAutoLogonPolicy(0) ; AutoLogonPolicy_Always=0, AutoLogonPolicy_OnlyIfBypassProxy=1, AutoLogonPolicy_Never=2
oHTTP.SetCredentials(User, Pass, 0) ;HTTPREQUEST_SETCREDENTIALS_FOR_SERVER=0, HTTPREQUEST_SETCREDENTIALS_FOR_PROXY=1
JoeWinograd
Posts: 1073
Joined: 10 Feb 2014, 20:00

Re: REST API basic authentication via COM WinHttp.WinHttpRequest.5.1

23 May 2018, 10:57

Hi Nextron,
Thanks for the quick reply. Here's my code with your suggestions in it:

Code: Select all

ResponseFile:="c:\temp\DocparserTest.txt"
FileDelete,%ResponseFile%
URL:="https://api.docparser.com/v1/ping"
APIkey:="this is my secret API key"
oHTTP:=ComObjCreate("WinHttp.WinHttpRequest.5.1")
If !IsObject(oHTTP)
{
  MsgBox,4112,Fatal Error,Unable to create HTTP object
  ExitApp
}
oHTTP.Open("GET",URL)
oHTTP.SetAutoLogonPolicy(0) ; AutoLogonPolicy_Always=0, AutoLogonPolicy_OnlyIfBypassProxy=1, AutoLogonPolicy_Never=2
oHTTP.SetCredentials(APIkey,"",0) ;HTTPREQUEST_SETCREDENTIALS_FOR_SERVER=0, HTTPREQUEST_SETCREDENTIALS_FOR_PROXY=1
oHTTP.Send()
Response:=oHTTP.ResponseText
FileAppend,%Response%,%ResponseFile%
Run,%ResponseFile%
ExitApp
The response is still this:

{"error":"please use HTTP Basic authentication"}

In your oHTTP.SetCredentials(User,Pass,0) line, I made User my secret API key and Pass null, because their doc says,
When using HTTP Basic Auth, use your secret API key as the "username" and leave the "password" blank.
I'm sure the API key is right. What else should I try? Thanks, Joe
User avatar
Nextron
Posts: 1274
Joined: 01 Oct 2013, 08:23
Location: Netherlands OS: Win7 x64 AHK: Unicode x32

Re: REST API basic authentication via COM WinHttp.WinHttpRequest.5.1

23 May 2018, 18:01

I think it's due to WinHttpRequest apparently only sending the credentials in response to a WWW-Authenticate header, whereas your site is sending a 400 - Bad Request response without such header.

Your best bet would be to send the authentication header manually:

Code: Select all

ResponseFile:="c:\temp\DocparserTest.txt"
FileDelete,%ResponseFile%
URL:="https://api.docparser.com/v1/ping"
APIkey:="this is my secret API key"
oHTTP:=ComObjCreate("WinHttp.WinHttpRequest.5.1")
If !IsObject(oHTTP)
{
  MsgBox,4112,Fatal Error,Unable to create HTTP object
  ExitApp
}
oHTTP.Open("GET",URL)
user:=APIkey, pass:=""
oHTTP.SetRequestHeader("Authorization", "Basic " Base64Encode(user ":" pass))
oHTTP.Send()
Response:=oHTTP.ResponseText
FileAppend,%Response%,%ResponseFile%
Run,%ResponseFile%
ExitApp


Base64Encode(String) ;by Uberi @ https://autohotkey.com/board/topic/5545-base64-coderdecoder/page-3#entry690930
{
    static CharSet := "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"
    VarSetCapacity(Output,Ceil(Length / 3) << 2)
    Index := 1, Length := StrLen(String)
    Loop, % Length // 3
    {
        Value := Asc(SubStr(String,Index,1)) << 16
            | Asc(SubStr(String,Index + 1,1)) << 8
            | Asc(SubStr(String,Index + 2,1))
        Index += 3
        Output .= SubStr(CharSet,(Value >> 18) + 1,1)
            . SubStr(CharSet,((Value >> 12) & 63) + 1,1)
            . SubStr(CharSet,((Value >> 6) & 63) + 1,1)
            . SubStr(CharSet,(Value & 63) + 1,1)
    }
    Length := Mod(Length,3)
    If Length = 0 ;no characters remain
        Return, Output
    Value := Asc(SubStr(String,Index,1)) << 10
    If Length = 1
    {
        Return, Output ;one character remains
            . SubStr(CharSet,(Value >> 12) + 1,1)
            . SubStr(CharSet,((Value >> 6) & 63) + 1,1) . "=="
    }
    Value |= Asc(SubStr(String,Index + 1,1)) << 2 ;insert the third character
    Return, Output ;two characters remain
        . SubStr(CharSet,(Value >> 12) + 1,1)
        . SubStr(CharSet,((Value >> 6) & 63) + 1,1)
        . SubStr(CharSet,(Value & 63) + 1,1) . "="
}
You can also just paste in a base64 encoded user:pass string so you can leave out the Base64Encode function.
JoeWinograd
Posts: 1073
Joined: 10 Feb 2014, 20:00

Re: REST API basic authentication via COM WinHttp.WinHttpRequest.5.1

23 May 2018, 19:25

Thank you so much, Nextron...works perfectly! Response text is:
{"msg":"pong"}

I then tried the URL to retrieve the parsers:
https://api.docparser.com/v1/parsers

Also worked a charm! Came back with the three parsers that I created under my account.

Next task is uploading documents (which requires a POST), then downloading the parsed data. I hope you won't mind some additional questions as I continue the project. Thanks again for getting me past the first stumbling block. Regards, Joe
JoeWinograd
Posts: 1073
Joined: 10 Feb 2014, 20:00

Re: REST API basic authentication via COM WinHttp.WinHttpRequest.5.1

23 May 2018, 22:24

Hi Nextron,

I can't get file upload to work. Their Upload Document From Local Path documentation says this:
Docparser allows you to upload documents from your local hard-drive with a multipart/form-data request. This is the same type of request a HTML form with a file upload field would send. The field name used for the document upload needs to be file.
It gives this example:

POST https://api.docparser.com/v1/document/upload/<PARSER_ID>

And this curl code:

Code: Select all

curl \
  -u <secret_api_key>: \
  -F "file=@/home/your/local/file.jpdf" \
  https://api.docparser.com/v1/document/upload/<PARSER_ID>
I know that I have the <PARSER_ID> right. Here's my code:

Code: Select all

oHTTP:=ComObjCreate("WinHttp.WinHttpRequest.5.1")
If !IsObject(oHTTP)
{
  MsgBox,4112,Fatal Error,Unable to create HTTP object
  ExitApp
}
UploadFile:="c:\temp\DocParserTest.pdf"
ResponseFileUpload:="c:\temp\DocparserUpload.txt"
FileDelete,%ResponseFileUpload%
URL:="https://api.docparser.com/v1/document/upload/<this is Parser ID - I am sure it is right"
oHTTP.Open("POST",URL)
UsernamePasswordBase64:="this is base64 encoded user:pass string - I know it is right because it works with the Basic Auth script"
oHTTP.SetRequestHeader("Authorization","Basic " UsernamePasswordBase64,"File " UploadFile)
oHTTP.Send()
Response:=oHTTP.ResponseText
FileAppend,%Response%,%ResponseFileUpload%
Run,%ResponseFileUpload%
ExitApp
The response text is:
{"error":"No file found. You can upload a file with a multipart\/form-data HTTP POST request by using <file> as the field name. Alternatively, you can also upload documents with a simple form-data HTTP POST request using the field names <file_content> and <file_name>. The file data sent through <file_content> is expected to be base64 encoded."}
I want to do <file>, not <file_content>. When the code above didn't work, I changed the SetRequestHeader line to:

Code: Select all

oHTTP.SetRequestHeader("Authorization","Basic " UsernamePasswordBase64,"Content-Type:multipart/form-data","File " UploadFile)
That also failed, and with the same response text. I'll very much appreciate your help on this. Regards, Joe
tmplinshi
Posts: 1278
Joined: 01 Oct 2013, 14:57

Re: REST API basic authentication via COM WinHttp.WinHttpRequest.5.1

24 May 2018, 00:25

You can use CreateFormData to create multipart/form-data. Example:

Code: Select all

UploadFile := "c:\temp\DocParserTest.pdf"
objParam := { "file": [UploadFile] }
CreateFormData(postData, hdr_ContentType, objParam)

UsernamePasswordBase64:="this is base64 encoded user:pass string - I know it is right because it works with the Basic Auth script"
URL := "https://api.docparser.com/v1/document/upload/<this is Parser ID - I am sure it is right"

oHTTP := ComObjCreate("WinHttp.WinHttpRequest.5.1")
oHTTP.Open("POST", URL)
oHTTP.SetRequestHeader("Authorization", "Basic " UsernamePasswordBase64)
oHTTP.SetRequestHeader("Content-Type", hdr_ContentType)
oHTTP.Send(postData)

ResponseFileUpload := "c:\temp\DocparserUpload.txt"
FileOpen(ResponseFileUpload, "w").Write(oHTTP.ResponseText)
Run,%ResponseFileUpload%
ExitApp
JoeWinograd
Posts: 1073
Joined: 10 Feb 2014, 20:00

Re: REST API basic authentication via COM WinHttp.WinHttpRequest.5.1

24 May 2018, 03:02

Hi tmplinshi,
The underlying code is much more complex than I had expected, but the function call makes it very easy to use, and it works perfectly! Response text was this:

{"id":"returned a unique Document ID here","file_size":84487,"pages":1,"quota_used":10,"quota_left":40,"quota_refill":"1970-01-01T00:00:00+00:00"}

Thank you very much! Nextron and you have me through the first two phases — authentication and upload. Next phase is download of the parsed data (an Excel file), but that will wait until (my) tomorrow, as it's 3am in my neck of the woods and time for me to call it a day. Thanks again for your help! Regards, Joe
JoeWinograd
Posts: 1073
Joined: 10 Feb 2014, 20:00

Re: REST API basic authentication via COM WinHttp.WinHttpRequest.5.1

24 May 2018, 12:54

This post is for anyone following this thread and interested in using AHK with Docparser. Their documentation says that there are three ways to retrieve the parsed data: create permanent download links, send parsed data with webhooks, and fetch parsed data with their API. I went for the latter. Here's a working script:

Code: Select all

oHTTP:=ComObjCreate("WinHttp.WinHttpRequest.5.1")
If !IsObject(oHTTP)
{
  MsgBox,4112,Fatal Error,Unable to create HTTP object
  ExitApp
}
ResponseFileDownload:="c:\temp\DocparserDownload.txt"
FileDelete,%ResponseFileDownload%
UsernamePasswordBase64:="this is your Base64-encoded username with a null password"
ParserID:="this is the unique ID of your parser"
DocumentID:="this is the unique ID of your document"
URL:="https://api.docparser.com/v1/results/" . ParserID . "/" . DocumentID
oHTTP.Open("GET",URL)
oHTTP.SetRequestHeader("Authorization","Basic " UsernamePasswordBase64)
oHTTP.Send()
Response:=oHTTP.ResponseText
FileAppend,%Response%,%ResponseFileDownload%
Run,%ResponseFileDownload%
ExitApp
The response text is a long string with key-value pairs (nested JSON objects). The beginning has some header info, such as "document_id":"1234567890abcdefghijklmnopqrstuv" and "file_name":"inputfile.pdf". Then it has the parsed data, such as "first_name":"JOHN","last_name":"SMITH","address":"123 MAIN ST","city":"PHILADELPHIA","state":"PA","zip_code":"19102-1234", etc.

In summary, this thread shows how to call the API to authenticate, upload the input PDF, and download the parsed data. My thanks to Nextron and tmplinshi, without whose help this would not have been possible. Regards, Joe
JoeWinograd
Posts: 1073
Joined: 10 Feb 2014, 20:00

Re: REST API basic authentication via COM WinHttp.WinHttpRequest.5.1

28 May 2018, 15:20

Here's another post for anyone following this thread and interested in using AHK with Docparser. My previous post shows how to download the parsed data as key-value pairs (nested JSON objects). It also mentions that there's some header data (before the actual parsed data from the PDF file) and it turns out that the header data contains a link to the actual Excel file that works perfectly with AHK's UrlDownloadToFile. The key is media_link_data and the value following it is the URL of the XLSX file that contains the parsed data. A minor modification of one of the scripts above is able to download the XLSX file, as shown in the tested script below:

Code: Select all

oHTTP:=ComObjCreate("WinHttp.WinHttpRequest.5.1")
If !IsObject(oHTTP)
{
  MsgBox,4112,Fatal Error,Unable to create HTTP object
  ExitApp
}
ExcelFileDownload:="c:\temp\DocparserDownload.xlsx"
FileDelete,%ExcelFileDownload%
ResponseFileDownload:="c:\temp\DocparserDownload.txt"
FileDelete,%ResponseFileDownload%
UsernamePasswordBase64:="this is your Base64-encoded username with a null password"
ParserID:="this is the unique ID of your parser"
DocumentID:="this is the unique ID of your document"
URL:="https://api.docparser.com/v1/results/" . ParserID . "/" . DocumentID
oHTTP.Open("GET",URL)
oHTTP.SetRequestHeader("Authorization","Basic " UsernamePasswordBase64)
oHTTP.Send()
Response:=oHTTP.ResponseText
FileAppend,%Response%,%ResponseFileDownload%
MediaLinkStr:="""media_link_data"":""" ; key for Excel URL
MediaLinkPos:=InStr(Response,MediaLinkStr)
If (MediaLinkPos=0)
{
  MsgBox,4112,Fatal Error,%MediaLinkStr% not found in response
  ExitApp
}
MediaLinkStrLen:=StrLen(MediaLinkStr)
ExcelLinkBegin:=MediaLinkPos+MediaLinkStrLen
ExcelLinkEnd:=InStr(Response,"""",,ExcelLinkBegin)-1
If (ExcelLinkEnd=0)
{
  MsgBox,4112,Fatal Error,Closing quote on Excel URL not found in response
  ExitApp
}
ExcelLinkLen:=ExcelLinkEnd-ExcelLinkBegin+1
ExcelLink:=SubStr(Response,ExcelLinkBegin,ExcelLinkLen)
ExcelLink:=StrReplace(ExcelLink,"\") ; backslash is escape char for forward slashes in URL - remove all
UrlDownloadToFile,%ExcelLink%,%ExcelFileDownload%
If (ErrorLevel<>0)
{
  MsgBox,4112,Fatal Error,Error Level=%ErrorLevel% trying to download:`n%ExcelLink%`nto:`n%ExcelFileDownload%
  ExitApp
}
; Excel file was downloaded
ExitApp
My plan is to turn all the code developed in this thread into functions, something along these lines:

Code: Select all

BasicAuthentication(UsernamePasswordBase64)
ListParsers(UsernamePasswordBase64)
UploadPDF(UsernamePasswordBase64,ParserID,Filename)
DownloadParsedData(UsernamePasswordBase64,ParserID,DocumentID,JSONfilename,XLSXfilename)
I also plan to make it more robust, with lots of error checking, including Try-Catch pairs where appropriate. But that's for the future with an unknown ETA. In the meantime, I hope that the code in here helps anyone looking to call the Docparser API with AHK. Regards, Joe
User avatar
kczx3
Posts: 668
Joined: 06 Oct 2015, 21:39

Re: REST API basic authentication via COM WinHttp.WinHttpRequest.5.1

28 May 2018, 20:21

Seems like a class might be better to group data that is to be used by all your functions
JoeWinograd
Posts: 1073
Joined: 10 Feb 2014, 20:00

Re: REST API basic authentication via COM WinHttp.WinHttpRequest.5.1

28 May 2018, 22:10

I've never coded a class...certainly willing to learn about it. But I'm curious...why would you recommend a class instead of functions?
tmplinshi
Posts: 1278
Joined: 01 Oct 2013, 14:57

Re: REST API basic authentication via COM WinHttp.WinHttpRequest.5.1

28 May 2018, 23:59

You should use json library to parse the response data. For example:

Code: Select all

obj := Jxon_Load(oHTTP.ResponseText)
MsgBox % obj.1.media_link_data
UrlDownloadToFile, % obj.1.media_link_data, %ExcelFileDownload%
JoeWinograd
Posts: 1073
Joined: 10 Feb 2014, 20:00

Re: REST API basic authentication via COM WinHttp.WinHttpRequest.5.1

29 May 2018, 02:34

Hi tmplinshi,
I was not aware of Jxon.ahk (or JSON.ahk). Thanks for the tip...Jxon works a charm! Regards, Joe
User avatar
kczx3
Posts: 668
Joined: 06 Oct 2015, 21:39

Re: REST API basic authentication via COM WinHttp.WinHttpRequest.5.1

29 May 2018, 07:34

JoeWinograd wrote:I've never coded a class...certainly willing to learn about it. But I'm curious...why would you recommend a class instead of functions?
First, because all those functions are operating on a common concept/purpose. Second, because they all accept some of the same parameters. So you would make a class and then when you create an instance of it, pass those shared parameters. Below is just pseudo code.

Code: Select all

myObj := new DocParser(username, password)
parsers := myObj.ListParsers()
for each, parser in parsers {
    MsgBox, % parser.ParserID
}

uploadedFile := myObj.UploadPDF(parsers[x].ParserID, filename)

myObj.DownloadParsedData(parsers[x].ParserID, uploadedFile.DocumentID, JSONfilename, XLSXfilename)
JoeWinograd
Posts: 1073
Joined: 10 Feb 2014, 20:00

Re: REST API basic authentication via COM WinHttp.WinHttpRequest.5.1

29 May 2018, 09:08

Hi kczx3,
Thanks for the explanation and the pseudo-code. I'll give it a good look when I move forward with the next phase of the project. Regards, Joe
JoeWinograd
Posts: 1073
Joined: 10 Feb 2014, 20:00

Re: REST API basic authentication via COM WinHttp.WinHttpRequest.5.1

30 May 2018, 17:46

Hi tmplinshi (or anyone else who cares to jump in),

I'm hoping that you can help me to understand this line of code in the snippet that you provided:

objParam := { "file": [UploadFile] }

In particular, I need to add parameters in addition to "file". For example, one of the params to the API is remote_id. I looked in the Objects doc, but didn't find it, so I experimented. I started with this:

Code: Select all

UploadFileRemoteID:="this is the remote_id"
objParam:={"file":[UploadFile],"remote_id":[UploadFileRemoteID]}
That did not work. Then I tried this:

objParam:={"file":[UploadFile],"remote_id":UploadFileRemoteID}

That worked, but I'd like to understand why, i.e., why are the square brackets required around [UploadFile] for the "file" param, but not around UploadFileRemoteID for the "remote_id" param. There are many other params in the API, so I need to understand, in general, how to pass them in objParam. Did I miss this in the AHK doc? Thanks again, Joe
tmplinshi
Posts: 1278
Joined: 01 Oct 2013, 14:57

Re: REST API basic authentication via COM WinHttp.WinHttpRequest.5.1

30 May 2018, 18:12

Hi JoeWinograd,

Only files are needed to enclose with [] in the objParam, this is how CreateFormData function understands if it is a string or you want to upload a file. There is an example and description in the CreateFormData topic.
JoeWinograd
Posts: 1073
Joined: 10 Feb 2014, 20:00

Re: REST API basic authentication via COM WinHttp.WinHttpRequest.5.1

30 May 2018, 18:22

this is how CreateFormData function understands if it is a string or you want to upload a file
Ah, now I get it!
There is an example and description in the CreateFormData topic.
I'll check it out...had been looking at the AHK doc. Thanks much! Regards, Joe

Return to “Ask For Help”

Who is online

Users browsing this forum: cadudesun, Gh0sTG0, Google [Bot], HeadHunter1987 and 66 guests