Convert string in clipboard from Unicode to specific codepage

[SVN]

I have to copy non-english text from a program that uses Unicode to another one that does not know what Unicode is (in my case the text should always be pasted as CP1251).

How such a conversion can be done with AHK (Unicode flavour)? Algorithm is something like this:
1) copy text to the clipboard normally;
2) in case Ctrl-V is hit is specific program, perform the said conversion of clipboard content prior to pasting.

I suppose it has everything to do with functions Strput/Strget, but I can not understand how they work.

[jeeswg]

- Did you try pasting text into the program using ctrl+v?
- The clipboard has many clipboard formats, i.e. different bits of data, all on the clipboard at the same time.
- So you can have some Unicode text, and ANSI text on the clipboard at the same time. And the program can check the numbers for each clipboard format, and see if it can find a clipboard format it can handle e.g. ANSI text. E.g. CF_TEXT := 0x1, CF_OEMTEXT := 0x7, CF_UNICODETEXT := 0xD.
- I would have thought that perhaps AHK Unicode would put multiple versions of the same string onto the clipboard, in different clipboard formats.
- The script below may or may not be useful. You can put text onto the clipboard, and replace characters not in the CP1251 codepage with question marks.
- If ctrl+v doesn't work, you can try SendInput or using SendMessage with WM_CHAR. Note: when you SendInput or use WM_CHAR, CRLFs send enter twice, so you may want to replace CRLFs with LFs before you send text.
- Btw I found StrGet/StrPut very confusing when I first started trying to use them. For example I thought that StrGet might be needed to 'get' the size of the necessary variable, but this is incorrect, you use StrPut to get the necessary size. Plus the use of 'smart' parameters is usually a bad thing, but with these functions it actually works very well.

Code: Select all

;StrPut can be used in AHK Unicode for UTF-16 text to ANSI/UTF-8 data
;use it with no address parameter to retrieve the size of a variable needed to hold the ANSI/UTF-8 data, the size returned includes a null
;use it with an address parameter to write ANSI/UTF-8 data to a variable
;StrGet can be used in AHK Unicode for ANSI/UTF-8 to UTF-16, to convert ANSI/UTF-8 data to a standard UTF-16 string that AHK Unicode can handle

;change the #IfWinActive line to match your program, e.g. use AutoHotkey's window spy to find out the class
#IfWinActive, ahk_class Notepad
;#IfWinActive, ahk_exe notepad.exe
q::
vText := Chr(8730) "abc" Chr(8730) ;Chr(8730) is the square root symbol
vSize := StrPut(vText, "CP1251")
VarSetCapacity(vTemp, vSize)
StrPut(vText, &vTemp, "CP1251")
Clipboard := StrGet(&vTemp, "CP1251")
;MsgBox, % Clipboard
SendInput, ^v
return
#IfWinActive

[SVN]

@jeeswg Thank you very much for your thorough explanation and the script.

Surely, the first thing I tried was to paste from the clipboard. That did not work out for the the case of program I needed to copy from. All non-english characters pasted were replaced with question marks. But I can see what you mean. If I try to copy text from browser, then everything looks just fine when pasted.

Needless to say I tried your script eagerly. Unfortunately, still no luck. It pastes «?abc?» (without quotes). I checked page https://msdn.microsoft.com/en-us/library/ms646276.aspx describing WM_CHAR... well, not sure I can put it to use. Looks like some hardcore programming stuff, at least for me (not exactly my field).

[jeeswg]

These are the characters in CP1251, only those characters should remain after you paste text into your program, all other characters should appear as question marks.
Windows-1251 - Wikipedia
https://en.wikipedia.org/wiki/Windows-1251
However, perhaps the program is set to be CP1252, in which case only the characters in CP1252 will remain, and won't be question marks.

So there are two questions:
- Have you ever seen CP1251 characters, i.e. Cyrillic characters, inside the program you use?
- Is there a way to make an ANSI program CP1251 compatible instead of CP1252.
Cheers.

[Helgef]

Code: Select all

Clipboard := StrGet(&vTemp, "CP1251")

See the remarks on strget. I suggest you try to strput to a variable, as you do, and then set the clipboard data via dllcall instead of using strget.

Cheers.

[jeeswg]

- A possible answer to 'Is there a way to make an ANSI program CP1251 compatible instead of CP1252.' Try searching the Internet for 'set system codepage'. This might be a solution.
- If you change one line in my script above to:
Clipboard := StrGet(&vTemp, "CP1252") ;1252 not 1251
That should put the right characters onto the clipboard, if your current system codepage is CP1252. Although when you paste them they will not appear Cyrillic, the character numbers should be correct.
- If you'd done a search for 'AutoHotkey WM_CHAR' you might have found some scripts. Unfortunately I didn't my see mine show up. There's an example here:
Double quote symbols appear as @ symbols in Command Prompt window - AutoHotkey Community
https://autohotkey.com/boards/viewtopic ... 18#p181518
- To get all 255 ANSI characters:

Code: Select all

;[based on: ANSI AND UNICODE CHARACTERS (STRGET AND STRPUT)]
;jeeswg's characters tutorial - AutoHotkey Community
;https://autohotkey.com/boards/viewtopic.php?t=26486

q:: ;get 255 ANSI characters (in AHK Unicode versions)
VarSetCapacity(vText, 255, 0)
Loop, 255
	NumPut(A_Index, vText, A_Index-1, "UChar")
;vOutput := StrGet(&vText, 255, "CP0")
vOutput := StrGet(&vText, 255, "CP1251")
MsgBox, % Clipboard := vOutput
return

[SVN]

Sorry for late answer.

Q:
Have you ever seen CP1251 characters, i.e. Cyrillic characters, inside the program you use?
A:
Yes, absolutely. I can type Cyrillic characters in it and it works just as good as typing Latin ones. Moreover, just like I said, I can copy Cyrillic text from, say, browser and it looks fine when pasted. Despite of this I guess the program is set to be CP1252 indeed (the creators talked about plans to develop a Unicode version, but apparently no one was much interested in this).

Q:
A possible answer to 'Is there a way to make an ANSI program CP1251 compatible instead of CP1252.' Try searching the Internet for 'set system codepage'. This might be a solution.
A:
I have already done it some time ago. It did not make any difference though (at least to solve this specific problem).

Q:
If you change one line in my script above to:
Clipboard := StrGet(&vTemp, "CP1252") ;1252 not 1251
A:
It did not help either. Changed nothing actually. It is still «?abc?» (no quotes) when pasted.

Q:
If you'd done a search for 'AutoHotkey WM_CHAR' you might have found some scripts.
A:
You are right, but... I simply don't have a slightest idea how to use all that information (I am not a programmer by any means and I don't possess necessary skills). I run your latest script though. The results of its execution are as follows:
!"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~ЂЃ‚ѓ„…†‡€‰Љ‹ЊЌЋЏђ‘’“”•–—˜™љ›њќћџ ЎўЈ¤Ґ¦§Ё©Є«¬­®Ї°±Ііґµ¶·ё№є»јЅѕїАБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯабвгдежзийклмнопрстуфхцчшщъыьэюя

But I can see a weak glimpse of hope. I included single Cyrillic character in your test text line, so it became
vText := Chr(8730) "abcы" Chr(8730) ;Chr(8730) is the square root symbol
Being pasted it looked like «?abcС‹?». So the letter «ы» had been replaced with «С‹». No idea what it can mean, but at least it is not that lax question mark.

[jeeswg]

So this program, you can type Cyrillic characters into it just fine. But you if you type Cyrillic characters into Notepad, copy those characters and paste into the program, the Cyrillic characters become question marks?

Here's a script to try that uses WM_CHAR:

Code: Select all

q:: ;test send characters
WinGet, hWnd, ID, A
ControlGetFocus, vCtlClassNN, % "ahk_id " hWnd
ControlGet, hCtl, Hwnd,, % vCtlClassNN, % "ahk_id " hWnd
if hCtl
	hWnd := hCtl
vText := "АаБбВвГг"
Loop, Parse, vText
	PostMessage, 0x102, % Ord(A_LoopField), 1,, % "ahk_id " hWnd ;WM_CHAR := 0x102
return

Another approach:

Code: Select all

w:: ;convert CP1251 to CP1252 (possibly the program will read the CP1252 as Cyrillic)
vText := "АаБбВвГг"
vSize := StrPut(vText, "CP1251")
VarSetCapacity(vTemp, vSize)
StrPut(vText, &vTemp, "CP1251")
Clipboard := StrGet(&vTemp, "CP1252")
SendInput, ^v
;MsgBox, % Clipboard
return

e:: ;convert thE CP1252 back to CP1251
vText := Clipboard
vSize := StrPut(vText, "CP1252")
VarSetCapacity(vTemp, vSize)
StrPut(vText, &vTemp, "CP1252")
MsgBox, % StrGet(&vTemp, "CP1251")
return

Another approach:

Code: Select all

r:: ;clear the clipboard and put ANSI text onto it
vText := "АаБбВвГг"
vSize := StrPut(vText, "CP1251")

;GMEM_ZEROINIT := 0x40, GMEM_MOVEABLE := 0x2
hBuf := DllCall("kernel32\GlobalAlloc", UInt,0x42, UPtr,vSize, Ptr)
pBuf := DllCall("kernel32\GlobalLock", Ptr,hBuf, Ptr)
StrPut(vText, pBuf, "CP1251")

;CF_LOCALE := 0x10 ;CF_UNICODETEXT := 0xD
;CF_OEMTEXT := 0x7 ;CF_TEXT := 0x1
hWnd := A_ScriptHwnd ? A_ScriptHwnd : WinExist("ahk_pid " DllCall("kernel32\GetCurrentProcessId", UInt))
DllCall("kernel32\GlobalUnlock", Ptr,hBuf)
DllCall("user32\OpenClipboard", Ptr,hWnd)
DllCall("user32\EmptyClipboard")
DllCall("user32\SetClipboardData", UInt,0x1, Ptr,hBuf, Ptr)
DllCall("user32\CloseClipboard")

SendInput, ^v
return

[Helgef]

@ jeeswg, you cannot use strget like that. (w::...) Edit: I think I misunderstood.

---

You can use sendInput,

Code: Select all

f1::sw("АаБбВвГг")
sw(byref str){
	local ss
	loop parse, str
		ss .= "{u+" format("{:x}",ord(A_LoopField)) "}"
	sendInput(ss)
}

It will result in WM_CHAR msg if IRCC.

---

For the clipboard, you can combine CF_TEXT and CF_LOCALE, like this, I wrote it in v2, I leave it as an exercise for the interested reader to ponder why I fails in v1

Code: Select all

; v2
ru := 0x419
enc:="CP1251"
localePutClip("АаБбВвГг", ru, enc)

localePutClip(str, locale, enc) {
	static GMEM_ZEROINIT := 0x0040
	static CF_TEXT := 1
	static CF_LOCALE := 16
	local size
	if !size := strput(str, enc)																				; Encode string
		throw exception("Fail, strput 1.")
	varsetcapacity(tStr, size:=(size+1)*2, 0)
	if !strput(str, &tStr, enc)
		throw exception("Fail, strput 2.")
	return setClipboardData([CF_LOCALE, CF_TEXT], [4, size], [&locale, &tStr])									; setClipboardData
}
setClipboardData(format, bytes, data){
	static GMEM_MOVEABLE := 0x0002
	static GMEM_ZEROINIT := 0x0040
	local lMem, e
	if !dllcall("OpenClipboard", "ptr", A_ScriptHwnd)															; OpenClipboard
		throw exception("OpenClipboard failed: " ErrorLevel "`n" A_LastError)
	if !dllcall("EmptyClipboard")																				; EmptyClipboard
		throw exception("EmptyClipboard failed: " ErrorLevel "`n" A_LastError)
	if !isobject(format)
		format := [format], bytes := [bytes], data := [data]		
	try
		for k, CF in format {
			if !hMem := dllcall("GlobalAlloc", "uint", GMEM_MOVEABLE | GMEM_ZEROINIT, "ptr", bytes[k], "ptr")	; GlobalAlloc
				throw exception("GlobalAlloc failed: " ErrorLevel "`n" A_LastError)
			if !lMem := dllcall("GlobalLock", "ptr", hMem, "ptr")												; GlobalLock
				throw exception("GlobalLock failed: " ErrorLevel "`n" A_LastError)
			if !dllcall("MSVCRT.dll\memcpy", "ptr", lMem, "ptr", data[k], "ptr", bytes[k], "cdecl ptr")			; memcpy
				throw exception("memmove failed: " ErrorLevel "`n" A_LastError)
			if dllcall("GlobalUnlock", "ptr", hMem)																; GlobalUnlock
				throw exception("GlobalUnlock failed: " ErrorLevel "`n" A_LastError)
			if !dllcall("SetClipboardData", "uint", CF, "ptr", hMem, "ptr")										; SetClipboardData
				throw exception("SetClipboardData failed: " ErrorLevel "`n" A_LastError)
			if hMem:=dllcall("GlobalFree", "ptr", hMem, "ptr")													; GlobalFree
				throw exception("GlobalFree failed: " ErrorLevel "`n" A_LastError)
		}
	catch e
		throw e
	finally
		if !dllcall("CloseClipboard")																			; CloseClipboard
			throw exception("CloseClipboard failed: " ErrorLevel "`n" A_LastError)
		
	return true
}

[jeeswg]

@Helgef: What do you think is wrong with the w hotkey subroutine? The text displayed by the MsgBox was as expected, and the e hotkey subroutine confirms that it can be reversed.

[SVN]

@jeeswg
1) Hitting «q» results in pasting «???°?‘?±?’???“??»;
2) Hitting «w» results in pasting «РђР°Р‘бВвГг»;
3) Hitting «e» results in popping up message «РђР°Р‘бВвГг»;
4) Hitting «r» results in pasting «РђР°Р‘бВвГг».

@Helgef
5) Hitting «F1» results in pasting «???°?‘?±?’???“??»;
6) Not quite understand how to use v2.

[jeeswg]

Does this work?

Code: Select all

q::
vText := "АаБбВвГг"
;vText := Clipboard
vText := StrReplace(vText, "`r`n", "`n")
SendInput, % "{Raw}" vText
return

- So: typing normally works, ctrl+c in Notepad and ctrl+v in the program doesn't work.
- If you do ctrl+c in the program and ctrl+v Notepad, does that work with Cyrillic characters?
- If you do ctrl+c in the program and ctrl+v into the program itself, does that work with Cyrillic characters?
- What do you get when you do ctrl+c in Notepad and do ctrl+v in the program, with АаБбВвГг, what does it look like?
- What is the ClassNN of the control you are pasting into, does pasting work differently in the other controls. Use AutoHotkey's window spy to find out the ClassNN.

[SVN]

Script pastes «???°?‘?±?’??».

Q:
So: typing normally works, ctrl+c in Notepad and ctrl+v in program doesn't work.
A:
Correct.

Q:If you do ctrl+c in the program and ctrl+v Notepad, does that work with Cyrillic characters?
A:
It does.

Q:
If you do ctrl+c in the program and ctrl+v into the program itself, does that work Cyrillic characters?
A:
It does.

Q:
What do you get when do ctrl+c in Notepad and do ctrl+v in the program, with АаБбВвГг, what does it look like?
A:
I get «????????».

Q:
What is the ClassNN of the control you are pasting into, does pasting work differently in the other controls. Use AutoHotkey's window spy to find out the ClassNN.
A:
It is «TRichEdit1». There is one more text control and it behaves identically.

[jeeswg]

Does this script work? The getting and/or the setting.

Code: Select all

q::
ControlGetText, vText, TRichEdit1, A
;Clipboard := vText
MsgBox, % vText
return

w::
vText := "АаБбВвГг"
ControlSetText, TRichEdit1, % vText, A
return

[SVN]

It pastes «РђР°Р‘бВвГг». However...

There are unintended consequences for everything we do indeed. You did help me to solve the problem, albeit in the most unexpected way.

My boss approached me and asked why we have not been making any progress here. I replied that the problem was much more complicated, than it had been believed before, and demonstrated this post. His reply was «Why should we waste our time as well as that of others? Let's retire this program and ask guys from our IT department to install something Unicode-capable as replacement.» (he conveniently forgot his stubborn refusal to do it before, that's the nature of bosses, I presume ). So the problem was not solved but fade away instead...

Let me thank you once again, I really appreciate your time and efforts you made.

[Helgef]

6) Not quite understand how to use v2.

Ok, to be fair, this is for v1,

Code: Select all

; v1
ru := 0x419
enc:="CP1251"
; str := "АаБбВвГг"
str := chr(0x44A) chr(0x44B) chr(0x44C) ; ъыь
localePutClip(str, ru, enc)

localePutClip(str, locale, enc) {
	static GMEM_ZEROINIT := 0x0040
	static CF_TEXT := 1
	static CF_LOCALE := 16
	local size, loc
	if !size := strput(str, enc)																				; Encode string
		throw exception("Fail, strput 1.")
	varsetcapacity(tStr, size:=(size+1)*2, 0)
	if !strput(str, &tStr, enc)
		throw exception("Fail, strput 2.")
	varsetcapacity(loc, 4,0)
	numput(locale, loc, "ushort")
	return setClipboardData([CF_LOCALE, CF_TEXT], [4, size], [&loc, &tStr])										; setClipboardData
}
setClipboardData(format, bytes, data){
	static GMEM_MOVEABLE := 0x0002
	static GMEM_ZEROINIT := 0x0040
	local lMem, e
	if !dllcall("OpenClipboard", "ptr", A_ScriptHwnd)															; OpenClipboard
		throw exception("OpenClipboard failed: " ErrorLevel "`n" A_LastError)
	if !dllcall("EmptyClipboard")																				; EmptyClipboard
		throw exception("EmptyClipboard failed: " ErrorLevel "`n" A_LastError)
	if !isobject(format)
		format := [format], bytes := [bytes], data := [data]		
	try
		for k, CF in format {
			if !hMem := dllcall("GlobalAlloc", "uint", GMEM_MOVEABLE | GMEM_ZEROINIT, "ptr", bytes[k], "ptr")	; GlobalAlloc
				throw exception("GlobalAlloc failed: " ErrorLevel "`n" A_LastError)
			if !lMem := dllcall("GlobalLock", "ptr", hMem, "ptr")												; GlobalLock
				throw exception("GlobalLock failed: " ErrorLevel "`n" A_LastError)
			if !dllcall("MSVCRT.dll\memcpy", "ptr", lMem, "ptr", data[k], "ptr", bytes[k], "cdecl ptr")			; memcpy
				throw exception("memmove failed: " ErrorLevel "`n" A_LastError)
			if dllcall("GlobalUnlock", "ptr", hMem)																; GlobalUnlock
				throw exception("GlobalUnlock failed: " ErrorLevel "`n" A_LastError)
			if !dllcall("SetClipboardData", "uint", CF, "ptr", hMem, "ptr")										; SetClipboardData
				throw exception("SetClipboardData failed: " ErrorLevel "`n" A_LastError)
			if hMem:=dllcall("GlobalFree", "ptr", hMem, "ptr")													; GlobalFree
				throw exception("GlobalFree failed: " ErrorLevel "`n" A_LastError)
		}
	catch e
		throw e
	finally
		if !dllcall("CloseClipboard")																			; CloseClipboard
			throw exception("CloseClipboard failed: " ErrorLevel "`n" A_LastError)
		
	return true
}

Edit: Run it, then try to paste with your keyboard.
Glad your problems faded away .

---

@ jeeswg, I will answer you later.

[SVN]

@Helgef Nope, it does not work either. What is pasted still includes question marks in place of Cyrillic glyphs.

[Helgef]

Are you sure that the program uses CP1251? According to this page, these also have to do with Cyrillic characters too,

Code: Select all

enc:="CP1251"
enc:="CP866"
enc:="CP866"
enc:="CP28595"
enc:="CP10007"
enc:="CP20866"
enc:="CP20880"
enc:="CP21025"
enc:="CP21866"

@ jeeswg, can you make a script which gets the clipboard data, formats and such, and prints info about it, then SVN can make a copy in the program, and we can see what is there. If we could replicate that content, it should work right? Since SVN can copy from the program and paste back into it without issues, if I understood correctly.

Cheers.

[jeeswg]

Code: Select all

АаБбВвГг ;text that SVN reported
АаБбВвГг ;what I get if I convert АаБбВвГг to UTF-8 bytes on my PC

So I think we are dealing with UTF-8.
- ControlGetText, vText, TRichEdit1, A
Did this work? What did it show?
- Yes Helgef, I was thinking we could press ctrl+c in Notepad, and inspect the clipboard. And we could press ctrl+c in the program, and inspect the clipboard. I was thinking of using Free Clipboard Viewer and/or NirSoft InsideClipboard, both are under 1 MB. So what I would recommend doing is running either of those utilities, pressing ctrl+c in Notepad or the program, and looking at the clipboard information revealed by the programs.

[EDIT:] The w subroutine below, is another candidate script for trying to solve this problem.

Code: Select all

q:: ;UTF-8 bytes to Unicode text
vText := "АаБбВвГг"
vSize := StrPut(vText, "CP1251")
VarSetCapacity(vTemp, vSize)
StrPut(vText, &vTemp, "CP1251")
MsgBox, % StrGet(&vTemp, "UTF-8") ;АаБбВвГг

vText := "АаБбВвГг"
vSize := StrPut(vText, "CP1252")
VarSetCapacity(vTemp, vSize)
StrPut(vText, &vTemp, "CP1252")
MsgBox, % StrGet(&vTemp, "UTF-8") ;АаБбВвГг
return

w:: ;Unicode text to UTF-8 bytes
vText := "АаБбВвГг"
vSize := StrPut(vText, "UTF-8")
VarSetCapacity(vTemp, vSize)
StrPut(vText, &vTemp, "UTF-8")
;MsgBox, % StrGet(&vTemp, "CP1251") ;АаБбВвГг
;MsgBox, % StrGet(&vTemp, "CP1252") ;АаБбВвГг
Clipboard := StrGet(&vTemp, "CP1251")
return

- Btw what is the equivalent of 'abc' in your language, is there a phrase like 'easy as abc'? абвг? абг? авг?

[EDIT:]
- Interesting story. Cheers. These are classic encoding, language, clipboard and ANSI/Unicode issues, worth a bit of investigation.
- Another approach.

Code: Select all

q:: ;send text using SendInput and ASC
vText := "АаБбВвГг"
vSize := StrPut(vText, "CP1251")
VarSetCapacity(vTemp, vSize)
StrPut(vText, &vTemp, "CP1251")
Loop, % vSize-1
{
	vNum := NumGet(&vTemp, A_Index-1, "UChar")
	SendInput, % "{ASC 0" vNum "}"
}
return

w:: ;send text using SendInput and ASC
vText := "ÀÁÂÃÄÅ"
vSize := StrPut(vText, "CP1252")
VarSetCapacity(vTemp, vSize)
StrPut(vText, &vTemp, "CP1252")
Loop, % vSize-1
{
	vNum := NumGet(&vTemp, A_Index-1, "UChar")
	SendInput, % "{ASC 0" vNum "}"
}
return

[Helgef]

Good investigation jeeswg

@Helgef: What do you think is wrong with the w hotkey subroutine? The text displayed by the MsgBox was as expected, and the e hotkey subroutine confirms that it can be reversed.

Sorry jeeswg, I think I misunderstood

Cheers