Alternate Data Stream Gibberish Topic is solved

Get help with using AutoHotkey and its commands and hotkeys
zcooler
Posts: 455
Joined: 11 Jan 2014, 04:59

Alternate Data Stream Gibberish  Topic is solved

11 Jun 2018, 10:06

Hi!

I started scripting in PowerShell to handle Alternate Data Streams (ADS) natively. Using the Cmdlets to add, get and clear the streams works nicely. What not works very well is to read the NTFS file property streams who previously was written by the IPropertyStorage.WriteMultiple COM interface with PowerShell cmdlets. For example I get gibberish when reading this:

Code: [Select all] [Download] GeSHi © Codebox Plus

PS C:\Users> $file = 'F:\Ny mapp\20180116_20-59-02_Discovery Channel_Gold Rush  Alaska - S08E03 - “Busted And Bushfixed”.ts'
Get-Content -Path $file -Stream SummaryInformation
਀Ȁ鿲累栐ꮑࠀ⬧동Ā 鿲累栐ꮑࠀ⬧동  ࠃ ࠀ Ā 䠀 倀 ̀ 堀 ؀ ꀀ Ȁ Ⰲ Ѐ 堂 Ԁ 簂 ऀ  Ȁ 뀄 ጀ ᴄ ἀ   匀 㠀䔀 ㌀ ⴀ ᰠ䈀甀猀琀攀搀 䄀渀搀 䈀甀猀栀昀椀砀攀搀ᴠ ἀ 숀 䘀爀 愀琀琀 欀甀渀渀愀 氀攀瘀攀爀攀爀愀 猀琀漀爀愀 琀爀愀渀猀瀀漀爀琀攀爀 最甀氀搀 戀攀栀瘀攀爀ꀀ最爀瘀愀爀渀愀 栀氀氀愀 椀最渀最
 猀椀渀 甀琀爀甀猀琀渀椀渀最⸀ 伀挀栀 猀 猀攀爀 最甀氀搀最爀瘀愀爀渀愀 琀椀氀氀戀愀欀愀 瀀 猀椀渀愀 瘀爀猀琀愀ꀀ洀椀猀猀搀攀渀 漀挀栀 洀攀猀琀 栀攀爀漀椀猀欀愀 椀渀猀愀琀猀攀爀⠀渀⤀ഀ਀䜀攀渀爀攀㨀 猀攀爀椀攀ഀ਀ ἀ ሀ 䜀漀氀搀 刀甀猀栀㨀 䄀氀愀猀欀愀 ἀ ฀ 䐀嘀䈀嘀椀攀眀攀爀 倀爀漀 ἀ ⼀ 䐀愀琀攀㨀 ㄀㘀⸀ ㄀⸀㈀ ㄀㠀 
㈀㄀㨀  㨀   ⴀ 䐀甀爀愀琀椀漀渀㨀  ㄀㨀  㨀   ἀ ഀ 䌀漀渀琀攀渀琀䤀䐀㨀  
I do wonder if anyone might have any ideas on how to get readable results? Applying PS output encoding does not work. The IPropertyStorage interface cannot be scrapped before its possible to read the NTSF file property it has written on thousands of recordings.
User avatar
nnnik
Posts: 3072
Joined: 30 Sep 2013, 01:01
Location: Germany

Re: Alternate Data Stream Gibberish

11 Jun 2018, 14:37

That allows for 2 possibilities A) wrong encoding B) this is not string data
Recommends AHK Studio
zcooler
Posts: 455
Joined: 11 Jan 2014, 04:59

Re: Alternate Data Stream Gibberish

11 Jun 2018, 14:59

What do you think about a third possibility, which is my theory? IPropertyStorage uses PROPVARIANT structure in the ReadMultiple and WriteMultiple methods and those are a real pain in the neck to understand. I think solving this may require lexikos level knowledge.
User avatar
nnnik
Posts: 3072
Joined: 30 Sep 2013, 01:01
Location: Germany

Re: Alternate Data Stream Gibberish

11 Jun 2018, 15:10

Oh sorry I ended up saying the wrong thing - corrected it.
It was supposed to say "this is not string data" instead of "this is not binary data".
Yeah that struct is not that difficult to understand. The vt value at the beginning of the struct tells you which type the union is going to have.
Actually turning that into something that you can use with NumPut is another matter though.
Recommends AHK Studio
zcooler
Posts: 455
Joined: 11 Jan 2014, 04:59

Re: Alternate Data Stream Gibberish

11 Jun 2018, 16:07

nnnik wrote:Actually turning that into something that you can use with NumPut is another matter though.
Well, that would require someone who actually knows what he/she is doing, which im clearly not in this area ;)
User avatar
Flipeador
Posts: 959
Joined: 15 Nov 2014, 21:31
GitHub: Flipeador
Location: Argentina
Contact:

Re: Alternate Data Stream Gibberish

15 Jun 2018, 07:34

Hello, Could you explain better?. I'm not sure what you're looking for.

Code: [Select all] [Expand] [Download] (Untitled.ahk)GeSHi © Codebox Plus



Does this have something to do with what you want to do?:
https://msdn.microsoft.com/en-us/library/windows/desktop/aa379964
https://msdn.microsoft.com/en-us/library/windows/desktop/aa380017
https://msdn.microsoft.com/en-us/library/windows/desktop/aa380342
Windws 1♂ Pro 64-Bits I make scripts for AHKv2 (my v2 compiler) & WIN_7+ Spanish Argentina SublimeText 3 & AHKv2 My GDI+ Library
zcooler
Posts: 455
Joined: 11 Jan 2014, 04:59

Re: Alternate Data Stream Gibberish

15 Jun 2018, 16:57

Hello!

Your test illustrates the problem pretty well. AHK can read the streams AHK writes. AHK can read ADS native streams written by powershell, but reads gibberish when attempting to read IPropertyStorage interface streams. Powershell can read streams created with the IPropertyStorage COM interface, but it's gibberish (see first post the chinese looking chars). Yes, your links does describe the mega-awkward IPropertyStorage interface. The software I wanna change now writes NTSF file properties with the IPropertyStorage interface and has gotten unstable in the write phase (memory corruption with a following crash on some systems). Hence I wanna scrap the IPropertyStorage write and read interface and go PS ADS native. But if PS can't read what the IPropertyStorage interface has written previously there is little point with pursuing such a change.

Its problematic to provide a sample with these IPropertyStorage streams cuz they easily disappear when downloaded.
User avatar
Flipeador
Posts: 959
Joined: 15 Nov 2014, 21:31
GitHub: Flipeador
Location: Argentina
Contact:

Re: Alternate Data Stream Gibberish

15 Jun 2018, 17:21

[...] but reads gibberish when attempting to read IPropertyStorage interface streams.

how do you get these "IPropertyStorage interface streams"?. How do you know that?. Could you upload a file or something?.

Edit: Can you see the streams using AlternateStreamView (or ADS Manager)?

Edit2: Here is a function to enumerate the streams.

Code: [Select all] [Expand] [Download] (Untitled.ahk)GeSHi © Codebox Plus

Windws 1♂ Pro 64-Bits I make scripts for AHKv2 (my v2 compiler) & WIN_7+ Spanish Argentina SublimeText 3 & AHKv2 My GDI+ Library
zcooler
Posts: 455
Joined: 11 Jan 2014, 04:59

Re: Alternate Data Stream Gibberish

16 Jun 2018, 01:22

Flipeador wrote:
[...] but reads gibberish when attempting to read IPropertyStorage interface streams.

how do you get these "IPropertyStorage interface streams"?.

By using the PowerShell cmdlet Get-Item:

Code: [Select all] [Download] GeSHi © Codebox Plus

Get-Item -Path 'F:\Ny mapp\20180116_20-59-02_Discovery Channel_Gold Rush  Alaska - S08E03 - “Busted And Bushfixed”.ts' -Stream *

Flipeador wrote:How do you know that?.
Look at my first code box. Cmdlet Get-Content retrieves the content of the SummaryInformation stream.

Flipeador wrote:Could you upload a file or something?.
Yes, I can try. Although this recording is 1GB in size and I have to find some way to preserve the streams, which normally are deleted once uploaded to a filesharing site (non-NTSF filesystem).

Flipeador wrote:Edit: Can you see the streams using AlternateStreamView (or ADS Manager)?
With ADS Manager yes and if watching the HEX preview the SummaryInformation stream is readable :)

Flipeador wrote:Edit2: Here is a function to enumerate the streams.

Code: [Select all] [Expand] [Download] (Untitled.ahk)GeSHi © Codebox Plus

Does not work. In fact it's harmful and deletes the $DATA stream which is the video. File goes from 1GB to 1kb using it :(
User avatar
nnnik
Posts: 3072
Joined: 30 Sep 2013, 01:01
Location: Germany

Re: Alternate Data Stream Gibberish

16 Jun 2018, 01:46

Any kind of editing of the file will destroy all Alternate Data Streams.
Recommends AHK Studio
zcooler
Posts: 455
Joined: 11 Jan 2014, 04:59

Re: Alternate Data Stream Gibberish

16 Jun 2018, 02:54

https://files.fm/u/bppxcvxy#_

If I managed to preserve the streams on this recording it should contain:
PS C:\Users\> Get-Item -Path 'F:\Ny mapp\20180116_20-59-02_Discovery Channel_Gold Rush Alaska - S08E03 - “Busted And Bushfixed”.ts' -Stream *

PSPath : Microsoft.PowerShell.Core\FileSystem::F:\Ny mapp\20180116_20-59-02_Discovery Channel_Gold Rush Alaska - S08E03 - “Busted And Bushfixed”.ts::$DATA
PSParentPath : Microsoft.PowerShell.Core\FileSystem::F:\Ny mapp
PSChildName : 20180116_20-59-02_Discovery Channel_Gold Rush Alaska - S08E03 - “Busted And Bushfixed”.ts::$DATA
PSDrive : F
PSProvider : Microsoft.PowerShell.Core\FileSystem
PSIsContainer : False
FileName : F:\Ny mapp\20180116_20-59-02_Discovery Channel_Gold Rush Alaska - S08E03 - “Busted And Bushfixed”.ts
Stream : :$DATA
Length : 1111831436

PSPath : Microsoft.PowerShell.Core\FileSystem::F:\Ny mapp\20180116_20-59-02_Discovery Channel_Gold Rush Alaska - S08E03 - “Busted And Bushfixed”.ts:DocumentSummaryInformat
ion
PSParentPath : Microsoft.PowerShell.Core\FileSystem::F:\Ny mapp
PSChildName : 20180116_20-59-02_Discovery Channel_Gold Rush Alaska - S08E03 - “Busted And Bushfixed”.ts:DocumentSummaryInformation
PSDrive : F
PSProvider : Microsoft.PowerShell.Core\FileSystem
PSIsContainer : False
FileName : F:\Ny mapp\20180116_20-59-02_Discovery Channel_Gold Rush Alaska - S08E03 - “Busted And Bushfixed”.ts
Stream : DocumentSummaryInformation
Length : 804

PSPath : Microsoft.PowerShell.Core\FileSystem::F:\Ny mapp\20180116_20-59-02_Discovery Channel_Gold Rush Alaska - S08E03 - “Busted And Bushfixed”.ts:SebiesnrMkudrfcoIaamtyk
dDa
PSParentPath : Microsoft.PowerShell.Core\FileSystem::F:\Ny mapp
PSChildName : 20180116_20-59-02_Discovery Channel_Gold Rush Alaska - S08E03 - “Busted And Bushfixed”.ts:SebiesnrMkudrfcoIaamtykdDa
PSDrive : F
PSProvider : Microsoft.PowerShell.Core\FileSystem
PSIsContainer : False
FileName : F:\Ny mapp\20180116_20-59-02_Discovery Channel_Gold Rush Alaska - S08E03 - “Busted And Bushfixed”.ts
Stream : SebiesnrMkudrfcoIaamtykdDa
Length : 148

PSPath : Microsoft.PowerShell.Core\FileSystem::F:\Ny mapp\20180116_20-59-02_Discovery Channel_Gold Rush Alaska - S08E03 - “Busted And Bushfixed”.ts:SummaryInformation
PSParentPath : Microsoft.PowerShell.Core\FileSystem::F:\Ny mapp
PSChildName : 20180116_20-59-02_Discovery Channel_Gold Rush Alaska - S08E03 - “Busted And Bushfixed”.ts:SummaryInformation
PSDrive : F
PSProvider : Microsoft.PowerShell.Core\FileSystem
PSIsContainer : False
FileName : F:\Ny mapp\20180116_20-59-02_Discovery Channel_Gold Rush Alaska - S08E03 - “Busted And Bushfixed”.ts
Stream : SummaryInformation
Length : 824

PSPath : Microsoft.PowerShell.Core\FileSystem::F:\Ny mapp\20180116_20-59-02_Discovery Channel_Gold Rush Alaska - S08E03 - “Busted And Bushfixed”.ts:{4c8cc155-6c1e-11d1-8e41
-00c04fb9386d}
PSParentPath : Microsoft.PowerShell.Core\FileSystem::F:\Ny mapp
PSChildName : 20180116_20-59-02_Discovery Channel_Gold Rush Alaska - S08E03 - “Busted And Bushfixed”.ts:{4c8cc155-6c1e-11d1-8e41-00c04fb9386d}
PSDrive : F
PSProvider : Microsoft.PowerShell.Core\FileSystem
PSIsContainer : False
FileName : F:\Ny mapp\20180116_20-59-02_Discovery Channel_Gold Rush Alaska - S08E03 - “Busted And Bushfixed”.ts
Stream : {4c8cc155-6c1e-11d1-8e41-00c04fb9386d}
Length : 0
User avatar
Flipeador
Posts: 959
Joined: 15 Nov 2014, 21:31
GitHub: Flipeador
Location: Argentina
Contact:

Re: Alternate Data Stream Gibberish

16 Jun 2018, 14:28

Does not work. In fact it's harmful and deletes the $DATA stream which is the video. File goes from 1GB to 1kb using it

Why does not it work? Could you explain? What version of AHK do you use?


I will be seeing, when I have time, everything else. :wave:
Windws 1♂ Pro 64-Bits I make scripts for AHKv2 (my v2 compiler) & WIN_7+ Spanish Argentina SublimeText 3 & AHKv2 My GDI+ Library
User avatar
nnnik
Posts: 3072
Joined: 30 Sep 2013, 01:01
Location: Germany

Re: Alternate Data Stream Gibberish

16 Jun 2018, 15:09

You only extract the $DATA stream types. I think the IProperty Stream is not a $DATA stream type.
Recommends AHK Studio
User avatar
Flipeador
Posts: 959
Joined: 15 Nov 2014, 21:31
GitHub: Flipeador
Location: Argentina
Contact:

Re: Alternate Data Stream Gibberish

16 Jun 2018, 15:39

Interesting, it will be fun to try to make this work. :)
Windws 1♂ Pro 64-Bits I make scripts for AHKv2 (my v2 compiler) & WIN_7+ Spanish Argentina SublimeText 3 & AHKv2 My GDI+ Library
zcooler
Posts: 455
Joined: 11 Jan 2014, 04:59

Re: Alternate Data Stream Gibberish

16 Jun 2018, 15:44

Flipeador wrote:
Does not work. In fact it's harmful and deletes the $DATA stream which is the video. File goes from 1GB to 1kb using it

Why does not it work? Could you explain? What version of AHK do you use?
The enumeration function works nicely but writing the streams like that causes mayhem with the $DATA stream and it's deleted. Try it with the file i linked for download. It's this part that goes wrong:

Code: [Select all] [Download] GeSHi © Codebox Plus

FileOpen(File, "w").Write("Stream #0")
FileOpen(File . ":stream1", "w").Write("Stream #1")
FileOpen(File . ":stream2", "w").Write("Stream #2")


Flipeador wrote:I will be seeing, when I have time, everything else. :wave:
Thanks :)

nnnik wrote:You only extract the $DATA stream types. I think the IProperty Stream is not a $DATA stream type.
Yes, the file im working with is a transport stream (.TS) and it seems all the video and audio are located to the $DATA stream (see the large Length value for that stream).

The IProperty Streams are:
DocumentSummaryInformation
SebiesnrMkudrfcoIaamtykdDa
SummaryInformation
...and they contain all sorts of EPG data for the tv-series, recordings date, duration and much more. This data is used when rebuilding the SQL recordings database.
Last edited by zcooler on 16 Jun 2018, 15:52, edited 2 times in total.
User avatar
nnnik
Posts: 3072
Joined: 30 Sep 2013, 01:01
Location: Germany

Re: Alternate Data Stream Gibberish

16 Jun 2018, 15:46

Code: [Select all] [Download] GeSHi © Codebox Plus

FileOpen(File, "w").Write("Stream #0") ;as soon as you change the main stream like this windows will delete all alternate data streams.
FileOpen(File . ":stream1", "w").Write("Stream #1")
FileOpen(File . ":stream2", "w").Write("Stream #2")
Recommends AHK Studio
User avatar
nnnik
Posts: 3072
Joined: 30 Sep 2013, 01:01
Location: Germany

Re: Alternate Data Stream Gibberish

16 Jun 2018, 15:54

It works with the rw flag though:

Code: [Select all] [Download] GeSHi © Codebox Plus

File := "test.dat"
FileOpen(File, "w").Write("Stream #0") ;as soon as you change the main stream like this windows will delete all alternate data streams.
FileOpen(File . ":stream1", "w").Write("Stream #1")
FileOpen(File . ":stream2", "w").Write("Stream #2")

Msgbox % FileOpen(File . ":stream1", "r").Read()
FileOpen(File, "rw").Write("Stream #0") ;replace w with rw or vice versa to see the different results
Msgbox % FileOpen(File . ":stream1", "r").Read()
Msgbox % FileOpen(File, "r").Read()
Recommends AHK Studio
User avatar
Flipeador
Posts: 959
Joined: 15 Nov 2014, 21:31
GitHub: Flipeador
Location: Argentina
Contact:

Re: Alternate Data Stream Gibberish

16 Jun 2018, 15:57

as soon as you change the main stream like this windows will delete all alternate data streams

Sure, all streams are deleted if you overwrite the file (main stream)...
Windws 1♂ Pro 64-Bits I make scripts for AHKv2 (my v2 compiler) & WIN_7+ Spanish Argentina SublimeText 3 & AHKv2 My GDI+ Library
User avatar
Flipeador
Posts: 959
Joined: 15 Nov 2014, 21:31
GitHub: Flipeador
Location: Argentina
Contact:

Re: Alternate Data Stream Gibberish

24 Jun 2018, 10:12

I'm going to put here the code that you have sent me by private message:

Just today I started to review it, and this is what I have so far:
Note: The code is not suitable for normal use, it is only a test.

Code: [Select all] [Expand] [Download] (Untitled.ahk)GeSHi © Codebox Plus


I will be reviewing it better, what I am seeing at the moment, is that there is a Stream that seems not to recover (3 of 4; I see one with size 0 in ADS Manager).
Windws 1♂ Pro 64-Bits I make scripts for AHKv2 (my v2 compiler) & WIN_7+ Spanish Argentina SublimeText 3 & AHKv2 My GDI+ Library
zcooler
Posts: 455
Joined: 11 Jan 2014, 04:59

Re: Alternate Data Stream Gibberish

24 Jun 2018, 10:58

I get the following error message executing that code:
---------------------------
proptest.ahk
---------------------------
Error: Memory limit reached (see #MaxMem in the help file).

Line#
036: {
037: propid := NumGet(&STATPROPSTG + A_PtrSize, "UInt")
039: VarSetCapacity(PROPSPEC, 2*A_PtrSize)
040: NumPut(1, &PROPSPEC, "UInt")
041: NumPut(propid, &PROPSPEC+A_PtrSize, "UInt")
043: VarSetCapacity(PROPVARIANT, A_PtrSize == 4 ? 16 : 24)
046: DllCall(NumGet(NumGet(IPropertyStorage+0)+3*A_PtrSize), "UPtr", IPropertyStorage, "UInt", 1, "UPtr", &PROPSPEC, "UPtr", &PROPVARIANT, "UInt")
---> 047: List .= StrGet(NumGet(&PROPVARIANT + 8, "UPtr"), "UTF-16") . "
--------------------
"
048: DllCall("Ole32.dll\PropVariantClear", "UPtr", &PROPVARIANT)
049: }
051: MsgBox,Str . "

" . List
052: }
053: Exit
054: Exit
054: Exit

The current thread will exit.
---------------------------
OK
---------------------------

Return to “Ask For Help”

Who is online

Users browsing this forum: Froggie, TAC109 and 41 guests