Jump to content

Sky Slate Blueberry Blackcurrant Watermelon Strawberry Orange Banana Apple Emerald Chocolate
Photo

VarZ - NT Native Data Compression


  • Please log in to reply
20 replies to this topic
SKAN
  • Administrators
  • 9115 posts
  • Last active:
  • Joined: 26 Dec 2005

Code Updated: The wrapper was rewritten for AHK_Lw 32-bit compatibility.


VarZ_Compress() validates the var header before it begins compression to prevent double compression.
If you alter the var header after compression, you can compress it again to lose extra bytes.

Here is an example to crunch ahk.bmp ( alpha bitmap ) from 73.1 KiB to 7.22KiB ( Info on ahk.bmp ) :
#SingleInstance, Force
SetWorkingDir %A_ScriptDir%

IfNotExist, ahk.bmp
 URLDownloadToFile, http://tinyurl.com/skanbox/Sample/ahk.bmp, ahk.bmp

FileGetSize, nSize, ahk.bmp            ; 74934 bytes
FileRead, BMP, *c ahk.bmp
nSize2 := VarZ_Compress( BMP, nSize )  ; 10279 bytes after primary compression
NumPut( 0x315F5A4C, BMP )              ; change header from "LZ_" to "LZ_1"
nSize3 := VarZ_Compress( BMP, nSize2 ) ; 7403 bytes after secondary decompression
VarZ_Save( BMP, nSize3, "ahk.bmp.lz_" )

[color=#FF0000]; Copy/paste VarZ wrapper[/color]


Retrieving / Displaying Alpha bitmap from compressed file

#SingleInstance, Force
SetWorkingDir %A_ScriptDir%  

IfNotExist, ahk.bmp.lz_
 URLDownloadToFile, http://tinyurl.com/skanbox/Sample/ahk.bmp.lz_, ahk.bmp.lz_

; Load and Uncompress buffer

FileRead, BMP, *c ahk.bmp.lz_          ; 7403 bytes
VarZ_Uncompress( BMP )                 ; 10279 bytes after secondary decompression

If ( NumGet( BMP ) = 0x315F5A4C ) {    ; LZ_1
     NumPut( 0x005F5A4C, BMP )         ; LZ_
     VarZ_Uncompress( BMP )            ; 74934 bytes after primary decompression
   }
   

; Load bitmap from buffer

hDC := DllCall( "GetDC", UInt,0 )
hBM := DllCall( "CreateDIBitmap", UInt,hDC, UInt,&BMP+14, UInt,0x4, UInt,&BMP+54
               , UInt,&BMP+14, UInt, NumGet( BMP,28 ) > 8 ? 0 : 1, UInt )
DllCall( "ReleaseDC", Uint,0, UInt,hDC )

Gui, Margin, 50, 50
Gui, Add, Picture, w240 h78 0xE hwndhwnd
SendMessage, 0x172, 0, hBM,, ahk_id %hwnd% ; STM_SETIMAGE
Gui, Show,, Var_Z Demo // Multi-level decompression
Return

GuiClose:
 ExitApp



VarZ_Uncompress( ByRef D ) {  ; Shortcode version of VarZ_Decompress() of VarZ 2.0 wrapper
; VarZ 2.0 by SKAN, 27-Sep-2012. http://www.autohotkey.com/community/viewtopic.php?t=45559
 IfNotEqual, A_Tab, % ID:=NumGet(D), IfNotEqual, ID, 0x5F5A4C,  Return 0, ErrorLevel := -1
 savedHash := NumGet(D,4), TZ := NumGet(D,10), DZ := NumGet(D,14)
 DllCall( "shlwapi\HashData", UInt,&D+8, UInt,DZ+10, UIntP,Hash, UInt,4 )
 IfNotEqual, Hash, %savedHash%, Return 0, ErrorLevel := -2
 VarSetCapacity( TD,TZ,0 ), NTSTATUS := DllCall( "ntdll\RtlDecompressBuffer", UInt
 , NumGet(D,8,"UShort"), UInt, &TD, UInt,TZ, UInt,&D+18, UInt,DZ, UIntP,Final, UInt )
 IfNotEqual, NTSTATUS, 0, Return 0, ErrorLevel := NTSTATUS
 VarSetCapacity( D,Final,0 ), DllCall( "RtlMoveMemory", UInt,&D, UInt,&TD, UInt,Final )
Return Final, VarSetCapacity( D,-1 )
}
Posted Image

HotKeyIt
  • Moderators
  • 7439 posts
  • Last active: Jun 22 2016 09:14 PM
  • Joined: 18 Jun 2008
That's really nice, result comes very close to zip :)

Should that method be build in probably?
Also I think default CompressionMode should be 0x2, because Maximum compression is really slow and eats up a lot of cpu.

I have changed below to use multi-level compression automatically and default CompressionMode to exclude COMPRESSION_ENGINE_MAXIMUM.
So we get the best out of it, smaller file size and faster compression :D
/* - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
__      __      ______
\ \    / /     |___  /           V A R Z  >>>  N A T I V E  D A T A  C O M P R E S S I O N
 \ \  / /_ _ _ __ / /            http://www.autohotkey.com/community/viewtopic.php?t=45559
  \ \/ / _` | '__/ /             Author: Suresh Kumar A N  (email: [email protected])
   \  / (_| | | / /__            Ver 2.0 | Created 19-Jun-2009 | Last Modified 27-Sep-2012
    \/ \__,_|_|/_____|           > http://tinyurl.com/skanbox/AutoHotkey/VarZ/2.0/VarZ.ahk
                                                  |
 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
*/

VarZ_Compress( ByRef Data, DataSize, CompressionMode = [color=#FF0000]0x2,RECURSIVE = 0[/color] ) { ; 0x100 = COMPRESSION_ENGINE_MAXIMUM / 0x2 = COMPRESSION_FORMAT_LZNT1

 Static STATUS_SUCCESS := 0x0,   HdrSz := 18

 If ( NumGet( Data ) = 0x005F5A4C )                           ; "LZ_" + Chr(0)
    Return 0, ErrorLevel := -1                                ; already compressed

 DllCall( "ntdll\RtlGetCompressionWorkSpaceSize"
        , UInt,  CompressionMode
        , UIntP, CompressBufferWorkSpaceSize
        , UIntP, CompressFragmentWorkSpaceSize )

 VarSetCapacity( CompressBufferWorkSpace, CompressBufferWorkSpaceSize )

 TempSize := VarSetCapacity( TempData, DataSize )             ; Workspace for Compress

 NTSTATUS := DllCall( "ntdll\RtlCompressBuffer"
                    , UInt,  CompressionMode
                    , UInt,  &Data                            ; Uncompressed data
                    , UInt,  DataSize
                    , UInt,  &TempData                        ; Compressed data
                    , UInt,  TempSize
                    , UInt,  CompressFragmentWorkSpaceSize
                    , UIntP, FinalCompressedSize              ; Compressed data size
                    , UInt,  &CompressBufferWorkSpace
                          ,  UInt )

 If ( NTSTATUS <> STATUS_SUCCESS  ||  FinalCompressedSize + HdrSz > DataSize )
    Return 0, ErrorLevel := ( NTSTATUS ? NTSTATUS : -2 )      ; unable to compress data
 
 VarSetCapacity( Data, FinalCompressedSize + HdrSz, 0 )       ; Renew variable capacity

 NumPut( 0x005F5A4C, Data )                                   ; "LZ_" + Chr(0)
 Numput( CompressionMode, Data, 8 )                           ; actually "UShort"
 NumPut( DataSize, Data, 10 )                                 ; Uncompressed data size
 NumPut( FinalCompressedSize, Data, 14 )                      ; Compressed data size

 DllCall( "RtlMoveMemory", UInt,  &Data + HdrSz               ; Target pointer
                         , UInt,  &TempData                   ; Source pointer
                         , UInt,  FinalCompressedSize )       ; Data length in bytes

 DllCall( "shlwapi\HashData", UInt,  &Data + 8                ; Read data pointer
                            , UInt,  FinalCompressedSize + 10 ; Read data size
                            , UInt,  &Data + 4                ; Write data pointer
                            , UInt,  4 )                      ; Write data length in bytes
[color=#FF0000] If !RECURSIVE && NumPut( 0x315F5A4C, Data, "UInt" ) ; Try extra compression
    If MultiCompressedSize:= VarZ_Compress(Data,FinalCompressedSize + HdrSz,CompressionMode,1)
     return MultiCompressedSize
    else NumPut( 0x005F5A4C, Data, "UInt" )[/color]
  Return FinalCompressedSize + HdrSz
}

VarZ_Uncompress( ByRef D ) {  ; Shortcode version of VarZ_Decompress() of VarZ 2.0 wrapper
; VarZ 2.0 by SKAN, 27-Sep-2012. http://www.autohotkey.com/community/viewtopic.php?t=45559
 IfNotEqual, A_Tab, % ID:=NumGet(D), IfNotEqual, ID, 0x5F5A4C,  Return 0, ErrorLevel := -1
 savedHash := NumGet(D,4), TZ := NumGet(D,10), DZ := NumGet(D,14)
 DllCall( "shlwapi\HashData", UInt,&D+8, UInt,DZ+10, UIntP,Hash, UInt,4 )
 IfNotEqual, Hash, %savedHash%, Return 0, ErrorLevel := -2
 VarSetCapacity( TD,TZ,0 ), NTSTATUS := DllCall( "ntdll\RtlDecompressBuffer", UInt
 , NumGet(D,8,"UShort"), UInt, &TD, UInt,TZ, UInt,&D+18, UInt,DZ, UIntP,Final, UInt )
 IfNotEqual, NTSTATUS, 0, Return 0, ErrorLevel := NTSTATUS
 VarSetCapacity( D,Final,0 ), DllCall( "RtlMoveMemory", UInt,&D, UInt,&TD, UInt,Final )
 [color=#FF0000]If NumGet(D)=0x315F5A4C && NumPut(0x005F5A4C,D)
  Return VarZ_Uncompress( D )[/color]
Return Final, VarSetCapacity( D,-1 )
}

;- -- - -- - -- - -- - -- - -- - -- - -- - -- - -- - -- - -- - -- - -- - -- - -- - -- - --

VarZ_Decompress( ByRef Data ) {

 Static STATUS_SUCCESS := 0x0,   HdrSz := 18
 
 If ( NumGet( Data ) <> 0x005F5A4C )                          ; "LZ_" + Chr(0)
    Return 0, ErrorLevel := -1                                ; not natively compressed

 DataSize := NumGet( Data, 14 )                               ; Compressed data size

 DllCall( "shlwapi\HashData", UInt,  &Data + 8                ; Read data pointer
                            , UInt,  DataSize + 10            ; Read data size
                            , UIntP, Hash                     ; Write data pointer
                            , UInt,  4 )                      ; Write data length in bytes

 If ( Hash <> NumGet( Data, 4 ) )                             ; Hash vs Saved hash
    Return 0, ErrorLevel := -2                                ; Hash failed = Data corrupt

 TempSize := NumGet( Data, 10 )                               ; Decompressed data size
 VarSetCapacity( TempData, TempSize, 0 )                      ; Workspace for Decompress

 NTSTATUS := DllCall( "ntdll\RtlDecompressBuffer"
                    , UInt,  NumGet( Data, 8, "UShort" )      ; Compression mode
                    , UInt,  &TempData                        ; Decompressed data
                    , UInt,  TempSize
                    , UInt,  &Data + HdrSz                    ; Compressed data
                    , UInt,  DataSize
                    , UIntP, FinalUncompressedSize            ; Decompressed data size
                           , UInt )

 If ( NTSTATUS <> STATUS_SUCCESS )
    Return 0, ErrorLevel := NTSTATUS                          ; Unable to decompress data

 VarSetCapacity( Data, FinalUncompressedSize, 0 )             ; Renew variable capacity

 DllCall( "RtlMoveMemory", UInt,  &Data                       ; Target pointer
                         , UInt,  &TempData                   ; Source pointer
                         , UInt,  FinalUncompressedSize )     ; Data length in bytes
 [color=#FF0000]If NumGet(Data)=0x315F5A4C && NumPut(0x005F5A4C,Data)
  Return VarZ_Uncompress( Data )[/color]
Return FinalUncompressedSize, VarSetCapacity( Data, -1 )
}

;- -- - -- - -- - -- - -- - -- - -- - -- - -- - -- - -- - -- - -- - -- - -- - -- - -- - --

VarZ_Load( ByRef Data, SrcFile ) {
 FileGetSize, DataSize, %SrcFile%
 IfNotEqual, ErrorLevel, 0, Return
 FileRead, Data, *c %SrcFile%
Return DataSize
}

;- -- - -- - -- - -- - -- - -- - -- - -- - -- - -- - -- - -- - -- - -- - -- - -- - -- - --

VarZ_Save( ByRef Data, DataSize, TrgFile ) {
 hFile :=  DllCall( "_lcreat", ( A_IsUnicode ? "AStr" : "Str" ),TrgFile, UInt,0 )
 IfLess, hFile, 1, Return "", ErrorLevel := 1
 nBytes := DllCall( "_lwrite", UInt,hFile, UInt,&Data, UInt,DataSize, UInt )
 DllCall( "_lclose", UInt,hFile )
Return nBytes
}

;- -- - -- - -- - -- - -- - -- - -- - -- - -- - -- - -- - -- - -- - -- End of VarZ wrapper
EDIT:
Fixed not to use Multi-compression if not possible.

  • Guests
  • Last active:
  • Joined: --
Thanks Skan. I only have one question (to myself: how could I have missed this?) :D

SKAN
  • Administrators
  • 9115 posts
  • Last active:
  • Joined: 26 Dec 2005
Dear HotKeyIt,

Thanks for the suggestions. :)
However, your recursive function will return 0 if secondary compression fails because of following validation.

If ( NTSTATUS <> STATUS_SUCCESS  ||  [color=#FF0000]FinalCompressedSize + HdrSz > DataSize[/color] )
    Return 0, ErrorLevel := ( NTSTATUS ? NTSTATUS : -2 )      ; unable to compress data


When user tries to compress a short string, the compressed data can actually become bigger than the original data owing
to the ovehead of 18 byte header prefixed by VarZ_Compress(). Also, AFAI tested, the secondary compression works with BITMAP but not with TEXT.
A single compression with 0x102 gives the best compression for TEXTual data.

:)
kWo4Lk1.png

HotKeyIt
  • Moderators
  • 7439 posts
  • Last active: Jun 22 2016 09:14 PM
  • Joined: 18 Jun 2008
I see, thanks for the info.
I also agree on CompressionMode, COMPRESSION_ENGINE_MAXIMUM will make sure that even small files can be compressed.
For multilevel compression we could check if second compression is possible and fall back if it is not:
If !RECURSIVE && NumPut( 0x315F5A4C, Data, "UInt" ) ; Try extra compression
    If MultiCompressedSize:= VarZ_Compress(Data,FinalCompressedSize + HdrSz,CompressionMode,1)
     return MultiCompressedSize
    else NumPut( 0x005F5A4C, Data, "UInt" )
  Return FinalCompressedSize + HdrSz
}


SKAN
  • Administrators
  • 9115 posts
  • Last active:
  • Joined: 26 Dec 2005
That is nice! :)

I have linked your version in Title post, HotKeyIt :)
I feel, double compression is better done outside the function..
..but agree with you that
UnCompress/Decompress functions should automatically double decompress when neccessary.
I actually use a console utility LZ_.exe ( find link in title post ) for compression, which I need to update.