Jump to content

Sky Slate Blueberry Blackcurrant Watermelon Strawberry Orange Banana Apple Emerald Chocolate
Photo

RFN variable storage and retrieval fast/high capacity


  • Please log in to reply
2 replies to this topic
JamixZol
  • Members
  • 54 posts
  • Last active: Jan 22 2014 04:35 AM
  • Joined: 26 Mar 2012
Here's a set of functions I wrote to store and retrieve large variable sets quickly and accurately.

I have been running into problems storing and dumping large sets of data from ini files, and had a need to do so on multiple machines with encrypted hard discs. This was causing IO issues with read/write times. I have tried ini, csv, local database all with their own set of issues and bugs. So I had this idea that a csv is perfect for this type of IO load, however commas and return lines need to be captured in this data.

So here's what I needed and what this satisfies:
1. write out/dump variable and their contents to a persistent file
2. auto archive these dumps
3. high capacity and low read times
4. Capture any 'normal' text characters including returns and commas etc

The what it does:
The file is written using a 'difficult to type' delimiter " ¦ "
Files are organized as such: VARIABLE_NAME¦VARIABLE_VALUE then simply strung together in the parse
When the file is read back in, variable names are assumed to be of an odd loop number, and the paired value to be of even
When variable are read back in from file, they are set global so as to 'pour' them back into the app

To generate a dump file:
First create a pipe delimited list of the variable to be captured: ie(LIST_OF_VARS=VAR1|VAR2|VAR3|VAR4|VAR5) note: omit any pipe at the end
Call the function: OUTPUT_TO_RFN(LIST_OF_VARS, "My RFN Output File.rfn", A_scriptdir . "\ARCHIVES\")
The "My RFN Output File.rfn" will be generated and if it already exists, the old moved to the specified archive folder and time stamped as a version
To overwrite the files without archive, just call the function without the archive folder parameter

To read the values back in:
Call the function: READ_FROM_RFN("My RFN Output File.rfn")
All variable found in the file will be pushed into the calling script as global variables

Here's the code:
; #INCLUDE RFN.ahk

/*
OUTPUT_TO_RFN(LIST_OF_FIELDS, FILE_NAME, ARCHIVE_FOLDER="")
LIST_OF_FIELDS: Pipe delimited list of variables to store
FILE_NAME: File name to store to
ARCHIVE_FOLDER: Name of folder to store archived file if it already exists(optional, archives will not be kept if omited)

*/
OUTPUT_TO_RFN(LIST_OF_VARS, FILE_NAME, ARCHIVE_FOLDER=""){
RFN_DEL= % chr(5)
CONTAINER= ; init the container
	Loop, parse,LIST_OF_VARS, "|" ; build the field value relationship dynamiclly, store in LEGACY_NOTES_FILE
	{
		CONTAINER.=A_LoopField . RFN_DEL . %A_LoopField% . RFN_DEL
	}
	StringTrimRight, CONTAINER,CONTAINER,1 ; trimm off the last delimiter
	
	;check for the archive folder
If(ARCHIVE_FOLDER)
	IfNotExist,%ARCHIVE_FOLDER%
		FileCreateDir, %ARCHIVE_FOLDER%
		
; check to see if the file exists before write and archive if requested
IfExist, %FILE_NAME%
{
	SplitPath,FILE_NAME,,,FILE_NAME_EXT,FILE_NAME_NOEXT
	If(ARCHIVE_FOLDER) ; only archive if requested
		FileMove,%FILE_NAME%, %ARCHIVE_FOLDER%\%FILE_NAME_NOEXT%_ARCHIVE_%A_MM%-%A_DD%-%A_YYYY%_at_%A_Hour%%A_Min%-%A_Sec%.%FILE_NAME_EXT%
	FileDelete, %FILE_NAME%
}
FileAppend,%CONTAINER%, %FILE_NAME%,UTF-8 ; UTF-8 seems to work best for this for some reason
}return

/*
READ_FROM_RFN(FILE_NAME)
FILE_NAME: File name to read back into variables

*/
READ_FROM_RFN(FILE_NAME){
local a := 1 ; used to push the dynamic global vars below (see: http://www.autohotkey.com/forum/topic19641.html) Thanks Lexikos! 
FileRead, CONTAINER, %FILE_NAME% ;Read the file into memory

	Loop, parse,CONTAINER, % chr(5) ; parse out the variable names and values expected(var names = odd numbers, var values = even numbers)
	{		
		;~ msgbox % A_Index A_LoopField
		; odd or even logic
	if Mod(A_Index,2) ; this is an odd number, assumed to be a variable name
		{
			LAST_VAR_NAME:=A_LoopField ; keep the name for value asignment
		}
	else ; this is an even number, assumed to be the value of the previous entry
		{
			%LAST_VAR_NAME%:=A_LoopField ; assign the value back into the variable set global above
		}
	}
}return

And a quick example:
#INCLUDE RFN.ahk

; list of variables to be handled
LIST_OF_VARS=ONE|TWO|THREE|FOUR|FIVE|SIX|SEVEN|EIGHT|NINE|TEN

	Loop, parse,LIST_OF_FIELDS,"|" ; Just putting some stuff in the vars to test with
	{
		%A_LoopField%:="`nPUT SOME STUFF " . A_Index
	}

FILE_NAME=TEST.RFN

ARCHIVE_FOLDER= %A_ScriptDir%\TEST ARCHIVE\

; Dump these variable to RFN file
OUTPUT_TO_RFN(LIST_OF_VARS, FILE_NAME, ARCHIVE_FOLDER)

; Open a file to read the variable back in
FileSelectFile, FILE_NAME,,%A_ScriptDir%,,(*.RFN)
READ_FROM_RFN(FILE_NAME)
ListVars
MsgBox % ONE TWO THREE FOUR FIVE SIX SEVEN EIGHT NINE TEN
return

Edit: changed so archive is optional

NOTES:
I'm using the .RFN extension just because it jives with the project this was written for, it can be anything really

I would expect these functions to run in any version of AHK and AHK_L

This was tested on WINXP 32 SP3, WIN 7 64 ULT, WIN 8 preview(winXbox lol)
AHK version: AHK_L v1.1.07.01
Tested in both VM and hardware machines

If anyone has figured out how to use a null character or something that can not be typed at all for this type of script, please let me know
Modifications are welcome, I just thought I'd share this as it has made my life easier :D

EDIT: Modified to use the chr(5) delimiter
EDIT: Fixed a bug in the file archive switch to allow paths correctly

  • Guests
  • Last active:
  • Joined: --
I regularly use Chr(5) and Chr(7) as newline and delimiter characters so I can dump data on one line and use them like so
StringSplit, Text, A_LoopField, % Chr(5)

; and/or

Loop, Parse, Text, % Chr(7)


JamixZol
  • Members
  • 54 posts
  • Last active: Jan 22 2014 04:35 AM
  • Joined: 26 Mar 2012
That's perfect! Thanks :D