StrUnused prototype function: find unused characters for use as delimiters/separators

Post your working scripts, libraries and tools
User avatar
jeeswg
Posts: 4280
Joined: 19 Dec 2016, 01:58
Location: UK

StrUnused prototype function: find unused characters for use as delimiters/separators

31 Jan 2018, 17:38

- A common issue I have with scriptwriting, is the need for unused characters as temporary characters for parsing.
- Also, if I have an array of key names or values, that I want to combine into a list, or perform a sort on, I need to know an unused character for these. Often I might like to use CRLFs, however, I need to confirm that they aren't in use.
- I have been curious as to how the AutoHotkey source code handles such problems, of sorting without having a delimiter available, although I don't yet know the answer.
- Btw what if a string contains literally every Unicode character. I had this problem a few times. [EDIT: In such a situation, a StrUnused function would return 0 characters, indicating that no unused characters were available.]

- It's tempting to write code from scratch, every time a function needs a temporary character.
- However, this often clutters up otherwise simple functions.
- It's also tempting to assume that say Chr(1), Chr(2) and Chr(3) are not in use. However, you may have made that exact same assumption earlier, and so those characters would be unavailable.

- Example of a function that needs unused characters:
proof of concept: Send what you mean - AutoHotkey Community
https://autohotkey.com/boards/viewtopic.php?f=6&t=40546

- The current approach is rather simple, grab text from all strings/object values, concatenate them, and perform a loop using InStr.
- I would welcome any suggestions for improving the function.
- I had wondered if such a function could be made more efficient via the AutoHotkey source code, e.g. if AutoHotkey knows where an object starts and ends, it could perform binary searches on the area to find unused characters. Perhaps you could specify to create an array with a special option, that collected the data close together in the address space.
- At the most basic level, I was curious if AutoHotkey kept information on the total length of all strings in an array, for use with StrJoin, to then prepare a variable with sufficient capacity to concatenate those strings. Something I also need for my current approach.
- Unfortunately also, there isn't an ObjCount function, which I could potentially use to guess the required capacity.

- Here is a prototype StrUnused function, together with examples showing the use of temporary characters when alphabetising/randomising a string. Thanks for reading.

Code: [Select all] [Expand] [Download] GeSHi © Codebox Plus

- [EDIT:] Some functionality added to the function, and some MsgBoxes added to the q hotkey subroutine.
- [EDIT:] Made the v# parameters ByRef, and made 65535 the maximum character to check for, as beyond that you get a pair of Unicode characters when you use Chr().
- [EDIT:] A StrUnused simple version, as a potential candidate AHK function:

Code: [Select all] [Expand] [Download] GeSHi © Codebox Plus

Last edited by jeeswg on 03 Feb 2018, 14:54, edited 1 time in total.
Helgef
Posts: 2900
Joined: 17 Jul 2016, 01:02
Contact:

Re: StrUnused prototype function: find unused characters for use as delimiters/separators

02 Feb 2018, 14:25

Hello jeeswg, thanks for sharing :thumbup: . That is thorough work. Why don't use a variadic declaration?

There is a funny function you can use for finding an unsed character in a string, wcsspn.

Cheers.
User avatar
jeeswg
Posts: 4280
Joined: 19 Dec 2016, 01:58
Location: UK

Re: StrUnused prototype function: find unused characters for use as delimiters/separators

02 Feb 2018, 15:00

- Thanks Helgef. Cheers for the function link, nice to see something related that is mainstream. Hmm, variadic declaration, you've reminded me, I need to add ByRef to each parameter, every parameter needs to be ByRef, Helgef. I.e. one of many steps to use as little memory as possible. [EDIT: I've added 'ByRef' above.]
- I wonder if it's remotely easy at present to do something like 'object get combined length of all key names/value strings'. And if a StrJoin function was added to AutoHotkey, if the way objects are currently defined, serves that purpose well, or whether changes are required to calculate the required string length. Alternatively, you could create a custom object that records key name lengths as they are created.

- Any official functions that I'd advocate for, I'd want them to be as simple possible, so possible straightforward functions that could be helpful would be:
- StrJoin (first parameter is the padding string, followed by a variadic parameter, so it can handle either multiple strings, or one linear array). You could specify multiple pad strings (cf. StrSplit) to go from ["a","b","c","d"] to a=b`r`nc=d. [See example in post lower down.]
- ObjCount (to return the number of keys in an array, useful for custom functions, for estimating the size of a string needed to concatenate key names/string values).
- ObjGetSize (some kind of vague size value that gives you a notion of how big the object is, this could help you create a buffer, even if it's an overestimate). Something specific like the length of all key names/value strings, would of course be useful, but seems a bit too specific for a built-in function, or perhaps not, it depends.
- StrUnused (first parameter is the number of characters to return, followed by a variadic parameter, so it can handle either multiple strings, or one linear array). [See StrUnused simple version in post higher up.]
Last edited by jeeswg on 03 Feb 2018, 15:01, edited 1 time in total.
User avatar
jeeswg
Posts: 4280
Joined: 19 Dec 2016, 01:58
Location: UK

Re: StrUnused prototype function: find unused characters for use as delimiters/separators

02 Feb 2018, 17:40

- Here's an example of allowing multiple pad strings in a StrJoin function, in a similar way to how StrSplit allows multiple delimiters.
- E.g. you can do n items per line, or present a list 'ini style'.
- The idea is that, for 3 pad strings, you append: item 1, pad string 1, item 2, pad string 2, item 3, pad string 3, item 4, pad string 1, item 5, pad string 2 . You alternate between pad strings.
- I haven't seen any StrJoin functions to compare it with, so I don't know if this is common or not.

Code: [Select all] [Expand] [Download] GeSHi © Codebox Plus

Last edited by jeeswg on 20 Feb 2018, 12:59, edited 3 times in total.
Helgef
Posts: 2900
Joined: 17 Jul 2016, 01:02
Contact:

Re: StrUnused prototype function: find unused characters for use as delimiters/separators

03 Feb 2018, 14:19

I haven't seen any StrJoin functions to compare it with, so I don't know if this is common or not.

There are many ways to do it, so there are probably many versions out there, and even more which have never been shared, due to lack of generality, an all-purpose strJoin is impossible to make. I think your version is very good though, as a general function. But I prefer to make it a method instead, i.e., str := array.toString(sep1,sep2,...).

Cheers for the function link

Did you try to make a strUnused function based on it? It might not be obvious how to do it, or maybe it is, or maybe my version doesn't work and I am making a fool of myself, I didn't give it much testing :lol:. I can share it later if you want. :ugeek:

Cheers.
User avatar
jeeswg
Posts: 4280
Joined: 19 Dec 2016, 01:58
Location: UK

Re: StrUnused prototype function: find unused characters for use as delimiters/separators

03 Feb 2018, 15:13

- Thanks. So you want to make everything in AutoHotkey object-y, as with your hotstring suggestions. Is that the plan? [EDIT: Btw I added to my comments there in the post below this one:]
Hotstring E(xecute) vs E(nd chars), Hotkey directives - AutoHotkey Community
https://autohotkey.com/boards/viewtopic.php?f=37&t=43404&p=197477#p197477
- I would be happy with ObjCount and StrJoin functions, even if no new methods were added. Although I would have nothing against having these being available: Obj.Count() or Obj.ToString(). I'm a bit reluctant ever to ask for new methods, because it seems like a big thing to add a new method to all arrays. But maybe it's not too bad.
- I do think that being able to redefine the array you get, when you do obj := {}, obj := [], could be a great advantage, but I don't know what the costs would be of implementing this.

- I haven't tried using the Winapi function. Often a problem with Winapi functions is that they stop at the first null character. I like functions to stop at a specific place, so that they can handle null characters, but don't go on forever. [EDIT: Actually that's also a limitation of InStr. Sometimes a trick is to add a temporary null, then restore the original character.]
- I have added a simpler StrUnused function to the OP, which I think is a reasonable candidate for a built-in AHK function. If people are willing to have it.
User avatar
Cerberus
Posts: 63
Joined: 12 Jan 2016, 15:46

Re: StrUnused prototype function: find unused characters for use as delimiters/separators

04 Feb 2018, 23:47

Hmm very interesting idea. I'm still not entirely sure, though, of a real-world scenario where you would really need this? I'm afraid I was unable to understand at a glance the issue in the post you linked to.

I sometimes use filler characters in e.g. the replace function in a text editor or some Tasker task on my phone, to concatenate strings or mark locations in regex (though I probably don't need too, but it's just easy to do); but then I just use a fixed string of characters rather than a single character. If you make the fixed string long enough, it will be practically unique, like #@xxqqxx@#. So do you really need a single character?

As to counting the number of keys in an object/array, I would use a for loop like so:

Code: [Select all] [Download] GeSHi © Codebox Plus

array := [4,5,6]
for key in array {
count ++
}
Msgbox % count ; displays "3"

Or is that not what you meant?
User
Posts: 125
Joined: 26 Jun 2017, 08:12

Re: StrUnused prototype function: find unused characters for use as delimiters/separators

05 Feb 2018, 07:58

I don't know if the below scripts have any usefulness here, but I will share them anyway!

Once I used to use "Long Unique Strings" as Delimiter, but I never liked such approach!

Then I wanted to find a way to use only 1 character as Delimiter, the problem was, it would have to be an unused character!

Then, someone introduced me regex "(*SKIP)(*F)"! Problem Solved! (Well, at least it is working well so far!)

In the examples below, # is used as Delimiter, but yet any # characters from the original string is preserved!

Example1, Saving Multi-Line Strings in a ".ini" key (LineFeed = #N)

Code: [Select all] [Expand] [Download] (Untitled.ahk)GeSHi © Codebox Plus



Example2, An effective way to switch strings, replace A with B and B with A

Code: [Select all] [Expand] [Download] (Untitled.ahk)GeSHi © Codebox Plus

Helgef
Posts: 2900
Joined: 17 Jul 2016, 01:02
Contact:

Re: StrUnused prototype function: find unused characters for use as delimiters/separators

06 Feb 2018, 05:54

- Thanks. So you want to make everything in AutoHotkey object-y, as with your hotstring suggestions. Is that the plan? [EDIT: Btw I added to my comments there in the post below this one:]

No, and I do not have a plan. Had you been interested, I'm sure you could have figured it out, or come up with a specific question. This is isn't the place to discuss it though.
it seems like a big thing to add a new method to all arrays.

I do not think it is, it just have to make sense to (motivate) the developer.

It make more sense to me to have a toStr method rather than a strJoin function, but only in the because it is more valuable to join elements of an array, rather than values of variables. I don't care if it is ObjToStr(obj, sep...) or obj.toStr(sep,...), only that it is not toStr([sep,...],obj*), the latter is wasteful if obj is big.

@Cerberus, a built-in ObjCount method would return the answer without counting, your method is fine and it is what we have to do currently.

@User, interesting, I'll take a look, thanks for sharing. :wave:

Cheers.
User avatar
jeeswg
Posts: 4280
Joined: 19 Dec 2016, 01:58
Location: UK

Re: StrUnused prototype function: find unused characters for use as delimiters/separators

07 Feb 2018, 10:06

- @Cerberus: Hmm, 'practically unique', but not *unique*. I did use 'qqq' for some things in Excel, before I used AutoHotkey, I can't remember the exact logical issues of when using such a string is safe or not.
- If you replace text, with the new text being shorter than or the same length as the needle, you don't have to calculate/estimate the buffer size for the outputted longer string. Also, if the new text and needle are the same length, e.g. you replace 'ABCD' with 'vUnused1 vUnused2 vUnused2 vUnused2', you can replace text directly by overwriting text, instead of using lots of memory moves, and then parse by vUnused1 (omitting any vUnused2 characters from substrings). I don't know if AutoHotkey uses these kinds of optimisations when replacing text.
- Anyway, when using big strings, optimisation can be a big advantage.

- I looked through scripts and grabbed some use StrUnused example cases:

Code: [Select all] [Expand] [Download] GeSHi © Codebox Plus


- Btw yes, you can use a for loop to count keys, but (a) it can be slow on big arrays, (b) something like MaxIndex/Length can't inform you of any gaps in a linear array, (c) it's the sort of used-all-the-time thing, you might expect to be a function, for convenience, and readability, to avoid cluttering a script/function with repetitive mundane code.
- Btw great avatar.

- @User: Nice to hear from you again. Those examples are very hard to follow, very clever though. Looking at code/text like that is one reason I avoid repeating characters as a method of escapement, and instead go for simple functions where you can specify a custom delimiter. I have a swap string A/B example earlier in this post.
- *SKIP and *F (aka *FAIL) are mentioned here:
pcre.txt
http://www.pcre.org/pcre.txt
- Btw what program do you use to create animated gifs? Is it freeware?

- @Helgef:
re. wish lists
User
Posts: 125
Joined: 26 Jun 2017, 08:12

Re: StrUnused prototype function: find unused characters for use as delimiters/separators

08 Feb 2018, 19:07

FreeStudio (FreeGIFMaker.exe)

Hard to follow?

Anyway, a function where custom Delimiter can be specified!

Code: [Select all] [Expand] [Download] (Untitled.ahk)GeSHi © Codebox Plus

User
Posts: 125
Joined: 26 Jun 2017, 08:12

Re: StrUnused prototype function: find unused characters for use as delimiters/separators

08 Feb 2018, 19:24

This one seems better:

Code: [Select all] [Expand] [Download] (Untitled.ahk)GeSHi © Codebox Plus


Return to “Scripts and Functions”

Who is online

Users browsing this forum: delseyyy and 15 guests