Jump to content

Sky Slate Blueberry Blackcurrant Watermelon Strawberry Orange Banana Apple Emerald Chocolate
Photo

[SOLVED] Remove both duplicates in a text file



  • Please log in to reply
18 replies to this topic
  • Guests
  • Last active:
  • Joined: --

I'm using AHK 1.0.47.6 (standard version).

...1.0.48.05 is newer & is still the "standard version". Then of course there's AHK_L, which is even newer, but non-standard.

Do you have any suggestions for improving the "Mark each line as old or new" part of the code, shown below?

...not yet. 1 thing I did notice tho...you Sort, then StringReplace...that may be fine, but it may also affect which lines are sorted where. I think I'd StringReplace, then Sort.

Ah yes, now I do have a recommendation...you are Loop, Read'ing the file you just wrote...all5.txt was just written from %var2%...so you could just Loop, Parse, var2 (but I'd rename that var)...also, you don't need to write all5.txt just to turn around & write all6.txt...keep processing the text in-memory, then dump it out to the file once you are done changing it. I'll try to write something up...

  • Guests
  • Last active:
  • Joined: --
OK, here's the new version...old.txt & new.txt no longer need to be UTF-8 (but they can be)...

;//
;// ---[ Set filenames here ]---
;//
file_old=old.txt
file_new=new.txt
file_output=unique.txt

;//
;// ---[ Read old & new job lists into vars ]---
;//
FileRead, old, %file_old%
FileRead, new, %file_new%

;//
;// ---[ Clean old & new job lists ]---
;//
ellipsis_char:=Chr(133)
StringReplace, old, old, %ellipsis_char%, ..., a
StringReplace, new, new, %ellipsis_char%, ..., a

;//
;// ---[ Merged & Sort the job lists ]---
;//
text:=old "`n" new
Sort, text

;//
;// ---[ Remove all sets of dupes ]---
;//
text:=RegExReplace(text, "m`a)^(.+)\R(\1(\R|$))+")

;//
;// ---[ Mark each line as old or new ]---
;//
Loop, Parse, text, `n, `r
{
	text_line:=A_LoopField
	Loop, Parse, old, `n, `r
	{
		old_line:=A_LoopField
		if (text_line==old_line) {
			text_line:="Old: "text_line
			;//text_line.=" - Old"
			text_output.="`n" text_line
		}
	}
	Loop, Parse, new, `n, `r
	{
		new_line:=A_LoopField
		if (text_line==new_line) {
			text_line:="New: " text_line
			;//text_line.=" - New"
			text_output.="`n" text_line
		}
	}
}

;// remove preceding new line
text_output:=SubStr(text_output, 2)

;//
;// ---[ Output to file ]---
;//
FileDelete, %file_output%
;//FileAppend, %text%, %file_output%
FileAppend, %text_output%, %file_output%


nimda
  • Members
  • 4368 posts
  • Last active: Aug 09 2015 02:36 AM
  • Joined: 26 Dec 2010
nitpick: Chris himself named AHK_L as the 'official version.' Basic is no longer developed, and never will be developed furter.

AHK_L has been current for a few years. This is why it is first on the d/l page

mikek
  • Members
  • 161 posts
  • Last active: Nov 09 2015 05:02 PM
  • Joined: 21 Nov 2008
I was able to significantly improve the program's speed using InStr instead of comparing line by line. So the program works great now!

Thank you everyone for your help.

- Mike