How to merge 2 text files?

Get help with using AutoHotkey (v1.1 and older) and its commands and hotkeys
Guest10
Posts: 578
Joined: 01 Oct 2013, 02:50

How to merge 2 text files?

12 May 2014, 20:09

i have 2 text files of the following constructs. as you see, there are several common strings (shared) in these files. how can i merger these files without any "duplicate" strings (all strings must appear only once in the new file and the new file must include all unique strings in file1 and file2)?:

file1:
136033EJ9
136033BJ9
136033AJ9
504698AH4
504698AG6
936033AJ8
936033BD0
54640AY38
13033WB44
72177MCB1
307144AB1
546279RG7
414738TC8
453419CG6
800767AW5
414738RB2
414738TA2
05714TAB8


file2:
453419CG6
800767AW5
414738RB2
414738TA2
05714TAB8
60535RAG3
804382DV8
417704GB3
804382DU0
13033WA45
389092AL1
63166MBX6
544566AN2
802576BM9
136033EJ9
136033BJ9
136033AJ9
Hamlet
Posts: 32
Joined: 02 Oct 2013, 09:55
Location: Seoul, Korea

Re: How to merge 2 text files?

12 May 2014, 21:13

Easy, Make the third file while checking each and every elements with Object.HasKey(Key) which gives you the uniquness.
Guest10
Posts: 578
Joined: 01 Oct 2013, 02:50

Re: How to merge 2 text files?

12 May 2014, 21:34

dug deep into my "archives" and came up with this: :lol: :ugeek:
(please advise if code could be further improved.)

Code: Select all

FileRead, file1, file1.txt
FileAppend, %file1%, file2.txt
FileRead, String, file2.txt

rawList := String
Sort, rawList, U
Loop, parse, rawList, `n, `r
ndl :=	"`am)^" A_LoopField "$"
, String := RegExReplace(String,ndl,"","",-1,RegExMatch(String,ndl)+StrLen(A_LoopField))

String := Trim(RegExReplace(String,"\v+","`n"))
FileAppend, %String%, file3.txt
Last edited by Guest10 on 12 May 2014, 22:05, edited 1 time in total.
Hamlet
Posts: 32
Joined: 02 Oct 2013, 09:55
Location: Seoul, Korea

Re: How to merge 2 text files?

12 May 2014, 21:36

Nop, that could be time consuming.
Specifically, If you have h.e.l.l. of BiG files, you have to wait millions of seconds.
Guest10
Posts: 578
Joined: 01 Oct 2013, 02:50

Re: How to merge 2 text files?

12 May 2014, 21:41

i am not well-versed with the use of Object.HasKey(Key).
Last edited by Guest10 on 17 May 2014, 08:23, edited 1 time in total.
Hamlet
Posts: 32
Joined: 02 Oct 2013, 09:55
Location: Seoul, Korea

Re: How to merge 2 text files?

12 May 2014, 21:43

Fine, read the manual. And try it again and again.
That's the only way. You know that.
User avatar
AlphaBravo
Posts: 586
Joined: 29 Sep 2013, 22:59

Re: How to merge 2 text files?

12 May 2014, 22:18

if order is not an issue :

Code: Select all

FileRead, F1, file1.txt
FileRead, F2, file2.txt
F3 := F1 "`r`n" F2
Sort, F3, U
FileAppend, %F3%, output.txt
Guest10
Posts: 578
Joined: 01 Oct 2013, 02:50

Re: How to merge 2 text files?

12 May 2014, 22:48

thank, no OBJECTion here! :lol: one minor note (will not affect my application) is a blank (empty) line always showing at the very top (looks like sorting starts with a blank line) as follows:
output file:
[blank line here]
05714TAB8
13033WA45
13033WB44
226360AC9
307144AB1
389092AL1
414738RB2
414738TA2
414738TC8
417704GB3
453419CG6
504698AG6
504698AH4
544566AN2
546279RG7
54640AY38
60535RAG3
63166MBX6
72177MCB1
800767AW5
802576BM9
804382DU0
804382DV8
936033AJ8
936033BD0
Hamlet
Posts: 32
Joined: 02 Oct 2013, 09:55
Location: Seoul, Korea

Re: How to merge 2 text files?

12 May 2014, 23:52

Next time, try not to waste someone else's precious time like this.
And, use a lot of "Thanks" to someone else.
I've participated your post twice(including this one).
You are saving too much "thanks" to the ones who put answers.
That should be not a good attitude.

(Finally, if possible, read your own codes again carefully.
I guess you do not have your own codes in your "archives".
Clearly you used the sort "U" options in your "archives" andy say thanks to someone for that.
That's just nonsenese.)
User avatar
PuzzledGreatly
Posts: 1303
Joined: 29 Sep 2013, 22:18

Re: How to merge 2 text files?

13 May 2014, 15:43

Hamlet, I don' think the attitude you have displayed in this thread is very friendly. It would have been nicer to see an example rather than put downs and criticism. If you feel a request for help is a waste of time then just don't respond. Thanks.
Guest10
Posts: 578
Joined: 01 Oct 2013, 02:50

Re: How to merge 2 text files?

13 May 2014, 16:14

an extensive (or should i say, dizzying) array of options to eliminate duplicate entries discussed here: http://www.autohotkey.com/board/topic/9 ... ith-regex/
Hamlet
Posts: 32
Joined: 02 Oct 2013, 09:55
Location: Seoul, Korea

Re: How to merge 2 text files?

13 May 2014, 17:01

If you use someone else's code, You MUST have explicitly mentioned that the code is someone else's and said THANK to that someone else's efforts for that. If you do not follow this simple general rules, You could be one of the thieves. and now You really are.
User avatar
PuzzledGreatly
Posts: 1303
Joined: 29 Sep 2013, 22:18

Re: How to merge 2 text files?

13 May 2014, 20:01

I've no idea what you are going on about and judging from your reply I'll probably ignore your future posts. Cheers
User avatar
joedf
Posts: 8940
Joined: 29 Sep 2013, 17:08
Location: Canada
Contact:

Re: How to merge 2 text files?

14 May 2014, 00:31

No more on this. Respect is essential for me and this community. If this gets worst, I will have to lock this thread.
Image Image Image Image Image
Windows 10 x64 Professional, Intel i5-8500, NVIDIA GTX 1060 6GB, 2x16GB Kingston FURY Beast - DDR4 3200 MHz | [About Me] | [About the AHK Foundation] | [Courses on AutoHotkey]
[ASPDM - StdLib Distribution] | [Qonsole - Quake-like console emulator] | [LibCon - Autohotkey Console Library]
Hamlet
Posts: 32
Joined: 02 Oct 2013, 09:55
Location: Seoul, Korea

Re: How to merge 2 text files?

14 May 2014, 09:42

Just figured out that COM Object Reference is little bit better than the HasKey method in some cases.
-You do not need to re-shuffle it to keep the original order (HasKey method sort it forcefully-alphabetical ordering).

Code: Select all

;	Thanks !!  jethrow 
;	COM Object Reference
;	http://www.autohotkey.com/board/topic/56987-com-object-reference-autohotkey-v11/?p=357746

a  := "cc aa bb"
b  := "zz cc aa ee bb dd ff"

c  :=  StrSplit( a " " b, A_Space )
d  :=  ComObjCreate( "Scripting.Dictionary" )

For  i  in  c
	d.Item( c[i] ) 

For  i  in  d
	e  .=  i  " "
	
MsgBox % e
Return
In any case, you must Read the Manual with your own eyes and fingers.
Fortunately, there is no way to escape from this stubborn truth. ^^
Last edited by Hamlet on 14 May 2014, 16:01, edited 4 times in total.
User avatar
joedf
Posts: 8940
Joined: 29 Sep 2013, 17:08
Location: Canada
Contact:

Re: How to merge 2 text files?

14 May 2014, 10:15

Nice use of COM there!
Image Image Image Image Image
Windows 10 x64 Professional, Intel i5-8500, NVIDIA GTX 1060 6GB, 2x16GB Kingston FURY Beast - DDR4 3200 MHz | [About Me] | [About the AHK Foundation] | [Courses on AutoHotkey]
[ASPDM - StdLib Distribution] | [Qonsole - Quake-like console emulator] | [LibCon - Autohotkey Console Library]
Hamlet
Posts: 32
Joined: 02 Oct 2013, 09:55
Location: Seoul, Korea

Re: How to merge 2 text files?

14 May 2014, 17:55

AHK has two Splits.
Do we need a Join?
- IMO, negative..
User avatar
FanaticGuru
Posts: 1906
Joined: 30 Sep 2013, 22:25

Re: How to merge 2 text files?

14 May 2014, 20:05

Hamlet wrote:Just figured out that COM Object Reference is little bit better than the HasKey method in some cases.
-You do not need to re-shuffle it to keep the original order (HasKey method sort it forcefully-alphabetical ordering).
It also has the advantage of being case sensitive which normal array keys are not. That makes ComObjCreate( "Scripting.Dictionary" ) pretty useful sometimes.

You can have two keys like "Cat" and "cat". That is not the case with normal object arrays.

FG
Hotkey Help - Help Dialog for Currently Running AHK Scripts
AHK Startup - Consolidate Multiply AHK Scripts with one Tray Icon
Hotstring Manager - Create and Manage Hotstrings
[Class] WinHook - Create Window Shell Hooks and Window Event Hooks
User avatar
PuzzledGreatly
Posts: 1303
Joined: 29 Sep 2013, 22:18

Re: How to merge 2 text files?

14 May 2014, 20:13

Thanks, Hamlet, for the com example.
User avatar
FanaticGuru
Posts: 1906
Joined: 30 Sep 2013, 22:25

Re: How to merge 2 text files?

14 May 2014, 20:14

Another method is to use a total overkill function

Code: Select all

a  := "cc aa bb"
b  := "zz cc aa ee bb dd ff"

MsgBox % RemoveDuplicate(a " " b, " ")

;{ RemoveDuplicate
; Fanatic Guru
; 2013 05 17
;
; FUNCTION to Remove Duplicate Elements from a String with Lots of Options
;
;------------------------------------------------
;
; Method:
; RemoveDuplicate(string, delim, opt, dupe)
;
; Parameters:
; 1) {string} string to search and remove duplicates from
; 2) {delim} a series of delimiters to be used seperated by |
;      delimiters can be more than one character
;      delim of "" causes each single character to be parsed with no delimiters
;      a | can be used as a delimiter by escaping ie. `|
;      number of delimiters limited to 11, can be increased in function
;      optional, default = "`n"
; 3) {opt} a combination of a number and single characters
;      {number} = the number of same duplicates to keep
;      F = keep First duplicate elements in string
;      L = keep Last duplicate elements in string
;      N = keep None of the duplicate elements in string
;      S = case Sensitive
;      I = case Insensitive
;      E = Empty elements in string are ignored and not removed
;      C = only consider Consecutive elements as duplicate
;      optional, default = "1FS"
; 4) {dupe} only remove duplicate elements that match dupe
;      multipy dupe can be seperated by |
;      optional, default = ""
;
; Returns:
; String with duplicates removed per options 
;
; Example:
; RemoveDuplicate(string,"`n|===|`||End","2SLEC","Cat|Dog|6")
;
RemoveDuplicate(string,delim:="`n",opt:="1FS",dupe:="")
{
	IfInString, opt, S	; Case Sensitive
		Opt_CS := true
	else
		Opt_CS := false
	IfInString, opt, I	; Case Insensitive
		Opt_CS := false
	else
		Opt_CS := true
	IfInString, opt, F	; Keep First Duplicate
		Opt_Last := false
	else
		Opt_Last := true
	IfInString, opt, L	; Keep Last Duplicate
		Opt_Last := true
	else
		Opt_Last := false
	IfInString, opt, E	; Ignore Empty Elements and do not remove them
		Opt_Empty := true
	else
		Opt_Empty := false
	IfInString, opt, C	; Only Save Consecutive Elements are Duplicates
		Opt_Con := true
	else
		Opt_Con := false
	Position := RegExMatch(opt,"P)[0-9]+",Length)
	if position
		Opt_Number := SubStr(opt, Position, Length)	; Number of Duplicates to keep
	else
		Opt_Number := 1		; If no numbers are given, not even zero, then default to 1
	IfInString, opt, N		; Keep None of the Duplicates
		Opt_Number :=0

	if delim
	{
		delimSymbols := "§¢¤¥©ª«®µ£¬¶"	; String must not contain these symbols, add more to increase # of delim allowed
		delim := RegExReplace(delim, "([^``])\|", "$1¦") ; change delimiter used in delim internally to "¦"
		StringSplit, delim_array, delim, "¦"
		if !(delim_array0 * 2 -1 >= StrLen(delim)) ; determine if delim contains multi-character delimiters
		{
			delim_multi := true
			Loop, %delim_array0%	; replace actual delimiters with symbol delimiters if multi-character
			{
				temp_delim := delim_array%A_Index%
				temp_delim_symbol := SubStr(delimSymbols,A_Index,1)
				string := RegExReplace(string, "\Q" temp_delim "\E", temp_delim_symbol)
				delim_parse .= temp_delim_symbol
			}
		}
		else
		{
			StringReplace, delim_parse, delim,¦,,All ; use delimiters as normal if single character
		}
	}
	
	if Opt_Last	; if last dupicates to be saved reverse string
	{
		rev_str := ""
		if delim ; if delim then reverse elements
		{
			string := string . "¶"	; delimiter added to end for when reversing to first element
			delim_parse .= "¶"
			Position := 0
			loop, parse, String, %delim_parse%
			{
				Position += StrLen(A_LoopField) + 1
				rev_str := A_LoopField . SubStr(String, Position, 1) . rev_str
			}
		}
		else ; if no delim then reverse single characters
		{
		VarSetCapacity(rev_str, n:=StrLen(string))
		Loop %n%
			rev_str .= SubStr(string, n--, 1)
		}
		string := rev_str
	}

	str := ""
	oList :=	{}
	CS := ComObjCreate("Scripting.Dictionary")	; Com Objects have case sensitive names unlike regular variables
	Position := 0
	Loop, parse, String, %delim_parse%
	{
		Position += StrLen(A_LoopField) + 1
		if delim
			current_delim := SubStr(String, Position, 1)
		if dupe
		{
			if (Opt_CS and !(RegExMatch(dupe,"(^|\|)" A_LoopField "($|\|)")))
			{
				str .=	A_LoopField current_delim
				Last_Element := A_LoopField
				continue
			}
			if (!Opt_CS and !(RegExMatch(dupe,"i)(^|\|)" A_LoopField "($|\|)")))
			{
				str .=	A_LoopField current_delim
				Last_Element := A_LoopField
				continue
			}
		}
		if Opt_Con
		{
			if Opt_CS
			{
				if (Last_Element == A_LoopField)
					Con_Count += 1
				else
					Con_Count := 0
			}
			else
			{
				if (Last_Element = A_LoopField)
					Con_Count += 1
				else
					Con_Count := 0
			}
			Last_Element := A_LoopField
			if (Opt_Empty and !A_LoopField)
			{
				str .=	A_LoopField current_delim
			}
			else
			{
					
				if (Opt_Number > Con_Count)
				{
					str .=	A_LoopField current_delim
					continue
				}
				else
				{
					continue
				}
			}
		}

		if (A_LoopField	!= "") ; element is not empty / null ie. not two delimiters together
		{
			if Opt_CS
			{
				if	!CS.item(A_LoopField)
				{
					CS.item(A_LoopField) := 1
					str .=	A_LoopField current_delim
				}
				else
				{
					if !Opt_Number
					{
						Dupe_Position := RegExMatch(str, "P)\Q" A_LoopField "\E", Dupe_Length)
						if Dupe_Position
						{
							if (StrLen(str) > Dupe_Position+Dupe_Length+1)
								str := SubStr(str, 1, Dupe_Position-1) . SubStr(str,Dupe_Position+Dupe_Length+1)
							else
								str := SubStr(str, 1, Dupe_Position-1)
						}
					}
					else
					{
						if (CS.item(A_LoopField) < Opt_Number)
						{
							CS.item(A_LoopField) += 1
							str .=	A_LoopField current_delim
						}
					}
				}
			}
			else
			{
				if	!oList[A_LoopField]
				{
					oList[A_LoopField] := 1
					str .=	A_LoopField current_delim
				}
				else
				{
					if !Opt_Number
					{
						Dupe_Position := RegExMatch(str, "P)\Q" A_LoopField "\E", Dupe_Length)
						if Dupe_Position
						{
							if (StrLen(str) > Dupe_Position+Dupe_Length+1)
								str := SubStr(str, 1, Dupe_Position-1) . SubStr(str,Dupe_Position+Dupe_Length+1)
							else
								str := SubStr(str, 1, Dupe_Position-1)
						}
					}
					else
					{
						If (oList[A_LoopField] < Opt_Number)
						{
							oList[A_LoopField] += 1
							str .=	A_LoopField current_delim
						}

					}
				}
			}
		}
		else	; process empty / null element where two delimiters are together
		{
			if	!Empty_Count
			{
				if !Opt_Empty
					Empty_Count := 1
				str .=	A_LoopField current_delim
			}
			else
			{
				if !Opt_Number
				{
					if !empty_str
					{
						Loop, parse, str, %delim_parse%
						{
							Empty_Position += StrLen(A_LoopField) + 1
							if A_LoopField
								Empty_str .= A_LoopField current_delim
						}
						str := Empty_str
					}
				}
				else
				{
					if (Empty_Count < Opt_Number)
					{
						Empty_Count += 1
						str .=	A_LoopField current_delim
					}
				}
			}
		}
	}

	if Opt_Last	; if last dupicates to be saved reverse string back to original order
	{
		rev_str := ""
		if delim
		{
			Position := 0
			loop, parse, Str, %delim_parse%
			{
				Position += StrLen(A_LoopField) + 1
				rev_str := A_LoopField . SubStr(Str, Position, 1) . rev_str
			}
			StringReplace, rev_str, rev_str, ¶,,All	; remove delimiter added to end for when reversing to first element
		}
		else
		{
		VarSetCapacity(rev_str, n:=StrLen(str))
		Loop %n%
			rev_str .= SubStr(str, n--, 1)
		}
		str := rev_str
	}
	
	if delim_multi	; if multi-character delimiters replaced with symbol delimiters
	{
		Loop, %delim_array0%	; replace symbol delimiters with actual delimiters to restore back to original
		{
			temp_delim := delim_array%A_Index%
			temp_delim_symbol := SubStr(delimSymbols,A_Index,1)
			StringReplace, str, str, %temp_delim_symbol%, %temp_delim%, All
		}
	}
	return str
}
;}
FG
Hotkey Help - Help Dialog for Currently Running AHK Scripts
AHK Startup - Consolidate Multiply AHK Scripts with one Tray Icon
Hotstring Manager - Create and Manage Hotstrings
[Class] WinHook - Create Window Shell Hooks and Window Event Hooks

Return to “Ask for Help (v1)”

Who is online

Users browsing this forum: CaseyMoon, Google [Bot], imstupidpleshelp, Rohwedder, ShatterCoder and 175 guests