Great function, thanks for posting.
I'd identified an error (optimization really) that was causing some issues when the first pixel was not unique on the screen. After the 2 for loops that check the individual pixels, I updated StartX to the last X value returned by PixelSearch (plus one so it increments the loop).
With the original implementation I found PixelSearch was repeatedly returning the same pixel until the increment eventually caught up; which could be very wasteful in some cases

.
For some people who are having trouble getting this to work on rectangular areas try the change below as well; I had one strange case where the function returned false negatives because the outer loop (length) was ending for some reason before the "line search" loop found the correct pixel.
Code:
FindMatrix(ScanStartX,ScanEndX,ScanStartY,ScanEndY,Dimension,Matrix)
{
StringSplit, Pix, Matrix, |
Length := ScanEndY - ScanStartY
pX = 0
pY = 0
;loop for per line of screenY
Loop, %Length%
{
StartX = %ScanStartX%
EndX = %ScanEndX%
StartY := ScanStartY + A_Index - 1
EndY := ScanStartY + A_Index - 1
;loop for searching inside a line
Loop
{
count = 0
MatchFound = 1
;change variance and 'Fast' if not suited
PixelSearch, pX, pY, %StartX%, %StartY%, %EndX%, %EndY%, %Pix1%, 0, Fast
IfNotEqual, ERRORLEVEL, 0, Break
sX = %pX%
sY = %pY%
;loop for matching
Loop, %Dimension%
{
Loop, %Dimension%
{
Count ++
PixelGetColor, CurrPix, %pX%, %pY%
IfNotEqual, Pix%Count%, %CurrPix%
MatchFound = 0
IfEqual, MatchFound, 0, Break
pX ++
}
IfEqual, MatchFound, 0, Break
pY ++
pX -= %Dimension%
}
;match found!
IfEqual, MatchFound, 1
{
Result = %sX% %sY%
Return Result
}
StartX := px + 1
DiffX := EndX - StartX
IfLess, DiffX, %Dimension%, Break
}
StartY ++
EndY ++
}
Return 0
}
Also note: while I'm not posting I also made an edit to allow for a "transparent" value in the matrix, as I had a need to search for patterns rather than exact squares / rectangles.