Rare Filecopy Issue

Get help with using AutoHotkey (v1.1 and older) and its commands and hotkeys
zcooler
Posts: 455
Joined: 11 Jan 2014, 04:59

Rare Filecopy Issue

11 Jan 2014, 05:21

Trying it on this site too.
Had this issue for years now but never been abled to figure out whats going wrong or why it happens.

Code: Select all

EPG_files_location := "D:\TV\Recording Service\EPG Info-Log Files"
Recording_database_backup_location_1 := "D:\AVI_VOB_BLU-RAY_AC3-tools\TV-tools\EPG Info-Log Files Backup"
FileCopy, %EPG_files_location%\*.*, %Recording_database_backup_location_1%\, 0 ;Backup of database
Sometimes (quite rare) all files in "EPG_files_location" gets copied to the D:\ root directory instead of to the correct location specified in the "Recording_database_backup_location_1" variable. Maybe someone have any clue upon why?
Anyone seen it before?
User avatar
joedf
Posts: 8959
Joined: 29 Sep 2013, 17:08
Location: Canada
Contact:

Re: Rare Filecopy Issue

11 Jan 2014, 13:01

Does the folder exist? How big are the files? Have you tried a file-loop?
Image Image Image Image Image
Windows 10 x64 Professional, Intel i5-8500, NVIDIA GTX 1060 6GB, 2x16GB Kingston FURY Beast - DDR4 3200 MHz | [About Me] | [About the AHK Foundation] | [Courses on AutoHotkey]
[ASPDM - StdLib Distribution] | [Qonsole - Quake-like console emulator] | [LibCon - Autohotkey Console Library]
zcooler
Posts: 455
Joined: 11 Jan 2014, 04:59

Re: Rare Filecopy Issue

11 Jan 2014, 13:44

joedf wrote:Does the folder exist? How big are the files? Have you tried a file-loop?
Yes the folder exists. Out of 100 copy sessions maybe 1 fails and it's then the copies ends up in the root directory. The size of each file is small (1kb), but the database files are quite many..at least 6000 in source folder and 11000 in destination folder. The Filecopy command resides in a file-loop yes.

My thoughts are that the Filecopy command can't cope in a certain situation (haven't seen any logic or pattern when the issue strikes). I suppose if having several recordings going at once and all of them finishes at the same time, several instances of the script with the file-loop gets executed simultaneously and I think that's the most common case when this issue occurs, but I have seen the issue occur even when only one recording finishes and executes the script (after recording task). So I don't know what's causing it.

Since I didn't expect anyone to help me enlighten this problem I changed my approach slightly. Instead of copying the whole database with 6000 files over to the destination backup folder and AHK gets forced to check the destination folder to NOT overwrite already copied files, I now only copy the database files dropped onto script to backup destination folder. I suspect that will work better. However the original issue persists if doing it according to my old way and I don't know if it could be a Filecopy command bug.
User avatar
joedf
Posts: 8959
Joined: 29 Sep 2013, 17:08
Location: Canada
Contact:

Re: Rare Filecopy Issue

11 Jan 2014, 15:32

Hmm ok, I see I will investigate soon... Maybe if you tried 'XCOPY'?
Image Image Image Image Image
Windows 10 x64 Professional, Intel i5-8500, NVIDIA GTX 1060 6GB, 2x16GB Kingston FURY Beast - DDR4 3200 MHz | [About Me] | [About the AHK Foundation] | [Courses on AutoHotkey]
[ASPDM - StdLib Distribution] | [Qonsole - Quake-like console emulator] | [LibCon - Autohotkey Console Library]
zcooler
Posts: 455
Joined: 11 Jan 2014, 04:59

Re: Rare Filecopy Issue

12 Jan 2014, 11:11

joedf wrote:Hmm ok, I see I will investigate soon... Maybe if you tried 'XCOPY'?
Hmm, XCOPY isn't a AHK command. I suppose you mean calling xcopy via comspec?
Well, I think i'll stick with my minor change of approach I did mention. Time will tell if I get files copied to root or not. Thanks joedf!
Morpheus wrote:You could try Skan's shellfileoperation()
http://www.autohotkey.com/forum/viewtop ... 249#133249
Well, here's the twist...the file-loop gets executed from a service application. The service account have limited permissions if comparing to the ordinary desktop account, so I can't call the shellfileoperation function which is using visible UI. That would mean script will stall. This fact can also have bearing on the topic issue. Only can't work out why the majority of the copy sessions goes well and a few goes bad.
User avatar
joedf
Posts: 8959
Joined: 29 Sep 2013, 17:08
Location: Canada
Contact:

Re: Rare Filecopy Issue

12 Jan 2014, 19:49

Hmm ok, add some error logging in your script, so that when it fails,
Log Errorlevel, A_LastError, etc. And thé command that the command that failed.. So that we may investigate further...
Image Image Image Image Image
Windows 10 x64 Professional, Intel i5-8500, NVIDIA GTX 1060 6GB, 2x16GB Kingston FURY Beast - DDR4 3200 MHz | [About Me] | [About the AHK Foundation] | [Courses on AutoHotkey]
[ASPDM - StdLib Distribution] | [Qonsole - Quake-like console emulator] | [LibCon - Autohotkey Console Library]
zcooler
Posts: 455
Joined: 11 Jan 2014, 04:59

Re: Rare Filecopy Issue

13 Jan 2014, 15:47

Aha, that will be interesting! Never used error logging before. It may take some time until the issue shows up though. Will report back when it does. Thanks joedf :)
zcooler
Posts: 455
Joined: 11 Jan 2014, 04:59

Re: Rare Filecopy Issue

15 Jan 2014, 15:56

joedf wrote:Hmm ok, add some error logging in your script, so that when it fails,
Log Errorlevel, A_LastError, etc. And thé command that the command that failed.. So that we may investigate further...
Ok, added some errorlogging and i get an error 80 every time using the Filecopy file-loop even when filecopy works and files arrives to the correct destination folder. Error 80 means:
ERROR_FILE_EXISTS
80 (0x50)
The file exists.
Dont understand the error. All sourcefiles already exists in the destination folder, except the ones dropped onto script (the ones that should be copied). The 0 paramter just tells FileCopy to not overwrite already existing files.
I admit this way of doing a backup isnt that smooth, but this was the method I chose when being totally new with AHK. I do have a better way now, but like to know what is going wrong with my old way.

Havent got any case yet when all source files ends up in the root instead of the destination folder. Wonder which error code that would bring.
zcooler
Posts: 455
Joined: 11 Jan 2014, 04:59

Re: Rare Filecopy Issue

19 Oct 2014, 13:42

The problem with files being copied to the D:\ root folder instead of to the destination folder is back even with the new method. Im stumped and lost for words. What the heck is happening here? It should not occur cuz i do use absolute paths.
Maybe, someone could have a look at this fileloop and let me know if seeing anything that might explain the issue?
The issue does occur quite rarely and I have still no clue what may trigger it. I've marked in the script where it sometimes goes wrong. Maybe some permission issue cuz this script is executed from the service account?

Code: Select all

EPG_files_location := "D:\TV\Recording Service\EPG Info-Log Files"
Recording_database_backup_location_1 := "D:\AVI_VOB_BLU-RAY_AC3-tools\TV-tools\EPG Info-Log Files Backup"
PropCopy_location := "D:\AVI_VOB_BLU-RAY_AC3-tools\TV-tools\DVBViewer\PropCopy_1_2_3\PropCopy.exe"

Loop %0%  ; For each parameter (or file dropped onto a script):
{
  Givenpath := %A_Index%  ; Fetch the contents of the variable whose name is contained in A_Index.
  Loop %GivenPath%, 1
  {
   Params := A_LoopFileLongPath
   PutItToTheSlaughter .= """" A_LoopFileLongPath """ "
  }
  ;msgbox % Params
  ;msgbox % PutItToTheSlaughter
  AddToBOF(RS_recording_folder . "\hiberscript_files.log", a_yyyy "-" a_mm "-" a_dd " " a_hour ":" a_min ":" a_sec " - " . Params . "`r`n")
  SplitPath, Params, OutFileName, OutDir,, OutNameNoExt
  FileCopy, %OutDir%\%OutNameNoExt%.log, %EPG_files_location%\ ;RS logfile will be copied to backupfolder
  FileMove, %OutDir%\%OutNameNoExt%.txt, %EPG_files_location%\ ;EPG information file will be moved to a subfolder
  Filemove, %OutDir%\%OutNameNoExt%.log, %EPG_files_location%\%OutNameNoExt%.ts ;RS Log file will be moved and renamed to .ts extension
  RunWait, %PropCopy_location% "%RS_recording_folder%\%OutNameNoExt%.ts" "%EPG_files_location%\%OutNameNoExt%.ts" ;Metadata/Fileinfo copy
  FileCopy, %EPG_files_location%\%OutNameNoExt%.*, %Recording_database_backup_location_1%\, 0 ;Backup of database ;>>>>>>>>>>> HERE IT SOMETIMES GOES WRONG
  refresh_recDb := URLToVar(Recording_Service_API_URL . "/tasks.html?task=RefreshDB&aktion=tasks") ;Refresh Recording Service Database
}
just me
Posts: 9458
Joined: 02 Oct 2013, 08:51
Location: Germany

Re: Rare Filecopy Issue

20 Oct 2014, 03:41

You might try to replace the one FileCopy with a File-Loop:

Code: Select all

#NoEnv
EPG_files_location := "D:\TV\Recording Service\EPG Info-Log Files"
Recording_database_backup_location_1 := "D:\AVI_VOB_BLU-RAY_AC3-tools\TV-tools\EPG Info-Log Files Backup"
PropCopy_location := "D:\AVI_VOB_BLU-RAY_AC3-tools\TV-tools\DVBViewer\PropCopy_1_2_3\PropCopy.exe"

Loop %0%  ; For each parameter (or file dropped onto a script):
{
  Givenpath := %A_Index%  ; Fetch the contents of the variable whose name is contained in A_Index.
  Loop %GivenPath%, 1
  {
   Params := A_LoopFileLongPath
   PutItToTheSlaughter .= """" A_LoopFileLongPath """ "
  }
  ;msgbox % Params
  ;msgbox % PutItToTheSlaughter
  AddToBOF(RS_recording_folder . "\hiberscript_files.log", a_yyyy "-" a_mm "-" a_dd " " a_hour ":" a_min ":" a_sec " - " . Params . "`r`n")
  SplitPath, Params, OutFileName, OutDir,, OutNameNoExt
  FileCopy, %OutDir%\%OutNameNoExt%.log, %EPG_files_location%\ ;RS logfile will be copied to backupfolder
  FileMove, %OutDir%\%OutNameNoExt%.txt, %EPG_files_location%\ ;EPG information file will be moved to a subfolder
  Filemove, %OutDir%\%OutNameNoExt%.log, %EPG_files_location%\%OutNameNoExt%.ts ;RS Log file will be moved and renamed to .ts extension
  RunWait, %PropCopy_location% "%RS_recording_folder%\%OutNameNoExt%.ts" "%EPG_files_location%\%OutNameNoExt%.ts" ;Metadata/Fileinfo copy
  Loop, %EPG_files_location%\%OutNameNoExt%.*
    FileCopy, %A_LoopFileLongPath%, %Recording_database_backup_location_1%\%A_LoopFileName%, 0 ; Backup of database
  refresh_recDb := URLToVar(Recording_Service_API_URL . "/tasks.html?task=RefreshDB&aktion=tasks") ;Refresh Recording Service Database
}
zcooler
Posts: 455
Joined: 11 Jan 2014, 04:59

Re: Rare Filecopy Issue

20 Oct 2014, 12:21

Thanks just me! I will try it and also throw some errorlevel logging on it by using your improved and excellent reverse logging function :D This issue has bugged me for years and i still havent the slightest idea what may cause it. Often I happen to discover it when working in the explorer and by then it could have gone days since the issue occcured and i have of course no recollection of the circumstances around the error. Im thinking of adding some kind of visible alert if/when errorlevel >= 1. However there is a problem. The script is launched in the service account and the MS permission limitations prevents me from using any kind of visible userinterface such as msgboxes, that would stall the afterrecording task that launches this script. I was thinking maybe it could work if script triggers some task in the windows scheduler. I definitely have to look up if that is a viable way to go.

Best regards
zcooler

Code: Select all

Loop, %EPG_files_location%\%OutNameNoExt%.* {
  	FileCopy, %A_LoopFileLongPath%, %Recording_database_backup_location_1%\%A_LoopFileName%, 0 ; Backup of database
    AddToBOF(A_ScriptDir . "\DatabaseCopyError.log", a_yyyy "-" a_mm "-" a_dd " " a_hour ":" a_min ":" a_sec " - " . A_LastError . "`r`n")
}
EDIT;::: WAIT, i have it!! I can pipe over the alert to a named pipe server ;)
zcooler
Posts: 455
Joined: 11 Jan 2014, 04:59

Re: Rare Filecopy Issue

28 Feb 2015, 15:26

Hi guys!

I got a FileCopy meltdown again even with the File-Loop where all files in source folder mysteriously gets copied to D:\ root folder. First time since October last year and changing to just me's File-Loop. All the File-Loop should do is copy three files (".ts", ".txt" and ".log") from source folder to destination folder (not root), instead AHK copies the whole source folder content (5000 files) to D:\ root. The database error log should have 3 entries when all three files are copied successfully and all with ErrorLevel 0. The log during the FileCopy meltdown was only one entry with ErrorLevel 0, cuz filecopy loop starts with the .log file and was breaked when a .log file was found in root. ErrorLevel on that single entry unfortunately doesnt give any clues why all source folder files were copied to root. The files in FileCopy-Loop when error striked was:
D:\TV\Recording Service\EPG Info-Log Files\20150226_22-17-43_TV6_Prometheus - ★★★★★★★☆☆☆.ts
D:\TV\Recording Service\EPG Info-Log Files\20150226_22-17-43_TV6_Prometheus - ★★★★★★★☆☆☆.txt
D:\TV\Recording Service\EPG Info-Log Files\20150226_22-17-43_TV6_Prometheus - ★★★★★★★☆☆☆.log

Now the peculiar thing is I have recorded many movies with these Unicode stars in the file name and it haven't gone wrong since October 2014. I bet these stars was involved then too. So, I wonder are we talking about a AHK FileCopy bug here? Some AHK internal DLL argument or special Unicode characters handling issue maybe? Why does the error strike so far in between?

This is the code i was using:
Spoiler
//zcooler
zcooler
Posts: 455
Joined: 11 Jan 2014, 04:59

Re: Rare Filecopy Issue

01 Mar 2015, 14:06

Im trying to avoid using the AHK FileCopy command and use the CopyFileEx function directly to see if that gives the same result. Don't exactly know which function AHK uses for FileCopy command? I do not think this is an AHK bug cuz, if that were the case this error would have shown itself years back, but then again I don't think many people are using Unicode stars in the filename like I do.

Tweaking a CopyFileEx function and inserting it in the script it looks like this:
Spoiler
I know there are params in there im not using. Im more interested if someone could help out optimizing the function in order to retrieve as much debug information out of it as possible, but suspect ErrorLevel is the only info that is returned. Does anyone have other ideas how to debug this?

regards
zcooler
zcooler
Posts: 455
Joined: 11 Jan 2014, 04:59

Re: Rare Filecopy Issue

04 Sep 2015, 12:59

I got the FileCopy error again today. Now 6800 files was wrongly copied to root. I'm sure the FileCopy command has a bug. Another case were peculiar stuff happens when Unicode stars is in the mix under specific situations. The difficulties to reproduce this error makes it far fetched expecting some kind of fix. No choice but to stop using FIleCopy and test how CopyFileEx will do the job.
zcooler
Posts: 455
Joined: 11 Jan 2014, 04:59

Re: Rare Filecopy Issue

20 Sep 2015, 15:43

CopyFileEx failed handling those Unicode stars today :( All those 6000+ files was wrongly copied to root. Hmm...I guess it's time to test out XCOPY and Skan's shellfileoperation() next.
zcooler
Posts: 455
Joined: 11 Jan 2014, 04:59

Re: Rare Filecopy Issue

23 Sep 2015, 14:02

Morpheus wrote:You could try Skan's shellfileoperation()
http://www.autohotkey.com/forum/viewtop ... 249#133249
Seems SKANs version doesnt support unicode. XCOPY and Robocopy seems to be ancient and I wont even try em to handle the unicode stars. Wow, I found this very promising function which I think is made by the great just me :D Im positive this is the solution to my problems :thumbup: It has been working nicely thus far.

Code: Select all

Loop, %EPG_files_location%\%OutNameNoExt%.* {
	FilesToCopy := [A_LoopFileLongPath]                                            ;ShellFileOperation SourceArray
	SHFileOperation(FilesToCopy, Recording_database_backup_location_1, 2)          ;ShellFileOperation
  }
; ==================================================================================================================================
; Copy or move files and folders using the SHFileOperation() function.
; SHFileOperation -> msdn.microsoft.com/en-us/library/bb762164(v=vs.85).aspx
; Parameters:
;     SourcesArray   -  Array of fully qualified source pathes.
;     TargetPath     -  Fully qualified path of the destination folder.
;     Operation      -  FO_MOVE = 1, FO_COPY = 2, other operations are not supported.
;     HWND           -  A handle to the window which will own the dialog.
;     Flags          -  Any combination of the FOF_ flags -> msdn.microsoft.com/en-us/library/bb759795(v=vs.85).aspx
;                       Default: FOF_NOCONFIRMMKDIR = 0x0200.
; Return values:
;     Returns 1 (True) if successful; otherwise 0 (False).
; ==================================================================================================================================
SHFileOperation(SourcesArray, TargetPath, Operation, HWND := 0, Flags := 0x0200) {
   Static TCS := A_IsUnicode ? 2 : 1 ; size of a TCHAR
   If Operation Not In 1,2
      Return False
   ; Count files and total string length
   TotalLength := 0
   Files := []
   For Each, FilePath In SourcesArray {
      If (Length := StrLen(FilePath))
         Files.Push({Path: FilePath, Len: Length + 1})
      TotalLength += Length
   }
   FileCount := Files.Length()
   If !(FileCount && TotalLength)
      Return
   ; Store the source pathes in Sources (the string must be double-null terminated)
   VarSetCapacity(Sources, (TotalLength + FileCount + 1) * TCS, 0)
   Offset := 0
   For Each, File In Files
      Offset += StrPut(File.Path, &Sources + Offset, File.Len) * TCS
   ; Store the target path in Target (the string must be double-null terminated)
   TargetLen := StrLen(TargetPath) + 2
   VarSetCapacity(Target, TargetLen * TCS, 0)
   Target := TargetPath
   ; Create and fill the SHFILEOPSTRUCT
   SHFOSLen := A_PtrSize * (A_PtrSize = 8 ? 7 : 8)
   VarSetCapacity(SHFOS, SHFOSLen, 0) ; SHFILEOPSTRUCT
   NumPut(HWND, SHFOS, 0, "UPtr") ; hwnd
   NumPut(Operation, SHFOS, A_PtrSize, "UInt") ; wFunc
   NumPut(&Sources, SHFOS, A_PtrSize * 2, "UPtr") ; pFrom
   NumPut(&Target, SHFOS, A_PtrSize * 3, "UPtr") ; pTo
   NumPut(Flags, SHFOS, A_PtrSize * 4, "UInt") ; fFlags
   If (A_IsUnicode)
      Return !DllCall("Shell32.dll\SHFileOperationW", "Ptr", &SHFOS, "Int")
   Else
      Return !DllCall("Shell32.dll\SHFileOperationA", "Ptr", &SHFOS, "Int")
 
}

Return to “Ask for Help (v1)”

Who is online

Users browsing this forum: filipemb, rc76, uchihito and 376 guests