Posted: 20th Jan 2023 16:43
Everything is in the question.

I want to build a small tool (Windows) that organize photos by date folders and I must be able to move files from source folder to destination one.
Is this possible ?

Regards,
Posted: 20th Jan 2023 17:08
either CreateFileFromMemblock() or MadBit's FileExplore plugin would work.
Posted: 20th Jan 2023 18:07
Yup, the memblock commands are handy for that. Here is a quick function I like to use:

+ Code Snippet
function CopyFile(fileName$,newName$)
    rem copy system using memblocks to retain exact 1:1 file data
    CreateMemblockFromFile(1,fileName$)
    CreateFileFromMemblock(newName$,1)
    DeleteMemblock(1)
endfunction


Then you can optionally delete the original if you just want to move it (and put it into a 'MoveFile' function for easy/repeat implementation).
Posted: 20th Jan 2023 18:54
Memblocks force app to load the file and save it again. it slower than real file copy.
Memblocks are ok when you handle few files, but when you must handle thousands of files, it is not adapted.
A true "Move file" should be used.

Thank you for your answer showing me that AppGameKit does not own true copy/move files functions for Windows.
Posted: 21st Jan 2023 1:37
Memblocks force app to load the file and save it again.


Of course, but any full copy operation is a read/load + write/save operation. So you will always have to load data and then save it out again whenever anything copies data. Is there a reason you expect it wouldn't/shouldn't be that way? What do you mean by 'real file copy'?

I manage about 1800 files with one of my projects and it does quite well. What are your performance results? To compare, I ran a test copying 100 files with that function I posted at 18 MB in size each with a result of 1.67 second (that's moving about 1.8 GB, so <100ms per 100 MB). Performing a 'copy *' operation from the command line is/was about the same.

I also ran a test with 1000 files at 1.8 MB in size each with a result of 1.173 second. So unless you are working with many thousands of files that are enormous (bigger than high mega-pixel JPG images at least), I would not expect performance to be much of an issue for you. And if you were trying to manage that many massive files, you'd probably encounter problems with any file copy method.

Also, be sure to use the file-to-memblock-to-file commands suggested and don't load the file(s) into an image memblock before you save them out. For maximum performance, you want to perform a direct byte-for-byte copy operation from source to destination with no memblock conversions or operations in the middle.

Did you try MadBit's plugin for a third method to compare with? I'd be interested to learn what methods you've tried and what your results are for comparison. Perhaps there is another factor I/we might need to consider.
Posted: 21st Jan 2023 5:54
knocked up some additional testing with madbit's FileExplore and Memblock methods (on my 2-3 yo lenovo laptop's internal SSD with only ~5GB free ).
2,262 Files @ 3.0 GB
files ranged from 1 KB .pngs to 44 MB .gifs (just noted a few ~400KB .pdfs in there if it matters).

FileExplore Copy: 39.58s
FileExplore Move: 0.92s

Memblock Copy: 44.94s
Memblock Move: 17.82s

WIN Copy/Paste ~ 13s ("details"/no thumbs)
WIN Delete ~7s

@SFSW,
can you run the (complete project + 64 bit exe) attached on your batch of files?
maybe the large gifs threw my results off?
ie, how did you achieve a much lower Elapsed avg with Memblocks than i?

otherwise, see anything wrong with my functions? and, there might be some more-useful functions in madbit's set vs manual copy method.
Posted: 21st Jan 2023 7:01
Cool test program you wrote there

I ran the test with the 100 large 18MB files (1.8 GB) and my results were:

'COPY' operation: 1.21 second
'COPY MEM operation: 2.02 seconds
'MOVE' operation: 0.02 second
'MOVE MEM' operation: 2.05 seconds

To get the 'MOVE MEM' operation to work, I just ported the copy function and added a delete command after the copy operation. I also changed the virtual button line, the updated code is as follows:

+ Code Snippet
If GetVirtualButtonPressed(MoveMemBUT) then MoveMemFiles()  // line 50

Function MoveMemFiles()
	ThisFolder = OpenRawFolder(SourceDIR$)
	TheseFiles = GetRawFolderNumFiles(ThisFolder)
	Start# = Timer()
	For x = 0 to TheseFiles
		ThisFile$ = GetRawFolderFileName(ThisFolder,x)
		If ThisFile$ &lt;&gt; ""
			CreateMemblockFromFile(1,"raw:"+SourceDIR$+"\" + ThisFile$)
			CreateFileFromMemblock("raw:"+DestDIR$+"\" + ThisFile$,1)
			DeleteMemblock(1)
            DeleteFile("raw:"+SourceDIR$+"\" + ThisFile$)
		EndIf
	next x
	CloseRawFolder(ThisFolder)

	Elapsed$ = STR(Timer()-Start#,2)
	
	Repeat
		Print( STR(TheseFiles) + " files copied.")
		Print(Elapsed$)
		Sync()
	Until GetPointerPressed()
EndFunction


As for results, this is with my main development system, which is pretty powerful. I suspect the ability to read/write data to/from memory has a lot to do with overall performance, which the laptop may be somewhat slow with (the Windows operations may also be somewhat slow also at around ~20 seconds total, but probably within expected results for such a device). Plus, if you're trying to move/copy 3 GB of data with only 5 GB free on an SSD, that could be a significant limiting factor... not to mention very hard on an SSD when it needs some room to load spread (cell balancing). I'd free up maybe at least 1/3 of the storage on your SSD, then try it again. If you still get similar results, see if you get better results maybe on a different system with better performance (particularly memory).

Either way, those results seem to confirm that FileExplore's copy operation is similar to internal memblock copying. Move has a distinct advantage (likely as well with Windows) with your system, likely attributable to method. But memblock copying/moving is still pretty quick, even on possibly slower machines, relative to Windows. Image files on their own should do pretty well, even larger ones copied in fairly large sets.

Edit: Running the test again with a wide variety of files (types, sizes, names, etc) and over 2800 files, the results are 2.79 seconds for FE's copy method, 0.47 second for FE's move method, 2.4 seconds for 'Copy Mem', and 1.79 for 'Move Mem'. However, I suspect some caching was involved in the latter, so I wouldn't put too much weight in the improved time for secondary operations. But overall, still really fast with more files and a wider variety of file types (including faster than native Windows copy and move operations, although move was fairly close). Windows copy operation was nearly 10 seconds, move was quicker and closer to the copy operations of the other tests.
Posted: 21st Jan 2023 7:33
hmm. adding the MemMove() and Move < Copy:

Memblock Copy: 44.94s
Memblock Move: 17.82s

re-running some of the others and my speeds are all over the place (last "copy" < 15 secs)... so, my data is garbage.

i did buy a 6 TB external ~xmas that i need to set up

thanks for testing/finishing/fixing it. i might expand it "some day".
Posted: 21st Jan 2023 17:26
I think if the program created new test files (large text files with random characters in them) each time, then started the timer to monitor copy duration, it would probably provide more accurate consistent results without caching offsets. But it's still valuable to gauge overall results, even with caching involved.
Posted: 21st Jan 2023 20:42
Here is an update version of the program that creates 1000 files with random line lengths for a more consistent test result:

+ Code Snippet
// Project: MoveFiles
// By: Virtual Nomad, updated by SFSW/Vice 2023-01-21 to incorporate Move Mem operation and building random initial files to avoid cache interference
// Created: 2023-01-20

// show all errors
SetErrorMode(2)

// set window properties
SetWindowTitle( "MoveFiles" )
SetWindowSize(1280,720, 0 )
SetWindowAllowResize( 0 ) // allow the user to resize the window

// set display properties
SetVirtualResolution( 1280,720 ) // doesn't have to match the window
SetOrientationAllowed( 1, 1, 1, 1 ) // allow both portrait and landscape on mobile devices
SetSyncRate( 30, 0 ) // 30fps instead of 60 to save battery
SetScissor( 0,0,0,0 ) // use the maximum available screen space, no black borders
UseNewDefaultFonts( 1 ) // since version 2.0.22 we can use nicer default fonts
SetPrintSize(24)
#import_plugin FileExplore as FE

CenterWindow()

GLOBAL SourceDIR$, DestDIR$, TotalFiles
	LoadPrefs()

// start by creating 100 large text files with random line lengths to avoid cache interference with test results
for i = 1 to 1000
if i=1 or mod(i,10)=0
    print("Removing any old files, please wait...")
    print(str(i/10)+"%")
    sync()
endif
s$="raw:"+SourceDIR$+"\"+"FileTest"+str(i)+".txt"
rem remove any old residual versions of the files first
if GetFileExists(s$)=1 then DeleteFile(s$)
rem check the destination folder also
s$="raw:"+DestDIR$+"\"+"FileTest"+str(i)+".txt"
if GetFileExists(s$)=1 then DeleteFile(s$)
next i

for i = 1 to 1000
if i=1 or mod(i,10)=0
    print("Building Random Files...")
    print(str(i/10)+"%")
    sync()
endif
rem now create the random text files
s$="raw:"+SourceDIR$+"\"+"FileTest"+str(i)+".txt"
OpenToWrite(1,s$)
for ii = 1 to 1000
ss$=spaces(random(100,200))
WriteLine(1,ss$)
next ii
CloseFile(1)
next i


TotalFiles=CountFiles(SourceDIR$)

GLOBAL FromBUT, ToBUT, CopyBUT, MoveBUT, CopyMemBUT, MoveMemBUT
	FromBUT = 1	:	ToBUT = 2	:	CopyBUT = 3	:	MoveBUT = 4
	CopyMemBUT = 5	:	MoveMemBUT = 6
	AddVirtualButton(FromBUT,32,688,64)		:	SetVirtualButtonText(FromBUT,"SOURCE$")
		SetVirtualButtonAlpha( FromBUT, 128 )
	AddVirtualButton(ToBUT,96,688,64)		:	SetVirtualButtonText(ToBUT,"DEST$")
		SetVirtualButtonAlpha( ToBUT, 128 )
	AddVirtualButton(CopyBUT,160,688,64)	:	SetVirtualButtonText(CopyBUT,"COPY")
		SetVirtualButtonAlpha( CopyBUT, 128 )
	AddVirtualButton(MoveBUT,224,688,64)	:	SetVirtualButtonText(MoveBUT,"MOVE")
		SetVirtualButtonAlpha( MoveBUT, 128 )
	AddVirtualButton(CopyMemBUT,288,688,64)	:	SetVirtualButtonText(CopyMemBUT,"COPY" + CHR(10) + "MEM")
		SetVirtualButtonAlpha( CopyMemBUT, 128 )
	AddVirtualButton(MoveMemBUT,352,688,64)	:	SetVirtualButtonText(MoveMemBUT,"MOVE" + CHR(10) + "MEM")
		SetVirtualButtonAlpha( MoveMemBUT, 128 )


do
    If GetVirtualButtonReleased(FromBUT) then SetSource()
    If GetVirtualButtonReleased(ToBUT) then SetDestination()
    If GetVirtualButtonPressed(CopyBUT) then CopyFiles()
	If GetVirtualButtonPressed(MoveBUT) then MoveFiles()
	If GetVirtualButtonPressed(CopyMemBUT) then CopyMemFiles()
	If GetVirtualButtonPressed(MoveMemBUT) then MoveMemFiles()

    Print( "From: " + SourceDIR$ + " @ " + STR(TotalFiles) + " files.")
    Print( "To: " + DestDIR$ )

    Sync()
loop

Function LoadPrefs()
	If GetFileExists("mfprefs.data")
		ThisFile = OpenToRead("mfprefs.data")
			SourceDIR$ = ReadLine(ThisFile)
			` TotalFiles = CountFiles(SourceDIR$)  ` wait until after any old files are cleared and new ones are created
			FE.SetWorkingDirectory(SourceDIR$)
			DestDIR$ = ReadLine(ThisFile)
		CloseFile(ThisFile)
	Else
		SourceDIR$ = GetReadPath()
		FE.SetWorkingDirectory(SourceDIR$ )
		DestDIR$ = GetWritePath()
	EndIf
EndFunction

Function SavePrefs()
	ThisFile = OpenToWrite("mfprefs.data")
		WriteLine(ThisFile, SourceDIR$)
		WriteLine(ThisFile, DestDIR$)
	CloseFile(ThisFile)
EndFunction

Function SetSource()
	ThisDIR$ = FE.ChooseFolderDialog("Select Source Folder", "")
	If ThisDIR$ &lt;&gt; ""
		SourceDIR$ = ThisDIR$
		SavePrefs()
		FE.SetWorkingDirectory(ThisDIR$)
		TotalFiles = CountFiles(ThisDIR$)
	EndIf
EndFunction

Function SetDestination()
	ThisDIR$ = FE.ChooseFolderDialog("Select Destination Folder", "")
	If ThisDIR$ &lt;&gt; ""
		DestDIR$ = ThisDIR$
		SavePrefs()
	EndIf
EndFunction

Function CountFiles(dir$)
	ThisFolder = OpenRawFolder(dir$)
		TotalFiles = GetRawFolderNumFiles( ThisFolder )
	CloseRawFolder(ThisFolder)
EndFunction TotalFiles

Function CopyFiles()
	TheseFiles = 0
	ThisFile$ = FE.GetFirstFile("*.*")
	Start# = Timer()
	Repeat
		FE.FileCopy(ThisFile$, DestDIR$+"\"+ThisFile$,0)
		INC TheseFiles
		ThisFile$ = FE.GetNextFile()
	Until ThisFile$ = ""
	
	Elapsed$ = STR(Timer()-Start#,2)
	
	Repeat
		Print( STR(TheseFiles) + " files copied.")
		Print(Elapsed$)
		Sync()
	Until GetPointerPressed()
	
	OpenBrowser(DestDIR$)
EndFunction

Function MoveFiles()
	TheseFiles = 0
	ThisFile$ = FE.GetFirstFile("*.*")
	Start# = Timer()
	Repeat
		FE.FileMove(ThisFile$, DestDIR$+"\"+ThisFile$)
		INC TheseFiles
		ThisFile$ = FE.GetNextFile()
	Until ThisFile$ = ""
	
	Elapsed$ = STR(Timer()-Start#,2)
	
	Repeat
		Print( STR(TheseFiles) + " files copied.")
		Print(Elapsed$)
		Sync()
	Until GetPointerPressed()
	
	OpenBrowser(DestDIR$)
EndFunction

Function CopyMemFiles()
	ThisFolder = OpenRawFolder(SourceDIR$)
	TheseFiles = GetRawFolderNumFiles(ThisFolder)
	Start# = Timer()
	For x = 0 to TheseFiles
		ThisFile$ = GetRawFolderFileName(ThisFolder,x)
		If ThisFile$ &lt;&gt; ""
			CreateMemblockFromFile(1,"raw:"+SourceDIR$+"\" + ThisFile$)
			CreateFileFromMemblock("raw:"+DestDIR$+"\" + ThisFile$,1)
			DeleteMemblock(1)
		EndIf
	next x
	CloseRawFolder(ThisFolder)

	Elapsed$ = STR(Timer()-Start#,2)
	
	Repeat
		Print( STR(TheseFiles) + " files copied.")
		Print(Elapsed$)
		Sync()
	Until GetPointerPressed()
	
	OpenBrowser(DestDIR$)
EndFunction

Function MoveMemFiles()
	ThisFolder = OpenRawFolder(SourceDIR$)
	TheseFiles = GetRawFolderNumFiles(ThisFolder)
	Start# = Timer()
	For x = 0 to TheseFiles
		ThisFile$ = GetRawFolderFileName(ThisFolder,x)
		If ThisFile$ &lt;&gt; ""
			CreateMemblockFromFile(1,"raw:"+SourceDIR$+"\" + ThisFile$)
			CreateFileFromMemblock("raw:"+DestDIR$+"\" + ThisFile$,1)
			DeleteMemblock(1)
            DeleteFile("raw:"+SourceDIR$+"\" + ThisFile$)
		EndIf
	next x
	CloseRawFolder(ThisFolder)

	Elapsed$ = STR(Timer()-Start#,2)
	
	Repeat
		Print( STR(TheseFiles) + " files copied.")
		Print(Elapsed$)
		Sync()
	Until GetPointerPressed()
EndFunction

Function CenterWindow()
	X = GetMaxDeviceWidth()/2.0 - GetWindowWidth()/2.0
	Y = GetMaxDeviceHeight()/2.0 - GetWindowHeight()/2.0
	SetWindowPosition( X,Y)
EndFunction


For me, I get pretty consistent times when running each method repeatedly. So this may be a better overall gauge of performance. I generally get in the 0.95 to 1.2 second range (few tenths slower for mem operations, so marginal difference). Caching may still get involved from time to time as occasionally, delays in retesting seem to slow down the initial delete operation significantly. But that would likely be true of any file management command, not just AGK's. Overall, all are comparatively very fast.
Posted: 21st Jan 2023 22:42
You may run into issues when you copy a file > 4GB
Posted: 28th Jan 2023 10:41
Thanks for sharing SFSW and VN
@Blink0k I think windows always had a problem with large file copying. hence why programs like unstoppable copier etcetera were created in the first place
https://www.majorgeeks.com/files/details/unstoppable_copier.html there are several variants of this style of program out there, tier 2 might be the
better approach if you are after true file copying, doesnt mean it won't hang on certain unpredictable interruptions and we have the ability to error
trap interrupts I predict there will always be some type of issue as with 90 percent of software out there.