experchange > delphi

skybuck2000 (12-05-18, 06:17 PM)
Hello,

My Windows Live Mail file disk image has become full (4 gigabytes).

I am now in the process of copieing these files over to a new file disk image of 8 gigabytes.

(I am also playing the Starwars Soundtrack at the same and buffering is at 30.000 milliseconds, so don't think that has much to do with it).

60.000 items like eml and nws files from windows live mail are being copiedfrom

drive P: to drive W:

(I set the new filedisk image/file system to gpt/ntfs and 512 bytes per sector.)

What I am noticing during this copy process is the following:

As the number of items remaining goes down from 60.000 to 50.000 the speed is constantly dropping.

It started at 5 megabyte/secs or even 15 megabytes/sec... and now it's dropping
to 4 mb/sec 3.9, 3.8, 3.7, 3.6, 3.5.... and so forth... every 100 files or so, the copieing speed drops with 100 kilobytes/sec which is quite severe !

It's now down to 1.90 megabytes/sec and continues to drop !

I am now starting to understand why windows live mail was so slow starting all these weeks/months.

It's apperently a very serious issue with NTFS itself... storing all these files into one or two folders makes NTFS very slow...

Probably because it has to find the file's entries somewhere.

Now I am not sure about my hypothesis above.

I have some further information:

The 2 terrabyte C: drive was defragged very recently, up to 5 passes.

So in all likelyhood to new 8 gigabytes should be pretty consolidated, big large chucnks... very little file fragmentation.

So the question is why is the speed so low and also why is the speed dropping ?

There are two "processes" at work:

"reading"
"writing"

At first I was also considering the possibility that it's perhaps a disk/platter issue... perhaps the cylinder/track it's on is slowing down... but I don't think it would be this extreme... so I'd think I can rule that out.

It has something to do with the file system itself.

So there are two possibilities:

1. Either the reading process is slowing down because the newer files are more fragmented.

This is a valid hypothesis, which could be de-bunked by first defragging the 4 gb "drive" and then repeating the copy process to a new 8 gb drive to see if the same performance degradation occurs.

If it does occur then I will consider it proven that NTFS slows down in a big way... the more files there are in the folder.

For now MY NEW HOPE lol... is that it is a file fragmentation issue on the reading drive... and not a file system issue on the writing drive/NTFS in general.

Otherwise this would be a somewhat sad/untested design of Windows Live Mail..

I highly recommend future e-mailing/newsgrouping software is tested with synthetic generation of millions of files to test what performance will be after years of usage and adjusting storage structures accordingly to handle such large ammount of files performance-wise efficiently if possible.

Maybe if I am lucky it was reading drive fragmentation and the new drive will performance better.

Further notes of importance: the defrag tool didn't really show that much file fragmentation on this P/reading drive if I remember correctly sub 20%.... maybe even a few percent.

I may report back later with my findings.

Bye for now,
and me the force be with you,
always ! =D
Skybuck Flying, with a not so flying copieing process, will still take 25 minutes... total time probably 1 hour ;) worth it though.

More testing/defragging/copieing will be done to shed some more light on this in coming days, stay tuned.
skybuck2000 (12-05-18, 06:59 PM)
The performance degradation is very real, at least with these two "virtual drives" which are based on a real drive which was defragmented recently, perhaps not entirely...

But the performance degradation is very real... same pattern is occuring.

First it was 60 megabyte/sec very briefly or so or 30... then 20... then itquickly when down to 8, 6, 5.9 and so forth.

Now some of that can be explained by "disk cache", "ram cache" etc.

But the rest of the time it seems a more severe issue with NTFS.

Unless Microsoft is completely retarded and cannot compute a "real speed/sec" thingy... I don't think their that retarded.

I hope they used a "moving average" or some kind of "sliding window" to compute "speed/sec".

I think Microsoft did use something like that, if correct then this does indicate a strange performance degradation with NTFS.

Which basically disqualifies NTFS somewhat as a decent storage medium.

Hence why I normally used FILE DISK IMAGES... in case I need to move large quanTITIES of files from one drive/computer to another.

NTFS just completely sucks at reading/writing large quanTITIES of files.

No real surprise there... though now in 2018/windows 7... still stucking atthis ?!

Can somebody please design/invent something better perhaps ?!

I am assuming it's an issue with NTFS and not IBM/Hitatchi harddisk reading/writing...

Performance is now down to 4 megabyte/sec and dropping like a BAD JAMES BOND movie...

I am not really amuzed by this and the files aren't even that big... mostly4 kb, 2kb, the occasional 3 megabyte photo file.

So all-in-all very strange performance degradation.

I am glad NTFS exists cause inventing something like that might be a bit hard/tricky but I do feel like there something be something better possible.... that doesn't degrade so badly.

In case you were wondering how I performed/confirmed this suspicion it was done/and currently is doing as follows:

1. I created another virtual disk with computer management...

2. Formatted it quickly.

3. Then performed the same kind of copy from drive to drive (testtesttest).

What's even more alarming is that I did set VCL media player buffering (assuming it's working to not 30 seconds.... but even 60 seconds... I also tried to 300 milliseconds setting, it's not really influecing this test in any big way, currently 60 seconds is being used).

The performance degradation is real and dropping.... right now 3.24 megabyte/sec ?!

This kinda makes me go WTF ?!? hmmmm....

Probably seeks per second limitation. Harddisk can do about 100 i/o operations...

So copieing/seeking all these tiny little shitty files... it's struggling with that...

But why is the bandwidth constantly dropping ? This seems a bit weird... it's like the NTFS accounting is GROWING... slowing it down significantly...

It's like NTFS has to see some kind of table first... find some kind of free entry... and dump/copy the file there... this should be done better somehow... but for example maintaining a pointer to the first free entry or so.... so it doesn't constantly have to seek this as well...

And/or I suspect it's going through some kind of list of entries/blocks/sectors... something like that...

As this list grows it slows down... which kinda sucks.

I have seen enough... the performance problem of copieing these shitty little files is real.

Let's compute some numbers.

100 x 2 kb = 400 kilobyte/sec copies if all files were 2kb... ok 2 megabytes is not so bad.

Also what is up with "system" creating vhd disks ?! it cannot be throttled ?! it doesn't even show up in resource monitor ?!

Bit strange...

I also want throttling for all kinds of bandwidth/i/o in windows next version, but that is a different topic !

Take care Luke and Han lol.

Bye,
Skybuck.
Similar Threads