DiskTune - why the big gap?

[cnmoore]

I tried DiskTune because you sounded so rational. 
I was surprised by the DiskTune results.

Please take a look: http://www.spywareinfoforum.com/index.php?showtopic=125397
Screenshots of DiskTune after it finished, and of subsequent Puran analysis.
It also has a .html page with Puran's statistics from its analysis.

The result is satisfactory but I am very curious about the big gap.  Is that deliberate, or did DiskTune possibly stop before it was finished?  It took over 5 hours and had finished this morning when I woke up.  (Does the program keep a log? I couldn't find any.)

[Joep]

Hello,

Thanks for trying DiskTune. Please keep in mind that it is kind of a hobby project that is safe to use and ready for use, but still needs some work. I have more plans for it.

The gap, yes. It is there intentionally. But tweaking the gap is also on my to-do-list. First thing DiskTune does during an optimize is creating the gap. To determine the gap size it tries to come up with a balance free space vs. used space, don't have the exact formula in my head right now, but I will have to dig into that again.

That gap, the fastest area of the selected volume, is used to place frequently used files, directory indexes and the MFT. The gap is intentionally larger than those components. It will allow Windows to store files you create (word processor, image files etc.) in the fast area, unfragmented. And it also allows DiskTune to move files that fragment over time and are therefore probably 'current files', into the fast area.

The whole idea behind it is to apply generic 'rules of thumb' for optimal file placement (fast access, prevent future fragmentation) rather than doing exhaustive analysis and all-the-time online/background monitoring. The idea is to create a condition under which it becomes likely that frequently used files will 'organically' move towards the fast side of the disk, and have enough free space available to allow unfragmented files to be created.

Logging:
- There is a limited debug log window which will show if the /dbg switch is added
- To see actual files that are candidate for movement, click the defrag/optimize/sys area labels next to the legend.

If you have suggestions or questions, please let me know.

[cnmoore]

Yes, the gap is very well thought out.  Nice.

Logging:
- There is a limited debug log window which will show if the /dbg switch is added
- To see actual files that are candidate for movement, click the defrag/optimize/sys area labels next to the legend.

I guess this means run it from command line with /dbg switch.  Are there any other command line switches?

Edit: Sorry, didn't read the manual enough.

I have a question.  When the PC was made, back in 2003, the MFT reserved area was put in a strange place.

Volume constants for drive: C:
total clusters: 12287708
free clusters : 4592003
mftreserved  : 11283776-12202144
end reserved  : 2500000

That's from your debug window.
As you can see, 'end reserved' is not within the 11283776-12202144, and the latter can be used for files.  However only Puran defragger sensibly disregards it.

How can I get rid of that huge reserved area near the end of the partition? NtfsMftZoneReservation is set to 0 in the registry but that has no effect.

C:\>fsutil fsinfo ntfsinfo c:
NTFS Volume Serial Number :       0x2c400d41400d12ee
Version :                         3.1
Number Sectors :                  0x0000000005dbf6e0
Total Clusters :                  0x0000000000bb7edc
Free Clusters  :                  0x00000000004612df
Total Reserved :                  0x0000000000000340
Bytes Per Sector  :               512
Bytes Per Cluster :               4096
Bytes Per FileRecord Segment    : 1024
Clusters Per FileRecord Segment : 0
Mft Valid Data Length :           0x000000000a4ec000
Mft Start Lcn  :                  0x000000000007278a
Mft2 Start Lcn :                  0x00000000002150bb
Mft Zone Start :                  0x0000000000ac2d40
Mft Zone End   :                  0x0000000000ba30a0

Fsutil agrees that neither Mft nor Mft 2 are actually within or even close to the Mft Zone.

defrag C: -a -v shows

Master File Table (MFT) fragmentation
    Total MFT size                      = 165 MB
    MFT record count                    = 149,878
    Percent MFT in use                  = 88
    Total MFT fragments                 = 4

[Joep]

Hello,

The size of the MFT reserved area can not be changed, it is set at the time you format the volume. You also can not force it to move although I have seen it move during my testing. Any software 'using' the defag API (like DiskTune) will be unable to move data into the MFT reserved area. At least that's my experience and also the experience of the author of MyDefrag. There's no real need to worry about the reserved area anyway and in Windows Vista MS changed some things with regards to the reserves area. I think it will move it closer towards the start of the disk, plus it will not reserve an area as large as XP did.

In case there's not enough free space to store files outside the MFT reserved area, Windows will store them inside that area. So, the space set aside for the MFT is not lost space.

But in the event that Windows needs to expand the MFT into the reserved area, during an optimize DiskTune will move the largest part of the MFT into the fastest disk area again.

Note: The end reserved  : 2500000 you see in the DiskTune log is not the same as the MFT reserved area, it is the end of the area DiskTune reserves for the system files etc..

[cnmoore]

Thanks for your answer, Joep.

I may have stumbled on a solution -  'Wipe free space removes MFT Zone'.
http://forum.piriform.com/index.php?s=&showtopic=21382&view=findpost&p=145105 I'm going to try that later.

In the meantime, your defragger is amazing!  I have a boot time measurement widget from http://donnedwards.openaccess.co.za/2007/06/great-defrag-shootout-all.html  Timer is at http://www.mustang.co.za/exe/RebootTimerSetup.exe.  It stores current time in registry, restarts PC, retrieves time.

After your optimize, my restart took 1 min 45 secs.  Compared to 2 mins 42 secs after MyDefrag 'Slow optimize', and 2 min 22 sec after mst Defrag.  Remarkable.  Makes sense though that people who do data recovery know really a lot about disk organization.  Boot time isn't actually very important to me as I only reboot a few times a week, but seems like a reasonable indication of speedy file management.

Your defragger is really slow doing either Optimize or Compact -  but running it overnight works for me.  And Defrag mode is very fast.

[Joep]

Hello,

That wipe free space creates a huge file. So that provokes the behavior I explained: The MFT reserved zone is not exclusive to the MFT, so at some point when Windows needs that space, it will just use it to store normal files. So, it may be an idea to add an option in DiskTune to create a huge file, simply to force Windows to relocate the MFT reserved zone. I will experiment with that.

Yeah, the way I thought DiskTune should work is when the computer isn't used anyway. So making it very fast (the optimize) wasn't a priority. The initial optimize will always be slow just because we have to move a lot of stuff. Problem is that DiskTune currently treats optimize always as if it was the first time it was run. So, I will see if can think of ways to improve that.

But so the reaults are promising at least. That's good to hear. Do not forget that mention that to that friend of yours who is editor for this major IT magazine :-)

[cnmoore]

Hello,

That wipe free space creates a huge file. So that provokes the behavior I explained: The MFT reserved zone is not exclusive to the MFT, so at some point when Windows needs that space, it will just use it to store normal files. So, it may be an idea to add an option in DiskTune to create a huge file, simply to force Windows to relocate the MFT reserved zone. I will experiment with that.

It only 'sort of' works.  It shrinks the Zone to a very small size (one square in Defraggler) at the end of the partition.  But it is restored to gigantic size after reboot.  However it is then at the end of the partition, beyond pagefile.sys, which is a much better place for it.

Yeah, the way I thought DiskTune should work is when the computer isn't used anyway. So making it very fast (the optimize) wasn't a priority. The initial optimize will always be slow just because we have to move a lot of stuff. Problem is that DiskTune currently treats optimize always as if it was the first time it was run. So, I will see if can think of ways to improve that.

That would be really nice.  Ideally it would just move any giant rarely used files to the end of the partition - Restore Points, service pack uninstalls and all that - and do some defragging.  It is a bit wearying to see it move everything on the partition if a person needs to re-optimize.

But so the results are promising at least. That's good to hear. Do not forget that mention that to that friend of yours who is editor for this major IT magazine :-)

I already mentioned it to the Defrag Shootout guy - https://www.blogger.com/comment.g?blogID=19298483&postID=8097402150450859683&page=1&pli=1  
Don't know yet how subsequent reboots will average out or how much variation there is between reboots.