GNOME Bugzilla – Bug 569921
dosfsck -n delays device scan
Last modified: 2020-01-20 17:59:38 UTC
Please describe the problem: When scanning for devices, gparted checks fat32 file systems using dosfsck -a. With very large, messy fat filesystems, this can take very long (up to severy minutes; can be reproduced by running dosfsck manually). Eventually, this occurs on every device rescan, and makes gparted quite hard to use. Steps to reproduce: 1. start gparted or 1. do anything that causes a rescan of devices Actual results: Scanning for devices takes several minutes to complete Expected results: There could be some kind of timeout with a suggestion to (temporarily) disable the checking of the offending partition. Does this happen every time? Other information: This occurs with the gparted version shipped with Ubuntu 8.10 and a 68 GB fat32 partition that is very low on free space and probably full of errors that dosfsck seems unable to correct (so -a results in very long delay each and every time). As a workaround, I kill the dosfsck process, which results in not being able to edit that partion - but at least I get to use gparted again...
Hi Pedric, Are you sure that the GParted application invoked the chkdsk operation? I suspect that GParted did NOT start the dosfsck process you encountered. Most likely it is started when GNU/Linux is booted. When the GParted application rescans devices, it does not invoke any check disk operations. You can check for the parent process by using the "ps -ef" command to list all of the processes, and then follow the chain of PPID's (Parent Process Identifiers) back to invoking PID (Process Identifier). For example, with GParted running and applying a "Check" operation on FAT32 file system, the following is displayed from "ps -ef" $ ps -ef UID PID PPID C STIME TTY TIME CMD <... stuff deleted ...> gedakc 7291 7121 0 09:40 ? 00:00:00 gnome-terminal gedakc 7313 7291 0 09:40 pts/1 00:00:00 bash gedakc 10921 7291 0 14:23 pts/2 00:00:00 bash root 10948 7313 0 14:23 pts/1 00:00:00 /bin/sh /usr/local/sbin/gparted root 10955 10948 0 14:23 pts/1 00:00:00 hal-lock --interface org.freedes root 10956 10955 2 14:23 pts/1 00:00:03 /usr/local/sbin/gpartedbin root 11192 10956 0 14:26 pts/1 00:00:00 sh -c nice -n 19 dosfsck -a -w - root 11193 11192 8 14:26 pts/1 00:00:00 dosfsck -a -w -v /dev/sdb3 gedakc 11194 10921 0 14:26 pts/2 00:00:00 ps -ef $ From the above listing, the dosfsck command is process 11193. Its parent process is 11192 which is the "sh -c nice -n 19 dosfsck -a ..." command. The parent for this process is 10956 which is the /usr/local/sbin/gpartedbin process. So in this case we can see that the gparted application is the grand parent of the dosfsck process.
Hi, this is the relevant output of ps -ef just after starting gparted. 1000 5639 1 0 23:13 ? 00:00:00 gksu /usr/sbin/gparted root 5640 5639 0 23:13 ? 00:00:00 /bin/sh /usr/sbin/gparted root 5644 5640 0 23:13 ? 00:00:00 hal-lock --interface org.freedeskdesktop.Hal.Device.Storage --exclusive --run /usr/sbin/gpartedbin root 5645 5644 15 23:13 ? 00:00:11 /usr/sbin/gpartedbin root 5661 5645 0 23:13 ? 00:00:00 sh -c dosfsck -a -v /dev/sda5 root 5662 5661 38 23:13 ? 00:00:27 dosfsck -a -v /dev/sda5 (it finished initial device scanning in 23:19...)
Thank you Pedric for the "ps -ef" listing. By following the process chain I can see that yes indeed, the dosfsck process is started by gparted. I noticed that this dosfsck command was missing the "-w" flag. It turns out that GParted runs "dosfsck -a -v /device" to determine the number of used sectors in the partition. Would you be able to post the output from "dosfsck -v /your-disk-device"? On a clean FAT32 partition, the output is as follows: $ sudo dosfsck -v /dev/sdb3 dosfsck 2.11 (12 Mar 2005) dosfsck 2.11, 12 Mar 2005, FAT32, LFN Checking we can access the last sector of the filesystem Boot sector contents: System ID "mkdosfs" Media byte 0xf8 (hard disk) 512 bytes per logical sector 16384 bytes per cluster 32 reserved sectors First FAT starts at byte 16384 (sector 32) 2 FATs, 32 bit entries 38977024 bytes per FAT (= 76127 sectors) Root directory start at cluster 2 (arbitrary size) Data area starts at byte 77970432 (sector 152286) 9744185 data clusters (159648727040 bytes) 63 sectors/track, 255 heads 0 hidden sectors 311966234 sectors total Checking for unused clusters. Checking free cluster summary. /dev/sdb3: 1 files, 1/9744185 clusters $
-v also took very long and produces the following output: $ dosfsck -v /dev/sda5 dosfsck 2.11 (12 Mar 2005) dosfsck 2.11, 12 Mar 2005, FAT32, LFN Checking we can access the last sector of the filesystem Boot sector contents: System ID "MSWIN4.1" Media byte 0xf8 (hard disk) 512 bytes per logical sector 65536 bytes per cluster 32 reserved sectors First FAT starts at byte 16384 (sector 32) 2 FATs, 32 bit entries 4480000 bytes per FAT (= 8750 sectors) Root directory start at cluster 2 (arbitrary size) Data area starts at byte 8976384 (sector 17532) 1119894 data clusters (73393373184 bytes) 63 sectors/track, 255 heads 30700278 hidden sectors 143363997 sectors total Checking for unused clusters. Checking free cluster summary. /dev/sda5: 232575 files, 1051022/1119894 clusters As mentionened before: big, old and messy... I found the problem not to occur when the partition is mounted - and of course, it occurs immediately when you unmount it from gparted to work with the partition.
To summarize, I believe we are dealing with two causes for this problem: 1) The root cause of the problem is the partition with the FAT32 file system that the dosfsck application finds errors within, but is unable to fix. This, along with the large, mostly full partition results in long delays for the dosfsck command to finish. 2) The secondary cause of the problem is GParted using dosfsck to determine the number of used sectors in the partition. Ideally, we solve (1), but if not, then we can consider addressing (2). Possible solutions to (1): ------------------------- A) Try to repair the file system with a newer version of dosfsck. The newest version of dosfstools that I am aware of is at: http://www.daniel-baumann.ch/software/dosfstools/ B) Try to repair the file system with another tool. If you have Windows on your computer, perhaps you could try fixing the problem using a Windows native tool, such as scandisk. Possible solutions to (2): ------------------------- C) Use another method to determine the number of used sectors in a partition containing a FAT32 file system. Perhaps you know of other tools that would perform this function? What are your thoughts? Are you able to try out any of the above ideas?
I still have windows (xp sp3) installed and performed a boot time scandisk operation of that partition, without any success concerning the dosfsck runtimes. I will, however, try the newer version of dosfsck. However, you should not be too concerned about my particular problem with that specific drive... Solving problem 2) should be more important... afaik dostools only provide mkfs und fsck for FAT, and nothing else...
Any luck with trying out the newer version of dosfsck?
Yes, but without luck. However, I did a quick hack on dosfsck to just print the boot sector information and refrain from checking - which I found to be an exercise in futility as I am typing this, as the number of used sectors is determined by the checking routine...
Thanks for checking out dosfsck. If you run across any other GNU/Linux tools for determining the number of used sectors in an FAT partition, I am all ears :-)
Restating this problem: The source of this problem is that GParted uses the "dosfsck -a -v" command to determine the number of used sectors in a FAT16/32 file system. This value is important to GParted when determining the smallest value to which a FAT16/32 file system can be shrunk. A similar problem exists for NTFS partitions. Gparted uses the "ntfsresize --info --force --no-progress-bar" command to determine the smallest size to which the NTFS file system can be shrunk. If the partition is mounted, then GParted can retrieve this information by another method (using a statvfs system call). Otherwise GParted must resort to other methods such as the above commands. This problem has also been reported in ubuntu as follows: https://bugs.launchpad.net/ubuntu/+source/gparted/+bug/121943?comments=all Does anyone know a better way to determine the number of used sectors in a FAT16 or FAT32 file system, or NTFS file system?
FYI: The "-a" option, which was removed in GParted 0.4.7, is now a "-n" option to fix bug #605175
Curtis, doesn't that mean this bug should be marked as fixed too?
Phillip, unfortunately in some situations the dosfsck command still takes an exceptionally long time to determine the number of used/unused sectors. If there is a better (and quicker) way to determine used/unused sectors for an unmounted FAT16/FAT32 file system then I am keen to learn about it.
You could read the fs directly. According to wikipedia, fat32 stores the free cluster count in the boot sector. For fat12/16 you could just scan the FAT and count the number of entries that are non zero. Didn't the change from -a to -n improve things drastically though? How long is exceptionally long when using -n? If that change mostly solved it then maybe this should still be marked as fixed, and a separate enhancement bug created to switch to reading the fs directly?
(In reply to comment #14) > You could read the fs directly. According to wikipedia, fat32 stores the free > cluster count in the boot sector. For fat12/16 you could just scan the FAT and > count the number of entries that are non zero. If I recall, the determination of free sectors involved following a chain of sectors through the file system. At least that's what my memory tells me when I was discussing such an option with another person (I thank Jan Claeys). To be honest though I have not looked deeply into this myself. > Didn't the change from -a to -n improve things drastically though? How long is > exceptionally long when using -n? If that change mostly solved it then maybe > this should still be marked as fixed, and a separate enhancement bug created to > switch to reading the fs directly? Not much. The "-n" flag does much the same thing but does not attempt to fix any errors found. The biggest improvement to shorten the scan time occurs if a user defragments the FAT file system first before using GParted.
> If I recall, the determination of free sectors involved following a chain of > sectors through the file system. At least that's what my memory tells me when > I was discussing such an option with another person (I thank Jan Claeys). You need to follow the chain if you want to count how many clusters a specific file is using ( but why bother when the directory entry tells you exactly how many bytes it is using ). If you just want to get a total count of how many are used/free, then you just need to walk the whole array and count the (non)zero entries. > Not much. The "-n" flag does much the same thing but does not attempt to fix > any errors found. I see, then yea, you probably want to read the fs directly to fix this one.
>> If I recall, the determination of free sectors involved following a chain of >> sectors through the file system. At least that's what my memory tells me when >> I was discussing such an option with another person (I thank Jan Claeys). > > You need to follow the chain if you want to count how many clusters a > specific file is using ( but why bother when the directory entry > tells you exactly how many bytes it is using ). If you just want to get > a total count of how many are used/free, then you just need to walk the > whole array and count the (non)zero entries. I think there were additional problems too. Jan Claeys, Do you recall what the challenges were with determining used/unused space in FAT file systems? If my memory serves me correct you had taken a look at this, but opted not to proceed further.
*** Bug 681331 has been marked as a duplicate of this bug. ***
Created attachment 374222 [details] [review] Switch to faster minfo and mdir to read FAT16/32 usage (v1) Hi Curtis, Here's a patchset v1 to replace the slow fsck.fat with the much faster minfo and mdir from mtools to resolve this issue. Thanks, Mike
Created attachment 374223 [details] [review] Switch to faster minfo and mdir to read FAT16/32 usage (v2) Hi Curtis, Here's patchset v2. Compared to v1 it makes a small coding improvement to parsing mdir output. No functional different. Thanks, Mike
Created attachment 374224 [details] [review] Switch to faster minfo and mdir to read FAT16/32 usage (v3) Hi Curtis, Here's patchset v3. Compared to v2 it makes a very minor change to the commit message for the first patch. Thanks, Mike
Thanks Mike for this performance enhancement patch set. My testing has gone well. I tried different amounts of files within an FAT32 file system and the values reported by the 1.0.0 release and this patch set were the same. I also tried resizing and each resize completed successfully. I have committed patch set (v3) to the git master branch for inclusion in the next release of GParted. The relevant git commits can be viewed at the following links: Switch to faster minfo and mdir to read FAT16/32 usage (#569921) https://gitlab.gnome.org/GNOME/gparted/commit/5a52b44071d86bd556451e1e22f5fbc3a1ae9000 Drop use of long ago removed udevinfo https://gitlab.gnome.org/GNOME/gparted/commit/3f15a66291af1f2db142edfe2a9219da219a29d5
Hello everyone, I can hardly remember the device/partition that this problem occurred with, as one would expect after such a long time. Anyways, thanks for your long term commitment to gparted! Christian (Pedric)
This enhancement was included in the GParted 1.1.0 release on January 20, 2020.