After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 569921 - dosfsck -n delays device scan
dosfsck -n delays device scan
Status: RESOLVED FIXED
Product: gparted
Classification: Other
Component: application
0.3.8
Other All
: Normal normal
: ---
Assigned To: Mike Fleetwood
gparted maintainers alias
: 681331 (view as bug list)
Depends on:
Blocks:
 
 
Reported: 2009-01-30 23:54 UTC by Pedric
Modified: 2020-01-20 17:59 UTC
See Also:
GNOME target: ---
GNOME version: ---


Attachments
Switch to faster minfo and mdir to read FAT16/32 usage (v1) (15.37 KB, patch)
2019-06-19 12:41 UTC, Mike Fleetwood
none Details | Review
Switch to faster minfo and mdir to read FAT16/32 usage (v2) (15.62 KB, patch)
2019-06-21 13:51 UTC, Mike Fleetwood
none Details | Review
Switch to faster minfo and mdir to read FAT16/32 usage (v3) (15.60 KB, patch)
2019-06-30 12:33 UTC, Mike Fleetwood
none Details | Review

Description Pedric 2009-01-30 23:54:27 UTC
Please describe the problem:
When scanning for devices, gparted checks fat32 file systems using dosfsck -a. With very large, messy fat filesystems, this can take very long (up to severy minutes; can be reproduced by running dosfsck manually). Eventually, this occurs on every device rescan, and makes gparted quite hard to use.

Steps to reproduce:
1. start gparted
or
1. do anything that causes a rescan of devices

Actual results:
Scanning for devices takes several minutes to complete

Expected results:
There could be some kind of timeout with a suggestion to (temporarily) disable the checking of the offending partition.

Does this happen every time?


Other information:
This occurs with the gparted version shipped with Ubuntu 8.10 and a 68 GB fat32 partition that is very low on free space and probably full of errors that dosfsck seems unable to correct (so -a results in very long delay each and every time). As a workaround, I kill the dosfsck process, which results in not being able to edit that partion - but at least I get to use gparted again...
Comment 1 Curtis Gedak 2009-02-03 21:37:51 UTC
Hi Pedric,

Are you sure that the GParted application invoked the chkdsk operation?

I suspect that GParted did NOT start the dosfsck process you encountered.  Most likely it is started when GNU/Linux is booted.  When the GParted application rescans devices, it does not invoke any check disk operations.


You can check for the parent process by using the "ps -ef" command to list all of the processes, and then follow the chain of PPID's (Parent Process Identifiers) back to invoking PID (Process Identifier).


For example, with GParted running and applying a "Check" operation on FAT32 file system, the following is displayed from "ps -ef"

$ ps -ef
UID        PID  PPID  C STIME TTY          TIME CMD
<... stuff deleted ...>
gedakc    7291  7121  0 09:40 ?        00:00:00 gnome-terminal
gedakc    7313  7291  0 09:40 pts/1    00:00:00 bash
gedakc   10921  7291  0 14:23 pts/2    00:00:00 bash
root     10948  7313  0 14:23 pts/1    00:00:00 /bin/sh /usr/local/sbin/gparted
root     10955 10948  0 14:23 pts/1    00:00:00 hal-lock --interface org.freedes
root     10956 10955  2 14:23 pts/1    00:00:03 /usr/local/sbin/gpartedbin
root     11192 10956  0 14:26 pts/1    00:00:00 sh -c nice -n 19 dosfsck -a -w -
root     11193 11192  8 14:26 pts/1    00:00:00 dosfsck -a -w -v /dev/sdb3
gedakc   11194 10921  0 14:26 pts/2    00:00:00 ps -ef
$

From the above listing, the dosfsck command is process 11193.
Its parent process is 11192 which is the "sh -c nice -n 19 dosfsck -a ..." command.
The parent for this process is 10956 which is the /usr/local/sbin/gpartedbin process.

So in this case we can see that the gparted application is the grand parent of the dosfsck process.
Comment 2 Pedric 2009-02-03 22:23:20 UTC
Hi,

this is the relevant output of ps -ef just after starting gparted.

1000      5639     1  0 23:13 ?        00:00:00 gksu /usr/sbin/gparted
root      5640  5639  0 23:13 ?        00:00:00 /bin/sh /usr/sbin/gparted
root      5644  5640  0 23:13 ?        00:00:00 hal-lock --interface org.freedeskdesktop.Hal.Device.Storage --exclusive --run /usr/sbin/gpartedbin 
root      5645  5644 15 23:13 ?        00:00:11 /usr/sbin/gpartedbin
root      5661  5645  0 23:13 ?        00:00:00 sh -c dosfsck -a -v /dev/sda5
root      5662  5661 38 23:13 ?        00:00:27 dosfsck -a -v /dev/sda5

(it finished initial device scanning in 23:19...)
Comment 3 Curtis Gedak 2009-02-04 16:14:42 UTC
Thank you Pedric for the "ps -ef" listing.  By following the process chain I can see that yes indeed, the dosfsck process is started by gparted.

I noticed that this dosfsck command was missing the "-w" flag.  It turns out that GParted runs "dosfsck -a -v /device" to determine the number of used sectors in the partition.


Would you be able to post the output from "dosfsck -v /your-disk-device"?


On a clean FAT32 partition, the output is as follows:

$ sudo dosfsck -v /dev/sdb3
dosfsck 2.11 (12 Mar 2005)
dosfsck 2.11, 12 Mar 2005, FAT32, LFN
Checking we can access the last sector of the filesystem
Boot sector contents:
System ID "mkdosfs"
Media byte 0xf8 (hard disk)
       512 bytes per logical sector
     16384 bytes per cluster
        32 reserved sectors
First FAT starts at byte 16384 (sector 32)
         2 FATs, 32 bit entries
  38977024 bytes per FAT (= 76127 sectors)
Root directory start at cluster 2 (arbitrary size)
Data area starts at byte 77970432 (sector 152286)
   9744185 data clusters (159648727040 bytes)
63 sectors/track, 255 heads
         0 hidden sectors
 311966234 sectors total
Checking for unused clusters.
Checking free cluster summary.
/dev/sdb3: 1 files, 1/9744185 clusters
$ 

Comment 4 Pedric 2009-02-04 20:17:34 UTC
-v also took very long and produces the following output:

$ dosfsck -v /dev/sda5
dosfsck 2.11 (12 Mar 2005)
dosfsck 2.11, 12 Mar 2005, FAT32, LFN
Checking we can access the last sector of the filesystem
Boot sector contents:
System ID "MSWIN4.1"
Media byte 0xf8 (hard disk)
       512 bytes per logical sector
     65536 bytes per cluster
        32 reserved sectors
First FAT starts at byte 16384 (sector 32)
         2 FATs, 32 bit entries
   4480000 bytes per FAT (= 8750 sectors)
Root directory start at cluster 2 (arbitrary size)
Data area starts at byte 8976384 (sector 17532)
   1119894 data clusters (73393373184 bytes)
63 sectors/track, 255 heads
  30700278 hidden sectors
 143363997 sectors total
Checking for unused clusters.
Checking free cluster summary.
/dev/sda5: 232575 files, 1051022/1119894 clusters

As mentionened before: big, old and messy...

I found the problem not to occur when the partition is mounted - and of course, it occurs immediately when you unmount it from gparted to work with the partition.
Comment 5 Curtis Gedak 2009-02-05 19:35:48 UTC
To summarize, I believe we are dealing with two causes for this problem:

1)  The root cause of the problem is the partition with the FAT32 file system that the dosfsck application finds errors within, but is unable to fix.  This, along with the large, mostly full partition results in long delays for the dosfsck command to finish.

2)  The secondary cause of the problem is GParted using dosfsck to determine the number of used sectors in the partition.

Ideally, we solve (1), but if not, then we can consider addressing (2).


Possible solutions to (1):
-------------------------
A)  Try to repair the file system with a newer version of dosfsck.
    The newest version of dosfstools that I am aware of is at:
    http://www.daniel-baumann.ch/software/dosfstools/

B)  Try to repair the file system with another tool.
    If you have Windows on your computer, perhaps you could try fixing the problem using a Windows native tool, such as scandisk.


Possible solutions to (2):
-------------------------
C)  Use another method to determine the number of used sectors in a partition containing a FAT32 file system.
    Perhaps you know of other tools that would perform this function?


What are your thoughts?  Are you able to try out any of the above ideas?






Comment 6 Pedric 2009-02-07 11:16:59 UTC
I still have windows (xp sp3) installed and performed a boot time scandisk operation of that partition, without any success concerning the dosfsck runtimes. I will, however, try the newer version of dosfsck.

However, you should not be too concerned about my particular problem with that specific drive... Solving problem 2) should be more important... afaik dostools only provide mkfs und fsck for FAT, and nothing else...
Comment 7 Curtis Gedak 2009-02-10 16:37:31 UTC
Any luck with trying out the newer version of dosfsck?
Comment 8 Pedric 2009-02-10 21:33:11 UTC
Yes, but without luck. However, I did a quick hack on dosfsck to just print the boot sector information and refrain from checking - which I found to be an exercise in futility as I am typing this, as the number of used sectors is determined by the checking routine...
Comment 9 Curtis Gedak 2009-02-11 18:10:20 UTC
Thanks for checking out dosfsck.

If you run across any other GNU/Linux tools for determining the number of used sectors in an FAT partition, I am all ears :-)
Comment 10 Curtis Gedak 2009-08-11 21:21:23 UTC
Restating this problem:

The source of this problem is that GParted uses the "dosfsck -a -v" command to determine the number of used sectors in a FAT16/32 file system.  This value is important to GParted when determining the smallest value to which a FAT16/32 file system can be shrunk.

A similar problem exists for NTFS partitions.  Gparted uses the "ntfsresize --info --force --no-progress-bar" command to determine the smallest size to which the NTFS file system can be shrunk.

If the partition is mounted, then GParted can retrieve this information by another method (using a statvfs system call).  Otherwise GParted must resort to other methods such as the above commands.

This problem has also been reported in ubuntu as follows:
https://bugs.launchpad.net/ubuntu/+source/gparted/+bug/121943?comments=all

Does anyone know a better way to determine the number of used sectors in a FAT16 or FAT32 file system, or NTFS file system?
Comment 11 Curtis Gedak 2010-01-14 18:09:40 UTC
FYI:

The "-a" option, which was removed in GParted 0.4.7, is now a "-n" option to fix bug #605175
Comment 12 Phillip Susi 2012-01-04 15:08:00 UTC
Curtis, doesn't that mean this bug should be marked as fixed too?
Comment 13 Curtis Gedak 2012-01-04 17:13:06 UTC
Phillip, unfortunately in some situations the dosfsck command still takes an exceptionally long time to determine the number of used/unused sectors.

If there is a better (and quicker) way to determine used/unused sectors for an unmounted FAT16/FAT32 file system then I am keen to learn about it.
Comment 14 Phillip Susi 2012-01-04 18:23:36 UTC
You could read the fs directly.  According to wikipedia, fat32 stores the free cluster count in the boot sector.  For fat12/16 you could just scan the FAT and count the number of entries that are non zero.

Didn't the change from -a to -n improve things drastically though?  How long is exceptionally long when using -n?  If that change mostly solved it then maybe this should still be marked as fixed, and a separate enhancement bug created to switch to reading the fs directly?
Comment 15 Curtis Gedak 2012-01-04 21:43:36 UTC
(In reply to comment #14)
> You could read the fs directly.  According to wikipedia, fat32 stores the free
> cluster count in the boot sector.  For fat12/16 you could just scan the FAT and
> count the number of entries that are non zero.

If I recall, the determination of free sectors involved following a chain of sectors through the file system.  At least that's what my memory tells me when I was discussing such an option with another person (I thank Jan Claeys).

To be honest though I have not looked deeply into this myself.

> Didn't the change from -a to -n improve things drastically though?  How long is
> exceptionally long when using -n?  If that change mostly solved it then maybe
> this should still be marked as fixed, and a separate enhancement bug created to
> switch to reading the fs directly?

Not much.  The "-n" flag does much the same thing but does not attempt to fix any errors found.

The biggest improvement to shorten the scan time occurs if a user defragments the FAT file system first before using GParted.
Comment 16 Phillip Susi 2012-01-04 23:16:26 UTC
> If I recall, the determination of free sectors involved following a chain of
> sectors through the file system.  At least that's what my memory tells me when
> I was discussing such an option with another person (I thank Jan Claeys).

You need to follow the chain if you want to count how many clusters a 
specific file is using ( but why bother when the directory entry 
tells you exactly how many bytes it is using ).  If you just want to get 
a total count of how many are used/free, then you just need to walk the 
whole array and count the (non)zero entries.

> Not much.  The "-n" flag does much the same thing but does not attempt to fix
> any errors found.

I see, then yea, you probably want to read the fs directly to fix this one.
Comment 17 Curtis Gedak 2012-01-06 16:50:46 UTC
>> If I recall, the determination of free sectors involved following a chain of
>> sectors through the file system.  At least that's what my memory tells me when
>> I was discussing such an option with another person (I thank Jan Claeys).
>
> You need to follow the chain if you want to count how many clusters a 
> specific file is using ( but why bother when the directory entry 
> tells you exactly how many bytes it is using ).  If you just want to get 
> a total count of how many are used/free, then you just need to walk the 
> whole array and count the (non)zero entries.

I think there were additional problems too.

Jan Claeys,

Do you recall what the challenges were with determining used/unused space in FAT file systems?

If my memory serves me correct you had taken a look at this, but opted not to proceed further.
Comment 18 Curtis Gedak 2013-10-05 15:45:41 UTC
*** Bug 681331 has been marked as a duplicate of this bug. ***
Comment 19 Mike Fleetwood 2019-06-19 12:41:16 UTC
Created attachment 374222 [details] [review]
Switch to faster minfo and mdir to read FAT16/32 usage (v1)

Hi Curtis,

Here's a patchset v1 to replace the slow fsck.fat with the much faster
minfo and mdir from mtools to resolve this issue.

Thanks,
Mike
Comment 20 Mike Fleetwood 2019-06-21 13:51:03 UTC
Created attachment 374223 [details] [review]
Switch to faster minfo and mdir to read FAT16/32 usage (v2)

Hi Curtis,

Here's patchset v2.  Compared to v1 it makes a small coding improvement
to parsing mdir output.  No functional different.

Thanks,
Mike
Comment 21 Mike Fleetwood 2019-06-30 12:33:22 UTC
Created attachment 374224 [details] [review]
Switch to faster minfo and mdir to read FAT16/32 usage (v3)

Hi Curtis,

Here's patchset v3.  Compared to v2 it makes a very minor change to the
commit message for the first patch.

Thanks,
Mike
Comment 22 Curtis Gedak 2019-07-05 17:02:35 UTC
Thanks Mike for this performance enhancement patch set.  My testing has gone well.  I tried different amounts of files within an FAT32 file system and the values reported by the 1.0.0 release and this patch set were the same.  I also tried resizing and each resize completed successfully.

I have committed patch set (v3) to the git master branch for inclusion in the next release of GParted.

The relevant git commits can be viewed at the following links:

Switch to faster minfo and mdir to read FAT16/32 usage (#569921)
https://gitlab.gnome.org/GNOME/gparted/commit/5a52b44071d86bd556451e1e22f5fbc3a1ae9000

Drop use of long ago removed udevinfo
https://gitlab.gnome.org/GNOME/gparted/commit/3f15a66291af1f2db142edfe2a9219da219a29d5
Comment 23 Pedric 2019-07-16 18:02:35 UTC
Hello everyone,

I can hardly remember the device/partition that this problem occurred with, as one would expect after such a long time. Anyways, thanks for your long term commitment to gparted!

Christian (Pedric)
Comment 24 Curtis Gedak 2020-01-20 17:59:38 UTC
This enhancement was included in the GParted 1.1.0 release on January 20, 2020.