Hi Matthias,

There is indeed a difference between the reported free space in many file systems depending upon whether these are mounted or not.  This was discovered while Mike Fleetwood was working on Bug #499202 - gparted does not see the difference if partition size differs from file system size.

At that time we decided to use a sliding scale of "significant unallocated space" before reporting this to the user.

Since you have encountered a situation where the warning is displayed incorrectly, then perhaps we need to adjust the calculation.


As background information, GParted uses both methods to calculate free space depending upon the mount status of the file system.  Using the statvfs call is much faster than calling a separate command to determine the free space.  Since we do not wish to unnecessarily mount or unmount partitions, we use the statvfs call when we can, but resort to calling a separate command when the file system is not mounted.

We have encountered situations where calling a separate command to determine file system usage takes an inordinate amount of time.  This has happened using the dosfsck, and ntfsresize commands.  See the following report:
Bug #569921 - dosfsck -n delays device scan
As such always using a separate command to determine file system usage is not desirable.


Please feel free to read through bug #499202 and let us know your thoughts on how we might improve upon handling this problem.

I have added Mike Fleetwood to the CC list for this bug report.

Hi Matthias,

You have provided most of the figures I need ...

Mounted file system size: 182331 (1K blocks)
Mounted file system free space: 116846 (1K blocks)
Unmounted file system size: 194560 (1K blocks)

Can you tell me these extra figures:

Unmounted file system free space -> "Free blocks:" from dump2fs -h command
Partition size -> Output from sfdisk -s /dev/sda4

Then I can repeat the calculations GParted is doing and see why it is showing unallocated space.

Hi Mike,

> dumpe2fs -h
Free blocks: 116846

> sfdisk -s /dev/sda4
194560

Here's the calculation which GParted is performing ...

PTN SZ  MNT   unallocated         FS SIZE         FS FREE           used
194560  yes  12229 (6.3%)  182331 (93.7%)  116846 (60.0%)  65485 (33.7%)
194560   no      0 (0.0%)  194560(100.0%)  116846 (60.0%)  77714 (39.9%)

(Figures in 1 KiB units)
Legend:
UPPER CASE - Figure supplied from sfdisk, dumpe2fs or statvfs()
             system call
lower case - Figure calculated as GParted would

GParted will be showing these figures when /boot (sda4) is mounted:
  Size:         190.00 MiB
  Used:         63.95 MiB   ( 34% )
  Unused:       114.11 MiB  ( 60% )
  Unallocated:  11.94 MiB   ( 6% )

1) The fact that dumpe2fs reports the file system size
   (dumpe2fs, Block count * Block size) matches the partition size
   (sfdisk -s) says the file system does fill the partition.

2) But the ext2 driver in the kernel reports the mounted file system
   size (stat, Blocks Total * Block size) as much smaller than the
   partition size.  182331 KiB rather than 194560 KiB.  This is
   intrinsic unallocated space GParted tries to avoid showing using a
   cutoff of 2 to 5%, depending on partition size.


My testing ...

# sfdisk -s /dev/sda10
194560
# mkfs.ext2 /dev/sda10
mke2fs 1.41.12 (17-May-2010)
...
# dumpe2fs -h /dev/sda10 | egrep 'Block count:|Free blocks:|Block size:'
dumpe2fs 1.41.12 (17-May-2010)
Block count:              194560
Free blocks:              186853
Block size:               1024
# mount /dev/sda10 /mnt/0
[root@rockover ~]# stat -f /mnt/0
  File: "/mnt/0"
    ID: 36008cbe1a40b0d2 Namelen: 255     Type: ext2/ext3
Block size: 1024       Fundamental block size: 1024
Blocks: Total: 188403     Free: 186853     Available: 177125
Inodes: Total: 48768      Free: 48757

So I only get a difference between a mounted file system size of
188403 KiB and a partition size of 194560 KiB, 3.2% unallocated space
and would therefore not be shown by GParted.


Matthias,

Can you:
1) Confirm figures shown in GParted match my prediction?
2) Tell me how old the file system is?
   dumpe2fs -h /dev/sda4 | egrep 'Filesystem created:'
3) Would you be able to create a new partition and file system and
   repeat my testing above?
4) What version of e2fsprogs do you have?
   dumpe2fs
5) What kernel version are you running?
   cat /proc/version


I have a theory that older versions of e2fsprogs may have created file
systems with larger intrinsic unallocated space.  I guess whatever the
outcome we'll have to look at increasing the threshold for intrinsic
unallocated space in GParted.

Created attachment 223497 [details]
dumpe2fs -h /dev/sda4

1)
Gparted shows in unmounted state:
Size: 190 MiB, Used: 75.89 MiB (40%), Unused 114.11 MiB (60%)
Gparted shows in mounted state:
Size: 190 MiB, Used: 63.95 MiB (34%), Unused 114.11 MiB (60%), Unallocated 11.94 MiB

Exactly as you calculated.

2)
dumpe2fs shows no "Filesystem created" field for that file system (but for others)

3)
I repeated your testing by creating a new ext2 file system and got exactly the same results as you did.
I afterwards compared the output of dumpe2fs between my old /boot and the newly created file system:

old /boot: Filesystem features:      filetype sparse_super
new fs: Filesystem features:      ext_attr resize_inode dir_index filetype sparse_super

old /boot: -
new fs: Filesystem flags:         signed_directory_hash 

old /boot: Inode count:              97344
new fs: Inode count:              48768

I attached the output of dumpe2fs for my (old) /boot.

4)
dumpe2fs 1.42 (29-Nov-2011)
5)
Linux version 3.5.3-030503-generic

Hi Matthias,

The situation is that yes your /boot file system does fill the partition *AND* it does contain 11.94 MiB (6.3%) unallocated space!  This is likely to have come about because the file system was created years ago using older mkfs.ext2.

The question is what do you want to do about it?
1) Nothing.
2) Unmount, shrink and grow the file system from the command line to see if it will reduce the unallocated space.
3) Backup, recreate and restore the file system.  As we have seen the latest mkfs.ext2 will create less intrinsic unallocated space.

Since this has happened for you I will be patching GParted to increase the threshold it uses to account for (hide) this larger amount of intrinsic unallocated space.

Mike

I tried shrink&enlarge and it didn't change.


What about using the e2fslib to obtain the block count of the device?
This should not be slower than vfsstat, but would be correct. It's also used by dumpe2fs.

Outline:

ext2fs fs;
ext2fs_open (device_name, flags, 0, 0, unix_io_manager, &fs);
struct ext2_super_block* super = fs->super;

block_count = super->s_blocks_count |
        (super->s_feature_incompat & EXT4_FEATURE_INCOMPAT_64BIT ?
        (__u64) super->s_blocks_count_hi << 32 : 0);

ext2fs_close(&fs);

(From the source code of dumpe2fs)

One could use dlsym to resolve the symbols at runtime, if one does not want to link to e2fslib.

I've worked out where the extra unallocated space (actually overhead) in
your file system has come from.

  Dumpe2fs figures:    Old (sda4) /boot    New figures
  Inode count:              97344          48768    *
  Block count:              194560         194560
  ...
  Block size:               1024           1024
  Fragment size:            1024           1024
  Blocks per group:         8192           8192
  Fragments per group:      8192           8192
  Inodes per group:         4056           2032     *
  Inode blocks per group:   507            254      **
  ...
  Inode size:               128            128

Your old (sda4) /boot file system has nearly twice as many blocks
dedicated to storing inode tables and can therefore store nearly twice
as many inodes as newly created ext2 file systems.  The kernel ext2
code which calculates the f_blocks value for the statvfs() system call
takes the size of the file system and subtracts overhead used by inode
tables, allocation bitmaps etc.  Hence the size of your file system is
reported smaller when mounted because it has less blocks for data.

  partition blocks:           194560
  block size:                 1024
  blocks per group:           8192
  old inode blocks per group: 507
  new inode blocks per group: 254
  delta blocks per group:     507-254 = 253
  block groups:               194560/8192 = 23.75 ~= 24
  extra overhead:             24*253 = 6072
  new blocks total:           188403
  estimated old blocks total: 188403-6072 = 182331
  actual old blocks total:    182331


I'm not sure using libe2fs gets us very far as GParted supports: btrfs,
ext*, fat*, hfs*, jfs, nilfs2, ntfs, reiserfs*, xfs and a few other file
systems.  They don't all have libraries and most of them report overhead
according to the f_blocks figure from the statvfs() system call.  I'm
still going to have to increase the threshold to ignore this larger
overhead.

Good point!

I see here that stat -f reports the total number of inodes for the file system.
We can use that number to increase the threshold dynamically.

I.e. if( num_blocks + num_inodes*inode_size/block_size + threshold < partition_size)
        /* unallocated space within the partition */

We don't know the inode size at that point, but we can conservatively calculate
with 128 bytes. At least for ext*, the inode size must be bigger-equal to 128 bytes. I'm not sure if that applies to other file systems, though.

Just to let you know I am working on a patchset which 1) allows the
method used to determine partition usage for active partitions to be
selected on a per file system basis; and 2) implements a specific 
partition usage method for ext2/3/4 which reads the file system size 
from the super block while mounted rather than use the inaccurate figure
from the statvfs() call.

(1) also means that determining usage of active LVM2 PVs won't have to 
be a special case any more as well as allowing (2).  

Confirmed with the ext4 email list that reading the super block for a
mounted ext2/3/4 file system is safe and the best way forward.
http://marc.info/?l=linux-ext4&r=1&b=201209&w=2

'Sounds good to me Mike.  Thanks for the heads up.

Created attachment 224340 [details] [review]
Gparted ext2/3/4 unallocated space fix (v1)

Hi Curtis,

Here's the fix patchset.  Hope that you think calling the same file
system specific set_used_sectors() when mounted and/or unmounted,
depending on the settings of fs.read and fs.read_online, is OK.

Thanks,
Mike

Created attachment 224617 [details] [review]
Gparted ext2/3/4 unallocated space fix (v2)

Hi curtis,

Here's version 2 of the patchset.  Version 1 works perfectly OK but I
decided to do this small update.

Second patch "Read file system size for mounted ext2/3/4 from superblock
(#683255)":
* Clarified comment at start of ext[234]::set_used_sectors()
* Updated condition in ext[234]::set_used_sectors()

Mike

*** Bug 685135 has been marked as a duplicate of this bug. ***

Hi Mike,

This patch set looks great.  I particularly like the way you made the mounted partition usage method selectable based on the file system.  This matches similar functionality in other areas of GParted.  :-)

The patch set in comment #14 has been committed to the git repository for inclusion in the next release of GParted.

The relevant git commits can be viewed at the following links:

Make mounted partition usage method selectable per file system (#683255)
http://git.gnome.org/browse/gparted/commit/?id=01150758c30ddfc49d03f1e2beeb22e93a768d44

Read file system size for mounted ext2/3/4 from superblock (#683255)
http://git.gnome.org/browse/gparted/commit/?id=3828019030f981cc5b3411badc40704f5525fdb3

Remove old #include <cerrno> from jfs and xfs modules
http://git.gnome.org/browse/gparted/commit/?id=e7bfd52a4d78ba1292f79f3e74156707760ca6cb

Remove old FIXME comment from reiserfs::read_label()
http://git.gnome.org/browse/gparted/commit/?id=eaeaebb421ca63750ad00a4fddc93560f7477410

This enhancement has been included in GParted 0.14.0 released on October 10, 2012.