Hi Phillip and Curtis,
I have git cloned the current gparted version and applyed the patch
I am currently working on external 1 tera usb drive, and moving ext4 partition, 121 GO with 32 go of data on it.
External usb drive is connected to vm ubuntu 13.10


Time remaining is around 4 or 5 hours : it seems to be much.
Does this come from using ext enabled by patch, or is it the same if gparted makes the move itself ?
Optimal block size is 16.00 MB

Larry

Same operation but on internal vm drive (I canceled the first one)
Moving 76 go partition with 12 go of data on it.
block size is 2.00 MB

Time remaining is 28 minutes.

In the previous mail I talk about "using ext"  : please read "using e2image" !

Larry

It sounds like you are still using the gparted internal copy method.  Do you have e2fsprogs version 1.42.9?

Hi Phillip,
I ran src/gpartedbin from the gparted patched directory.
What do you mean by internal ? 

e2fsprogs version is 1.42.8-1ubuntu1
1.42.9 is not in my the ubuntu repo.

What do you expect me to do ?

Move operation is done. I save de details, if needed.

Larry

The internal algorithm where gparted itself just copies the whole partition instead of running an external program ( e2image in this case ).  You will need to manually install a newer e2fsprogs or upgrade to Ubuntu 14.04.

Hi Larry,

Thank you for beginning to test this patch.  It sounds like you have confirmed that the old internal copy/move code still works with the patch applied.

Since you are experienced with Virtual Machines, it might be easiest to try Phillips suggestion of installing a pre-release of ubuntu 14.04 because this includes the required e2fsprogs v1.42.9.

For this patch we probably don't need it tested on all distros, but one or two extra would be good to ensure that the old copy/move code continues to function properly.

Curtis

Ok, done
* ubuntu 14.04 and gparted from git
* e2fsprogs 1.42.9

* e2image runs the moving operation
* As expected by Phillip : no info during resize, nor progress bar
* I dont know the time it took

I save de details in case.

Larry

If you run it on a large filesystem with hardly any data on it, you should notice it takes hardly any time ;)

Larry, would you be able to test with ext2, ext3, and ext4 partitions that contain data as well?

Perhaps also try some move/copies using Cylinder alignment too.

Thanks,
Curtis

Hi guys !
I made all the tests :
- ext2/3/4 : moving 11 go data. It took more or less 5 minutes
- I set cylinder option
- I save the details if there is a need.

It seems to work well.

Is this all what you need as test ?

Larry

11 gb of data on a filesystem that was how large?

ext2 : 35.57 gb --> moving 11.43 gb
ext3 : 35.25 gb --> moving 11.55 gb
ext4 : 53.78 gb --> moving 10.81 gb

btw, when launching gpartedbin from terminal, i have some error lines (about 40 lines) like this one :
(gpartedbin:2898): GLib-CRITICAL **: Source ID 115 was not found when attempting to remove it
ID number grows up

Larry

Thank you Larry for the extra testing and noticing the new "GLib-CRITICAL" message.

Regarding the "GLib-CRITICAL" message, this messsage seems unrelated to this patch.  The message appears even with gparted-0.18.0 without patches.  Interestingly, the message does not appear with Ubuntu 13.10 or earlier so it looks like something changed with the GLib library in Ubuntu 14.04.


Phillip,

From looking at the man page for e2image, the "-p" command flag sounds like it will output text based progress.

Was there a reason this flag was omitted from the patch, or is this something we can add to improve progress feedback?

Thanks,
Curtis

I left it out of this patch in case you wanted to apply it before applying the progress feedback patches, but yes, adding that flag on top of the other patches will give progress indication.

I also made sure to make it cancel safe: if the first SIGINT comes in after the point of no return, it prints a warning rather than bailing out and leaving you with a corrupt fs.  Hitting force cancel to send a second SIGINT will stop it.

Hi Phillip,

Please get the code layout and commit messages closer to the GParted
standard style.

In fact I've never see any source layout an if statement like this:
	if ( condition )
		statement_1;
	else statement_2;

Like this with the else condition statement on a separate line and
indented the same amount as the then condition statement:
	if ( condition )
		statement_1;
	else
		statement_2;

When a statement is wrapped on to multiple lines indent using tabs to
a common vertical for the code block, use spaces for further indent on
continuation lines.  This is so that if someone has their tab width
other than the standard 8 the code is still laid out correctly.
	statement( ...
	           continuation further indented with space )
^TAB	^SPACES^^^^

GParted commit messages are like this with the bug number in the summary
line and bug description in the title.  If necessary open the GNOME bug
a description of the change you are going to do and post the patch as
a subsequent update:
--8<-- commit message --8<--
Use e2image to move/copy (#721516)

Use e2image features added in e2fsprogs 1.42.9 to move/copy an ext[234]
file system more efficiently, by skipping unused blocks.  If e2image is
not available or doesn't have the required features fall back to using
GParted's internal whole partition block copy.

Bug #721516 - Use e2image to move/copy ext[234] filesystems
--8<--

Thanks,
Mike

Created attachment 270596 [details] [review]
0001-Use-e2image-to-move-copy-721516.patch

I've made the new line after else and log changes, but I am not sure how to get emacs to do indenting like that, and I looked around and it doesn't look like any of the existing gparted code does that.  It is always tab tab tab, space space.  Sometimes it lines up with the first line, and sometimes it ends up significantly to the left, only being indented one or two more tabs than the first line.

Hi Phillip,

Not using Emacs myself I don't know, but the first result from Google is
this which appears to explain how to do it:

http://www.emacswiki.org/SmartTabs


Hi Larry,

As you seem to be testing this can you make sure you test partition moving,
both none overlapping and overlapping in both directions.

Thanks,
Mike

Hi Phillip,

Please also correct else statement layout in ext2:move().

(In reply to comment #17)

> Hi Larry,
> As you seem to be testing this can you make sure you test partition moving,
> both none overlapping and overlapping in both directions.

- This means three operations for each fs, right ?

- The vm drive I use is 200 GB, so I first need to shrink each partition to get enough free space for avoid overlapping.
Does partition size make any problem ? You dont mention anything about partition size, so I guess it doesnt matter.

- What about cylinder/mega option ?

Larry

Hi Mike,

Thanks for the tip about Smart Tabs for emacs.  I use a combination of Eclipse and Emacs when programming and have always had to futz with the indentation.

I'm running kubuntu 12.04 on my development box which has emacs 23.3+1-1ubuntu9.2.  To use smart-tabs-mode, I downloaded both cl-lib.el and smart-tabs-mode.el and saved these into the directories indicated in the following ~/.emacs customization file snippet:

;; ---------
;; SmartTabs
;;   http://www.emacswiki.org/SmartTabs
;;   http://blog.lovolabs.com/2013/03/emacs-smart-tabs-i-prefer-tabs-for.html
;;   https://stackoverflow.com/questions/20678847/cannot-load-cl-lib-at-emacs-startup
;; ---------
;;
;; add required cl-lib needed by smart tabs
(add-to-list 'load-path "~/.emacs.d/lisp/cl-lib/")
(require 'cl-lib)

;; load smart tabs
(load "~/.emacs.d/lisp/smart-tabs-mode")

;; use smart tabs for all the predefined languages
(smart-tabs-insinuate 'c 'c++ 'java 'javascript 'cperl 'python 'ruby 'nxml)


Phillip,

On the topic of indentation, we have been trying to use the indentation method described by Smart Tabs to ensure that the code displays properly in all code editors, regardless of tab size.  Having said that, there is much code in GParted that does not follow this alignment scheme and uses a mish-mash of spaces and tabs, and sometimes even spaces immediately in front of tabs.

This makes for some "interesting" block indentation and bracket alignment in sections of the source code.  Longer term we would like to clean this up bit by bit.  As such for changes and new code we are try to follow the alignment scheme outlined in the smart tabs.

I slip up on this indentation occasionally too which is another reason why it really helps to have additional eyes looking at the code such as you and Mike.

On the topic of e2image progress indication, I tested adding "-p" to the e2image commands and this worked well.  I could then see the progress of the e2image command when I expanded the details in the GParted Applying Pending Operations dialog.  I find this progress information valuable even if it is not tied to a graphical progress bar.

If there are no concerns about adding this flag, then please do add "-p" to the e2image commands.


Larry,

>> As you seem to be testing this can you make sure you test partition moving,
>> both none overlapping and overlapping in both directions.
>
>- This means three operations for each fs, right ?

I think this would mean four operations.  Ideally performed with each of ext[234].  For example:

  1)  Move ext2 right such that a portion overlaps the original partition.
  2)  Move ext2 left  such that a portion overlaps the original partition.
  3)  Move ext2 right such that no portion overlaps the original partition.
  4)  Move ext2 left  such that no portion overlaps the original partition.

Since you have already tested Cylinder and MiB alignment I believe that these alignment tests do not need to be done again.

Hope that helps,
Curtis

(In reply to comment #20)

> Larry,
> I think this would mean four operations.  Ideally performed with each of
> ext[234].  For example:
> 
>   1)  Move ext2 right such that a portion overlaps the original partition.
>   2)  Move ext2 left  such that a portion overlaps the original partition.
>   3)  Move ext2 right such that no portion overlaps the original partition.
>   4)  Move ext2 left  such that no portion overlaps the original partition.
> Hope that helps,
> Curtis

Hi Curtis,
All is done.
Absolutely no error reported, but I still save each gparted-details for each operation and each fs. I thing I wont post 12 replies, one for each file, but I can mail them if you wish. Let me know.

Larry

Hi Larry,

Thank you for running all those tests.  Now I think we need to wait for the patch to be reformatted using the suggestions in earlier comments.

If you have more time for testing, I noticed that Phillip posted an additional patch for the following report:

Bug 467925 - gparted: add progress bar during operation

Curtis

Created attachment 270733 [details] [review]
Use e2image to move/copy ext2/3/4 file systems (v3)

Hi All,

Here's patch v3 of Phillip's patch.  Makes the following changes
compared to v2 from comment #16:

1) Only runs e2image to test it's capabilities if it's found in the
   path.  This prevents these errors written to the console by GParted
   if e2image doesn't exist:
      Failed to execute child process "e2image" (No such file or
      directory)

2) Fixes previously mentioned smart tab/space indent issue (from
   comment #15)

3) Fixes else statement layout issue in ext2::move() (mentioned in
   comment #18)

4) Adds "-p" progress option to move and copy (comment #20).

Thanks,
Mike

Hi All,
Testing the last new patch v3.
I notice the -p option add the detail progression of the move operation, if opening details box. It works well.

Is there any other special test to run about this patch ?

Larry

Created attachment 270827 [details] [review]
Use e2image to move/copy ext2/3/4 file systems (v4)

Hi All,

While reviewing patch v3 I discovered that some of the command lines were built using String::ucompose.  Unfortunately String::ucompose can introduce problems due to different locales using different number formats.  As such I have removed the use of String::ucompose from patch v3.

I also noticed the use of String::ucompose when writing or clearing labels for XFS file systems.  Hence I included a small patch to correct this usage, though admittedly it would not likely have created a big problem if the number format in a label was different.  See the patch description for more details on the problem.


Mike,

Thank you for cleaning up and posting patch v3.  When you have time please review patch v4.

Larry,

Thank you for testing patch v3.  If you have time would you be able to perform a few move/resize tests with ext[234] using this new patch v4?

Also please test setting and clearing the label on an XFS file system.

Thanks,
Curtis

(In reply to comment #25)

> 
> Larry,
> 
> Thank you for testing patch v3.  If you have time would you be able to perform
> a few move/resize tests with ext[234] using this new patch v4?
> 
> Also please test setting and clearing the label on an XFS file system.
> 
> Thanks,
> Curtis

Ok Curtis,
One thing : test have to be run on ubuntu 14.04, I guess, because of the e2fsprogs version, right ?

Larry

Hi Larry,

Yes, testing is needed on ubuntu 14.04 because it has the required e2fsprogs 1.42.9 version.

It is also useful to test on a distro that has an older version of e2fsprogs that includes e2image, and perhaps an even older distro that doesn't include e2image.

Curtis

You can download the version of e2fsprogs from 14.04 and install it on 13.10 if you want.

(In reply to comment #27)
> and perhaps an even older distro that doesn't include
> e2image.

@ Curtis
If I though, the system check if e2image is available and if not gparted does the trick itself. 
Am I wrong ?

@ Phillip
To dwnld this e2fsprogs version I need to add an entry in the apt.sources file ?
Or is there an other way to get it ?

(In reply to comment #29)
> If I though, the system check if e2image is available and if not gparted does
> the trick itself. 
> Am I wrong ?

You are correct Larry.  The GParted code should gracefully handle this situation all by itself.  The key is to test that GParted does correctly fall back to using it's own move/copy algorithm when the e2image version is less than 1.42.9, or when the e2image program is not available.

If you wish you can simulate a missing e2image by finding where the e2image program is (which e2image), and then temporarily moving it to a different name (sudo mv /sbin/e2image /sbin/e2image.bak).  After testing the e2image command can be restored back to it's original name.

> To dwnld this e2fsprogs version I need to add an entry in the apt.sources file
> ?
> Or is there an other way to get it ?

There are a variety of ways to get the latest e2fsprogs.  I find that setting up a VM for an OS that already has the required version is the easiest.  In fact Ubuntu 14.04 just released the beta1 image a few days ago.  See:

First Round of Ubuntu 14.04 Beta Releases are Available for Download
http://www.howtogeek.com/183514/first-round-of-ubuntu-14.04-beta-releases-are-available-for-download/

Other options include downloading and installing the package (.deb or .rpm) directly, or downloading the source tarball and compiling and installing.

In order to keep track of all my VMs, I try _not_ to install versions higher than those in the repositories.  In this way I ensure that when I test on a distro that I am testing with the exact versions the distro includes.

Hope that helps.  :-)

Curtis

You can look it up on packages.ubuntu.com, or there is also a way on launchpad.net, but the former is easier.

Hi All,

Reviewed patchset V4 from comment #25.  Passed review and some testing.

But, when testing some overlapping moves of a 6 GiB ext4 file system in
a virtual machine I found GParted's internal block copy roughly 3 times
faster than e2image's copy.  So e2image was only faster when the fs was
less that 33% full.  After that GParted's internal block copy was
faster.  Did anyone else notice this?

Also GParted's internal block copy can be cancelled and rolled back,
leaving the file system intact.  Force cancelling an e2image copy will
leave the file system corrupted.  There is however a warning in the
output of e2image that this will happen if you proceed and it is press
Cancel, wait 5 seconds press Force Cancel.

Mike

Hi Mike,

Thanks for reviewing and performing some more tests.  Based on your observations I performed some quick tests with a 50% full ext4 1 GiB partition and a move overlap of 66 MiB.  I was surprised to see GParted internal moves were taking about the same amount of time as e2image moves (about 5 seconds).


Larry,

Would you be able to perform overlapping move tests with an ext4 file system that contains files?

Please record the time for moves to the left and to the right using e2image.
Then rename /sbin/e2image to /sbin/e2image.orig and run the tests again to time the moves using GParted's internal copy algorithm.

If you can use some larger partition sizes that might help too.

Thanks,
Curtis

IIRC, the way I implemented it in e2image it will perform poorly if moving to the right by only a slight amount.  In other words, almost entirely overlapping.  This is because if you are moving say, 1 MiB to the right, then it has to read at most 1 MiB, write it, then seek backwards and repeat.  Moving a significant distance to the right, or any amount to the left should perform quite nicely.

I would think that 66 MiB would be plenty to avoid the seek penalty and since e2image is only copying the used blocks, should be quite a bit faster.

Hi Phillip,

I too anticipated that e2image would be much faster.  A closer look at Larry's test results should help us get to the bottom of this mystery.


Larry,

If I recall correctly, you saved all of the gparted_details.htm log files.  If so, it might be easier to summarize the results of these into some sort of chart.

Perhaps a chart with headings such as the following:

Copy Type:  e2image or GParted
Partition Size:
File system %full:
Move direction:  left or right
Move distance (MiB):
Elapsed time:

Please feel free to come up with a chart of your own choosing.  We are trying to learn more about how long e2image takes to move a partition compared to GParted's internal algorithm.

Thanks,
Curtis

(In reply to comment #25)
> Created an attachment (id=270827) [details] [review]
> Use e2image to move/copy ext2/3/4 file systems (v4)

> Larry,
> 
> Thank you for testing patch v3.  If you have time would you be able to perform
> a few move/resize tests with ext[234] using this new patch v4?

@Curtis to not scatter me, I give here my results about those tests (either ext2/3/4 with about 40% of data) :
ubuntu 14.04 e2fsprogs 1.42.9 e2image runs the move
ubuntu 13.10 e2fsprogs 1.42.8 move using internal algorithm
fedora 18 e2fsprogs 1.42.5-1 move using internal algorithm
centos 6.5 e2fsprogs 1.41.12-18 move using internal algorithm
fedora 20 e2fsprogs 1.42.8-3 (1.42.9 coming with FC21 ->2014 august) move using internal algorithm

> 
> Also please test setting and clearing the label on an XFS file system.
I did not perform this test yet.
______________

Curtis
About the Copy Type, I gonna perform overlapping move tests with an ext4 file system that contains files, and report them into the chart you ask.

Larry

Hi All,
Here are my tests :
* Perform overlapping move tests with an ext4 file system that contains files.
* OS : Ubuntu 14.04 beta --> e2fsprogs 1.42.9
* Learning more about how long e2image takes to move a partition compared to GParted's internal algorithm


chart 
Copy Type:  e2image / GParted (e2image disabled)
Partition Size: 
File system %full: 
Move direction:  to the left / to the right
Move distance :
Elapsed time (move only / complete operation):

*******Copy Type :  e2image*******

Partition Size: 26.06 GiB
File system %full: 41% (15 GiB)
Move direction:  to the right
Move distance : 10.04 GiB (21067776)
Elapsed time (move only / complete operation) : 8:06 / 8:09

Partition Size: 26.06 GiB
File system %full: 41% (15 GiB)
Move direction:  to the left
Move distance : 5.72 GiB (12015616)
Elapsed time (move only / complete operation): 4:47 / 4:50

Partition Size: 26.06 GiB
File system %full: 78% (19 GiB)
Move direction:  to the left
Move distance : 15.06 GiB (31588352)
Elapsed time (move only / complete operation): 7:26 / 7:28

Partition Size: 26.06 GiB
File system %full: 78% (19 GiB)
Move direction:  to the right
Move distance : 12.91 GiB (27076608)
Elapsed time (move only / complete operation): 7:20 / 7:22


*******Copy Type: GParted (= e2image disabled)*******

Partition Size: 26.06 GiB
File system %full:  78% (19 GiB)
Move direction:  to the left
Move distance : 12.91 GiB (27076608)
Elapsed time (move only / complete operation): 9:56 / 9:57

Partition Size: 26.06 GiB
File system %full:  78% (19 GiB)
Move direction:  to the right
Move distance : 19.77 GiB (41461760)
Elapsed time (move only / complete operation): 10:55 / 10:58

Partition Size: 26.06 GiB
File system %full: 36% (9.4 GiB)
Move direction:  to the left
Move distance : 4.84 GiB (10156032)
Elapsed time (move only / complete operation): 9:52 / 9:53

Partition Size: 26.06 GiB
File system %full: 36% (9.4 GiB)
Move direction:  to the right
Move distance : 5.78 GiB (12128256)
Elapsed time (move only / complete operation): 11:54 / 11:57


Hope this help
Larry

Hi All,

Looking at Larry's figures e2image is faster because it only copies used
blocks, though there is quite a lot of variability with both methods.

I've done some more testing and here are my setup this time with
figures:

Host:
	CentOS 6.5 x86-64
	16 GiB RAM
	xfs file system
	VirtualBox 4.3.8
Guest:
	Fedora 20
	updated with e2fsprogs 1.41.9
	5.0 GiB ext4 file system
	2.21 GiB used (44%)
Operation: Move left by 500 MiB
e2image		0:25 (min:sec)
Internal	2:03
Operation: Move right by 500 MiB
Internal	1:01
e2image		0:23
Operation: Move left by 100 MiB
e2image		0:26
Internal	1:11
Operation: Move right by 100 MiB
Internal	0:58
e2image		0:25

These agree with Larry's figures.  e2image is faster because it only has
to copy used blocks.

Yesterday when I was testing the host file system containing the VM was
ext4.  This makes a difference because VirtualBox specifically warns
that for ext4 it can't do direct writing and can only write to the file
system buffer cache.  This can make the guest I/O speed very variable as
sometimes it's all absorbed by the host cache and other times write back
is forced part way through the operation.

I'm ready to commit the patchset now, unless there are any objections.

Thanks,
Mike

For VM testing you can avoid the performance problems and variability by using an lvm volume instead of a file as the backing store for the virtual disk.

I believe you can mostly get there with a disk file if you make sure to preallocate all of the space so it doesn't get fragmented with eg. fallocate.  Also at least qemu has options to disable the use of the host buffer cache and use direct IO, and using the virtio disk interface is better than emulating ide.

Hi All,

Thank you Larry and Mike for testing and summarizing the results.  Both of your testing results show that e2image is most often faster than the GParted internal algorithm.  In my testing I recall e2image being faster, and not worse than the internal algorithm.

Phillip,

Thank you for the testing suggestions.  For my VMs I create an extra virtual disk for running test operations.  I think this should be representative of the real world, but I know that sometimes a person just has to test on bare metal.  :-)

Mike,

I agree that the patch set is now ready.  Please feel free to commit the patch set.  

Regards,
Curtis

The question is in *how* you create the virtual disk for testing.  It can be backed by a sparse file, a qcow2 imgage, a real disk, or an lvm volume.  The sparseness of a sparse file or qcow2 will cause all sorts of performance weirdness with this kind of load.  You can use fallocate to force a regular file to not be sparse, and hopefully, not too fragmented.

Hi Phillip,

I'm using VMWare Player to create and run my VMs.  I've been using this software to create the virtual disk.  I suspect that it creates a sparse file that grows as the virtual disk is filled with data.

Regarding performance testing I repeat steps a few times, such as moving a partition the same distance back and forth the same amount.  By doing this I believe the underlying file size should become allocated leading to similar performance times for subsequent tests.  At least this has been my experience so far.

Curtis

It has been years since I've looked at vmware, but iirc, it has its own proprietary file format that is similar to qcow2 that is used by qemu and xen.  It should also have an option though, to turn that off and just use a flat file.

Repeating the same test multiple times should get the results to be consistent, but they may be consistently bad compared to the alternatives.

Hi,

The following commits have been applied to the GParted GIT repository
for inclusion in the next release:

Use e2image to move/copy ext[234] file systems (#721516)
https://git.gnome.org/browse/gparted/commit/?id=86111fe12a26d23d9fc2a9e2d19281290ecaf985

Avoid using String::ucompose to build command lines
https://git.gnome.org/browse/gparted/commit/?id=7b1011237ca8921163f1cc576f9df7047e337c31

Thanks,
Mike

This enhancement was included in the GParted 0.19.0 release on June 10, 2014.