I saw that the original code was written by Jason so I'm adding him to CC because I don't understand the need for scaling the calibration.

(also adding the person Josh, who first reported this on the gnome-c-c list)

The original code is actually from xinput_calibrator; I just forked and translated it to C. From what I understand, the scaling is used to account for whatever calibration is currently in effect.

For example, assume a 1000px wide screen and a 1000dx wide sensor with 10% spill (i.e. 0px == 100dx; 1000px == 900dx). To make our lives easier, we'll also assume that the calibration targets are at the exact corners of the screen. Prior to calibration xmin==0, xmax==1000. The sampled values are xmin'==100, xmax'==900. The code calculates the scale factor as xscale=(xmax-xmin)/1000==1.0 and uses this and the pre-existing offset to obtain xmin=xmin'*xscale+xmin==100, xmax=xmax*xscale+xmin==900 (exactly as we expect). On a subsequent calibration, the sampled values are xmin'==0, xmax'==1000. Now the scale is xscale=(xmax-xmin)/1000==0.8 which when used results in xmin=xmin'*xscale+xmin==100, xmax=xmax*xscale+xmin==900 (exactly as we need).

I find it interesting that the (I assume) original bug report[1] mentions that xinput_calibrator does not use the correct device by default, but will provide a proper calibration when given the right one. It sounds to me like the GNOME calibrator is somehow being fed the wrong device ID on startup.

[1]: https://bugs.launchpad.net/ubuntu/+source/gnome-control-center/+bug/1163107

I think I see what you are saying about the scaling being useful.  Sadly, I have not had a chance to really explore this code, yet.  Hoping to have that time this next week.

What I have seen is this.  If I open up dconf, search for wacom, and chop about 5000 off each of the high values of the area config, then I will never be able to get a calibrated device using the config panel.  Similarly, if I feed these values into xinput_calibrator, I will also not get a valid calibration.  (By default, xinput_calibration starts with "reset" values for the area.)

The thing I would like to know is whether not using the old values will work or not. Assuming that this would be the same as an initial state and that it would work, we could remove that code. Unless I didn't understand its real importance...

Any progress on this? As an artist, this makes ubuntu derivatives unusable for me.

It's fixed in GNOME 3.10, which resets the tablet calibration to the default values before showing the calibration UI. If the calibration is cancelled or unsuccessful, the old values will be reapplied.

See also bug 707784