After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 662454 - Incorrect word counts
Incorrect word counts
Status: RESOLVED FIXED
Product: damned-lies
Classification: Infrastructure
Component: general
unspecified
Other Linux
: Normal normal
: ---
Assigned To: damned-lies Maintainer(s)
damned-lies Maintainer(s)
Depends on:
Blocks:
 
 
Reported: 2011-10-22 12:45 UTC by Gabor Kelemen
Modified: 2012-06-12 18:43 UTC
See Also:
GNOME target: ---
GNOME version: ---


Attachments
Screenshot of the problem (82.37 KB, image/png)
2011-10-22 12:45 UTC, Gabor Kelemen
  Details
a screenshot that shows the problem (77.73 KB, image/png)
2012-04-28 05:52 UTC, Jiro Matsuzawa
  Details
a patch for this bug (547 bytes, patch)
2012-04-28 05:56 UTC, Jiro Matsuzawa
none Details | Review

Description Gabor Kelemen 2011-10-22 12:45:23 UTC
Created attachment 199720 [details]
Screenshot of the problem

The new word counting feature reports incorrect values for my gnome-help translation. 

Strings: 98%   2470    35     0  
Words:  95%  51252  1195  1287 

How can the 0 untranslated strings be 1287 words long?
Aside from that, the 1195 words for the number of fuzzy strings is correct.
Comment 1 Gil Forcada 2011-12-24 23:23:41 UTC
The problem is that I can not reproduce it locally unfortunately.

Any pointers on how to reproduce it would be helpful.
Comment 2 Chris Leonard 2012-03-28 03:04:23 UTC
51252 translated 

plus

1195  fuzzy 

equals the 

52447 total words shown in screen shot for the POT.

The persistence of the 1287 in a field that should be zero is inexplicable and apparently not reproducible.

As this bug was filed 5 months ago, can you provide another example fo thsi behavior?  Or perhaps is was a temporary abberation.
Comment 3 Gabor Kelemen 2012-03-28 07:41:53 UTC
Not reproducible? Sadly it is.

Please take a look at http://l10n.gnome.org/languages/hu/gnome-3-4/ui/ and any of the modules, they almost all (those changed recently) show the same symptom:

http://l10n.gnome.org/vertimus/empathy/master/po/hu
4332     0   441

http://l10n.gnome.org/vertimus/epiphany/gnome-3-4/po/hu
1757     0   623

etc.

And other languages too:
http://l10n.gnome.org/vertimus/epiphany/gnome-3-4/po/fr
1757     0     1

http://l10n.gnome.org/vertimus/epiphany/gnome-3-4/po/de
1757     0   623

http://l10n.gnome.org/vertimus/epiphany/gnome-3-4/po/ca
1757     0  3647
Comment 4 Jiro Matsuzawa 2012-04-28 05:52:51 UTC
Created attachment 213011 [details]
a screenshot that shows the problem

Hi all,

I've attached a screenshot that shows this bug. Please refer to it.
It shows:
Strings: 100% 1204 0 0
Words: 97% 3812 0 111

I've debugged and determined the cause of the problem.
The following diff is a patch:
===================================
diff --git a/stats/utils.py b/stats/utils.py
index 40b8982..10999a6 100644
--- a/stats/utils.py
+++ b/stats/utils.py
@@ -333,7 +333,7 @@ def po_file_stats(pofile, msgfmt_checks=True):
         input_file = pofile
 
         if has_toolkit:
-            status = pocount.calcstats_old(pofile)
+            status = pocount.calcstats(pofile)
             res['fuzzy_words'] = status['fuzzysourcewords']
             res['translated_words'] = status['translatedsourcewords']
             res['untranslated_words'] = status['untranslatedsourcewords']
===================================

You should use calcstats() instead of calcstats_old().
The calcstats_old() function counts obsolete messages as untranslataed.
So, the total count of valid words are incorrect.

Thank you.
Comment 5 Jiro Matsuzawa 2012-04-28 05:56:11 UTC
Created attachment 213012 [details] [review]
a patch for this bug

I've attached the patch.
Please review it.
Comment 6 Claude Paroz 2012-04-28 07:31:36 UTC
I remember we had problems with calcstats, see http://git.gnome.org/browse/damned-lies/commit/?id=827c8fcfb

I can try to temporarily reenable the new calcstats method on l10n.gnome.org, but we might encounter the same issues again.
Comment 7 Claude Paroz 2012-04-28 07:47:56 UTC
Here's the traceback I got from using the calcstats method:

Traceback (most recent call last):
  • File "/var/www/djamnedlies/stats/management/commands/update-stats.py", line 39 in handle
    branch.update_stats(options['force'])
  • File "/var/www/djamnedlies/stats/models.py", line 460 in update_stats
    pot_stats = utils.po_file_stats(potfile, msgfmt_checks=False)
  • File "/var/www/djamnedlies/stats/utils.py", line 339 in po_file_stats
    status = pocount.calcstats(pofile)
  • File "/usr/local/www/gnomeweb/python-packages/lib/python2.6/dist-packages/translate/tools/pocount.py", line 81 in calcstats
    statscache = statsdb.StatsCache()
  • File "/usr/local/www/gnomeweb/python-packages/lib/python2.6/dist-packages/translate/storage/statsdb.py", line 337 in __new__
    return make_database(statsfile)
  • File "/usr/local/www/gnomeweb/python-packages/lib/python2.6/dist-packages/translate/storage/statsdb.py", line 316 in make_database
    cache.create()
  • File "/usr/local/www/gnomeweb/python-packages/lib/python2.6/dist-packages/translate/storage/statsdb.py", line 144 in decorated_f
    result = f(self, *args, **kwargs)
  • File "/usr/local/www/gnomeweb/python-packages/lib/python2.6/dist-packages/translate/storage/statsdb.py", line 342 in create
    self.file_totals = FileTotals(self.cur)
  • File "/usr/local/www/gnomeweb/python-packages/lib/python2.6/dist-packages/translate/storage/statsdb.py", line 189 in __init__
    translatedtargetwords   INTEGER NOT NULL);""")
OperationalError: disk I/O error

Comment 8 Jiro Matsuzawa 2012-04-28 13:23:02 UTC
(In reply to comment #6)
> I remember we had problems with calcstats, see
> http://git.gnome.org/browse/damned-lies/commit/?id=827c8fcfb
> 
> I can try to temporarily reenable the new calcstats method on l10n.gnome.org,
> but we might encounter the same issues again.

Hmm,
In fact, there is mismatch between string statistics and words statitics. 
I think it is a problem of the pocount which counts obsolete units.
I report that on http://bugs.locamotion.org [1].

[1] http://bugs.locamotion.org/show_bug.cgi?id=2159
Comment 9 Gil Forcada 2012-06-07 17:43:17 UTC
New update on this.

I asked the sysadmins to apply the patch made by Jiro (and already committed on translation-toolkit) and as of yesterday is already on l10n.gnome.org, see https://bugzilla.gnome.org/show_bug.cgi?id=677587

For some reason the bug is still present. I made quite a few commits yesterday and today and all the translations still counts the obsolete strings on the word counting.

Maybe there is another patch that is missing. I will ask the translate-toolkit team to create a new release.
Comment 10 F Wolff 2012-06-11 09:49:24 UTC
I don't see any confirmation in bug 677587 that the fix has been deployed. The latest toolkit does have this fixed (I verified on some specific files that currently the problem on damned lies), so I'm just wondering if the fix was maybe applied to an unused copy of the toolkit, or maybe not yet deployed.
Comment 11 Claude Paroz 2012-06-11 11:37:12 UTC
I confirm (sorry for the delay) that DL is not using the system translate-toolkit but a custom one.
I just applied the patch and restarted the app, so now you can check again if this makes any difference.
Comment 12 Gil Forcada 2012-06-12 18:43:33 UTC
I can confirm that it seems to work:

I just uploaded this translation which shows words at 100%:
http://l10n.gnome.org/vertimus/gtksourceview/gnome-3-4/po/ca@valencia

Even if the file has obsolete translations:
http://l10n.gnome.org/POT/gtksourceview.gnome-3-4/gtksourceview.gnome-3-4.ca@valencia.po

Hurray for Jiro!!