I don't quite get the "generate NEWS from appdata.xml" part. What would that achieve?

I thought that it could be easier to maintain the releases information in the appdata.xml and then generate the NEWS file, because XML->TXT is easier then parsing the a TXT file.

The current content of NEWS would then go into the corresponding <description /> elements, I assume.

Yes, have a look in the org.gnome.Recipes.appdata.xml.in from Gnome Recipes:

https://git.gnome.org/browse/recipes/tree/data/appdata/org.gnome.Recipes.appdata.xml.in

One more reason to add the <releases>-Tag to the gimp.appdata.xml.in:

GNOME Software has now a recently released applications list in the overview page.

Upstream applications have to do two things to be featured there:

1. Have an upstream release within the last 2 months (will be reduced as the number of apps increases)
2. Have upstream release notes in the AppData file

https://bugzilla.gnome.org/show_bug.cgi?id=779839

Are there any examples of NEWS files being generated from appdata files?

Created attachment 352024 [details]
gimp.appdata.xml.in with release information

I've added the release information from the NEWS file from the last release to the gimp.appdata.xml.in.

Created attachment 352025 [details]
First start of a XSLT file to create a text file from the XML file

I've tried to create a XSLT file to create a text file from the XML file. But I'm not an expert...

> Are there any examples of NEWS files being generated from appdata files?

Not that I know, but I tried it myself (It works 50% and creats a html and not a txt file) and now I think it should not to hard for someone with real XSLT knowledge. 

Gimp does btw. something similar with the authors xml file. (Written by Sven Neumann)

Even better, it looks like we can use "appstream-util appdata-to-news" like hughsie does it in the gnome-color-manager:

https://github.com/GNOME/gnome-color-manager/blob/master/Makefile.am

Created attachment 352030 [details]
gimp.appdata.xml.in with release information

I've added the complete release information from the 2.9 NEWS file to the gimp.appdata.xml.in.

I think that makes a lot of sense, but would require whoever runs
make dist have appstream-foo installed.

Created attachment 352592 [details]
gimp.appdata.xml.in with release information

New version of the gimp.appdata.xml.in with all the release information from the 2.9 NEWS file.

Created attachment 352593 [details]
XSLT file to create the NEWS file from the  gimp.appdata.xml.in

Created attachment 352594 [details]
The NEWS file created from the  gimp.appdata.xml.in

This file was created with:
saxonb-xslt gimp.appdata.xml.in gimp.appdata.xml.in.xsl > NEWS

The good news is:
Now we don't need appstream-foo to create the NEWS file.

The bad news is:
"appstream-util validate-strict gimp.appdata.xml.in" will fail with this errors:
 
• style-invalid         : <li> is too long [Layer modes' code is now refactored to separate gamma-corrected variations from the linear light ones, legacy modes have been preserved to support older XCF files]
• style-invalid         : <li> is too long [Never dither when converting to a higher bit depth, or to anything more than 16 bits; don't offer dithering options in these cases]
• style-invalid         : <li> is too long [Pre-initialize some babl-based pixel format conversion routines at start-up to decrease the initial lazy intialization cost for some interactions]
• style-invalid         : <li> is too long [Easily add or remove layer masks by clicking either the layer preview or a new button in the Layers dialog]
• style-invalid         : <li> is too long [Enhance the Layer Attributes dialog to provide the single UI for setting layer's name, blending mode, opacity, and offset, toggling visibility, link status, various locks, color tags]
• style-invalid         : <li> is too long [For all paint tools, when switching to the color picker mode, don't rely on the exact modifier being pressed or released. Instead, check if only the right modifier is pressed after *each* modifier change, and switch to color picking if it is]
• style-invalid         : <li> is too short [Many new icons]
• style-invalid         : <li> is too long [Toolbox buttons do not grab focus anymore, which used to break usage of the Tab key and other canvas-related shortcuts after changing tools with a pointing device click]
• style-invalid         : <li> is too long [Extend the text along the tangent of the last path stroke, when the length of the path is shorter than the width of the text to be warped]
• style-invalid         : <li> is too long [New Interface / Dialog Defaults page in the Preferences dialog to control defaults values of various dialogs]
• style-invalid         : <li> is too long [Color Picker doesn't snap to anything when picking colors anymore, always draws the outline of the averaged region (not only while the mouse is being pressed), and supports arbitrary range of values regardless of the actual image color precision]
• style-invalid         : <li> is too long [For bitmap brushes, GIMP now caches hardness and disables dynamic change of hardness to improve painting performance. Bitmap brushes also don't get clipped anymore, when hardness is less than 100]
• style-invalid         : <li> is too long [Load PSD files with layers which contain additional information with a length that is not divisible by 4]
• style-invalid         : <li> is too long [Print resolution can now be reset to defaults, new values for the same image is actually used now without reloading the image]
• style-invalid         : <li> is too long [Enhance the raw (not the digital negative) files plugin: introduce exporting settings and remember them across sessions, make the plugin invokable non-interactively, and more]
• style-invalid         : <li> is too long [Depends on LCMS2.7+ now, because earlier LCMS2 releases caused problems with certain GIMP blend modes]
• style-invalid         : <li> is too long [Depends on libpng 1.6.25+ to avoid warnings about known incorrect sRGB profiles, also avoid various vulnerabilities]
• style-invalid         : <li> is too long [Updated: Catalan, Czech, Danish, Finnish, French, German, Greek, Hungarian, Icelandic, Italian, Occitan, Polish, Portuguese, Russian, Scottish Gaelic, Serbian, Slovak, Slovenian, Spanish, Swedish]
• style-invalid         : Too many <p> tags for a good description [14/3]
• style-invalid         : <p> is too short [Color Management:]
• style-invalid         : <p> is too short [Now color-managed:]
• style-invalid         : <li> is too short [Color Picker tool]
• style-invalid         : <li> is too short [Color selectors]
• style-invalid         : <li> is too short [DND widget]
• style-invalid         : <li> is too short [Painting tools]
• style-invalid         : <li> is too long [Add a per-image "is color managed" switch and show the image's "is color managed" state in the window title string]
• style-invalid         : <li> is too long [Add "Optimize" options for display and soft-proofing to optionally speed-up rendering at the cost of color fidelity]
• style-invalid         : <li> is too long [Initialize fontconfig in the background to be able to show a pulsing progress bar when rebuilding the list of fonts (typically, the first time GIMP runs)]
• style-invalid         : <p> is too short [Menus:]
• style-invalid         : <p> is too short [Fuzzy Select:]
• style-invalid         : <li> is too short [Improve performance]
• style-invalid         : <li> is too short [Add mirror symmetry]
• style-invalid         : <li> is too short [Add Colorize mode]
• style-invalid         : <li> is too short [Improve blend math]
• style-invalid         : <p> is too short [Preview:]
• style-invalid         : <p> is too short [Build/win:]
• style-invalid         : <li> is too short [Disable webkitgtk]
• style-invalid         : <p> is too short [Windows:]
• style-invalid         : Too many <p> tags for a good description [40/3]
• style-invalid         : <p> is too short [The core got completely ported to GEGL, listing all changes involved would be too much, therefore we summarize the changes in app/ per source subdirectory:]
• style-invalid         : <li> is too short [Add LCH layer modes]
• style-invalid         : <li> is too long [Add much more options to painting, like the ability to lock tool options to brush defaults, and to zoom the brush with the canvas]
• style-invalid         : <li> is too short [Translation updates]
• style-invalid         : Too many <p> tags for a good description [19/3]

The good news is:
"appstream-util validate-relax gimp.appdata.xml.in" runs fine.

*** Bug 791900 has been marked as a duplicate of this bug. ***

Hello,

The NEWS generation from the appdata is an interesting approach. I am only annoyed a bit by the fact we would insert <release> information for development and RC releases with this approach, as discussed in bug 791900.

Most distributions are not meant to distribute development releases and that would just confuse people to show a listing of development releases when all they can install is a stable release.

So I proposed to appstream to add a "type" property to add some semantics to the release listing, so that software applications can choose/filter what they display:
https://github.com/ximion/appstream/pull/158

In the meantime, I welcome patches to add 2.8.x <release> tags in our appdata, and we can prepare for the 2.10 <release> tag.

But I propose to hold a little bit for adding development <release> tags which will show in software installers as well.

Making it a 2.10 blocker because we *MUST* have at least a <release> tag for the stable 2.10 when it is out.

As for the automation script and whether or not we add tags for development releases, that is still to be decided though.

Ok so I at least added 2.8 <release> tags. No description, just dates and versions.

Also I realized that the description needs to be localized so we cannot wait for the last minute to add one. I don't think it needs to be an overly long description though. A few lines with the bigger points of a release should be enough IMO.

commit 3595ef9dc51c6d4a1676043fb6ef4cd02cb71b63 (HEAD -> master, origin/master, origin/HEAD)
Author: Jehan <jehan@girinstud.io>
Date:   Wed Dec 27 22:11:57 2017 +0100

    desktop: add <release> tags for 2.8.x.
    
    Did a bit of archeology to get all the dates back.
    I didn't add any description. Not sure if we should bother that much for
    older versions (but most probably for upcoming version, would be
    useful).
    I also wanted to add URLs to the various news for each release, but
    appstream-util would fail, telling that hyperlinks are not allowed
    inside <p>. Unfortunately <release> tag cannot have a <url> child (nor a
    property) according to appstream docs. So for now, I'll just stick to
    date listing.

 desktop/gimp.appdata.xml.in | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

This may have broken distcheck builds - unless the current breakage of the gimp-2-8 build job on build.gimp.org is caused by the build environment there.

(In reply to Michael Schumacher from comment #22)
> This may have broken distcheck builds - unless the current breakage of the
> gimp-2-8 build job on build.gimp.org is caused by the build environment
> there.

Looks unrelated.
If I look at the console output (https://build.gimp.org/job/gimp-2.8/738/console), I see weird "Permission denied" errors but not much related to desktop/gimp.appdata.xml.in.

Also it looks like we don't check for validity of the appdata in the gimp-2-8 branch (unlike in master) so even if the file was completely broken, it could not break the build anyway.

I have also added a first draft of our <release> description for 2.10 (commented-out for now) in desktop/gimp.appdata.xml.in.

Please, Schumaml, Prokoudine, Patdavid or others, help me to get the right text there. I believe it doesn't have to be exhaustive, just have the gist of what 2.10 is about. Maybe it is even too long already.

And finally I added a checkpoint to fill in this <release> in our release howto. So we won't forget it.

With these, this report is not a blocker anymore. I still leave it opened for the 2 remaining point:
* Do we add development releases?
* Do we go for automatization of NEWS from Appstream metadata?

commit a017668b09065231d575505ea19c668005c2e7ef
Author: Jehan <jehan@girinstud.io>
Date:   Wed Dec 27 23:04:26 2017 +0100

    desktop: add (commented-out) release tag for 2.10.
    
    Let's start from here. I don't think it needs to be too long, just a
    summary of the most awesome changes, I'd say.
    Let's leave it commented out for now, until the description is finished
    and ready for localization. This is the description which may appear in
    software installers using appstream metadata.

 desktop/gimp.appdata.xml.in | 44 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 44 insertions(+)

commit 803917110d55db7ae97b24fd9e8cbef3c224b872
Author: Jehan <jehan@girinstud.io>
Date:   Wed Dec 27 23:16:11 2017 +0100

    devel-docs: add point for <release> appstream data in release-howto.

 devel-docs/release-howto.txt | 9 +++++++++
 1 file changed, 9 insertions(+)

Ok, so the idea of generating NEWS from the appdata was interesting, but I was a bit unsure about whether both files really have the same purpose. We use NEWS as a quite exhaustive summaries of feature changes (or noteworthy changes in general, even when sometimes not visible features). That's still easier to read than `git log`, yet not of interest for everyone, can have quite technical points, and is (very) long.
On the other hand, appdata is GUI-visible data for instance in GNOME Software or other similar installer software to tell what a new version is about.

So I asked the appstream project if they thought that appdata could be used the same was as we used NEWS: https://github.com/ximion/appstream/issues/171
Part of the answer from hughskie:

> I would expect the release info to be readable (so less than say 4 paragraphs)
> and it should be designed for a non-technical user. The primary purpose is so
> the user can decide "do I want to do this update" and the secondary purpose
> IMHO is for marketing. I don't think it's terribly useful to see every change
> that happened, and this is a case where maybe the NEWS page and the appdata
> could diverge.

That seems quite decisive to me that appadata <release> tags cannot really replace our NEWS file because we would just pollute the visual output in GNOME Software, KDE Discover and others.

Therefore I added simple <release> tags with no description for all the 2.9.x release (with a type="development" property, which has now been added to the spec after I asked for it; see https://github.com/ximion/appstream/pull/158). But I didn't add your proposed contents, Tobias. I'm sorry, I think this must have been time-consuming work you did, but this really doesn't look like the kind of contents appstream people have in mind according to their answer above.

I have added the contents for 2.9.8 as proposed by Nate Graham in bug 791900 though, which is more like what appdata seems to be made for. Feel free to propose similar short summaries for other 2.9.x releases.

We are of course welcoming new arguments if you disagree (then just reopen the bug report) and still think we should merge both NEWS and appdata. This was indeed a quite interesting and tempting idea and I liked the generation part of one file from the other.
The problem here is really: how to do so while having acceptable release notes visible in GNOME Software/KDE Discover/etc.

Thanks anyway for the determination and the report.

commit 6d26f78cd6919294918f5ef5055f50e806d64193
Author: Nathaniel Graham <pointedstick@zoho.com>
Date:   Thu Feb 1 03:28:58 2018 +0100

    desktop: added release information to the AppStream file...
    
    ... pre-populated with release notes from version 2.9.8.

 desktop/org.gimp.GIMP.appdata.xml.in.in | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

commit 78f251b0a745219b2f119dc093b2744049d25f17
Author: Jehan <jehan@girinstud.io>
Date:   Thu Feb 1 03:37:57 2018 +0100

    Bug 779839 - Add <releases>-Tag to the gimp.appdata.xml.in.
    
    - appstream-util returns a "style-invalid" error: "<ul> cannot start a
      description [(null)]". So I add a <p> introduction to the 2.9.8
      <release> tag. This was part of unit test failure on the appdata file.
    - I also add a type property for 2.9.8. This is a new property which I
      proposed and which just got accepted in the appstream specification:
      https://github.com/ximion/appstream/pull/158
    - I add <release> tags for all previous 2.9.x releases. No description
      for these, just a type property. But feel free to propose patches
      adding short non-technical description for these.
    
    Note: it was originally proposed in the bug report to use the appdata
    file in place of NEWS (and have this one generated from appdata). But
    after discussion with appstream project, appdata is expected to be
    concise, non-technical and more "marketing" than exhaustive. This is
    quite a different usage than NEWS which is more an exhaustive summary of
    new features and major changes. So these 2 files will likely remain
    distinct.

 desktop/org.gimp.GIMP.appdata.xml.in.in | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

commit 5fc91e92e1b1e2ac1a2c61d3d8ef10bad4a15e3e (HEAD -> master, origin/master, origin/HEAD)
Author: Jehan <jehan@girinstud.io>
Date:   Thu Feb 1 03:51:55 2018 +0100

    desktop: fix the appstream unit tests.
    
    The first validation command was actually failing but the test was not,
    because only the last return value is taken into account, obviously. Add
    a '&&' between the commands.
    Also test against the built files, not the templates (in particular
    because `appstream-util` doesn't like .in.in templates, and anyway it's
    always better to test against the finale file).
    Finally move to "validate-relax" test for the time being.
    "validate-strict" actually makes a bunch of errors, but I can't make the
    time to look at these now. Let's just settle with basic validation at
    least.

 desktop/Makefile.am     | 2 +-
 desktop/test-appdata.sh | 6 ++++--
 2 files changed, 5 insertions(+), 3 deletions(-)

P.S.: maybe you could provide a screenshot of what the release listing would look like with huge <release> tags in GNOME Software or KDE Discover or other installers? I don't see anything in my current GNOME Software, maybe it's not implemented yet or not in my version of GNOME.

When I see the appdata from Recipes in comment 4, it really looks like they also just provide small concise release notes.

This being said, it is possible that after 2.10, as we are planning to release new features more often, our release notes may become much smaller and maybe we can start considering merging both files then.