GNOME Bugzilla – Bug 330879
ID3 tags displayed with wrong encoding
Last modified: 2006-03-03 13:04:25 UTC
When playing MP3 files with the xine-lib backend, non-ASCII ID3 tags are displayed garbled. Despite the locale being UTF-8 and the tags being encoded in UTF-8, they are displayed as if there's some ISO8859-1 conversion somewhere. For example "Poupée" becomes "Poupée". The same appears in the properties page. The encoding for OGG metadata is correct.
Try removing the smb plugin from xine-lib: rm /usr/lib/xine/plugins/1.1.2/xineplug_inp_smb.so As well as the smb plugin from nautilus: rm /usr/lib/gnome-vfs-2.0/modules/libsmb.so and restart nautilus Does the metadata appear properly?
Sorry, it doesn't change anything. However, I notice the xine-lib version is 1.0.1, could it be a bug fixed in a more recent version?
Could you please give me the output of locale, as well as the file that's causing troubles, and the command-line output of nautilus with /apps/totem/debug enabled?
1) The output of locale: LANG=fr_FR.UTF-8 LC_CTYPE="fr_FR.UTF-8" LC_NUMERIC=C LC_TIME="fr_FR.UTF-8" LC_COLLATE="fr_FR.UTF-8" LC_MONETARY="fr_FR.UTF-8" LC_MESSAGES="fr_FR.UTF-8" LC_PAPER="fr_FR.UTF-8" LC_NAME="fr_FR.UTF-8" LC_ADDRESS="fr_FR.UTF-8" LC_TELEPHONE="fr_FR.UTF-8" LC_MEASUREMENT="fr_FR.UTF-8" LC_IDENTIFICATION="fr_FR.UTF-8" LC_ALL= 2) A small problematic file : http://malsain.org/~joss/orgasme.mp3 3) The output of nautilus with debugging enabled : load_plugins: skipping unreadable plugin directory /home/joss/.xine/plugins. load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_vo_out_xshm.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/post/xineplug_post_goom.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/post/xineplug_post_audio_filters.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/post/xineplug_post_audio_filters.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/post/xineplug_post_switch.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/post/xineplug_post_visualizations.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/post/xineplug_post_visualizations.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/post/xineplug_post_visualizations.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/post/xineplug_post_mosaico.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/post/xineplug_post_planar.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/post/xineplug_post_planar.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/post/xineplug_post_planar.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/post/xineplug_post_planar.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/post/xineplug_post_planar.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/post/xineplug_post_planar.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/post/xineplug_post_planar.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/post/xineplug_post_planar.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/post/xineplug_post_tvtime.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_dmx_mpeg_ts.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_dmx_qt.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_inp_gnome_vfs.so foundload_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_dmx_games.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_dmx_games.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_dmx_games.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_dmx_games.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_dmx_games.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_dmx_games.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_dmx_games.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_dmx_games.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_dmx_games.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_dmx_games.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_dmx_games.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_decode_w32dll.so foundload_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_decode_w32dll.so foundload_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_decode_a52.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_decode_bitplane.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_ao_out_none.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_dmx_mpeg.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_vo_out_dxr3.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_vo_out_dxr3.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_decode_speex.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_dmx_matroska.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_ao_out_alsa.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_decode_dts.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_dmx_asf.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_decode_real_audio.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_dmx_avi.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_dmx_mpeg_elem.so foundload_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_dmx_sputext.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_decode_real.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_dmx_fli.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_ao_out_arts.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_dmx_flv.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_dmx_audio.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_dmx_audio.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_dmx_audio.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_dmx_audio.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_dmx_audio.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_dmx_audio.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_dmx_audio.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_dmx_audio.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_dmx_audio.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_dmx_audio.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_dmx_audio.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_dmx_audio.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_dmx_audio.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_dmx_audio.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_dmx_audio.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_inp_dvb.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_inp_dvd.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_decode_spucmml.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_decode_dxr3_video.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_dmx_iff.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_decode_mad.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_vo_out_vidix.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_vo_out_vidix.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_decode_mpc.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_dmx_mpeg_pes.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_decode_nsf.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_inp_rtsp.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_decode_dvaudio.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_dmx_mng.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_inp_stdin_fifo.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_decode_rgb.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_dmx_ogg.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_dmx_ogg.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_dmx_nsv.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_decode_spucc.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_inp_mms.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_decode_spudvb.so foundload_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_inp_net.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_decode_spu.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_decode_gsm610.so foundload_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_dmx_pva.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_flac.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_flac.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_inp_file.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_inp_pnm.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_inp_v4l.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_inp_v4l.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_decode_faad.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_inp_pvr.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xiparams.c:OpenConfFile() - Unable to open configuration file "/home/joss/.smb/smb.conf": Aucun fichier ou répertoire de ce type params.c:OpenConfFile() - Unable to open configuration file "/etc/samba/smb.conf": No such file or directory Using netbios name ARRAKIS. Using workgroup WORKGROUP. neplug_inp_rtp.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_inp_smb.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_vo_out_none.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_decode_yuv.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_decode_mpeg2.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_inp_vcd.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_decode_lpcm.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_decode_image.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_ao_out_esd.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_decode_ff.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_decode_ff.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_decode_ff.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_decode_qt.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_decode_qt.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_decode_sputext.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_inp_vcdo.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_vo_out_caca.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_dmx_real.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_dmx_yuv4mpeg2.so foundload_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_ao_out_oss.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_decode_dxr3_spu.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_dmx_slave.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_dmx_image.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_dmx_yuv_frames.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_dmx_rawdv.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_ao_out_file.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_vo_out_aa.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_vo_out_fb.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_inp_http.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_vo_out_xv.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_decode_theora.so foundload_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_vo_out_opengl.so foundload_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_vo_out_sdl.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_inp_cdda.so found load_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_decode_vorbis.so foundload_plugins: plugin /usr/lib/xine/plugins/1.0.1/xineplug_dmx_mpeg_block.so found init class succeeded gnome_vfs init_input_class audio_out: thread created video_out: thread created xine_stream_new xine: found input plugin : file input plugin load_plugins: probing demux 'anx' load_plugins: probing demux 'mpeg-ts' load_plugins: probing demux 'quicktime' load_plugins: probing demux 'wve' load_plugins: probing demux 'idcin' load_plugins: probing demux 'ipmovie' load_plugins: probing demux 'vqa' load_plugins: probing demux 'wc3movie' load_plugins: probing demux 'roq' load_plugins: probing demux 'str' load_plugins: probing demux 'film' load_plugins: probing demux 'smjpeg' load_plugins: probing demux 'fourxm' load_plugins: probing demux 'vmd' load_plugins: probing demux 'matroska' ebml: invalid master element load_plugins: probing demux 'asf' load_plugins: probing demux 'avi' load_plugins: probing demux 'fli' load_plugins: probing demux 'flashvideo' load_plugins: probing demux 'aud' load_plugins: probing demux 'aiff' load_plugins: probing demux 'cdda' load_plugins: probing demux 'flac' load_plugins: probing demux 'nsf' load_plugins: probing demux 'realaudio' load_plugins: probing demux 'snd' load_plugins: probing demux 'voc' load_plugins: probing demux 'vox' load_plugins: probing demux 'wav' load_plugins: probing demux 'mod' TEST mod decode load_plugins: probing demux 'iff' load_plugins: probin (the output really stops here) When reading the file with totem, a similar output is produced, but it isn't truncated. Interesting lines include: xine: found demuxer plugin: MPEG audio demux plugin demux_mpgaudio: ID3V2.3 tag
I can reproduce the problem (it even makes the goom viz crash on my machine...). My guess is that there's either a bug in xine-lib's tag parser, or that the file is badly tagged (ie. claims an ISO8859-1 tag when it's a UTF-8 one).
Apparently there is no such thing as an encoding for ID3 tags, even for ID3v2. most applications (but happily, not all) assume they are encoded in iso-8859-1. It looks like this is the case for totem, whether it's using the xine or gstreamer backend. At least XMMS and Beep use the current locale instead, which is of course much better. The most sane choice would probably be to always assume UTF8, and to fall back to iso-8859-1 if it isn't valid UTF8.
Of course there is. The first byte of the id3 tag frame represents the encoding: http://www.id3.org/id3v2-00.txt " Frames that allow different types of text encoding have a text encoding description byte directly after the frame size. If ISO-8859-1 is used this byte should be $00, if unicode is used it should be $01. " Your string is in UTF-8 (hence the "Ã"), I don't think xine-lib or Totem is interpreting it properly (or it's double-encoded, or something).
Indeed, thanks for the link. It looks like there is no way to encode strings in UTF-8 when following the ID3 standard, as Unicode strings should be encoded in UCS-2. However I don't know of any Linux application that understand UCS-2 encoded tags. Internationalization of ID3 tags looks like a horrible mess at the moment.
Not quite. The link I gave you was for ID3v2. Your MP3 is ID3v2.3. From http://www.id3.org/id3v2.3.0.txt " Frames that allow different types of text encoding have a text encoding description byte directly after the frame size. If ISO-8859-1 is used this byte should be $00, if Unicode is used it should be $01. " And from the gdb log below you can see that xine-lib recognises your MP3 as having ISO-8859-1 text, not Unicode. Please fix the tags in your MP3s to say that they are in UTF-8, when the encoding is such. $ gdb ./metadata-test GNU gdb Red Hat Linux (6.3.0.0-1.98rh) Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-redhat-linux-gnu"...Using host libthread_db library "/lib/libthread_db.so.1". (gdb) b id3.c:523 No source file named id3.c. Make breakpoint pending on future shared library load? (y or [n]) y Breakpoint 1 (id3.c:523) pending. (gdb) run orgasme.mp3 Starting program: /home/hadess/Projects/Cvs/totem/src/metadata-test orgasme.mp3 Reading symbols from shared object read from target memory...done. Loaded system supplied DSO at 0x83f000 [Thread debugging using libthread_db enabled] [New Thread -1208215888 (LWP 2907)] <snip> Breakpoint 2, id3v23_parse_tag (input=0x8a22f90, stream=0x89f9a88, mp3_frame_header=0xbfbe1da4 "ID3\003 ") at id3.c:525 525 _x_meta_info_set_generic(stream, XINE_META_INFO_TITLE, buf + 1, id3_encoding[enc]); (gdb) step _x_meta_info_set_generic (stream=0x89f9a88, info=0, str=0x8a34131 "Répondeur orgasme", enc=0x970002 "ISO-8859-1") at info_helper.c:290
ID3v2.3 doesn't allow UTF-8 strings more than ID3v2.0: "All Unicode strings [UNICODE] use 16-bit unicode 2.0 (ISO/IEC 10646-1:1993, UCS-2)." There's only a handful of software that can write such UCS-2 tags, and anyway totem doesn't understand them.
Sorry, it should have said "Please fix the tags in your MP3s to say that they are in Unicode, when the encoding is such." UCS-2 should work, xine-lib should convert it to UTF-8 for applications to use.
Well, that's what I've tested yesterday. Neither totem-xine nor totem-gstreamer seem to understand UCS-2 encoded tags.
Ah, but it seems ID3v2.4 has now the possibility to encode the tags in UTF-8. I guess I should just wait so that enough software implement ID3v2.4...
Created attachment 60559 [details] [review] xine-lib-dont-convert-utf8-strings.patch I added this work-around in version 1.16 of src/xine-engine/info_helper.c in xine-lib. It fixes your problem (UTF-8 data, but marked as ISO8859-1)