GNOME Bugzilla – Bug 766214
gst-launch: Does not support non-UTF8 shell
Last modified: 2018-11-03 12:34:35 UTC
Writing stream to file with local characters in filename fails on windows to open file with that name. Same filename works fine on linux. example of problem: gst-launch-1.0 videotestsrc ! filesink location=tést.raw fails with WARNING: erroneous pipeline: no element "filesink" works fine: gst-launch-1.0 videotestsrc ! filesink location=test.raw Problem is in function gst_fopen in file gstfilesink.c, this function expect its argument filename to be in UTF8, but in fact filename is in ASCII and character é is represented as single BYTE 0xE9. Converting filename to UTF8 before calling g_utf8_to_utf16 solves the problem. gchar *ufilename = g_locale_to_utf8(filename, -1, NULL, NULL, NULL); wchar_t *wfilename = g_utf8_to_utf16 (ufilename, -1, NULL, NULL, NULL);
You need to provide all filenames (and all strings in general) to GStreamer (and GLib) as UTF8.
(In reply to Sebastian Dröge (slomo) from comment #1) > You need to provide all filenames (and all strings in general) to GStreamer > (and GLib) as UTF8. I think you missed the point. Adding g_local_to_utf8() in gst-launch-1.0 seems to make a lot of sense to me. We can't assume the shell will run in UTF-8. Normally the GOption stuff will deal with that, but we don't ask it to extract the remaining options, instead we get it ourself (for legacy reason).
Hmm, though I can't reproduce this issue on Linux, I wonder why ... export LANG=fr.IS8851-2 gst-launch-1.0 fakesrc ! filesink location=allé Need some research.
(In reply to Nicolas Dufresne (stormer) from comment #3) > Hmm, though I can't reproduce this issue on Linux, I wonder why ... > > export LANG=fr.IS8851-2 > gst-launch-1.0 fakesrc ! filesink location=allé > > Need some research. Linux does not convert strings in function gst_fopen. This issue affects only Windows. Windows code converts input string to UTF16 to use _wfopen. Linux code uses fopen.
Also see bug #350034 for background
(In reply to Tim-Philipp Müller from comment #5) > Also see bug #350034 for background Looks identical, shall we mark this one as duplicate ?
If the problem is the argument parsing it's probably not the exact same thing, or is it?
I don't know, by reflex I was expecting this bug to be the the entire pipeline passed to gst_parse_launch() to not be UTF-8 (which would be a bug). That still feels possible, since we don't properly use GOption to get that string. Later on, I would expect that UTF-8 to be converted by glib to UTF-16 for opening the file. Does that make sense ? Am I missing something ?
Yes, I think we should probably use G_OPTION_REMAINING to collect the remaining arguments?
On second thought, it seems like we can't really win. Because on *nix filenames are byte arrays and need to be passed through as-is (must not do any conversion), while on windows they are UTF-8 and we need to convert. But also, that only applies to filenames and not any of the other arguments (text='Hällo') so it's pretty difficult to handle this right across the board, because we don't know which arguments are filenames and which not.
IMHO all arguments should be converted to UTF8 before we get them. All strings we set on elements should be UTF8 and then converted to whatever makes sense in the context and on the system.
That means location=foo will not work correctly on *nix systems with gst-launch any more (regression), and possibly we're breaking the interface of filesrc's location property too. Don't know if that's a good idea. We could do: 1) not care ("gst-launch-1.0 is a debugging tool") 2) convert all args to UTF-8 on Windows only, which would fix this problem and leave how things work on *nix (including things that don't work right on *nix if terminal is not UTF-8) 3) convert args to UTF-8 on Windows and Linux, but special case filename-type properties we know about from a whitelist such as location=xyz and leave those unconverted on *nix and convert on Windows, which would probably make things work right thing in 99% of the cases.
Passing non-UTF8 through G_TYPE_STRING seems like calling for problems :) Wasn't there some GLib UTF-8-to-filename-encoding function?
-- GitLab Migration Automatic Message -- This bug has been migrated to freedesktop.org's GitLab instance and has been closed from further activity. You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.freedesktop.org/gstreamer/gstreamer/issues/171.