GNOME Bugzilla – Bug 729195
videotestsrc: undefined behaviour in left-shift
Last modified: 2014-05-01 20:21:20 UTC
In line 1137 of videotestsrc.c there is a left-shift by 24 applied to a guint8. The behaviour is undefined.
Created attachment 275426 [details] [review]
fix by typecasting value before left-shifting
The two options were changing the type to guint32 in vts_color_struct or casting to guint32 when necessary. This struct has more uses and could potentially be hundreds of instances. So type casting is less intrusive.
The problem, as slomo reminded me, is that a guint8 gets promoted to an int (gint32). Which means there could be sign bit flipping happening without the type casting.
Pushed the patch: