There are several instances where a user may want to enter values that use fractions smaller than a currency's default smallest currency unit (or smallest fraction). A few reasons I can think of:
1) You have a loan that trades in small fractions of a currency (see bug 427948).
2) You want to split a bill in half (because you are only responsible for half of the bill), but the bill total is an odd number of your currency's smallest unit.
3) Your currency definition no longer exists in current releases of GnuCash (see bug 408444).
4) Your currency's fraction traded has changed since GnuCash was last released (or it is otherwise incorrect in the GnuCash source code).

To deal with these situations, a user should be able to change the settings for a currency as needed (or to perhaps create their own currency with a smaller fraction).

In the "Security Editor" (under the "Tools Menu" in GnuCash 2.4.4) there is an option to "Show National Currencies". When this checkbox is enabled, you can select your currency and click "Edit". This brings up a dialog box that says "Currency Information". However, all the fields in this dialog are grayed out, not allowing any changes. Seeing as this dialog already exists, it seems it would be relatively easy to allow the user to edit some (or all) of the fields.

On the back-end this doesn't seem like it should be very difficult to change. I was able to change my XML file manually by adding the line "<cmdty:fraction>1000</cmdty:fraction>", and GnuCash seemed to correctly work with the currency using three decimal places. However, this change was overwritten when I saved the file in GnuCash and reopened it (resulting in some display issues where GnuCash would show $17.680 but the balance would reflect more precision), so this is not a long-term solution). Seems like an XML tag that tells GnuCash *not* to revert the value would be the only back-end change necessary. Of course, I am new to GnuCash, and not a developer, so don't take my word for that.