Except for the Activities overview & System Settings, it looks like all search boxes that do "live search" display the search icon on the left. For consistency, System Settings should therefore set the search icon as "primary", not "secondary".

----
Below copied (with minor additions) from: https://bugs.launchpad.net/ubuntu/+source/unity/+bug/828567/comments/8

There is an important difference between search fields where you have to press Enter or click a button to search, and search fields -- like in the Dash -- that perform a live search as you type. Where you have to click a button, the search icon is that button. Where the search is live, the search icon exists only to show that the field is a search field.

That explains some of the inconsistency between search fields in Ubuntu software, but not all.

Program ................. Search icon .. Clear button

Search fields that do live search:
Unity Dash .............. left ......... left
Shotwell ................ left ......... right
Banshee ................. left ......... right
Evolution ............... left ......... right
Rhythmbox ............... no ........... right
System Settings ......... right ........ right
Ubuntu Software Center .. left ......... right
Chromium Preferences .... no ........... right
Shell Overview .......... right ........ right
Gedit ................... left ......... no
Evince .................. no ........... no

Search fields that don't do live search:
Chromium address field .. left ......... no
Nautilus ................ right ........ no
Firefox ................. right ........ no
Thunderbird ............. right ........ no

From those examples I can see only two clear conclusions:
(1) Unity's Dash is inconsistent with everything else in having its clear button on the left.
(2) System Settings is inconsistent with everything else in having live search with a search icon on the right.