With libxslt 0.10.0, given xo.xsl:

<?xml version='1.0'?>
<xsl:stylesheet version="1.0" 
  xmlns:xsl="http://www.w3.org/1999/XSL/Transform">

<xsl:output method="xml"
  doctype-public="-//W3C//DTD XHTML 1.1//EN"
  doctype-system="http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd"/>

<xsl:template match="/">
  <o/>
</xsl:template>

</xsl:stylesheet>

the command 

xsltproc xo.xsl xo.xsl

produces

<?xml version="1.0"?>
<!DOCTYPE o PUBLIC "-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<!DOCTYPE o PUBLIC "-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<o/>

(happens with any .xml file as input, not just xo.xsl).

That should be a bug, no?

 Steve