When trying to validate (e.g. using xmllint --valid) the following 
invalid document, libxml seems to hang in some endless loop eating 
more and more memory and finally crashes with 
realloc failed !nSegmentation fault

(I had .5 G physical plus 1G virtual memory, so I don't think that this is 
an effect of to few memory)

---
<!DOCTYPE x [
<!ELEMENT x (a+ | ((b), (c?, d*)+)+)>
<!ELEMENT a EMPTY>
<!ELEMENT b EMPTY>
<!ELEMENT c EMPTY>
<!ELEMENT d EMPTY>
]>
<x><b/><a/></x>
---

Of course the form of the content model is a bit strange and
can be replaced by the equivalent
<!ELEMENT x (a+ | ((b), (c | d )*)+)>
in which case xmllint perfectly finds, that the document is invalid.

It seems that the realloc failes in valid.c:vstateVPush since this is
the only error message missing the \ before the 'n' (so typos can be
helpful ;-). I didn't manage to track the problem down, though.

As I said, the content model is a bit strange, but it is perfectly
legal.
Actually I came across this problem, when playing around with
the xmlValidGetValidElements function on some TEI-document using
the xteilite.dtd which contains the element declaration
<!ELEMENT publicationStmt (p+ | ((publisher | distributor | authority) , 
                                 (pubPlace?, address?, idno*,
availability?, date?)+)+)>

I checked this with libxml 2.4.17 and 2.4.18.