From campbell at mumble.net  Wed Jul 26 21:08:02 2006
From: campbell at mumble.net (Taylor R Campbell)
Date: Wed, 26 Jul 2006 21:08:02 +0000
Subject: [s-xml-rpc-devel] bug: S-XML-RPC sends unencoded strings with
	invalid characters
Message-ID: <20060726210803.7F70598937@pluto.mumble.net>

Well-formed XML is not allowed to include character entities that are
not legal XML characters, according to the definition of character
entity references in section 4.1 of the XML 1.0 specification:

   [66] CharRef ::= '&#' [0-9]+ ';' | '&#x' [0-9a-fA-F]+ ';'

   Well-formedness constraint:  Legal Character
   - Characters referred to using character references MUST match the
   - production for Char.

The production for Char is defined in section 2.2:

  [2] Char ::= #x9 | #xA | #xD | [#x20-#xD7FF] | [#xE000-#xFFFD] |
               [#x10000-#x10FFFF]

Strings with any other characters ought to be base64-encoded.  I just
encountered a problem when lisppaste sent an XML-RPC response with a
paste that contains form feed characters to client with an extremely
strict XML parser.


From scaekenberghe at common-lisp.net  Fri Jul 28 13:18:16 2006
From: scaekenberghe at common-lisp.net (Sven Van Caekenberghe)
Date: Fri, 28 Jul 2006 15:18:16 +0200
Subject: [s-xml-rpc-devel] bug: S-XML-RPC sends unencoded strings with
	invalid characters
In-Reply-To: <20060726210803.7F70598937@pluto.mumble.net>
References: <20060726210803.7F70598937@pluto.mumble.net>
Message-ID: <4A677826-7FDF-4F81-9D66-0E2619C7CA87@common-lisp.net>

Hi Taylor,

Thanks for the feedback!

Could you please be a bit more specific ? It has been a while since I  
have been reading these specs ;-)

The function s-xml:print-string-xml does a bit of escaping and is  
used from s-xml-rpc::encode-xml-rpc-value, what exactly are they  
doing wrong ?
Could you give some concrete example, a CL listener session maybe ?
I know that s-xml-rpc is used by a couple of other projects/people,  
so changing the string encoding must be done carefully.

Regards,

Sven


On 26 Jul 2006, at 23:08, Taylor R Campbell wrote:

> Well-formed XML is not allowed to include character entities that are
> not legal XML characters, according to the definition of character
> entity references in section 4.1 of the XML 1.0 specification:
>
>    [66] CharRef ::= '&#' [0-9]+ ';' | '&#x' [0-9a-fA-F]+ ';'
>
>    Well-formedness constraint:  Legal Character
>    - Characters referred to using character references MUST match the
>    - production for Char.
>
> The production for Char is defined in section 2.2:
>
>   [2] Char ::= #x9 | #xA | #xD | [#x20-#xD7FF] | [#xE000-#xFFFD] |
>                [#x10000-#x10FFFF]
>
> Strings with any other characters ought to be base64-encoded.  I just
> encountered a problem when lisppaste sent an XML-RPC response with a
> paste that contains form feed characters to client with an extremely
> strict XML parser.
> _______________________________________________
> s-xml-rpc-devel site list
> s-xml-rpc-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/s-xml-rpc-devel



From campbell at mumble.net  Fri Jul 28 16:10:17 2006
From: campbell at mumble.net (Taylor R Campbell)
Date: Fri, 28 Jul 2006 16:10:17 +0000
Subject: [s-xml-rpc-devel] bug: S-XML-RPC sends unencoded strings with
	invalid characters
In-Reply-To: <4A677826-7FDF-4F81-9D66-0E2619C7CA87@common-lisp.net>
	(scaekenberghe@common-lisp.net)
Message-ID: <20060728161018.AC5C898072@pluto.mumble.net>

   Date: Fri, 28 Jul 2006 15:18:16 +0200
   From: Sven Van Caekenberghe <scaekenberghe at common-lisp.net>

   Could you please be a bit more specific ? It has been a while since I  
   have been reading these specs ;-)

   The function s-xml:print-string-xml does a bit of escaping and is  
   used from s-xml-rpc::encode-xml-rpc-value, what exactly are they  
   doing wrong ?

S-XML:PRINT-STRING-XML does do certain escaping, but some code points
are invalid even if escaped as entity references.  The particular
problem I had was that lisppaste was sending me the contents of a
paste with an ASCII form feed character, i.e. U+0C, which is not
allowed in well-formed XML, even as an entity reference -- that is,
the XML specification forbids &#xC;.  To get around this, any XML-RPC
message containing such a character (or any character below U+0C that
is not U+09, U+0A, or U+0D) must be base64-encoded first.

   Could you give some concrete example, a CL listener session maybe ?
   I know that s-xml-rpc is used by a couple of other projects/people,  
   so changing the string encoding must be done carefully.

CL-USER> (s-xml:print-xml-string (string (code-char #x0C)))
"&#xC;"
CL-USER> (s-xml-rpc::encode-xml-rpc-value (string (code-char #x0C)) t)
<value><string>&#xC;</string></value>
"</value>"
CL-USER> 

S-XML:PRINT-XML-STRING should signal an error if any such characters
are encountered, and S-XML-RPC::ENCODE-XML-RPC-VALUE should instead
base64-encode the string and generate this output:

<value><base64>DA==</base64></value>


From scaekenberghe at common-lisp.net  Fri Jul 28 18:49:26 2006
From: scaekenberghe at common-lisp.net (Sven Van Caekenberghe)
Date: Fri, 28 Jul 2006 20:49:26 +0200
Subject: [s-xml-rpc-devel] bug: S-XML-RPC sends unencoded strings with
	invalid characters
In-Reply-To: <20060728161018.AC5C898072@pluto.mumble.net>
References: <20060728161018.AC5C898072@pluto.mumble.net>
Message-ID: <EF138543-FC08-4D72-9EF4-1B1AC9DB909B@common-lisp.net>


On 28 Jul 2006, at 18:10, Taylor R Campbell wrote:

> S-XML:PRINT-STRING-XML does do certain escaping, but some code points
> are invalid even if escaped as entity references.  The particular
> problem I had was that lisppaste was sending me the contents of a
> paste with an ASCII form feed character, i.e. U+0C, which is not
> allowed in well-formed XML, even as an entity reference -- that is,
> the XML specification forbids &#xC;.  To get around this, any XML-RPC
> message containing such a character (or any character below U+0C that
> is not U+09, U+0A, or U+0D) must be base64-encoded first.
>
> CL-USER> (s-xml:print-xml-string (string (code-char #x0C)))
> "&#xC;"
> CL-USER> (s-xml-rpc::encode-xml-rpc-value (string (code-char #x0C)) t)
> <value><string>&#xC;</string></value>
> "</value>"
> CL-USER>
>
> S-XML:PRINT-XML-STRING should signal an error if any such characters
> are encountered, and S-XML-RPC::ENCODE-XML-RPC-VALUE should instead
> base64-encode the string and generate this output:
>
> <value><base64>DA==</base64></value>

Where exactly in the XML-RPC spec does it say this ? I never saw  
something like that.
 From my point of view, the XML-RPC spec is softer and more down to  
earth than others.
This is both a strong as well as a weak point. I also find their use  
of XML as being simplified as well.

If what you say is correct (and it looks that way), then I like XML  
even less than before ;-)
This would make any serialization of string overly complex, for any  
language or encoding.
Applying this change would kill some of my code is border cases.
Maybe we could add something like a 'strict' flag to toggle the  
behavior you suggest.

Sven



From campbell at mumble.net  Fri Jul 28 18:59:15 2006
From: campbell at mumble.net (Taylor R Campbell)
Date: Fri, 28 Jul 2006 18:59:15 +0000
Subject: [s-xml-rpc-devel] bug: S-XML-RPC sends unencoded strings with
	invalid characters
In-Reply-To: <EF138543-FC08-4D72-9EF4-1B1AC9DB909B@common-lisp.net>
	(scaekenberghe@common-lisp.net)
Message-ID: <20060728185916.B82A598072@pluto.mumble.net>

   Date: Fri, 28 Jul 2006 20:49:26 +0200
   From: Sven Van Caekenberghe <scaekenberghe at common-lisp.net>

   Where exactly in the XML-RPC spec does it say this ? I never saw  
   something like that.
    From my point of view, the XML-RPC spec is softer and more down to  
   earth than others.
   This is both a strong as well as a weak point. I also find their use  
   of XML as being simplified as well.

It's not the XML-RPC specification that mandates this; it is simply
what must be done to accomodate clauses in the XML specification.
That is, <value><string>&#C;</string></value> is malformed XML, while
<value><base64>DA==</base64></value> is well-formed, and since XML-RPC
is encoded in XML, the only way to encode the value of the Lisp code
(STRING (CODE-CHAR #X0C)) in XML-RPC is to use base64.

   If what you say is correct (and it looks that way), then I like XML  
   even less than before ;-)
   This would make any serialization of string overly complex, for any  
   language or encoding.
   Applying this change would kill some of my code is border cases.
   Maybe we could add something like a 'strict' flag to toggle the  
   behavior you suggest.

I'm no big fan of XML myself, but the XML specification is very
specific about this, and it's much more stringent than its
predecessors about these details.  Much as I dislike XML, I think that
it is at least important to try to comply with it, or at least to
follow the universal internet guideline of being lenient in what is
accepted and conservative in what is produced.  (I had to fix MIT
Scheme's XML-RPC support, by the way -- it, too, was failing to encode
strings properly, although its XML parser refused the &#xC; from
lisppaste.)