From edi at agharta.de Wed May 22 15:40:10 2013 From: edi at agharta.de (Edi Weitz) Date: Wed, 22 May 2013 17:40:10 +0200 Subject: A bug in functon parse-content-type. In-Reply-To: References: <1FA61ECA-59A8-4A0B-AE98-3FBE900BA448@flownet.com> Message-ID: If I'm not mistaken, the slash is a "separator" and thus not a token constituent according to RFC 2616 which means "path=/foo" is not legal input for READ-NAME-VALUE-PAIR. On Wed, May 22, 2013 at 5:27 PM, Ron Garret wrote: > Very likely Jingtao's code is calling READ-NAME-VALUE-PAIR without being wrapped in this macro > > But there's still a bug in READ-NAME-VALUE-PAIR: > > ? (WITH-INPUT-FROM-VECTOR (S (MAP '(VECTOR (UNSIGNED-BYTE 8)) 'CHAR-CODE "path=/foo")) > (chunga:with-character-stream-semantics > (CHUNGA:READ-NAME-VALUE-PAIR S))) > ("path" . "") > > On May 22, 2013, at 8:19 AM, Edi Weitz wrote: > >> On Wed, May 22, 2013 at 4:18 PM, Ron Garret wrote: >>> I found a bug in CHUNGA:READ-NAME-VALUE-PAIR. >> >> It's not quite clear to me yet what the bug is supposed to be. >> >> The documentation clearly says that calls to READ-NAME-VALUE-PAIR and >> friends must be wrapped with this macro: >> >> http://weitz.de/chunga/#with-character-stream-semantics >> >> (You might argue that this isn't very user-friendly, but Chunga wasn't >> really intended to be used that way.) > From jingtaozf at gmail.com Thu May 23 00:52:17 2013 From: jingtaozf at gmail.com (Jingtao Xu) Date: Thu, 23 May 2013 08:52:17 +0800 Subject: A bug in functon parse-content-type. In-Reply-To: References: <1FA61ECA-59A8-4A0B-AE98-3FBE900BA448@flownet.com> Message-ID: Hi All, 1. The function `read-name-value-pair' is called by ` parse-content-type' in hunchentoo/util.lisp,not by my codes. 2. the slash is a token constituent in java/ruby implementation,and I think some web client/server treat it as a token constituent too, but I am waiting for the hunchentoot log to give us a live example. With Best Regards, jingtao On Wed, May 22, 2013 at 11:40 PM, Edi Weitz wrote: > If I'm not mistaken, the slash is a "separator" and thus not a token > constituent according to RFC 2616 which means "path=/foo" is not legal > input for READ-NAME-VALUE-PAIR. > > On Wed, May 22, 2013 at 5:27 PM, Ron Garret wrote: >> Very likely Jingtao's code is calling READ-NAME-VALUE-PAIR without being wrapped in this macro >> >> But there's still a bug in READ-NAME-VALUE-PAIR: >> >> ? (WITH-INPUT-FROM-VECTOR (S (MAP '(VECTOR (UNSIGNED-BYTE 8)) 'CHAR-CODE "path=/foo")) >> (chunga:with-character-stream-semantics >> (CHUNGA:READ-NAME-VALUE-PAIR S))) >> ("path" . "") >> >> On May 22, 2013, at 8:19 AM, Edi Weitz wrote: >> >>> On Wed, May 22, 2013 at 4:18 PM, Ron Garret wrote: >>>> I found a bug in CHUNGA:READ-NAME-VALUE-PAIR. >>> >>> It's not quite clear to me yet what the bug is supposed to be. >>> >>> The documentation clearly says that calls to READ-NAME-VALUE-PAIR and >>> friends must be wrapped with this macro: >>> >>> http://weitz.de/chunga/#with-character-stream-semantics >>> >>> (You might argue that this isn't very user-friendly, but Chunga wasn't >>> really intended to be used that way.) >> From edi at agharta.de Wed May 22 15:19:15 2013 From: edi at agharta.de (Edi Weitz) Date: Wed, 22 May 2013 17:19:15 +0200 Subject: A bug in functon parse-content-type. In-Reply-To: <1FA61ECA-59A8-4A0B-AE98-3FBE900BA448@flownet.com> References: <1FA61ECA-59A8-4A0B-AE98-3FBE900BA448@flownet.com> Message-ID: On Wed, May 22, 2013 at 4:18 PM, Ron Garret wrote: > I found a bug in CHUNGA:READ-NAME-VALUE-PAIR. It's not quite clear to me yet what the bug is supposed to be. The documentation clearly says that calls to READ-NAME-VALUE-PAIR and friends must be wrapped with this macro: http://weitz.de/chunga/#with-character-stream-semantics (You might argue that this isn't very user-friendly, but Chunga wasn't really intended to be used that way.) From edi at agharta.de Sun May 5 13:02:31 2013 From: edi at agharta.de (Edi Weitz) Date: Sun, 5 May 2013 15:02:31 +0200 Subject: test In-Reply-To: <83241367758429@web18d.yandex.ru> References: <83241367758429@web18d.yandex.ru> Message-ID: It seems that all cl.net mailing lists still exist but the corresponding web frontends have vanished because the list server was changed. Which means that tons of links like the one you mentioned now point to nowhere. My understanding was that Drew wanted to do something about this, but I haven't heard from him for quite a while. There's a way to subscribe to the list without the need to go through a web browser, but I'm reluctant to update two dozen websites before this is settled. Cheers, Edi. On Sun, May 5, 2013 at 2:53 PM, Anton Vodonosov wrote: > Testing if the list exists, in response to this question on comp.lang.list > > Christoph-Simon Senjak: >> Hi all. >> >> I just went over http://weitz.de/hunchentoot/#mail and the list that is >> linked there is a dead link. I could not find the mailing list anywhere >> else. Does it still exist? >> >> Best regards >> CSS > From ron at flownet.com Wed May 22 06:12:45 2013 From: ron at flownet.com (Ron Garret) Date: Tue, 21 May 2013 23:12:45 -0700 Subject: A bug in functon parse-content-type. In-Reply-To: References: Message-ID: I was able to reproduce the problem -- sort of: ? (WITH-INPUT-FROM-VECTOR (s (map '(VECTOR (UNSIGNED-BYTE 8)) 'char-code "x=/y")) (chunga:read-name-value-pair s)) > Error: Read character #\/, but expected #\=. The reason I say "sort of" is that I encountered this bit of rather extreme weirdness: ? l1 (WITH-INPUT-FROM-VECTOR (S (MAP '(VECTOR (UNSIGNED-BYTE 8)) 'CHAR-CODE "x=/y")) (CHUNGA:READ-NAME-VALUE-PAIR S)) ? l2 (WITH-INPUT-FROM-VECTOR (S (MAP '(VECTOR (UNSIGNED-BYTE 8)) 'CHAR-CODE "x=/y")) (CHUNGA:READ-NAME-VALUE-PAIR S)) ? (equal l1 l2) T ? (eval l1) > Error: Read character #\/, but expected #\=. > While executing: CHUNGA::SIGNAL-UNEXPECTED-CHARS, in process Listener(6). > Type cmd-. to abort, cmd-\ for a list of available restarts. > Type :? for other options. 1 > ? (eval l2) ("x" . "") I'm still scratching my head over that one. rg On May 21, 2013, at 10:15 PM, Jingtao Xu wrote: > Hi Hans, > > As hunchentoot have not logged them to message.log,I could not give > out the content-type header to you. > And this bug don't appear always,so I have no way reproduce this bug > at this time. > > But I have made some patch to chunga and hunchentoot by cloning the repository: > https://github.com/jingtaozf/chunga/commits/master > https://github.com/jingtaozf/hunchentoot/commits/master > > > With Best Regards, > jingtao. > > > On Wed, May 22, 2013 at 1:03 PM, Hans H?bner wrote: >> Jingtao, >> >> thank you for your bug report. Can you please share the actual value of the >> Content-type header that triggered the bug? >> >> Thanks, >> Hans >> >> >> On Wed, May 22, 2013 at 4:28 AM, Jingtao Xu wrote: >>> >>> Hi All, >>> >>> My website use hunchentoot and allow some external website post url >>> request to hunchentoot server(which is necessary),and I found an error >>> log in message.log: >>> >>> --------------------------------------------------- >>> Error when reading POST parameters from body: Corrupted Content-Type >>> header: >>> Read character #\/, but expected #\=. >>> --------------------------------------------------- >>> >>> I found this bug is caused in function parse-content-type when it want >>> to get parameters >>> by calling function read-name-value-pairs then read-name-value-pair,in >>> lisp function read-name-value-pair,there exist an assert which raise >>> this exception: >>> >>> ------------------------------------------ >>> (when (or value-required-p >>> (eql (peek-char* stream nil) #\=)) >>> (assert-char stream #\=) >>> ------------------------------------------ >>> >>> And this will make hunchentoot thread stop the request processing and >>> exit,which is wrong. >>> >>> When I have a deep investigation,I found that this bug should be >>> caused by package chuga >>> when try to read-token,its function token-char-p will call function >>> separatorp and allow character #\/ >>> as a token character,which is not practical well,because both java and >>> ruby and many other launguage don't allow them as a token >>> character,more details you can get in a discussion: >>> >>> http://bugs.python.org/issue2193 >>> >>> and a java implemention of token-char-p is like this: >>> >>> ------------------------------------------------------------- >>> >>> private static final String tspecials = ",; "; >>> >>> private boolean isToken(String value) { >>> int len = value.length(); >>> for (int i = 0; i < len; i++) { >>> char c = value.charAt(i); >>> if (c < 0x20 || c >= 0x7f || tspecials.indexOf(c) != -1) >>> return false; >>> } >>> return true; >>> } >>> ------------------------------------------------------------- >>> >>> Anyway, hunchentoot should not raise an exception in >>> parse-content-type,which make my site could not receive user's >>> request. >>> And to be more compatible with other web client/server, hunchentoot >>> should read token >>> like java/ruby. >>> >>> Hope some patch can be done to package chunga and hunchentoot to fix my >>> issue. >>> >>> With Best Regards, >>> jingtao. >>> >> > From bhyde at pobox.com Sun May 5 13:01:02 2013 From: bhyde at pobox.com (Ben Hyde) Date: Sun, 5 May 2013 09:01:02 -0400 Subject: test In-Reply-To: <83241367758429@web18d.yandex.ru> References: <83241367758429@web18d.yandex.ru> Message-ID: <9B456840-3AF7-46A7-AC1F-C70A837F9FC2@pobox.com> Yes. Meanwhile common-lisp.net is suffering these days. On May 5, 2013, at 8:53 AM, Anton Vodonosov wrote: > Testing if the list exists, in response to this question on comp.lang.list > > Christoph-Simon Senjak: >> Hi all. >> >> I just went over http://weitz.de/hunchentoot/#mail and the list that is >> linked there is a dead link. I could not find the mailing list anywhere >> else. Does it still exist? >> >> Best regards >> CSS > From jingtaozf at gmail.com Wed May 22 02:28:39 2013 From: jingtaozf at gmail.com (Jingtao Xu) Date: Wed, 22 May 2013 10:28:39 +0800 Subject: A bug in functon parse-content-type. Message-ID: Hi All, My website use hunchentoot and allow some external website post url request to hunchentoot server(which is necessary),and I found an error log in message.log: --------------------------------------------------- Error when reading POST parameters from body: Corrupted Content-Type header: Read character #\/, but expected #\=. --------------------------------------------------- I found this bug is caused in function parse-content-type when it want to get parameters by calling function read-name-value-pairs then read-name-value-pair,in lisp function read-name-value-pair,there exist an assert which raise this exception: ------------------------------------------ (when (or value-required-p (eql (peek-char* stream nil) #\=)) (assert-char stream #\=) ------------------------------------------ And this will make hunchentoot thread stop the request processing and exit,which is wrong. When I have a deep investigation,I found that this bug should be caused by package chuga when try to read-token,its function token-char-p will call function separatorp and allow character #\/ as a token character,which is not practical well,because both java and ruby and many other launguage don't allow them as a token character,more details you can get in a discussion: http://bugs.python.org/issue2193 and a java implemention of token-char-p is like this: ------------------------------------------------------------- private static final String tspecials = ",; "; private boolean isToken(String value) { int len = value.length(); for (int i = 0; i < len; i++) { char c = value.charAt(i); if (c < 0x20 || c >= 0x7f || tspecials.indexOf(c) != -1) return false; } return true; } ------------------------------------------------------------- Anyway, hunchentoot should not raise an exception in parse-content-type,which make my site could not receive user's request. And to be more compatible with other web client/server, hunchentoot should read token like java/ruby. Hope some patch can be done to package chunga and hunchentoot to fix my issue. With Best Regards, jingtao. From hans.huebner at gmail.com Wed May 22 05:33:53 2013 From: hans.huebner at gmail.com (=?ISO-8859-1?Q?Hans_H=FCbner?=) Date: Wed, 22 May 2013 07:33:53 +0200 Subject: A bug in functon parse-content-type. In-Reply-To: References: Message-ID: Jingtao, before I make any changes to Chunga, I need to have an understanding what the problem is and how it can be triggered. As you have already invested quite some time in reporting this bug (thanks for that), please go the extra mile and determine what the real cause is and how the bug that you have found is triggered. Thanks, -Hans On Wed, May 22, 2013 at 7:15 AM, Jingtao Xu wrote: > Hi Hans, > > As hunchentoot have not logged them to message.log,I could not give > out the content-type header to you. > And this bug don't appear always,so I have no way reproduce this bug > at this time. > > But I have made some patch to chunga and hunchentoot by cloning the > repository: > https://github.com/jingtaozf/chunga/commits/master > https://github.com/jingtaozf/hunchentoot/commits/master > > > With Best Regards, > jingtao. > > > On Wed, May 22, 2013 at 1:03 PM, Hans H?bner > wrote: > > Jingtao, > > > > thank you for your bug report. Can you please share the actual value of > the > > Content-type header that triggered the bug? > > > > Thanks, > > Hans > > > > > > On Wed, May 22, 2013 at 4:28 AM, Jingtao Xu wrote: > >> > >> Hi All, > >> > >> My website use hunchentoot and allow some external website post url > >> request to hunchentoot server(which is necessary),and I found an error > >> log in message.log: > >> > >> --------------------------------------------------- > >> Error when reading POST parameters from body: Corrupted Content-Type > >> header: > >> Read character #\/, but expected #\=. > >> --------------------------------------------------- > >> > >> I found this bug is caused in function parse-content-type when it want > >> to get parameters > >> by calling function read-name-value-pairs then read-name-value-pair,in > >> lisp function read-name-value-pair,there exist an assert which raise > >> this exception: > >> > >> ------------------------------------------ > >> (when (or value-required-p > >> (eql (peek-char* stream nil) #\=)) > >> (assert-char stream #\=) > >> ------------------------------------------ > >> > >> And this will make hunchentoot thread stop the request processing and > >> exit,which is wrong. > >> > >> When I have a deep investigation,I found that this bug should be > >> caused by package chuga > >> when try to read-token,its function token-char-p will call function > >> separatorp and allow character #\/ > >> as a token character,which is not practical well,because both java and > >> ruby and many other launguage don't allow them as a token > >> character,more details you can get in a discussion: > >> > >> http://bugs.python.org/issue2193 > >> > >> and a java implemention of token-char-p is like this: > >> > >> ------------------------------------------------------------- > >> > >> private static final String tspecials = ",; "; > >> > >> private boolean isToken(String value) { > >> int len = value.length(); > >> for (int i = 0; i < len; i++) { > >> char c = value.charAt(i); > >> if (c < 0x20 || c >= 0x7f || tspecials.indexOf(c) != -1) > >> return false; > >> } > >> return true; > >> } > >> ------------------------------------------------------------- > >> > >> Anyway, hunchentoot should not raise an exception in > >> parse-content-type,which make my site could not receive user's > >> request. > >> And to be more compatible with other web client/server, hunchentoot > >> should read token > >> like java/ruby. > >> > >> Hope some patch can be done to package chunga and hunchentoot to fix my > >> issue. > >> > >> With Best Regards, > >> jingtao. > >> > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hans.huebner at gmail.com Wed May 22 05:03:54 2013 From: hans.huebner at gmail.com (=?ISO-8859-1?Q?Hans_H=FCbner?=) Date: Wed, 22 May 2013 07:03:54 +0200 Subject: A bug in functon parse-content-type. In-Reply-To: References: Message-ID: Jingtao, thank you for your bug report. Can you please share the actual value of the Content-type header that triggered the bug? Thanks, Hans On Wed, May 22, 2013 at 4:28 AM, Jingtao Xu wrote: > Hi All, > > My website use hunchentoot and allow some external website post url > request to hunchentoot server(which is necessary),and I found an error > log in message.log: > > --------------------------------------------------- > Error when reading POST parameters from body: Corrupted Content-Type > header: > Read character #\/, but expected #\=. > --------------------------------------------------- > > I found this bug is caused in function parse-content-type when it want > to get parameters > by calling function read-name-value-pairs then read-name-value-pair,in > lisp function read-name-value-pair,there exist an assert which raise > this exception: > > ------------------------------------------ > (when (or value-required-p > (eql (peek-char* stream nil) #\=)) > (assert-char stream #\=) > ------------------------------------------ > > And this will make hunchentoot thread stop the request processing and > exit,which is wrong. > > When I have a deep investigation,I found that this bug should be > caused by package chuga > when try to read-token,its function token-char-p will call function > separatorp and allow character #\/ > as a token character,which is not practical well,because both java and > ruby and many other launguage don't allow them as a token > character,more details you can get in a discussion: > > http://bugs.python.org/issue2193 > > and a java implemention of token-char-p is like this: > > ------------------------------------------------------------- > > private static final String tspecials = ",; "; > > private boolean isToken(String value) { > int len = value.length(); > for (int i = 0; i < len; i++) { > char c = value.charAt(i); > if (c < 0x20 || c >= 0x7f || tspecials.indexOf(c) != -1) > return false; > } > return true; > } > ------------------------------------------------------------- > > Anyway, hunchentoot should not raise an exception in > parse-content-type,which make my site could not receive user's > request. > And to be more compatible with other web client/server, hunchentoot > should read token > like java/ruby. > > Hope some patch can be done to package chunga and hunchentoot to fix my > issue. > > With Best Regards, > jingtao. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jingtaozf at gmail.com Sat May 25 10:38:17 2013 From: jingtaozf at gmail.com (Jingtao Xu) Date: Sat, 25 May 2013 18:38:17 +0800 Subject: A bug in functon parse-content-type. In-Reply-To: References: <1FA61ECA-59A8-4A0B-AE98-3FBE900BA448@flownet.com> Message-ID: Hi all, I found the content type header which raise the bug in my message.log generated by hunchentoot. It happened when hunchentoot get following content type header: ----------------------------------------------------------------------------------------- application/x-www-form-urlencoded; text/html; charset=UTF-8 ----------------------------------------------------------------------------------------- I noticed that in package drakma's file read.lisp,function 'get-content-type' also assumed "/" as a token separator. I hope package chunga/drakma/hunchentoot could accept such content type header without raising an exception,As Edl said,a new special variable similar to *accept-bogus-eols* or *treat-semicolon-as-continuation* which only assume " ,;" as token separator may be a good idea and will fix my question. Any way, RFC standard is not well fit with the read world. Thanks very much. WIth Best Regards, jingtao. On Thu, May 23, 2013 at 2:01 PM, Edi Weitz wrote: > I'm not the maintainer anymore, but my take is that if some Ruby or > Java client misinterprets the RFC I wouldn't change Hunchentoot's (or > rather Chunga's) default behavior because of that. I'd rather > introduce a new special variable similar to *accept-bogus-eols* or > *treat-semicolon-as-continuation*. > > Just my .02 Euros, > Edi. > > > > On Thu, May 23, 2013 at 2:52 AM, Jingtao Xu wrote: >> Hi All, >> >> 1. The function `read-name-value-pair' is called by ` >> parse-content-type' in hunchentoo/util.lisp,not by my codes. >> 2. the slash is a token constituent in java/ruby implementation,and I >> think some web client/server treat it as a token constituent too, >> but I am waiting for the hunchentoot log to give us a live example. >> >> With Best Regards, >> jingtao >> >> >> On Wed, May 22, 2013 at 11:40 PM, Edi Weitz wrote: >>> If I'm not mistaken, the slash is a "separator" and thus not a token >>> constituent according to RFC 2616 which means "path=/foo" is not legal >>> input for READ-NAME-VALUE-PAIR. >>> >>> On Wed, May 22, 2013 at 5:27 PM, Ron Garret wrote: >>>> Very likely Jingtao's code is calling READ-NAME-VALUE-PAIR without being wrapped in this macro >>>> >>>> But there's still a bug in READ-NAME-VALUE-PAIR: >>>> >>>> ? (WITH-INPUT-FROM-VECTOR (S (MAP '(VECTOR (UNSIGNED-BYTE 8)) 'CHAR-CODE "path=/foo")) >>>> (chunga:with-character-stream-semantics >>>> (CHUNGA:READ-NAME-VALUE-PAIR S))) >>>> ("path" . "") >>>> >>>> On May 22, 2013, at 8:19 AM, Edi Weitz wrote: >>>> >>>>> On Wed, May 22, 2013 at 4:18 PM, Ron Garret wrote: >>>>>> I found a bug in CHUNGA:READ-NAME-VALUE-PAIR. >>>>> >>>>> It's not quite clear to me yet what the bug is supposed to be. >>>>> >>>>> The documentation clearly says that calls to READ-NAME-VALUE-PAIR and >>>>> friends must be wrapped with this macro: >>>>> >>>>> http://weitz.de/chunga/#with-character-stream-semantics >>>>> >>>>> (You might argue that this isn't very user-friendly, but Chunga wasn't >>>>> really intended to be used that way.) >>>> >> From avodonosov at yandex.ru Sun May 5 12:53:49 2013 From: avodonosov at yandex.ru (Anton Vodonosov) Date: Sun, 05 May 2013 16:53:49 +0400 Subject: test Message-ID: <83241367758429@web18d.yandex.ru> Testing if the list exists, in response to this question on comp.lang.list Christoph-Simon Senjak: > Hi all. > > I just went over http://weitz.de/hunchentoot/#mail and the list that is > linked there is a dead link. I could not find the mailing list anywhere > else. Does it still exist? > > Best regards > CSS From ron at flownet.com Wed May 22 15:27:43 2013 From: ron at flownet.com (Ron Garret) Date: Wed, 22 May 2013 08:27:43 -0700 Subject: A bug in functon parse-content-type. In-Reply-To: References: <1FA61ECA-59A8-4A0B-AE98-3FBE900BA448@flownet.com> Message-ID: Very likely Jingtao's code is calling READ-NAME-VALUE-PAIR without being wrapped in this macro But there's still a bug in READ-NAME-VALUE-PAIR: ? (WITH-INPUT-FROM-VECTOR (S (MAP '(VECTOR (UNSIGNED-BYTE 8)) 'CHAR-CODE "path=/foo")) (chunga:with-character-stream-semantics (CHUNGA:READ-NAME-VALUE-PAIR S))) ("path" . "") On May 22, 2013, at 8:19 AM, Edi Weitz wrote: > On Wed, May 22, 2013 at 4:18 PM, Ron Garret wrote: >> I found a bug in CHUNGA:READ-NAME-VALUE-PAIR. > > It's not quite clear to me yet what the bug is supposed to be. > > The documentation clearly says that calls to READ-NAME-VALUE-PAIR and > friends must be wrapped with this macro: > > http://weitz.de/chunga/#with-character-stream-semantics > > (You might argue that this isn't very user-friendly, but Chunga wasn't > really intended to be used that way.) From hans.huebner at gmail.com Wed May 22 07:32:52 2013 From: hans.huebner at gmail.com (=?ISO-8859-1?Q?Hans_H=FCbner?=) Date: Wed, 22 May 2013 09:32:52 +0200 Subject: A bug in functon parse-content-type. In-Reply-To: References: Message-ID: I'm interested in seeing an actual content-type header value that triggers the problem - Raising an uncaught error in response to some user input is certainly undesirable and warrants a fix, but I would like to understand what valid content types are handled incorrectly. On Wed, May 22, 2013 at 8:35 AM, Jingtao Xu wrote: > I have add a log to hunchentoot so I can get the content header when it > happens, > > But I think Ron have reproduced the problem: > > ? (WITH-INPUT-FROM-VECTOR (s (map '(VECTOR (UNSIGNED-BYTE 8)) > 'char-code "x=/y")) > > With Best Regards, > jingtao. > > > On Wed, May 22, 2013 at 1:33 PM, Hans H?bner > wrote: > > Jingtao, > > > > before I make any changes to Chunga, I need to have an understanding what > > the problem is and how it can be triggered. As you have already invested > > quite some time in reporting this bug (thanks for that), please go the > extra > > mile and determine what the real cause is and how the bug that you have > > found is triggered. > > > > Thanks, > > -Hans > > > > > > On Wed, May 22, 2013 at 7:15 AM, Jingtao Xu wrote: > >> > >> Hi Hans, > >> > >> As hunchentoot have not logged them to message.log,I could not give > >> out the content-type header to you. > >> And this bug don't appear always,so I have no way reproduce this bug > >> at this time. > >> > >> But I have made some patch to chunga and hunchentoot by cloning the > >> repository: > >> https://github.com/jingtaozf/chunga/commits/master > >> https://github.com/jingtaozf/hunchentoot/commits/master > >> > >> > >> With Best Regards, > >> jingtao. > >> > >> > >> On Wed, May 22, 2013 at 1:03 PM, Hans H?bner > >> wrote: > >> > Jingtao, > >> > > >> > thank you for your bug report. Can you please share the actual value > of > >> > the > >> > Content-type header that triggered the bug? > >> > > >> > Thanks, > >> > Hans > >> > > >> > > >> > On Wed, May 22, 2013 at 4:28 AM, Jingtao Xu > wrote: > >> >> > >> >> Hi All, > >> >> > >> >> My website use hunchentoot and allow some external website post url > >> >> request to hunchentoot server(which is necessary),and I found an > error > >> >> log in message.log: > >> >> > >> >> --------------------------------------------------- > >> >> Error when reading POST parameters from body: Corrupted Content-Type > >> >> header: > >> >> Read character #\/, but expected #\=. > >> >> --------------------------------------------------- > >> >> > >> >> I found this bug is caused in function parse-content-type when it > want > >> >> to get parameters > >> >> by calling function read-name-value-pairs then > read-name-value-pair,in > >> >> lisp function read-name-value-pair,there exist an assert which raise > >> >> this exception: > >> >> > >> >> ------------------------------------------ > >> >> (when (or value-required-p > >> >> (eql (peek-char* stream nil) #\=)) > >> >> (assert-char stream #\=) > >> >> ------------------------------------------ > >> >> > >> >> And this will make hunchentoot thread stop the request processing and > >> >> exit,which is wrong. > >> >> > >> >> When I have a deep investigation,I found that this bug should be > >> >> caused by package chuga > >> >> when try to read-token,its function token-char-p will call function > >> >> separatorp and allow character #\/ > >> >> as a token character,which is not practical well,because both java > and > >> >> ruby and many other launguage don't allow them as a token > >> >> character,more details you can get in a discussion: > >> >> > >> >> http://bugs.python.org/issue2193 > >> >> > >> >> and a java implemention of token-char-p is like this: > >> >> > >> >> ------------------------------------------------------------- > >> >> > >> >> private static final String tspecials = ",; "; > >> >> > >> >> private boolean isToken(String value) { > >> >> int len = value.length(); > >> >> for (int i = 0; i < len; i++) { > >> >> char c = value.charAt(i); > >> >> if (c < 0x20 || c >= 0x7f || tspecials.indexOf(c) != -1) > >> >> return false; > >> >> } > >> >> return true; > >> >> } > >> >> ------------------------------------------------------------- > >> >> > >> >> Anyway, hunchentoot should not raise an exception in > >> >> parse-content-type,which make my site could not receive user's > >> >> request. > >> >> And to be more compatible with other web client/server, hunchentoot > >> >> should read token > >> >> like java/ruby. > >> >> > >> >> Hope some patch can be done to package chunga and hunchentoot to fix > my > >> >> issue. > >> >> > >> >> With Best Regards, > >> >> jingtao. > >> >> > >> > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jingtaozf at gmail.com Wed May 22 05:15:13 2013 From: jingtaozf at gmail.com (Jingtao Xu) Date: Wed, 22 May 2013 13:15:13 +0800 Subject: A bug in functon parse-content-type. In-Reply-To: References: Message-ID: Hi Hans, As hunchentoot have not logged them to message.log,I could not give out the content-type header to you. And this bug don't appear always,so I have no way reproduce this bug at this time. But I have made some patch to chunga and hunchentoot by cloning the repository: https://github.com/jingtaozf/chunga/commits/master https://github.com/jingtaozf/hunchentoot/commits/master With Best Regards, jingtao. On Wed, May 22, 2013 at 1:03 PM, Hans H?bner wrote: > Jingtao, > > thank you for your bug report. Can you please share the actual value of the > Content-type header that triggered the bug? > > Thanks, > Hans > > > On Wed, May 22, 2013 at 4:28 AM, Jingtao Xu wrote: >> >> Hi All, >> >> My website use hunchentoot and allow some external website post url >> request to hunchentoot server(which is necessary),and I found an error >> log in message.log: >> >> --------------------------------------------------- >> Error when reading POST parameters from body: Corrupted Content-Type >> header: >> Read character #\/, but expected #\=. >> --------------------------------------------------- >> >> I found this bug is caused in function parse-content-type when it want >> to get parameters >> by calling function read-name-value-pairs then read-name-value-pair,in >> lisp function read-name-value-pair,there exist an assert which raise >> this exception: >> >> ------------------------------------------ >> (when (or value-required-p >> (eql (peek-char* stream nil) #\=)) >> (assert-char stream #\=) >> ------------------------------------------ >> >> And this will make hunchentoot thread stop the request processing and >> exit,which is wrong. >> >> When I have a deep investigation,I found that this bug should be >> caused by package chuga >> when try to read-token,its function token-char-p will call function >> separatorp and allow character #\/ >> as a token character,which is not practical well,because both java and >> ruby and many other launguage don't allow them as a token >> character,more details you can get in a discussion: >> >> http://bugs.python.org/issue2193 >> >> and a java implemention of token-char-p is like this: >> >> ------------------------------------------------------------- >> >> private static final String tspecials = ",; "; >> >> private boolean isToken(String value) { >> int len = value.length(); >> for (int i = 0; i < len; i++) { >> char c = value.charAt(i); >> if (c < 0x20 || c >= 0x7f || tspecials.indexOf(c) != -1) >> return false; >> } >> return true; >> } >> ------------------------------------------------------------- >> >> Anyway, hunchentoot should not raise an exception in >> parse-content-type,which make my site could not receive user's >> request. >> And to be more compatible with other web client/server, hunchentoot >> should read token >> like java/ruby. >> >> Hope some patch can be done to package chunga and hunchentoot to fix my >> issue. >> >> With Best Regards, >> jingtao. >> > From hans.huebner at gmail.com Fri May 3 09:52:33 2013 From: hans.huebner at gmail.com (=?ISO-8859-1?Q?Hans_H=FCbner?=) Date: Fri, 3 May 2013 11:52:33 +0200 Subject: Missing client-as-string for Lispworks In-Reply-To: References: Message-ID: Thanks for the patch. I made a new release (1.2.18) containing the fix. -Hans On Fri, May 3, 2013 at 11:01 AM, Raymond Wiker wrote: > I see that there is a recent change that means that client-as-string is > called from create-request- > handler-thread. This breaks for Lispworks users, as client-as-string is > only defined for platforms that use usocket (or strictly speaking, for > platforms which are not lispworks :-) > > The following implementation of client-as-string seems to do what's > required for Lispworks: > > #+:lispworks > (defun client-as-string (socket) > "A helper function which returns the client's address and port as a > string and tries to act robustly in the presence of network problems." > (multiple-value-bind (address port) > (comm:get-socket-peer-address socket) > (when (and address port) > (format nil "~A:~A" > (comm:ip-address-string address) > port)))) > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rwiker at gmail.com Wed May 22 05:43:44 2013 From: rwiker at gmail.com (Raymond Wiker) Date: Wed, 22 May 2013 07:43:44 +0200 Subject: A bug in functon parse-content-type. In-Reply-To: References: Message-ID: Hm... this may or may not be relevant, but I had a problem somewhere in the intersection of drakma, chunga and (possibly) hunchentoot, which turned out to be caused by drakma treating "application/xml" as a binary format. While checking my mailbox to aid my recollection of this, I came across https://github.com/edicl/hunchentoot/issues/1 . For *my* problem, the solution was simply to add (pushnew (cons "application" "xml") drakma:*text-content-types* :test 'equalp) in the startup block of my code. Again, I'm not sure if any of this is relevant :-) On Wed, May 22, 2013 at 7:15 AM, Jingtao Xu wrote: > Hi Hans, > > As hunchentoot have not logged them to message.log,I could not give > out the content-type header to you. > And this bug don't appear always,so I have no way reproduce this bug > at this time. > > But I have made some patch to chunga and hunchentoot by cloning the > repository: > https://github.com/jingtaozf/chunga/commits/master > https://github.com/jingtaozf/hunchentoot/commits/master > > > With Best Regards, > jingtao. > > > On Wed, May 22, 2013 at 1:03 PM, Hans H?bner > wrote: > > Jingtao, > > > > thank you for your bug report. Can you please share the actual value of > the > > Content-type header that triggered the bug? > > > > Thanks, > > Hans > > > > > > On Wed, May 22, 2013 at 4:28 AM, Jingtao Xu wrote: > >> > >> Hi All, > >> > >> My website use hunchentoot and allow some external website post url > >> request to hunchentoot server(which is necessary),and I found an error > >> log in message.log: > >> > >> --------------------------------------------------- > >> Error when reading POST parameters from body: Corrupted Content-Type > >> header: > >> Read character #\/, but expected #\=. > >> --------------------------------------------------- > >> > >> I found this bug is caused in function parse-content-type when it want > >> to get parameters > >> by calling function read-name-value-pairs then read-name-value-pair,in > >> lisp function read-name-value-pair,there exist an assert which raise > >> this exception: > >> > >> ------------------------------------------ > >> (when (or value-required-p > >> (eql (peek-char* stream nil) #\=)) > >> (assert-char stream #\=) > >> ------------------------------------------ > >> > >> And this will make hunchentoot thread stop the request processing and > >> exit,which is wrong. > >> > >> When I have a deep investigation,I found that this bug should be > >> caused by package chuga > >> when try to read-token,its function token-char-p will call function > >> separatorp and allow character #\/ > >> as a token character,which is not practical well,because both java and > >> ruby and many other launguage don't allow them as a token > >> character,more details you can get in a discussion: > >> > >> http://bugs.python.org/issue2193 > >> > >> and a java implemention of token-char-p is like this: > >> > >> ------------------------------------------------------------- > >> > >> private static final String tspecials = ",; "; > >> > >> private boolean isToken(String value) { > >> int len = value.length(); > >> for (int i = 0; i < len; i++) { > >> char c = value.charAt(i); > >> if (c < 0x20 || c >= 0x7f || tspecials.indexOf(c) != -1) > >> return false; > >> } > >> return true; > >> } > >> ------------------------------------------------------------- > >> > >> Anyway, hunchentoot should not raise an exception in > >> parse-content-type,which make my site could not receive user's > >> request. > >> And to be more compatible with other web client/server, hunchentoot > >> should read token > >> like java/ruby. > >> > >> Hope some patch can be done to package chunga and hunchentoot to fix my > >> issue. > >> > >> With Best Regards, > >> jingtao. > >> > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jingtaozf at gmail.com Wed May 29 05:44:41 2013 From: jingtaozf at gmail.com (Jingtaqo Xu) Date: Wed, 29 May 2013 13:44:41 +0800 Subject: A bug in functon parse-content-type. In-Reply-To: References: <1FA61ECA-59A8-4A0B-AE98-3FBE900BA448@flownet.com> Message-ID: Hi Hans, The whole thing I want is a a stable hunchentoot server which will be compatible with other web clients and a stable drakma client which will be be compatible with other web servers,whether the web clients/servers follows http protocols well or not should not be the reason which makes hunchentoot/drakma failed directly. I hope you can understand that if drakma/hunchentoot failed directly, my commercial business will fail too. I must say that the codes from Edi has very high qulities,and I have high respect to Edi and you for that. For the case of this question, I hope chunga/drakma/hunchentoot could accept a special feature or a speical variable to make them accept the content type header which not follows http protocols well,like cl-http does. ----------------------------------------------------------------------------- (parse-mime-content-type-header "application/x-www-form-urlencoded; text/html; charset=UTF-8") ? ?==> (:APPLICATION :X-WWW-FORM-URLENCODED :CHARSET :UTF-8) ----------------------------------------------------------------------------- I think your solution(request-with-bad-content-type) will be a little trivial for me. If you accept my suggestion, I can give you a patch for these three packages(chunga/drakma/hunchentoot). With Best Regards, At Sun, 26 May 2013 08:04:15 +0200, Hans H?bner wrote: > > [1 ] > > [2 ] > Jingtao, > > please refer to?http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.7, it clearly > describes that a media type consists of exactly one type/subtype indicator followed by > optional attribute=value pairs. ?The content type that you have presented is not valid > according to these rules. ? Neither a lax parser like the one in CL-HTTP nor the fact > that a large site sends these bogus headers makes them valid. ?I do not want to include > code in Hunchentoot that tries to interpret such bogus data. > > However, if you cannot get your trading partner to fix their client, I can offer this > solution: > > (defclass request-with-bad-content-type (hunchentoot:request) > ? ()) > > (defmethod hunchentoot:header-in :around ((name (eql :content-type)) (request > request-with-bad-content-type)) > ? (alexandria:when-let (content-type (call-next-method)) > ? ? (ppcre:regex-replace-all "^([^/]+/[^/]+); *[^/]+/[^/;]+" content-type "\\1"))) > > You'll then have to use the :request-class argument to your acceptor instantiation to > make it use the request-with-bad-content-type class. ?You also want to review the regular > expression carefully and maybe profile your application to see whether you need to cache > or otherwise improve performance. > > -Hans > > On Sun, May 26, 2013 at 5:07 AM, Jingtao Xu wrote: > > Hi Hans, > > I don't agree with you to say that this content type header is just bogus. > As the content-type is sent by the largest B2B/B2C site in china, it > must have a reason. > > And if you try cl-http, you can find that cl-http will parse such > content type correctly. > > ----------------------------------------------------------------------------- > (parse-mime-content-type-header "application/x-www-form-urlencoded; > text/html; charset=UTF-8") > ? ?==> (:APPLICATION :X-WWW-FORM-URLENCODED :CHARSET :UTF-8) > ----------------------------------------------------------------------------- > > You can find the definition in cl-http/server/headers.lisp > ----------------------------------------------------------------------------- > (define-header-type :content-type-header (:header) > ? :parse-function parse-mime-content-type-header > ? :print-function print-mime-content-type-header) > ----------------------------------------------------------------------------- > > Even this content-type header is bogus(actually I don't think so), > hunchentoot/drakma should parse > the header without raising an error if one special variable like * > accept-bogus-content-type* is true. > > With Best Regards, > jingtao. > > On Sat, May 25, 2013 at 8:11 PM, Hans H?bner wrote: > > Jingtao, > > > > the content-type header "application/x-www-form-urlencoded; text/html; > > charset=UTF-8" is just bogus. ?I do not want to include code that makes > > Hunchentoot work with clearly broken clients. ?Better error reporting would > > be acceptable, though. > > > > -Hans > > > > > > On Sat, May 25, 2013 at 12:38 PM, Jingtao Xu wrote: > >> > >> Hi all, > >> > >> I found the content type header which raise the bug in my message.log > >> generated by hunchentoot. > >> It happened when hunchentoot get following content type header: > >> > >> > >> > ----------------------------------------------------------------------------------------- > >> application/x-www-form-urlencoded; text/html; charset=UTF-8 > >> > >> > ----------------------------------------------------------------------------------------- > >> > >> I noticed that in package drakma's file read.lisp,function > >> 'get-content-type' > >> also assumed "/" as a token separator. > >> > >> I hope package chunga/drakma/hunchentoot could accept such content type > >> header > >> without raising an exception,As Edl said,a new special variable > >> similar to *accept-bogus-eols* or > >> *treat-semicolon-as-continuation* which only assume " ,;" as token > >> separator may be a good idea and will fix my question. > >> > >> Any way, RFC standard is not well fit with the read world. > >> > >> Thanks very much. > >> > >> WIth Best Regards, > >> jingtao. > >> > >> > >> On Thu, May 23, 2013 at 2:01 PM, Edi Weitz wrote: > >> > I'm not the maintainer anymore, but my take is that if some Ruby or > >> > Java client misinterprets the RFC I wouldn't change Hunchentoot's (or > >> > rather Chunga's) default behavior because of that. ?I'd rather > >> > introduce a new special variable similar to *accept-bogus-eols* or > >> > *treat-semicolon-as-continuation*. > >> > > >> > Just my .02 Euros, > >> > Edi. > >> > > >> > > >> > > >> > On Thu, May 23, 2013 at 2:52 AM, Jingtao Xu wrote: > >> >> Hi All, > >> >> > >> >> 1. The function `read-name-value-pair' is called by ` > >> >> parse-content-type' in hunchentoo/util.lisp,not by my codes. > >> >> 2. the slash is a token constituent in java/ruby implementation,and I > >> >> think some web client/server treat it as a token constituent too, > >> >> ? ? but I am waiting for the hunchentoot log to give us a live example. > >> >> > >> >> With Best Regards, > >> >> jingtao > >> >> > >> >> > >> >> On Wed, May 22, 2013 at 11:40 PM, Edi Weitz wrote: > >> >>> If I'm not mistaken, the slash is a "separator" and thus not a token > >> >>> constituent according to RFC 2616 which means "path=/foo" is not legal > >> >>> input for READ-NAME-VALUE-PAIR. > >> >>> > >> >>> On Wed, May 22, 2013 at 5:27 PM, Ron Garret wrote: > >> >>>> Very likely Jingtao's code is calling READ-NAME-VALUE-PAIR without > >> >>>> being wrapped in this macro > >> >>>> > >> >>>> But there's still a bug in READ-NAME-VALUE-PAIR: > >> >>>> > >> >>>> ? (WITH-INPUT-FROM-VECTOR (S (MAP '(VECTOR (UNSIGNED-BYTE 8)) > >> >>>> 'CHAR-CODE "path=/foo")) > >> >>>> ? (chunga:with-character-stream-semantics > >> >>>> ? ? ? (CHUNGA:READ-NAME-VALUE-PAIR S))) > >> >>>> ("path" . "") > >> >>>> > >> >>>> On May 22, 2013, at 8:19 AM, Edi Weitz wrote: > >> >>>> > >> >>>>> On Wed, May 22, 2013 at 4:18 PM, Ron Garret wrote: > >> >>>>>> I found a bug in CHUNGA:READ-NAME-VALUE-PAIR. > >> >>>>> > >> >>>>> It's not quite clear to me yet what the bug is supposed to be. > >> >>>>> > >> >>>>> The documentation clearly says that calls to READ-NAME-VALUE-PAIR > >> >>>>> and > >> >>>>> friends must be wrapped with this macro: > >> >>>>> > >> >>>>> ?http://weitz.de/chunga/#with-character-stream-semantics > >> >>>>> > >> >>>>> (You might argue that this isn't very user-friendly, but Chunga > >> >>>>> wasn't > >> >>>>> really intended to be used that way.) > >> >>>> > >> >> > > > > > > From jingtaozf at gmail.com Wed May 22 06:35:48 2013 From: jingtaozf at gmail.com (Jingtao Xu) Date: Wed, 22 May 2013 14:35:48 +0800 Subject: A bug in functon parse-content-type. In-Reply-To: References: Message-ID: I have add a log to hunchentoot so I can get the content header when it happens, But I think Ron have reproduced the problem: ? (WITH-INPUT-FROM-VECTOR (s (map '(VECTOR (UNSIGNED-BYTE 8)) 'char-code "x=/y")) With Best Regards, jingtao. On Wed, May 22, 2013 at 1:33 PM, Hans H?bner wrote: > Jingtao, > > before I make any changes to Chunga, I need to have an understanding what > the problem is and how it can be triggered. As you have already invested > quite some time in reporting this bug (thanks for that), please go the extra > mile and determine what the real cause is and how the bug that you have > found is triggered. > > Thanks, > -Hans > > > On Wed, May 22, 2013 at 7:15 AM, Jingtao Xu wrote: >> >> Hi Hans, >> >> As hunchentoot have not logged them to message.log,I could not give >> out the content-type header to you. >> And this bug don't appear always,so I have no way reproduce this bug >> at this time. >> >> But I have made some patch to chunga and hunchentoot by cloning the >> repository: >> https://github.com/jingtaozf/chunga/commits/master >> https://github.com/jingtaozf/hunchentoot/commits/master >> >> >> With Best Regards, >> jingtao. >> >> >> On Wed, May 22, 2013 at 1:03 PM, Hans H?bner >> wrote: >> > Jingtao, >> > >> > thank you for your bug report. Can you please share the actual value of >> > the >> > Content-type header that triggered the bug? >> > >> > Thanks, >> > Hans >> > >> > >> > On Wed, May 22, 2013 at 4:28 AM, Jingtao Xu wrote: >> >> >> >> Hi All, >> >> >> >> My website use hunchentoot and allow some external website post url >> >> request to hunchentoot server(which is necessary),and I found an error >> >> log in message.log: >> >> >> >> --------------------------------------------------- >> >> Error when reading POST parameters from body: Corrupted Content-Type >> >> header: >> >> Read character #\/, but expected #\=. >> >> --------------------------------------------------- >> >> >> >> I found this bug is caused in function parse-content-type when it want >> >> to get parameters >> >> by calling function read-name-value-pairs then read-name-value-pair,in >> >> lisp function read-name-value-pair,there exist an assert which raise >> >> this exception: >> >> >> >> ------------------------------------------ >> >> (when (or value-required-p >> >> (eql (peek-char* stream nil) #\=)) >> >> (assert-char stream #\=) >> >> ------------------------------------------ >> >> >> >> And this will make hunchentoot thread stop the request processing and >> >> exit,which is wrong. >> >> >> >> When I have a deep investigation,I found that this bug should be >> >> caused by package chuga >> >> when try to read-token,its function token-char-p will call function >> >> separatorp and allow character #\/ >> >> as a token character,which is not practical well,because both java and >> >> ruby and many other launguage don't allow them as a token >> >> character,more details you can get in a discussion: >> >> >> >> http://bugs.python.org/issue2193 >> >> >> >> and a java implemention of token-char-p is like this: >> >> >> >> ------------------------------------------------------------- >> >> >> >> private static final String tspecials = ",; "; >> >> >> >> private boolean isToken(String value) { >> >> int len = value.length(); >> >> for (int i = 0; i < len; i++) { >> >> char c = value.charAt(i); >> >> if (c < 0x20 || c >= 0x7f || tspecials.indexOf(c) != -1) >> >> return false; >> >> } >> >> return true; >> >> } >> >> ------------------------------------------------------------- >> >> >> >> Anyway, hunchentoot should not raise an exception in >> >> parse-content-type,which make my site could not receive user's >> >> request. >> >> And to be more compatible with other web client/server, hunchentoot >> >> should read token >> >> like java/ruby. >> >> >> >> Hope some patch can be done to package chunga and hunchentoot to fix my >> >> issue. >> >> >> >> With Best Regards, >> >> jingtao. >> >> >> > > > From edi at agharta.de Thu May 23 06:01:51 2013 From: edi at agharta.de (Edi Weitz) Date: Thu, 23 May 2013 08:01:51 +0200 Subject: A bug in functon parse-content-type. In-Reply-To: References: <1FA61ECA-59A8-4A0B-AE98-3FBE900BA448@flownet.com> Message-ID: I'm not the maintainer anymore, but my take is that if some Ruby or Java client misinterprets the RFC I wouldn't change Hunchentoot's (or rather Chunga's) default behavior because of that. I'd rather introduce a new special variable similar to *accept-bogus-eols* or *treat-semicolon-as-continuation*. Just my .02 Euros, Edi. On Thu, May 23, 2013 at 2:52 AM, Jingtao Xu wrote: > Hi All, > > 1. The function `read-name-value-pair' is called by ` > parse-content-type' in hunchentoo/util.lisp,not by my codes. > 2. the slash is a token constituent in java/ruby implementation,and I > think some web client/server treat it as a token constituent too, > but I am waiting for the hunchentoot log to give us a live example. > > With Best Regards, > jingtao > > > On Wed, May 22, 2013 at 11:40 PM, Edi Weitz wrote: >> If I'm not mistaken, the slash is a "separator" and thus not a token >> constituent according to RFC 2616 which means "path=/foo" is not legal >> input for READ-NAME-VALUE-PAIR. >> >> On Wed, May 22, 2013 at 5:27 PM, Ron Garret wrote: >>> Very likely Jingtao's code is calling READ-NAME-VALUE-PAIR without being wrapped in this macro >>> >>> But there's still a bug in READ-NAME-VALUE-PAIR: >>> >>> ? (WITH-INPUT-FROM-VECTOR (S (MAP '(VECTOR (UNSIGNED-BYTE 8)) 'CHAR-CODE "path=/foo")) >>> (chunga:with-character-stream-semantics >>> (CHUNGA:READ-NAME-VALUE-PAIR S))) >>> ("path" . "") >>> >>> On May 22, 2013, at 8:19 AM, Edi Weitz wrote: >>> >>>> On Wed, May 22, 2013 at 4:18 PM, Ron Garret wrote: >>>>> I found a bug in CHUNGA:READ-NAME-VALUE-PAIR. >>>> >>>> It's not quite clear to me yet what the bug is supposed to be. >>>> >>>> The documentation clearly says that calls to READ-NAME-VALUE-PAIR and >>>> friends must be wrapped with this macro: >>>> >>>> http://weitz.de/chunga/#with-character-stream-semantics >>>> >>>> (You might argue that this isn't very user-friendly, but Chunga wasn't >>>> really intended to be used that way.) >>> > From hans.huebner at gmail.com Sun May 26 06:04:15 2013 From: hans.huebner at gmail.com (=?ISO-8859-1?Q?Hans_H=FCbner?=) Date: Sun, 26 May 2013 08:04:15 +0200 Subject: A bug in functon parse-content-type. In-Reply-To: References: <1FA61ECA-59A8-4A0B-AE98-3FBE900BA448@flownet.com> Message-ID: Jingtao, please refer to http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.7, it clearly describes that a media type consists of exactly one type/subtype indicator followed by optional attribute=value pairs. The content type that you have presented is not valid according to these rules. Neither a lax parser like the one in CL-HTTP nor the fact that a large site sends these bogus headers makes them valid. I do not want to include code in Hunchentoot that tries to interpret such bogus data. However, if you cannot get your trading partner to fix their client, I can offer this solution: (defclass request-with-bad-content-type (hunchentoot:request) ()) (defmethod hunchentoot:header-in :around ((name (eql :content-type)) (request request-with-bad-content-type)) (alexandria:when-let (content-type (call-next-method)) (ppcre:regex-replace-all "^([^/]+/[^/]+); *[^/]+/[^/;]+" content-type "\\1"))) You'll then have to use the :request-class argument to your acceptor instantiation to make it use the request-with-bad-content-type class. You also want to review the regular expression carefully and maybe profile your application to see whether you need to cache or otherwise improve performance. -Hans On Sun, May 26, 2013 at 5:07 AM, Jingtao Xu wrote: > Hi Hans, > > I don't agree with you to say that this content type header is just bogus. > As the content-type is sent by the largest B2B/B2C site in china, it > must have a reason. > > And if you try cl-http, you can find that cl-http will parse such > content type correctly. > > > ----------------------------------------------------------------------------- > (parse-mime-content-type-header "application/x-www-form-urlencoded; > text/html; charset=UTF-8") > ==> (:APPLICATION :X-WWW-FORM-URLENCODED :CHARSET :UTF-8) > > ----------------------------------------------------------------------------- > > You can find the definition in cl-http/server/headers.lisp > > ----------------------------------------------------------------------------- > (define-header-type :content-type-header (:header) > :parse-function parse-mime-content-type-header > :print-function print-mime-content-type-header) > > ----------------------------------------------------------------------------- > > Even this content-type header is bogus(actually I don't think so), > hunchentoot/drakma should parse > the header without raising an error if one special variable like * > accept-bogus-content-type* is true. > > > With Best Regards, > jingtao. > > On Sat, May 25, 2013 at 8:11 PM, Hans H?bner > wrote: > > Jingtao, > > > > the content-type header "application/x-www-form-urlencoded; text/html; > > charset=UTF-8" is just bogus. I do not want to include code that makes > > Hunchentoot work with clearly broken clients. Better error reporting > would > > be acceptable, though. > > > > -Hans > > > > > > On Sat, May 25, 2013 at 12:38 PM, Jingtao Xu > wrote: > >> > >> Hi all, > >> > >> I found the content type header which raise the bug in my message.log > >> generated by hunchentoot. > >> It happened when hunchentoot get following content type header: > >> > >> > >> > ----------------------------------------------------------------------------------------- > >> application/x-www-form-urlencoded; text/html; charset=UTF-8 > >> > >> > ----------------------------------------------------------------------------------------- > >> > >> I noticed that in package drakma's file read.lisp,function > >> 'get-content-type' > >> also assumed "/" as a token separator. > >> > >> I hope package chunga/drakma/hunchentoot could accept such content type > >> header > >> without raising an exception,As Edl said,a new special variable > >> similar to *accept-bogus-eols* or > >> *treat-semicolon-as-continuation* which only assume " ,;" as token > >> separator may be a good idea and will fix my question. > >> > >> Any way, RFC standard is not well fit with the read world. > >> > >> Thanks very much. > >> > >> WIth Best Regards, > >> jingtao. > >> > >> > >> On Thu, May 23, 2013 at 2:01 PM, Edi Weitz wrote: > >> > I'm not the maintainer anymore, but my take is that if some Ruby or > >> > Java client misinterprets the RFC I wouldn't change Hunchentoot's (or > >> > rather Chunga's) default behavior because of that. I'd rather > >> > introduce a new special variable similar to *accept-bogus-eols* or > >> > *treat-semicolon-as-continuation*. > >> > > >> > Just my .02 Euros, > >> > Edi. > >> > > >> > > >> > > >> > On Thu, May 23, 2013 at 2:52 AM, Jingtao Xu > wrote: > >> >> Hi All, > >> >> > >> >> 1. The function `read-name-value-pair' is called by ` > >> >> parse-content-type' in hunchentoo/util.lisp,not by my codes. > >> >> 2. the slash is a token constituent in java/ruby implementation,and I > >> >> think some web client/server treat it as a token constituent too, > >> >> but I am waiting for the hunchentoot log to give us a live > example. > >> >> > >> >> With Best Regards, > >> >> jingtao > >> >> > >> >> > >> >> On Wed, May 22, 2013 at 11:40 PM, Edi Weitz wrote: > >> >>> If I'm not mistaken, the slash is a "separator" and thus not a token > >> >>> constituent according to RFC 2616 which means "path=/foo" is not > legal > >> >>> input for READ-NAME-VALUE-PAIR. > >> >>> > >> >>> On Wed, May 22, 2013 at 5:27 PM, Ron Garret > wrote: > >> >>>> Very likely Jingtao's code is calling READ-NAME-VALUE-PAIR without > >> >>>> being wrapped in this macro > >> >>>> > >> >>>> But there's still a bug in READ-NAME-VALUE-PAIR: > >> >>>> > >> >>>> ? (WITH-INPUT-FROM-VECTOR (S (MAP '(VECTOR (UNSIGNED-BYTE 8)) > >> >>>> 'CHAR-CODE "path=/foo")) > >> >>>> (chunga:with-character-stream-semantics > >> >>>> (CHUNGA:READ-NAME-VALUE-PAIR S))) > >> >>>> ("path" . "") > >> >>>> > >> >>>> On May 22, 2013, at 8:19 AM, Edi Weitz wrote: > >> >>>> > >> >>>>> On Wed, May 22, 2013 at 4:18 PM, Ron Garret > wrote: > >> >>>>>> I found a bug in CHUNGA:READ-NAME-VALUE-PAIR. > >> >>>>> > >> >>>>> It's not quite clear to me yet what the bug is supposed to be. > >> >>>>> > >> >>>>> The documentation clearly says that calls to READ-NAME-VALUE-PAIR > >> >>>>> and > >> >>>>> friends must be wrapped with this macro: > >> >>>>> > >> >>>>> http://weitz.de/chunga/#with-character-stream-semantics > >> >>>>> > >> >>>>> (You might argue that this isn't very user-friendly, but Chunga > >> >>>>> wasn't > >> >>>>> really intended to be used that way.) > >> >>>> > >> >> > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jingtaozf at gmail.com Sun May 26 03:07:47 2013 From: jingtaozf at gmail.com (Jingtao Xu) Date: Sun, 26 May 2013 11:07:47 +0800 Subject: A bug in functon parse-content-type. In-Reply-To: References: <1FA61ECA-59A8-4A0B-AE98-3FBE900BA448@flownet.com> Message-ID: Hi Hans, I don't agree with you to say that this content type header is just bogus. As the content-type is sent by the largest B2B/B2C site in china, it must have a reason. And if you try cl-http, you can find that cl-http will parse such content type correctly. ----------------------------------------------------------------------------- (parse-mime-content-type-header "application/x-www-form-urlencoded; text/html; charset=UTF-8") ==> (:APPLICATION :X-WWW-FORM-URLENCODED :CHARSET :UTF-8) ----------------------------------------------------------------------------- You can find the definition in cl-http/server/headers.lisp ----------------------------------------------------------------------------- (define-header-type :content-type-header (:header) :parse-function parse-mime-content-type-header :print-function print-mime-content-type-header) ----------------------------------------------------------------------------- Even this content-type header is bogus(actually I don't think so), hunchentoot/drakma should parse the header without raising an error if one special variable like * accept-bogus-content-type* is true. With Best Regards, jingtao. On Sat, May 25, 2013 at 8:11 PM, Hans H?bner wrote: > Jingtao, > > the content-type header "application/x-www-form-urlencoded; text/html; > charset=UTF-8" is just bogus. I do not want to include code that makes > Hunchentoot work with clearly broken clients. Better error reporting would > be acceptable, though. > > -Hans > > > On Sat, May 25, 2013 at 12:38 PM, Jingtao Xu wrote: >> >> Hi all, >> >> I found the content type header which raise the bug in my message.log >> generated by hunchentoot. >> It happened when hunchentoot get following content type header: >> >> >> ----------------------------------------------------------------------------------------- >> application/x-www-form-urlencoded; text/html; charset=UTF-8 >> >> ----------------------------------------------------------------------------------------- >> >> I noticed that in package drakma's file read.lisp,function >> 'get-content-type' >> also assumed "/" as a token separator. >> >> I hope package chunga/drakma/hunchentoot could accept such content type >> header >> without raising an exception,As Edl said,a new special variable >> similar to *accept-bogus-eols* or >> *treat-semicolon-as-continuation* which only assume " ,;" as token >> separator may be a good idea and will fix my question. >> >> Any way, RFC standard is not well fit with the read world. >> >> Thanks very much. >> >> WIth Best Regards, >> jingtao. >> >> >> On Thu, May 23, 2013 at 2:01 PM, Edi Weitz wrote: >> > I'm not the maintainer anymore, but my take is that if some Ruby or >> > Java client misinterprets the RFC I wouldn't change Hunchentoot's (or >> > rather Chunga's) default behavior because of that. I'd rather >> > introduce a new special variable similar to *accept-bogus-eols* or >> > *treat-semicolon-as-continuation*. >> > >> > Just my .02 Euros, >> > Edi. >> > >> > >> > >> > On Thu, May 23, 2013 at 2:52 AM, Jingtao Xu wrote: >> >> Hi All, >> >> >> >> 1. The function `read-name-value-pair' is called by ` >> >> parse-content-type' in hunchentoo/util.lisp,not by my codes. >> >> 2. the slash is a token constituent in java/ruby implementation,and I >> >> think some web client/server treat it as a token constituent too, >> >> but I am waiting for the hunchentoot log to give us a live example. >> >> >> >> With Best Regards, >> >> jingtao >> >> >> >> >> >> On Wed, May 22, 2013 at 11:40 PM, Edi Weitz wrote: >> >>> If I'm not mistaken, the slash is a "separator" and thus not a token >> >>> constituent according to RFC 2616 which means "path=/foo" is not legal >> >>> input for READ-NAME-VALUE-PAIR. >> >>> >> >>> On Wed, May 22, 2013 at 5:27 PM, Ron Garret wrote: >> >>>> Very likely Jingtao's code is calling READ-NAME-VALUE-PAIR without >> >>>> being wrapped in this macro >> >>>> >> >>>> But there's still a bug in READ-NAME-VALUE-PAIR: >> >>>> >> >>>> ? (WITH-INPUT-FROM-VECTOR (S (MAP '(VECTOR (UNSIGNED-BYTE 8)) >> >>>> 'CHAR-CODE "path=/foo")) >> >>>> (chunga:with-character-stream-semantics >> >>>> (CHUNGA:READ-NAME-VALUE-PAIR S))) >> >>>> ("path" . "") >> >>>> >> >>>> On May 22, 2013, at 8:19 AM, Edi Weitz wrote: >> >>>> >> >>>>> On Wed, May 22, 2013 at 4:18 PM, Ron Garret wrote: >> >>>>>> I found a bug in CHUNGA:READ-NAME-VALUE-PAIR. >> >>>>> >> >>>>> It's not quite clear to me yet what the bug is supposed to be. >> >>>>> >> >>>>> The documentation clearly says that calls to READ-NAME-VALUE-PAIR >> >>>>> and >> >>>>> friends must be wrapped with this macro: >> >>>>> >> >>>>> http://weitz.de/chunga/#with-character-stream-semantics >> >>>>> >> >>>>> (You might argue that this isn't very user-friendly, but Chunga >> >>>>> wasn't >> >>>>> really intended to be used that way.) >> >>>> >> >> > > From hans.huebner at gmail.com Sat May 25 12:11:09 2013 From: hans.huebner at gmail.com (=?ISO-8859-1?Q?Hans_H=FCbner?=) Date: Sat, 25 May 2013 14:11:09 +0200 Subject: A bug in functon parse-content-type. In-Reply-To: References: <1FA61ECA-59A8-4A0B-AE98-3FBE900BA448@flownet.com> Message-ID: Jingtao, the content-type header "application/x-www-form-urlencoded; text/html; charset=UTF-8" is just bogus. I do not want to include code that makes Hunchentoot work with clearly broken clients. Better error reporting would be acceptable, though. -Hans On Sat, May 25, 2013 at 12:38 PM, Jingtao Xu wrote: > Hi all, > > I found the content type header which raise the bug in my message.log > generated by hunchentoot. > It happened when hunchentoot get following content type header: > > > ----------------------------------------------------------------------------------------- > application/x-www-form-urlencoded; text/html; charset=UTF-8 > > ----------------------------------------------------------------------------------------- > > I noticed that in package drakma's file read.lisp,function > 'get-content-type' > also assumed "/" as a token separator. > > I hope package chunga/drakma/hunchentoot could accept such content type > header > without raising an exception,As Edl said,a new special variable > similar to *accept-bogus-eols* or > *treat-semicolon-as-continuation* which only assume " ,;" as token > separator may be a good idea and will fix my question. > > Any way, RFC standard is not well fit with the read world. > > Thanks very much. > > WIth Best Regards, > jingtao. > > > On Thu, May 23, 2013 at 2:01 PM, Edi Weitz wrote: > > I'm not the maintainer anymore, but my take is that if some Ruby or > > Java client misinterprets the RFC I wouldn't change Hunchentoot's (or > > rather Chunga's) default behavior because of that. I'd rather > > introduce a new special variable similar to *accept-bogus-eols* or > > *treat-semicolon-as-continuation*. > > > > Just my .02 Euros, > > Edi. > > > > > > > > On Thu, May 23, 2013 at 2:52 AM, Jingtao Xu wrote: > >> Hi All, > >> > >> 1. The function `read-name-value-pair' is called by ` > >> parse-content-type' in hunchentoo/util.lisp,not by my codes. > >> 2. the slash is a token constituent in java/ruby implementation,and I > >> think some web client/server treat it as a token constituent too, > >> but I am waiting for the hunchentoot log to give us a live example. > >> > >> With Best Regards, > >> jingtao > >> > >> > >> On Wed, May 22, 2013 at 11:40 PM, Edi Weitz wrote: > >>> If I'm not mistaken, the slash is a "separator" and thus not a token > >>> constituent according to RFC 2616 which means "path=/foo" is not legal > >>> input for READ-NAME-VALUE-PAIR. > >>> > >>> On Wed, May 22, 2013 at 5:27 PM, Ron Garret wrote: > >>>> Very likely Jingtao's code is calling READ-NAME-VALUE-PAIR without > being wrapped in this macro > >>>> > >>>> But there's still a bug in READ-NAME-VALUE-PAIR: > >>>> > >>>> ? (WITH-INPUT-FROM-VECTOR (S (MAP '(VECTOR (UNSIGNED-BYTE 8)) > 'CHAR-CODE "path=/foo")) > >>>> (chunga:with-character-stream-semantics > >>>> (CHUNGA:READ-NAME-VALUE-PAIR S))) > >>>> ("path" . "") > >>>> > >>>> On May 22, 2013, at 8:19 AM, Edi Weitz wrote: > >>>> > >>>>> On Wed, May 22, 2013 at 4:18 PM, Ron Garret wrote: > >>>>>> I found a bug in CHUNGA:READ-NAME-VALUE-PAIR. > >>>>> > >>>>> It's not quite clear to me yet what the bug is supposed to be. > >>>>> > >>>>> The documentation clearly says that calls to READ-NAME-VALUE-PAIR and > >>>>> friends must be wrapped with this macro: > >>>>> > >>>>> http://weitz.de/chunga/#with-character-stream-semantics > >>>>> > >>>>> (You might argue that this isn't very user-friendly, but Chunga > wasn't > >>>>> really intended to be used that way.) > >>>> > >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rwiker at gmail.com Fri May 3 09:01:57 2013 From: rwiker at gmail.com (Raymond Wiker) Date: Fri, 3 May 2013 11:01:57 +0200 Subject: Missing client-as-string for Lispworks Message-ID: I see that there is a recent change that means that client-as-string is called from create-request- handler-thread. This breaks for Lispworks users, as client-as-string is only defined for platforms that use usocket (or strictly speaking, for platforms which are not lispworks :-) The following implementation of client-as-string seems to do what's required for Lispworks: #+:lispworks (defun client-as-string (socket) "A helper function which returns the client's address and port as a string and tries to act robustly in the presence of network problems." (multiple-value-bind (address port) (comm:get-socket-peer-address socket) (when (and address port) (format nil "~A:~A" (comm:ip-address-string address) port)))) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ron at flownet.com Wed May 22 06:22:42 2013 From: ron at flownet.com (Ron Garret) Date: Tue, 21 May 2013 23:22:42 -0700 Subject: A bug in functon parse-content-type. In-Reply-To: References: Message-ID: <7E9CD889-6C1E-4542-84A1-1F1B09342612@flownet.com> Found it: ? l1 (WITH-INPUT-FROM-VECTOR (S (MAP '(VECTOR (UNSIGNED-BYTE 8)) 'CHAR-CODE "x=/y")) (CHUNGA:READ-NAME-VALUE-PAIR S)) ? (eval l1) ("x" . "") ? (eval l1) > Error: Read character #\/, but expected #\=. > While executing: CHUNGA::SIGNAL-UNEXPECTED-CHARS, in process Listener(6). > Type cmd-. to abort, cmd-\ for a list of available restarts. > Type :? for other options. 1 > ? chunga::*char-buffer* NIL ? (eval l1) ("x" . "") ? chunga::*char-buffer* #\/ ? (eval l1) > Error: Read character #\/, but expected #\=. On May 21, 2013, at 11:12 PM, Ron Garret wrote: > I was able to reproduce the problem -- sort of: > > ? (WITH-INPUT-FROM-VECTOR (s (map '(VECTOR (UNSIGNED-BYTE 8)) 'char-code "x=/y")) > (chunga:read-name-value-pair s)) >> Error: Read character #\/, but expected #\=. > > The reason I say "sort of" is that I encountered this bit of rather extreme weirdness: > > ? l1 > (WITH-INPUT-FROM-VECTOR (S (MAP '(VECTOR (UNSIGNED-BYTE 8)) 'CHAR-CODE "x=/y")) (CHUNGA:READ-NAME-VALUE-PAIR S)) > ? l2 > (WITH-INPUT-FROM-VECTOR (S (MAP '(VECTOR (UNSIGNED-BYTE 8)) 'CHAR-CODE "x=/y")) (CHUNGA:READ-NAME-VALUE-PAIR S)) > ? (equal l1 l2) > T > ? (eval l1) >> Error: Read character #\/, but expected #\=. >> While executing: CHUNGA::SIGNAL-UNEXPECTED-CHARS, in process Listener(6). >> Type cmd-. to abort, cmd-\ for a list of available restarts. >> Type :? for other options. > 1 > > ? (eval l2) > ("x" . "") > > I'm still scratching my head over that one. > > rg > > On May 21, 2013, at 10:15 PM, Jingtao Xu wrote: > >> Hi Hans, >> >> As hunchentoot have not logged them to message.log,I could not give >> out the content-type header to you. >> And this bug don't appear always,so I have no way reproduce this bug >> at this time. >> >> But I have made some patch to chunga and hunchentoot by cloning the repository: >> https://github.com/jingtaozf/chunga/commits/master >> https://github.com/jingtaozf/hunchentoot/commits/master >> >> >> With Best Regards, >> jingtao. >> >> >> On Wed, May 22, 2013 at 1:03 PM, Hans H?bner wrote: >>> Jingtao, >>> >>> thank you for your bug report. Can you please share the actual value of the >>> Content-type header that triggered the bug? >>> >>> Thanks, >>> Hans >>> >>> >>> On Wed, May 22, 2013 at 4:28 AM, Jingtao Xu wrote: >>>> >>>> Hi All, >>>> >>>> My website use hunchentoot and allow some external website post url >>>> request to hunchentoot server(which is necessary),and I found an error >>>> log in message.log: >>>> >>>> --------------------------------------------------- >>>> Error when reading POST parameters from body: Corrupted Content-Type >>>> header: >>>> Read character #\/, but expected #\=. >>>> --------------------------------------------------- >>>> >>>> I found this bug is caused in function parse-content-type when it want >>>> to get parameters >>>> by calling function read-name-value-pairs then read-name-value-pair,in >>>> lisp function read-name-value-pair,there exist an assert which raise >>>> this exception: >>>> >>>> ------------------------------------------ >>>> (when (or value-required-p >>>> (eql (peek-char* stream nil) #\=)) >>>> (assert-char stream #\=) >>>> ------------------------------------------ >>>> >>>> And this will make hunchentoot thread stop the request processing and >>>> exit,which is wrong. >>>> >>>> When I have a deep investigation,I found that this bug should be >>>> caused by package chuga >>>> when try to read-token,its function token-char-p will call function >>>> separatorp and allow character #\/ >>>> as a token character,which is not practical well,because both java and >>>> ruby and many other launguage don't allow them as a token >>>> character,more details you can get in a discussion: >>>> >>>> http://bugs.python.org/issue2193 >>>> >>>> and a java implemention of token-char-p is like this: >>>> >>>> ------------------------------------------------------------- >>>> >>>> private static final String tspecials = ",; "; >>>> >>>> private boolean isToken(String value) { >>>> int len = value.length(); >>>> for (int i = 0; i < len; i++) { >>>> char c = value.charAt(i); >>>> if (c < 0x20 || c >= 0x7f || tspecials.indexOf(c) != -1) >>>> return false; >>>> } >>>> return true; >>>> } >>>> ------------------------------------------------------------- >>>> >>>> Anyway, hunchentoot should not raise an exception in >>>> parse-content-type,which make my site could not receive user's >>>> request. >>>> And to be more compatible with other web client/server, hunchentoot >>>> should read token >>>> like java/ruby. >>>> >>>> Hope some patch can be done to package chunga and hunchentoot to fix my >>>> issue. >>>> >>>> With Best Regards, >>>> jingtao. >>>> >>> >> > From ron at flownet.com Wed May 22 14:18:57 2013 From: ron at flownet.com (Ron Garret) Date: Wed, 22 May 2013 07:18:57 -0700 Subject: A bug in functon parse-content-type. In-Reply-To: References: Message-ID: <1FA61ECA-59A8-4A0B-AE98-3FBE900BA448@flownet.com> On May 21, 2013, at 11:35 PM, Jingtao Xu wrote: > But I think Ron have reproduced the problem: Well, sort of. I found a bug in CHUNGA:READ-NAME-VALUE-PAIR. But upon further investigation it appears that function is not called by Hunchentoot as far as I can tell. Hunchentoot uses RFC2388:PARSE-HEADER to parse headers, and that function doesn't invoke CHUNGA:READ-NAME-VALUE-PAIR. My guess at the moment is that CHUNGA:READ-NAME-VALUE-PAIR is being invoked by Jingtao's code, not Hunchentoot. Jingtao: can you provide a backtrace from the point of error? rg