From travis at crosswirecorp.com Thu Nov 3 17:01:57 2005 From: travis at crosswirecorp.com (Travis Cross) Date: Thu, 03 Nov 2005 12:01:57 -0500 Subject: [tbnl-devel] Jpg/png upload/download issues while using cvs SBCL and "stand-alone" TBNL In-Reply-To: References: Message-ID: <436A4285.2000302@crosswirecorp.com> Ignas Mikalajunas wrote: >>Thanks for the info. Does that mean one has to add this to *FEATURES* >>before building SBCL or just before using TBNL? > > Just before using tbnl. My experience is that :tbnl-bivalent-streams must be added to *FEATURES* before TBNL is _built_. The source code seems to support this notion as the #:+/- checks will occur at compile time. It seems that :tbnl-bivalent-streams could be added to features automatically for SBCL by adding :sbcl to the code block on line 309 of specials.lisp: --- specials.lisp.orig 2005-11-03 11:57:39.000000000 -0500 +++ specials.lisp 2005-11-03 11:58:05.000000000 -0500 @@ -309,7 +309,8 @@ (eval-when (:compile-toplevel :load-toplevel :execute) #+(or (and :allegro :allegro-version>= (version>= 6 0)) :lispworks4.3 - :lispworks4.4) + :lispworks4.4 + :sbcl) (pushnew :tbnl-bivalent-streams *features*)) #+:sbcl I would not worry tremendously about breaking older versions of SBCL because 1) TBNL is already broken on older versions of SBCL and 2) most SBCL users track pretty close to the latest release. Cheers, -- Travis From travis at crosswirecorp.com Thu Nov 3 17:13:41 2005 From: travis at crosswirecorp.com (Travis Cross) Date: Thu, 03 Nov 2005 12:13:41 -0500 Subject: [tbnl-devel] Fixing binary uploads in SBCL / TBNL / mod_lisp2 Message-ID: <436A4545.6090201@crosswirecorp.com> Running SBCL 0.9.6 with TBNL 0.8.4 and Apache 2.0.54 with mod_lisp2, the image download examples work as expected, but uploading any binary file results in a 500 Internal Server Error being throw by Apache. No error is reported in Lisp, but the Apache log file error_log shows the following entry: [error] (70014)End of file found: error reading from Lisp Setting apache to log in debug mode doesn't provide any additional insight, but some testing and modification of the mod_lisp2.c code showed that the error is starting at line 588 where mod_lisp attempts to read the first of the headers. This leads back to read_lisp_line(), then deeper to fill_input_buffer(), where the failure seems to originate at the call to: RELAY_ERROR (((length = (sizeof (buffer->data))), (apr_recv (socket, (buffer->data), (&length))))); With *DEBUG-MODE* set to t, *COMMAND* contains seemingly plausible headers (shown below). I am not familiar enough with the internals to have much of an idea of what is causing the breakdown, though the C code seems to suggest that lisp is sending a different amount of data than is promised. I'm happy to dig into either the TBNL, KMRCL, or mod_lisp2.c code and come up with a patch for this, but it would be tremendously helpful if someone has a good idea of where to start and what the problem might be. Cheers, -- Travis *COMMAND* after failed file upload: (("content-stream" . #) ("server-protocol" . "HTTP/1.1") ("method" . "POST") ("url" . "/tbnl/test/upload.html") ("content-type" . "multipart/form-data; boundary=---------------------------130786897319483305451172113725") ("content-length" . "281039") ("server-ip-addr" . "127.0.0.1") ("server-ip-port" . "80") ("remote-ip-addr" . "127.0.0.1") ("script-filename" . "/usr/htdocs") ("remote-ip-port" . "48073") ("server-id" . "tbnl") ("server-baseversion" . "Apache/2.0.54") ("modlisp-version" . "1.2") ("modlisp-major-version" . "2") ("Host" . "localhost") ("User-Agent" . "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.10) Gecko/20050829 Firefox/1.0.6") ("Accept" . "text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5") ("Accept-Language" . "en-us,en;q=0.5") ("Accept-Encoding" . "gzip,deflate") ("Accept-Charset" . "ISO-8859-1,utf-8;q=0.7,*;q=0.7") ("Keep-Alive" . "300") ("Connection" . "keep-alive") ("Referer" . "http://localhost/tbnl/test/upload.html") ("Cookie" . "pumpkin=barking") ("Authorization" . "Basic bmFub29rOmlnbG9v") ("Content-Type" . "multipart/form-data; boundary=---------------------------130786897319483305451172113725") ("Content-Length" . "281039")) From edi at agharta.de Thu Nov 3 23:53:35 2005 From: edi at agharta.de (Edi Weitz) Date: Fri, 04 Nov 2005 00:53:35 +0100 Subject: [tbnl-devel] Jpg/png upload/download issues while using cvs SBCL and "stand-alone" TBNL In-Reply-To: <436A4285.2000302@crosswirecorp.com> (Travis Cross's message of "Thu, 03 Nov 2005 12:01:57 -0500") References: <436A4285.2000302@crosswirecorp.com> Message-ID: On Thu, 03 Nov 2005 12:01:57 -0500, Travis Cross wrote: > My experience is that :tbnl-bivalent-streams must be added to > *FEATURES* before TBNL is _built_. > > The source code seems to support this notion as the #:+/- checks > will occur at compile time. > > It seems that :tbnl-bivalent-streams could be added to features > automatically for SBCL by adding :sbcl to the code block on line 309 > of specials.lisp: > > --- specials.lisp.orig 2005-11-03 11:57:39.000000000 -0500 > +++ specials.lisp 2005-11-03 11:58:05.000000000 -0500 > @@ -309,7 +309,8 @@ > (eval-when (:compile-toplevel :load-toplevel :execute) > #+(or (and :allegro :allegro-version>= (version>= 6 0)) > :lispworks4.3 > - :lispworks4.4) > + :lispworks4.4 > + :sbcl) > (pushnew :tbnl-bivalent-streams *features*)) > > #+:sbcl > > I would not worry tremendously about breaking older versions of SBCL > because 1) TBNL is already broken on older versions of SBCL and 2) > most SBCL users track pretty close to the latest release. OK, thanks. I'll add that for the next release. I think there are still problems, though - see other message. Cheers, Edi. From edi at agharta.de Thu Nov 3 23:57:22 2005 From: edi at agharta.de (Edi Weitz) Date: Fri, 04 Nov 2005 00:57:22 +0100 Subject: [tbnl-devel] Fixing binary uploads in SBCL / TBNL / mod_lisp2 In-Reply-To: <436A4545.6090201@crosswirecorp.com> (Travis Cross's message of "Thu, 03 Nov 2005 12:13:41 -0500") References: <436A4545.6090201@crosswirecorp.com> Message-ID: On Thu, 03 Nov 2005 12:13:41 -0500, Travis Cross wrote: > Running SBCL 0.9.6 with TBNL 0.8.4 and Apache 2.0.54 with mod_lisp2, > the image download examples work as expected, Hmm, I just tried with SBCL 0.9.6 in TBNL's "stand-alone" mode and only the first image download example worked for me. > but uploading any binary file results in a 500 Internal Server Error > being throw by Apache. No error is reported in Lisp, but the Apache > log file error_log shows the following entry: > > [error] (70014)End of file found: error reading from Lisp > > Setting apache to log in debug mode doesn't provide any additional > insight, but some testing and modification of the mod_lisp2.c code > showed that the error is starting at line 588 where mod_lisp > attempts to read the first of the headers. This leads back to > read_lisp_line(), then deeper to fill_input_buffer(), where the > failure seems to originate at the call to: > > RELAY_ERROR > (((length = (sizeof (buffer->data))), > (apr_recv (socket, (buffer->data), (&length))))); > > With *DEBUG-MODE* set to t, *COMMAND* contains seemingly plausible > headers (shown below). > > I am not familiar enough with the internals to have much of an idea > of what is causing the breakdown, though the C code seems to suggest > that lisp is sending a different amount of data than is promised. > > I'm happy to dig into either the TBNL, KMRCL, or mod_lisp2.c code > and come up with a patch for this, but it would be tremendously > helpful if someone has a good idea of where to start and what the > problem might be. I don't really have an idea but I'd advise you to track this down without Apache - use TBNL without a back-end. I'm pretty sure the problem is TBNL not properly supporting SBCL's Unicode branch and it's not related to Apache or mod_lisp. You might want to experiment with different locale settings and see if it makes a difference. Cheers, Edi. From jdcal at yahoo.com Sat Nov 12 17:43:28 2005 From: jdcal at yahoo.com (Jeff Caldwell) Date: Sat, 12 Nov 2005 09:43:28 -0800 (PST) Subject: [tbnl-devel] tbnl patch Message-ID: <20051112174328.74480.qmail@web50613.mail.yahoo.com> Hello, Here is a tiny patch for tbnl. The motivation is that I'm working with blocks of dispatch table entries that are added and withdrawn as modules without restarting tbnl. I'd also like reflective access to the dispatch table e.g. to show a web page of installed dispatch handlers. To affect the base distrubution as little as possible, I took the loop through the dispatch table that funcalls each entry and moved that loop into a defmethod. Now I can specialize on the method and have my own dispatch mechanisms. Jeff Caldwell __________________________________ Yahoo! Mail - PC Magazine Editors' Choice 2005 http://mail.yahoo.com -------------- next part -------------- A non-text attachment was scrubbed... Name: diff-tbnl-0.8.4 Type: application/octet-stream Size: 1584 bytes Desc: 819789442-diff-tbnl-0.8.4 URL: From adam.thorsen at gmail.com Mon Nov 14 12:49:29 2005 From: adam.thorsen at gmail.com (Adam Thorsen) Date: Mon, 14 Nov 2005 05:49:29 -0700 Subject: [tbnl-devel] sbcl 0.9.6 socket issue? Message-ID: <290f54b70511140449r16c03002k40c0bc6c845d1ed8@mail.gmail.com> (This is to all of the sbcl users on the list -- I realize sbcl is unsupported by the maintainer) Versions: SBCL 0.9.6 TBNL 0.8.4 mod_lisp Platform: gentoo x86 Indications: After running (tbnl:start-tbnl), tbnl functions properly for about 30 seconds, then refuses to serve the default page (or any page). A stack trace (which follows) seems to me to indicate some issue with kmrcl. (note: bivalent streams has been added to features). 0: (SB-UNIX::SIGINT-HANDLER # # #.(SB-SYS:INT-SAP #XB7C63D30)$ 1: (SB-SYS:INVOKE-INTERRUPTION #) 2: ("foreign function: call_into_lisp") 3: ("foreign function: funcall3") 4: ("foreign function: interrupt_handle_now") 5: ("foreign function: #x8052BB9") 6: ("bogus stack frame") 7: (SB-SYS:WAIT-UNTIL-FD-USABLE 9 :INPUT NIL) 8: (SB-IMPL::REFILL-BUFFER/FD #) 9: (SB-IMPL::INPUT-CHAR/ASCII # NIL NIL) 10: (READ-LINE # NIL NIL #) 11: (TBNL::GET-REQUEST-DATA) 12: (TBNL::LISTEN-FOR-REQUEST # TBNL::PROCESS-REQUEST) 13: ((LAMBDA NIL)) 14: (KMRCL::START-SOCKET-SERVER #) 15: (KMRCL::START-SOCKET-SERVER #) 16: (KMRCL::MAKE-SOCKET-SERVER #) 17: (KMRCL::LISTENER-STARTUP #) 18: (KMRCL:INIT/LISTENER # :START) 19: (SB-INT:EVAL-IN-LEXENV (TBNL:START-TBNL) #) 20: (SWANK::EVAL-REGION "(tbnl:start-tbnl) " T) From edi at agharta.de Mon Nov 14 14:31:41 2005 From: edi at agharta.de (Edi Weitz) Date: Mon, 14 Nov 2005 15:31:41 +0100 Subject: [tbnl-devel] tbnl patch In-Reply-To: <20051112174328.74480.qmail@web50613.mail.yahoo.com> (Jeff Caldwell's message of "Sat, 12 Nov 2005 09:43:28 -0800 (PST)") References: <20051112174328.74480.qmail@web50613.mail.yahoo.com> Message-ID: On Sat, 12 Nov 2005 09:43:28 -0800 (PST), Jeff Caldwell wrote: > Here is a tiny patch for tbnl. The motivation is that I'm working > with blocks of dispatch table entries that are added and withdrawn > as modules without restarting tbnl. I'd also like reflective access > to the dispatch table e.g. to show a web page of installed dispatch > handlers. Perhaps this would be something for the "contrib" directory of TBNL? > To affect the base distrubution as little as possible, I took the > loop through the dispatch table that funcalls each entry and moved > that loop into a defmethod. Now I can specialize on the method and > have my own dispatch mechanisms. Thanks Jeff, I've just made a new release with these changes. ChangeLog: Version 0.8.5 2005-11-14 Added generic function DISPATCH-REQUEST (thanks to Jeff Caldwell) Download: Cheers, Edi. From Alan-Shields at omrf.ouhsc.edu Tue Nov 15 18:26:30 2005 From: Alan-Shields at omrf.ouhsc.edu (Alan Shields) Date: Tue, 15 Nov 2005 12:26:30 -0600 Subject: [tbnl-devel] Merged patches to Araneida (with one change) Message-ID: <20051115182630.GX29109@inquisitor.omrf.org> Hello everyone, As Edi is out for a few days, he asked me to email the devel list instead. I've merged in the patches to Araneida to enable TBNL to work. I made a few modifications, however. I'd like someone to test out the changes to make sure they work. To test out the changes, do: darcs get http://code.microarray.omrf.org/darcs/alan-araneida-testing/ On the TBNL-side, instead of setting araneida::*standard-araneida* to nil, you need to set araneida:*araneida-mode* to :tbnl . Let me know if it works! Alan Shields From edi at agharta.de Fri Nov 18 15:46:17 2005 From: edi at agharta.de (Edi Weitz) Date: Fri, 18 Nov 2005 16:46:17 +0100 Subject: [tbnl-devel] Merged patches to Araneida (with one change) In-Reply-To: <20051115182630.GX29109@inquisitor.omrf.org> (Alan Shields's message of "Tue, 15 Nov 2005 12:26:30 -0600") References: <20051115182630.GX29109@inquisitor.omrf.org> Message-ID: Hi Alan! I'm back home in Hamburg. On Tue, 15 Nov 2005 12:26:30 -0600, Alan Shields wrote: > I've merged in the patches to Araneida to enable TBNL to work. I > made a few modifications, however. I'd like someone to test out the > changes to make sure they work. > > To test out the changes, do: > darcs get http://code.microarray.omrf.org/darcs/alan-araneida-testing/ > > On the TBNL-side, instead of setting araneida::*standard-araneida* > to nil, you need to set araneida:*araneida-mode* to :tbnl . I tested this briefly but without real success - or let's just say that I'm confused. When I tried the TBNL/Araneida combo as described in the TBNL docs it complained about REQUEST-UNPARSED-BODY not being external in the ARANEIDA package. Does that mean I'm not supposed to use the code in araneida.lisp in TBNL anymore? Also, I couldn't find any mention of ARANEIDA::*STANDARD-ARANEIDA* in TBNL anywhere. As you might have guessed by now I've never used Araneida myself, I basically just copied what Bob Hutchinson submitted. I hope he'll jump in and tell me how to change TBNL's code and docs to match the new Araneida. Also, do I need to publish a new download URI for Araneida? Cheers, Edi. From edi at agharta.de Fri Nov 18 15:50:42 2005 From: edi at agharta.de (Edi Weitz) Date: Fri, 18 Nov 2005 16:50:42 +0100 Subject: [tbnl-devel] New version 0.8.6 Message-ID: ChangeLog: Version 0.8.6 2005-11-18 Restored original stream-based code for multipart/form-data parsing (got lost somehow) Wrap REMOTE-ADDR with IGNORE-ERRORS (just in case) Download: Cheers, Edi. From hutch at recursive.ca Fri Nov 18 22:22:46 2005 From: hutch at recursive.ca (Bob Hutchison) Date: Fri, 18 Nov 2005 17:22:46 -0500 Subject: [tbnl-devel] Merged patches to Araneida (with one change) In-Reply-To: References: <20051115182630.GX29109@inquisitor.omrf.org> Message-ID: Hi, I've got this on my queue which at the moment is a little long. I'll try to have a look over the weekend. Sorry about that, esp. since Alan took the time to make these changed. Cheers, Bob On Nov 18, 2005, at 10:46 AM, Edi Weitz wrote: > Hi Alan! > > I'm back home in Hamburg. > > On Tue, 15 Nov 2005 12:26:30 -0600, Alan Shields Shields at omrf.ouhsc.edu> wrote: > >> I've merged in the patches to Araneida to enable TBNL to work. I >> made a few modifications, however. I'd like someone to test out the >> changes to make sure they work. >> >> To test out the changes, do: >> darcs get http://code.microarray.omrf.org/darcs/alan-araneida- >> testing/ >> >> On the TBNL-side, instead of setting araneida::*standard-araneida* >> to nil, you need to set araneida:*araneida-mode* to :tbnl . > > I tested this briefly but without real success - or let's just say > that I'm confused. When I tried the TBNL/Araneida combo as described > in the TBNL docs it complained about REQUEST-UNPARSED-BODY not being > external in the ARANEIDA package. Does that mean I'm not supposed to > use the code in araneida.lisp in TBNL anymore? > > Also, I couldn't find any mention of ARANEIDA::*STANDARD-ARANEIDA* in > TBNL anywhere. > > As you might have guessed by now I've never used Araneida myself, I > basically just copied what Bob Hutchinson submitted. I hope he'll > jump in and tell me how to change TBNL's code and docs to match the > new Araneida. Also, do I need to publish a new download URI for > Araneida? > > Cheers, > Edi. > _______________________________________________ > tbnl-devel site list > tbnl-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/tbnl-devel ---- Bob Hutchison -- blogs at Recursive Design Inc. -- Raconteur -- From ignas.mikalajunas at gmail.com Mon Nov 28 22:18:08 2005 From: ignas.mikalajunas at gmail.com (Ignas Mikalajunas) Date: Tue, 29 Nov 2005 00:18:08 +0200 Subject: [tbnl-devel] Content length with multibyte character encodings Message-ID: Content length is calculated by calling (length content) which produces wrong results with unicode characters in the string. Piso on #lisp proposed a solution - using (length (string-to-octets string :external-format :utf-8)) which translates to just (length (string-to-octets string :external-format)) in the code. The true way to solve this would be using (file-string-length), but the function is not working properly on sbcl yet. So could you please fix the (send-output), because with current setup browsers that strictly adhere to the content-lenght (IE 6.0, Opera) would trim 1 character of the responses body for each UTF-8 character in it. Ignas From edi at agharta.de Tue Nov 29 09:14:59 2005 From: edi at agharta.de (Edi Weitz) Date: Tue, 29 Nov 2005 10:14:59 +0100 Subject: [tbnl-devel] New version 0.8.7 (Was: Content length with multibyte character encodings) In-Reply-To: (Ignas Mikalajunas's message of "Tue, 29 Nov 2005 00:18:08 +0200") References: Message-ID: On Tue, 29 Nov 2005 00:18:08 +0200, Ignas Mikalajunas wrote: > Content length is calculated by calling (length content) which > produces wrong results with unicode characters in the string. Piso > on #lisp proposed a solution - using (length (string-to-octets > string :external-format :utf-8)) which translates to just (length > (string-to-octets string :external-format)) in the code. I won't do that because it's most likely a terrible performance hog if you convert each page to octets be default (assuming that most users already send octets). I also don't understand why (length (string-to-octets string :external-format :utf-8)) translates to (length (string-to-octets string :external-format)) > The true way to solve this would be using (file-string-length), > but the function is not working properly on sbcl yet. Huh? How is that supposed to work (even if it would work on SBCL)? *TBNL-STREAM* is a binary stream which accepts octets, isn't it? > So could you please fix the (send-output), IMHO there's nothing to "fix" because TBNL works as expected. The docs clearly say that you're supposed to send octets, see for example here: Note that the UTF-8 example that comes with TBNL sends a correct header. FWIW, I've just released a new version where you can manually set the CONTENT-LENGTH slot of the REPLY object. If it is not NIL TBNL won't bother to compute the content length so you can set it to any value you want. Note, though, that you'll run into trouble w.r.t. TBNL/Apache interaction if you set a wrong value there. > because with current setup browsers that strictly adhere to the > content-lenght (IE 6.0, Opera) would trim 1 character of the > responses body for each UTF-8 character in it. Nope, that's not how UTF-8 works. Cheers, Edi. From ignas.mikalajunas at gmail.com Tue Nov 29 11:24:44 2005 From: ignas.mikalajunas at gmail.com (Ignas Mikalajunas) Date: Tue, 29 Nov 2005 13:24:44 +0200 Subject: [tbnl-devel] Re: New version 0.8.7 (Was: Content length with multibyte character encodings) In-Reply-To: References: Message-ID: > On Tue, 29 Nov 2005 00:18:08 +0200, Ignas Mikalajunas wrote: > > > Content length is calculated by calling (length content) which > > produces wrong results with unicode characters in the string. Piso > > on #lisp proposed a solution - using (length (string-to-octets > > string :external-format :utf-8)) which translates to just (length > > (string-to-octets string :external-format)) in the code. > > I won't do that because it's most likely a terrible performance hog if > you convert each page to octets be default (assuming that most users > already send octets). Sorry i was not aware of that. If i understand you correctly the right way is converting all of my pages (they all are utf-8) to octets before sending them to tbnl? > I also don't understand why > > (length (string-to-octets string :external-format :utf-8)) > > translates to > > (length (string-to-octets string :external-format)) Because the first one is cl-user:string-to-octets and the second one is tbnl:string-to-octets. > > because with current setup browsers that strictly adhere to the > > content-lenght (IE 6.0, Opera) would trim 1 character of the > > responses body for each UTF-8 character in it. > > Nope, that's not how UTF-8 works. What i meant was: (length "????") returns 4 though (lenght (string-to-octests "????")) is 8. Which means that tbnl would try to fit an 8 octet body with a content length of 4 and IE/Opera would display that as "??". That's how it works on SBCL. Ignas From edi at agharta.de Tue Nov 29 14:58:30 2005 From: edi at agharta.de (Edi Weitz) Date: Tue, 29 Nov 2005 15:58:30 +0100 Subject: [tbnl-devel] Re: New version 0.8.7 In-Reply-To: (Ignas Mikalajunas's message of "Tue, 29 Nov 2005 13:24:44 +0200") References: Message-ID: On Tue, 29 Nov 2005 13:24:44 +0200, Ignas Mikalajunas wrote: > Sorry i was not aware of that. If i understand you correctly the > right way is converting all of my pages (they all are utf-8) to > octets before sending them to tbnl? Yep. Either that or (with the new version) figure out the octet length with other, less expensive means and setting it directly - if SBCL allows you to return a random Unicode string to TBNL. >> I also don't understand why >> >> (length (string-to-octets string :external-format :utf-8)) >> >> translates to >> >> (length (string-to-octets string :external-format)) > > Because the first one is cl-user:string-to-octets and the second one > is tbnl:string-to-octets. Ah, OK. Assuming you meant :UTF-8 instead of :EXTERNAL-FORMAT in the second form it's rather the other way around, though. A call to TBNL::STRING-TO-OCTETS will be translated to a call to the corresponding function in SB-EXT. >> > because with current setup browsers that strictly adhere to the >> > content-lenght (IE 6.0, Opera) would trim 1 character of the >> > responses body for each UTF-8 character in it. >> >> Nope, that's not how UTF-8 works. > > What i meant was: > (length "????") returns 4 > though (lenght (string-to-octests "????")) is 8. > Which means that tbnl would try to fit an 8 octet body with a > content length of 4 and IE/Opera would display that as "??". That's > how it works on SBCL. But you said "each" character. UTF-8 is a variable-length encoding where one character can have any length from one to six octets. For example, if your characters are all within the ASCII charset you won't lose any octets at all. Cheers, Edi.