From nowhere.man at levallois.eu.org Sat Dec 1 11:45:22 2007 From: nowhere.man at levallois.eu.org (Pierre THIERRY) Date: Sat, 1 Dec 2007 12:45:22 +0100 Subject: [hunchentoot-devel] Re: Hunchentoot's query parameter seperator In-Reply-To: References: Message-ID: <20071201114522.GA6064@(none)> Scribit Edi Weitz dies 22/11/2007 hora 10:30: > So, to everyone on the list - what's your experience with semicolons > as query string separators? Is this normal or esoteric? And, even > more interesting, does any client out there actually use this > convention? I've seen them even recently, and they're quite useful, as said already (no escaping hell). > Seems OK, but it might break existing applications. Any client-side application that doesn't send them to a server escaped is buggy, and any server-side application that parse the query string without using them a separator as well. If I have a form with a text field name foo and type "a=1;b=2&c=3" in it, my current browser, Firefox, will send the following query string: foo=a%3D1%3Bb%3D2%26c%3D3 In a PHP program, then $_GET['foo'] would evaluate as "a=1;b=2&c=3". If arg_separator.input in php.ini contains #\;, then it would parse the following query strng as defining two parameters, a and b: foo=a%3D1%3Bb%3D2%26c%3D3;b=bar Note that the default value is "&" and here is the commented line in my default php.ini: #arg_separator.input = ";&" Making Hunchentoot configurable on this point would be the best, I think. Quickly, Pierre -- nowhere.man at levallois.eu.org OpenPGP 0xD9D50D8A -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From edi at agharta.de Sat Dec 1 15:20:56 2007 From: edi at agharta.de (Edi Weitz) Date: Sat, 01 Dec 2007 16:20:56 +0100 Subject: [hunchentoot-devel] Re: Hunchentoot's query parameter seperator In-Reply-To: <20071201114522.GA6064@(none)> (Pierre THIERRY's message of "Sat, 1 Dec 2007 12:45:22 +0100") References: <20071201114522.GA6064@(none)> Message-ID: On Sat, 1 Dec 2007 12:45:22 +0100, Pierre THIERRY wrote: > If arg_separator.input in php.ini contains #\;, then it would parse > the following query strng as defining two parameters, a and b: > > foo=a%3D1%3Bb%3D2%26c%3D3;b=bar > > Note that the default value is "&" and here is the commented line in > my default php.ini: > > #arg_separator.input = ";&" > > Making Hunchentoot configurable on this point would be the best, I > think. I fully agree. That's what I said as well. Except that my point is that right I now I don't need it and I'm busy with other things anyway, so it has a very low priority for me. Patches welcome, of course. http://weitz.de/patches.html Edi. From dhf at panix.com Wed Dec 5 04:44:27 2007 From: dhf at panix.com (David H. Friedman) Date: Tue, 4 Dec 2007 23:44:27 -0500 (EST) Subject: [hunchentoot-devel] Problem with flexi-stream-bound on openmcl In-Reply-To: References: <2B7F55EC-DBCF-4E1F-8BC7-C671BF1BB5FF@gmail.com> Message-ID: Hi, Was the issue in ever solved? I'm having a similar problem with SBCL 1.0.12 (both x86 and x86_64) with flexi-streams-0.13.1 and hunchentoot-0.14.7 I wiped all .fasls, did (require :hunchentoot-test) and on all attempts to access localhost:3000 got "Error while processing connection: The function (SETF HUNCHENTOOT::FLEXI-STREAM-BOUND) is undefined." (Yes, it is looking in the hunchentoot package instead of flexi-streams.) Used to work; the problem seems to have developed fter I upgraded to sbcl 1.0.12. I'm going to downgrade sbcl and try again, but I thought I'd ask if anyone else has had (or resolved) the problem. Thanks, -- David From edi at agharta.de Wed Dec 5 04:54:20 2007 From: edi at agharta.de (Edi Weitz) Date: Wed, 05 Dec 2007 05:54:20 +0100 Subject: [hunchentoot-devel] Problem with flexi-stream-bound on openmcl In-Reply-To: (David H. Friedman's message of "Tue, 4 Dec 2007 23:44:27 -0500 (EST)") References: <2B7F55EC-DBCF-4E1F-8BC7-C671BF1BB5FF@gmail.com> Message-ID: On Tue, 4 Dec 2007 23:44:27 -0500 (EST), "David H. Friedman" wrote: > I'm having a similar problem with SBCL 1.0.12 (both x86 and x86_64) > with flexi-streams-0.13.1 and hunchentoot-0.14.7 > > I wiped all .fasls, did (require :hunchentoot-test) and on all > attempts to access localhost:3000 got > > "Error while processing connection: The function (SETF HUNCHENTOOT::FLEXI-STREAM-BOUND) is undefined." Are you sure that you only have /one/ copy of each library lying around? Maybe you have an old version somewhere which got installed via Debian's package system or somesuch and which is compiled and loaded instead of the one you think you're using. If you're sure this isn't the problem, wipe /all/ fasl files again, and show us the output generated by (asdf:oos 'asdf:load-op :hunchentoot) Edi. From dhf at panix.com Wed Dec 5 06:13:35 2007 From: dhf at panix.com (David H. Friedman) Date: Wed, 5 Dec 2007 01:13:35 -0500 (EST) Subject: [hunchentoot-devel] Problem with flexi-stream-bound on openmcl In-Reply-To: References: <2B7F55EC-DBCF-4E1F-8BC7-C671BF1BB5FF@gmail.com> Message-ID: On Wed, 5 Dec 2007, Edi Weitz wrote: > On Tue, 4 Dec 2007 23:44:27 -0500 (EST), "David H. Friedman" wrote: > >> I'm having a similar problem with SBCL 1.0.12 (both x86 and x86_64) >> with flexi-streams-0.13.1 and hunchentoot-0.14.7 >> >> I wiped all .fasls, did (require :hunchentoot-test) and on all >> attempts to access localhost:3000 got >> >> "Error while processing connection: The function (SETF HUNCHENTOOT::FLEXI-STREAM-BOUND) is undefined." > > Are you sure that you only have /one/ copy of each library lying > around? Maybe you have an old version somewhere which got installed > via Debian's package system or somesuch and which is compiled and > loaded instead of the one you think you're using. > > If you're sure this isn't the problem, wipe /all/ fasl files again, > and show us the output generated by > > (asdf:oos 'asdf:load-op :hunchentoot) > > Edi. Yes, of course that turned out to be the problem: there was an older copy of flexi-streams, and sbcl-1.0.12 changed the library directory search order. Thank you for your help (and for all the terrific libraries.) -- David From download at hpc.unm.edu Thu Dec 6 17:37:52 2007 From: download at hpc.unm.edu (Jim Prewett) Date: Thu, 6 Dec 2007 10:37:52 -0700 (MST) Subject: [hunchentoot-devel] how to diagnose output problem? Message-ID: Hello, I'm new to Hunchentoot. :) I'm working on an app that uses the Demo Allegro Lisp, the AGraph package, and Hunchentoot for the front-end. When I run on MacOS, everything seems great! When I run on Linux, things are not so good. I get *NO* output from Hunchentoot when I put certain records into the database. I don't have any idea if this is a Hunchentoot bug, a Agraph/Franz bug, or a bug in my understanding :) I don't see any errors at the REPL - I've not hit the limitations of the demo ACL. The function I use, when I put (warn ...) statements in, clearly finds all of the records I've requested and thinks its done all of its work (it believes its output the entire webpage), but I get no output :P Can anyone give me suggestions on how to tell /where/ I'm having the problem? I got hunchentoot and friends from asdf-install this morning on both systems (Mac and Linux). I'm running the latest ACL/AGraph. I've set *SHOW-LISP-BACKTRACES-P* and *SHOW-LISP-ERRORS-P* to t and don't see any errors. Here is a transcript of using telnet to request the page: # telnet localhost 80 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. GET /list?user=download HTTP/1.1 200 OK Content-Type: text/html; charset=iso-8859-1 Date: Thu, 06 Dec 2007 15:41:26 GMT Server: Hunchentoot 0.14.7 Connection: Close Set-Cookie: hunchentoot-session=7%3a355b027548e5bd694b3dd0b7b477b018; path=/ Connection closed by foreign host. Thanks for any ideas, Jim James E. Prewett Jim at Prewett.org download at hpc.unm.edu Systems Team Leader LoGS: http://www.hpc.unm.edu/~download/LoGS/ Designated Security Officer OpenPGP key: pub 1024D/31816D93 HPC Systems Engineer III UNM HPC 505.277.8210 From ctdean at sokitomi.com Thu Dec 6 18:32:49 2007 From: ctdean at sokitomi.com (Chris Dean) Date: Thu, 06 Dec 2007 10:32:49 -0800 Subject: [hunchentoot-devel] how to diagnose output problem? In-Reply-To: (Jim Prewett's message of "Thu, 6 Dec 2007 10:37:52 -0700 (MST)") References: Message-ID: Jim Prewett writes: > When I run on MacOS, everything seems great! When I run on Linux, things > are not so good. I get *NO* output from Hunchentoot when I put certain > records into the database. I don't have any idea if this is a Hunchentoot > bug, a Agraph/Franz bug, or a bug in my understanding :) I don't see any > errors at the REPL - I've not hit the limitations of the demo ACL. ... > I've set *SHOW-LISP-BACKTRACES-P* and *SHOW-LISP-ERRORS-P* to t and don't > see any errors. Try also setting *catch-errors-p* to nil and see if that helps. You can also look at the hunchentoot log and see if there is any hint in there (unlikely, but it may understanding what's going on). Cheers, Chris Dean From gwking at metabang.com Thu Dec 6 20:43:25 2007 From: gwking at metabang.com (Gary King) Date: Thu, 6 Dec 2007 15:43:25 -0500 Subject: [hunchentoot-devel] how to diagnose output problem? In-Reply-To: References: Message-ID: <77B5FA1A-2EA2-4B2E-8FA6-3176A50B106F@metabang.com> Hi Jim, I'm very new to Hunchentoot but wonder if there is a way to mimic an HTTP request. If there is, what happens if you call your dispatch function manually? -- Gary Warren King, metabang.com Cell: (413) 559 8738 Fax: (206) 338-4052 gwkkwg on Skype * garethsan on AIM From emailmac at gmail.com Thu Dec 6 21:02:18 2007 From: emailmac at gmail.com (Mac Chan) Date: Thu, 6 Dec 2007 13:02:18 -0800 Subject: [hunchentoot-devel] how to diagnose output problem? In-Reply-To: <77B5FA1A-2EA2-4B2E-8FA6-3176A50B106F@metabang.com> References: <77B5FA1A-2EA2-4B2E-8FA6-3176A50B106F@metabang.com> Message-ID: <4877ae640712061302p5789d3fehdf18784382a5aa17@mail.gmail.com> Hi Gary, On Dec 6, 2007 12:43 PM, Gary King wrote: > If there is, what happens if you call your dispatch > function manually? Many Hunchentoot functions assume that a number of dynamic variables to be bound when called. *request* is one of them, which your dispatch function will need. Search the list for a hack that I put together for saving various special variables for "replaying" the traffic. > I'm very new to Hunchentoot but wonder if there is a way to mimic an > HTTP request. You can try this http://lukego.livejournal.com/6154.html (and use Firebug for capturing a sample request) HTH - Mac From edi at agharta.de Thu Dec 6 23:45:09 2007 From: edi at agharta.de (Edi Weitz) Date: Fri, 07 Dec 2007 00:45:09 +0100 Subject: [hunchentoot-devel] how to diagnose output problem? In-Reply-To: (Chris Dean's message of "Thu, 06 Dec 2007 10:32:49 -0800") References: Message-ID: On Thu, 06 Dec 2007 10:32:49 -0800, Chris Dean wrote: > Try also setting *catch-errors-p* to nil and see if that helps. If it doesn't, there are also tools like WireShark. From edi at agharta.de Thu Dec 6 23:48:03 2007 From: edi at agharta.de (Edi Weitz) Date: Fri, 07 Dec 2007 00:48:03 +0100 Subject: [hunchentoot-devel] how to diagnose output problem? In-Reply-To: (Jim Prewett's message of "Thu, 6 Dec 2007 10:37:52 -0700 (MST)") References: Message-ID: On Thu, 6 Dec 2007 10:37:52 -0700 (MST), Jim Prewett wrote: > Here is a transcript of using telnet to request the page: > # telnet localhost 80 > Trying 127.0.0.1... > Connected to localhost. > Escape character is '^]'. > GET /list?user=download > HTTP/1.1 200 OK > Content-Type: text/html; charset=iso-8859-1 > Date: Thu, 06 Dec 2007 15:41:26 GMT > Server: Hunchentoot 0.14.7 > Connection: Close > Set-Cookie: hunchentoot-session=7%3a355b027548e5bd694b3dd0b7b477b018; path=/ > > Connection closed by foreign host. This doesn't look like "NO output" to me. The headers are sent, aren't they? Are you sure your handler is actually sending a body? Have you checked if the test webpage that comes with Hunchentoot works? From gwking at metabang.com Fri Dec 7 04:04:19 2007 From: gwking at metabang.com (Gary King) Date: Thu, 6 Dec 2007 23:04:19 -0500 Subject: [hunchentoot-devel] how to diagnose output problem? In-Reply-To: <4877ae640712061302p5789d3fehdf18784382a5aa17@mail.gmail.com> References: <77B5FA1A-2EA2-4B2E-8FA6-3176A50B106F@metabang.com> <4877ae640712061302p5789d3fehdf18784382a5aa17@mail.gmail.com> Message-ID: <4B61FF20-7149-42A5-8412-CFB6038F655B@metabang.com> Hi Marc, Cool on both counts. I look forward to experimenting. Thanks, -- Gary Warren King, metabang.com Cell: (413) 559 8738 Fax: (206) 338-4052 gwkkwg on Skype * garethsan on AIM From download at hpc.unm.edu Fri Dec 7 07:27:55 2007 From: download at hpc.unm.edu (Jim Prewett) Date: Fri, 7 Dec 2007 00:27:55 -0700 (MST) Subject: [hunchentoot-devel] how to diagnose output problem? In-Reply-To: References: Message-ID: > This doesn't look like "NO output" to me. The headers are sent, > aren't they? Are you sure your handler is actually sending a body? > Have you checked if the test webpage that comes with Hunchentoot > works? Right, thats what is especially making me scratch my head - I am getting a response to my http request, it just doesn't have any of the fun webpage type contents I was expecting. I've not exhaustively gone through the Hunchentoot test pages, but a quick check says that everything is working OK. How might I check that my handler is actually sending a body? Like I said previously, putting the 'wrong' record in my database causes me to go from the output I expect to only getting the headers from the webserver - all that I can currently identify about the record being 'wrong' is that it has an "@" character in a text string, FWIW. Thanks, Jim From edi at agharta.de Fri Dec 7 08:11:40 2007 From: edi at agharta.de (Edi Weitz) Date: Fri, 07 Dec 2007 09:11:40 +0100 Subject: [hunchentoot-devel] how to diagnose output problem? In-Reply-To: (Jim Prewett's message of "Fri, 7 Dec 2007 00:27:55 -0700 (MST)") References: Message-ID: On Fri, 7 Dec 2007 00:27:55 -0700 (MST), Jim Prewett wrote: > How might I check that my handler is actually sending a body? Make sure that it returns a string and not NIL. (Unless you're using SEND-HEADERS.) From hutch at recursive.ca Fri Dec 14 22:03:55 2007 From: hutch at recursive.ca (Bob Hutchison) Date: Fri, 14 Dec 2007 17:03:55 -0500 Subject: [hunchentoot-devel] Google search result Message-ID: <35EE1A82-9609-4305-B545-B8815268E295@recursive.ca> Hi Edi, Debating whether to mention this or not... Have you noticed the text that shows up when you search for Hunchentoot in google: "HUNCHENTOOT - The Common Lisp web server formerly known as TBNL Web server coded in Common Lisp; supports HTTP/1.1 chunking, persistent connections (keep-alive), and SSL; works only with LispWorks." The 'works only with LispWorks' bit is maybe something you don't want there? Cheers, Bob ---- Bob Hutchison -- tumblelog at http://www.recursive.ca/so/ Recursive Design Inc. -- weblog at http://www.recursive.ca/hutch http://www.recursive.ca/ -- works on http://www.raconteur.info/cms-for-static-content/home/ From gwking at metabang.com Fri Dec 14 22:18:38 2007 From: gwking at metabang.com (Gary King) Date: Fri, 14 Dec 2007 17:18:38 -0500 Subject: [hunchentoot-devel] Google search result In-Reply-To: <35EE1A82-9609-4305-B545-B8815268E295@recursive.ca> References: <35EE1A82-9609-4305-B545-B8815268E295@recursive.ca> Message-ID: IIRC, there is some way to ping Google and get them to spider your page more quickly... On Dec 14, 2007, at 5:03 PM, Bob Hutchison wrote: > Hi Edi, > > Debating whether to mention this or not... > > Have you noticed the text that shows up when you search for > Hunchentoot in google: > > "HUNCHENTOOT - The Common Lisp web server formerly known as TBNL > Web server coded in Common Lisp; supports HTTP/1.1 chunking, > persistent connections (keep-alive), and SSL; works only with > LispWorks." > > The 'works only with LispWorks' bit is maybe something you don't > want there? > > Cheers, > Bob > > ---- > Bob Hutchison -- tumblelog at http://www.recursive.ca/so/ > Recursive Design Inc. -- weblog at http://www.recursive.ca/hutch > http://www.recursive.ca/ -- works on http://www.raconteur.info/cms-for-static-content/home/ > > > _______________________________________________ > tbnl-devel site list > tbnl-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/tbnl-devel -- Gary Warren King, metabang.com Cell: (413) 559 8738 Fax: (206) 338-4052 gwkkwg on Skype * garethsan on AIM From edi at agharta.de Fri Dec 14 22:22:24 2007 From: edi at agharta.de (Edi Weitz) Date: Fri, 14 Dec 2007 23:22:24 +0100 Subject: [hunchentoot-devel] Google search result In-Reply-To: <35EE1A82-9609-4305-B545-B8815268E295@recursive.ca> (Bob Hutchison's message of "Fri, 14 Dec 2007 17:03:55 -0500") References: <35EE1A82-9609-4305-B545-B8815268E295@recursive.ca> Message-ID: On Fri, 14 Dec 2007 17:03:55 -0500, Bob Hutchison wrote: > Debating whether to mention this or not... > > Have you noticed the text that shows up when you search for > Hunchentoot in google: > > "HUNCHENTOOT - The Common Lisp web server formerly known as TBNL Web > server coded in Common Lisp; supports HTTP/1.1 chunking, persistent > connections (keep-alive), and SSL; works only with LispWorks." > > The 'works only with LispWorks' bit is maybe something you don't > want there? Didn't see that before. Of course, the information that Hunchentoot only works with LispWorks is more than a year old. Says something about Google... Now, if I only knew where Google gets these tidbits from. From hutch at recursive.ca Fri Dec 14 22:29:22 2007 From: hutch at recursive.ca (Bob Hutchison) Date: Fri, 14 Dec 2007 17:29:22 -0500 Subject: [hunchentoot-devel] Google search result In-Reply-To: References: <35EE1A82-9609-4305-B545-B8815268E295@recursive.ca> Message-ID: On 14-Dec-07, at 5:22 PM, Edi Weitz wrote: > On Fri, 14 Dec 2007 17:03:55 -0500, Bob Hutchison > wrote: > >> Debating whether to mention this or not... >> >> Have you noticed the text that shows up when you search for >> Hunchentoot in google: >> >> "HUNCHENTOOT - The Common Lisp web server formerly known as TBNL Web >> server coded in Common Lisp; supports HTTP/1.1 chunking, persistent >> connections (keep-alive), and SSL; works only with LispWorks." >> >> The 'works only with LispWorks' bit is maybe something you don't >> want there? > > Didn't see that before. Of course, the information that Hunchentoot > only works with LispWorks is more than a year old. Says something > about Google... > > Now, if I only knew where Google gets these tidbits from. I think if you put a in the header you'll get what you want. The Hunchentoot page was cached by Google on Dec 8, 2007 which is pretty recent. Cheers, Bob > > _______________________________________________ > tbnl-devel site list > tbnl-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/tbnl-devel ---- Bob Hutchison -- tumblelog at http://www.recursive.ca/so/ Recursive Design Inc. -- weblog at http://www.recursive.ca/hutch http://www.recursive.ca/ -- works on http://www.raconteur.info/cms-for-static-content/home/ From edi at agharta.de Fri Dec 14 22:31:02 2007 From: edi at agharta.de (Edi Weitz) Date: Fri, 14 Dec 2007 23:31:02 +0100 Subject: [hunchentoot-devel] Google search result In-Reply-To: (Gary King's message of "Fri, 14 Dec 2007 17:18:38 -0500") References: <35EE1A82-9609-4305-B545-B8815268E295@recursive.ca> Message-ID: On Fri, 14 Dec 2007 17:18:38 -0500, Gary King wrote: > IIRC, there is some way to ping Google and get them to spider your > page more quickly... The text is /not/ a result of spidering - it never appeared on my webpage in this form. Google spiders weitz.de pretty often anyway. For example, Googlebot looks at the Hunchentoot page about once per day: edi at nanook:/var/log/apache$ nslookup 66.249.67.206 Server: 81.169.163.104 Address: 81.169.163.104#53 Non-authoritative answer: 206.67.249.66.in-addr.arpa name = crawl-66-249-67-206.googlebot.com. Authoritative answers can be found from: edi at nanook:/var/log/apache$ cat weitz_access.log.1 weitz_access.log | grep -i 66.249.67.206 | grep "GET /hunchentoot/ " 66.249.67.206 - - [02/Dec/2007:12:10:00 +0100] "GET /hunchentoot/ HTTP/1.1" 200 133631 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 66.249.67.206 - - [03/Dec/2007:12:05:52 +0100] "GET /hunchentoot/ HTTP/1.1" 200 133631 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 66.249.67.206 - - [03/Dec/2007:16:40:12 +0100] "GET /hunchentoot/ HTTP/1.1" 200 133631 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 66.249.67.206 - - [04/Dec/2007:18:51:13 +0100] "GET /hunchentoot/ HTTP/1.1" 200 133631 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 66.249.67.206 - - [05/Dec/2007:16:16:03 +0100] "GET /hunchentoot/ HTTP/1.1" 200 133631 "-" "DoCoMo/1.0/N505i/c20/TB/W20H10 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)" 66.249.67.206 - - [05/Dec/2007:22:05:52 +0100] "GET /hunchentoot/ HTTP/1.1" 200 133631 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 66.249.67.206 - - [05/Dec/2007:22:30:16 +0100] "GET /hunchentoot/ HTTP/1.1" 200 133631 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 66.249.67.206 - - [07/Dec/2007:05:49:10 +0100] "GET /hunchentoot/ HTTP/1.1" 200 133631 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 66.249.67.206 - - [07/Dec/2007:08:09:28 +0100] "GET /hunchentoot/ HTTP/1.1" 200 133631 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 66.249.67.206 - - [08/Dec/2007:08:41:29 +0100] "GET /hunchentoot/ HTTP/1.1" 200 133631 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 66.249.67.206 - - [09/Dec/2007:17:13:07 +0100] "GET /hunchentoot/ HTTP/1.1" 200 133631 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 66.249.67.206 - - [11/Dec/2007:14:32:29 +0100] "GET /hunchentoot/ HTTP/1.1" 200 133631 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 66.249.67.206 - - [12/Dec/2007:15:36:36 +0100] "GET /hunchentoot/ HTTP/1.1" 200 133631 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 66.249.67.206 - - [13/Dec/2007:16:05:52 +0100] "GET /hunchentoot/ HTTP/1.1" 200 133631 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 66.249.67.206 - - [14/Dec/2007:15:32:27 +0100] "GET /hunchentoot/ HTTP/1.1" 200 133631 "-" "DoCoMo/1.0/N505i/c20/TB/W20H10 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)" 66.249.67.206 - - [14/Dec/2007:19:47:44 +0100] "GET /hunchentoot/ HTTP/1.1" 200 133631 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" From hutch at recursive.ca Fri Dec 14 22:33:15 2007 From: hutch at recursive.ca (Bob Hutchison) Date: Fri, 14 Dec 2007 17:33:15 -0500 Subject: [hunchentoot-devel] Google search result In-Reply-To: References: <35EE1A82-9609-4305-B545-B8815268E295@recursive.ca> Message-ID: <97B239DD-9F44-4C6A-8949-7230299403BB@recursive.ca> On 14-Dec-07, at 5:18 PM, Gary King wrote: > IIRC, there is some way to ping Google and get them to spider your > page more quickly... Yes there is... I've got the ability to completely forget how to do that within seconds of leaving that page. I think it is in the google webmaster pages. You can register a site. Ah... it is all linked to off this page: Edi, if you did that a while ago that might be where the description is coming from. Cheers, Bob > > > On Dec 14, 2007, at 5:03 PM, Bob Hutchison wrote: > >> Hi Edi, >> >> Debating whether to mention this or not... >> >> Have you noticed the text that shows up when you search for >> Hunchentoot in google: >> >> "HUNCHENTOOT - The Common Lisp web server formerly known as TBNL >> Web server coded in Common Lisp; supports HTTP/1.1 chunking, >> persistent connections (keep-alive), and SSL; works only with >> LispWorks." >> >> The 'works only with LispWorks' bit is maybe something you don't >> want there? >> >> Cheers, >> Bob >> >> ---- >> Bob Hutchison -- tumblelog at http://www.recursive.ca/so/ >> Recursive Design Inc. -- weblog at http://www.recursive.ca/hutch >> http://www.recursive.ca/ -- works on http://www.raconteur.info/cms-for-static-content/home/ >> >> >> _______________________________________________ >> tbnl-devel site list >> tbnl-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/tbnl-devel > > -- > Gary Warren King, metabang.com > Cell: (413) 559 8738 > Fax: (206) 338-4052 > gwkkwg on Skype * garethsan on AIM > > > > > _______________________________________________ > tbnl-devel site list > tbnl-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/tbnl-devel ---- Bob Hutchison -- tumblelog at http://www.recursive.ca/so/ Recursive Design Inc. -- weblog at http://www.recursive.ca/hutch http://www.recursive.ca/ -- works on http://www.raconteur.info/cms-for-static-content/home/ From edi at agharta.de Fri Dec 14 23:14:59 2007 From: edi at agharta.de (Edi Weitz) Date: Sat, 15 Dec 2007 00:14:59 +0100 Subject: [hunchentoot-devel] Google search result In-Reply-To: <97B239DD-9F44-4C6A-8949-7230299403BB@recursive.ca> (Bob Hutchison's message of "Fri, 14 Dec 2007 17:33:15 -0500") References: <35EE1A82-9609-4305-B545-B8815268E295@recursive.ca> <97B239DD-9F44-4C6A-8949-7230299403BB@recursive.ca> Message-ID: On Fri, 14 Dec 2007 17:33:15 -0500, Bob Hutchison wrote: > Edi, if you did that a while ago that might be where the description > is coming from. No, I didn't. However, it is pretty easy to see where their information is coming from. Copy the "Web server ... only with LispWorks" text and paste /that/ into Google's search box. The first thing Google will find is the "Google Directory" entry for Hunchentoot. So, someone (not me) added this entry more than a year ago and never bothered to update it. From edi at agharta.de Sat Dec 15 00:00:33 2007 From: edi at agharta.de (Edi Weitz) Date: Sat, 15 Dec 2007 01:00:33 +0100 Subject: [hunchentoot-devel] Google search result In-Reply-To: (Bob Hutchison's message of "Fri, 14 Dec 2007 17:29:22 -0500") References: <35EE1A82-9609-4305-B545-B8815268E295@recursive.ca> Message-ID: On Fri, 14 Dec 2007 17:29:22 -0500, Bob Hutchison wrote: > I think if you put a in the header you'll get what you want. I've done that now. Let's see what comes out of it. From lstaflin at gmail.com Sat Dec 15 14:32:17 2007 From: lstaflin at gmail.com (Lennart Staflin) Date: Sat, 15 Dec 2007 15:32:17 +0100 Subject: [hunchentoot-devel] Google search result In-Reply-To: References: <35EE1A82-9609-4305-B545-B8815268E295@recursive.ca> Message-ID: It seems that dmoz uses that particular phrase. -- //Lennart Staflin From jeffrey at cunningham.net Mon Dec 17 15:36:56 2007 From: jeffrey at cunningham.net (Jeff Cunningham) Date: Mon, 17 Dec 2007 07:36:56 -0800 Subject: [hunchentoot-devel] Having trouble with create-folder-dispatcher-and-handler Message-ID: <47669798.5010500@cunningham.net> I haven't used this handler for awhile, but I do recall it worked the last time I did. Now, when I set up a simple test case it fails, generating an Internal Server Error (500) with this error message in the log: [error] [Hunchentoot] The value NIL is not of type (MOD 536870911) I isolated it as far as the enough-url function, where it seems that the mismatch is returning NIL and the subseq doesn't like it. My test case is: (create-folder-dispatcher-and-handler "/test/" #p"/home/hunchentoot/test/" "text/plain") Both folders exist and are world readable. I am running SBCL 1.0.12. I tried checking what the arguments are to enough-url when it fails - they are: "url=/test/ url-prefix=/test/ which, of course, explains the NIL and the failure of the subseq. I don't understand how this handler is supposed to work, which it makes it difficult for me to troubleshoot further. Any idea why this doesn't work? --Jeff From hans at huebner.org Mon Dec 17 15:54:31 2007 From: hans at huebner.org (=?ISO-8859-1?Q?Hans_H=FCbner?=) Date: Mon, 17 Dec 2007 16:54:31 +0100 Subject: [hunchentoot-devel] Having trouble with create-folder-dispatcher-and-handler In-Reply-To: <47669798.5010500@cunningham.net> References: <47669798.5010500@cunningham.net> Message-ID: Can you please try with the attached patch? Thanks, Hans On Dec 17, 2007 4:36 PM, Jeff Cunningham wrote: > I haven't used this handler for awhile, but I do recall it worked the > last time I did. Now, when I set up a simple test case it fails, > generating an Internal Server Error (500) with this error message in the > log: > > [error] [Hunchentoot] The value NIL is not of type (MOD 536870911) > > I isolated it as far as the enough-url function, where it seems that the > mismatch is returning NIL and the subseq doesn't like it. My test case is: > > (create-folder-dispatcher-and-handler "/test/" > #p"/home/hunchentoot/test/" "text/plain") > > Both folders exist and are world readable. I am running SBCL 1.0.12. > > I tried checking what the arguments are to enough-url when it fails - > they are: > > "url=/test/ > url-prefix=/test/ > > which, of course, explains the NIL and the failure of the subseq. I > don't understand how this handler is supposed to work, which it makes it > difficult for me to troubleshoot further. Any idea why this doesn't work? > > --Jeff > > _______________________________________________ > tbnl-devel site list > tbnl-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/tbnl-devel > -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: patch.txt URL: From jeffrey.k.cunningham at boeing.com Mon Dec 17 17:56:06 2007 From: jeffrey.k.cunningham at boeing.com (Cunningham, Jeff) Date: Mon, 17 Dec 2007 09:56:06 -0800 Subject: [Fwd: [hunchentoot-devel] Having trouble with create-folder-dispatcher-and-handler] Message-ID: <4766B836.2090300@boeing.com> > Can you please try with the attached patch? > > Thanks, > Hans > /> /> -------------- next part -------------- > Index: misc.lisp > =================================================================== > --- misc.lisp (revision 2282) > +++ misc.lisp (working copy) > @@ -180,7 +180,10 @@ > (defun enough-url (url url-prefix) > "Returns the relative portion of URL relative to URL-PREFIX, similar > to what ENOUGH-NAMESTRING does for pathnames." > - (subseq url (mismatch url url-prefix))) > + (let ((start (mismatch url url-prefix))) > + (if start > + (subseq url start) > + ""))) > > (defun create-folder-dispatcher-and-handler (uri-prefix base-path &optional content-type) > "Creates and returns a dispatch function which will dispatch to a I tried the patch. It is still does not work, but the error message changes to: Bad Request Your browser sent a request that this server could not understand. --Jeff -------------- next part -------------- An HTML attachment was scrubbed... URL: From hans at huebner.org Mon Dec 17 18:05:36 2007 From: hans at huebner.org (=?ISO-8859-1?Q?Hans_H=FCbner?=) Date: Mon, 17 Dec 2007 19:05:36 +0100 Subject: [Fwd: [hunchentoot-devel] Having trouble with create-folder-dispatcher-and-handler] In-Reply-To: <4766B836.2090300@boeing.com> References: <4766B836.2090300@boeing.com> Message-ID: The change to enough-url modifies it so that it behaves the same as enough-namestring (which was previously used) does. What I am not sure about is whether the create-folder-dispatcher-and-handler dispatcher and handler can handle an empty path. What you'd propably want to see is a directory listing? Can you fetch files if you specify the complete URL? -Hans On Dec 17, 2007 6:56 PM, Cunningham, Jeff wrote: > > > Can you please try with the attached patch? > > > > Thanks, > > Hans > > > > > > -------------- next part -------------- > > Index: misc.lisp > > =================================================================== > > --- misc.lisp (revision 2282) > > +++ misc.lisp (working copy) > > @@ -180,7 +180,10 @@ > > (defun enough-url (url url-prefix) > > "Returns the relative portion of URL relative to URL-PREFIX, similar > > to what ENOUGH-NAMESTRING does for pathnames." > > - (subseq url (mismatch url url-prefix))) > > + (let ((start (mismatch url url-prefix))) > > + (if start > > + (subseq url start) > > + ""))) > > > > (defun create-folder-dispatcher-and-handler (uri-prefix base-path > &optional content-type) > > "Creates and returns a dispatch function which will dispatch to a > > > I tried the patch. It is still does not work, but the error message changes > to: > > Bad Request > Your browser sent a request that this server could not understand. > > --Jeff > > _______________________________________________ > tbnl-devel site list > tbnl-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/tbnl-devel > From jeffrey.k.cunningham at boeing.com Mon Dec 17 18:16:12 2007 From: jeffrey.k.cunningham at boeing.com (Cunningham, Jeff) Date: Mon, 17 Dec 2007 10:16:12 -0800 Subject: [Fwd: [hunchentoot-devel] Having trouble withcreate-folder-dispatcher-and-handler] In-Reply-To: References: <4766B836.2090300@boeing.com> Message-ID: <4766BCEC.9050004@boeing.com> Hans H?bner wrote: > The change to enough-url modifies it so that it behaves the same as > enough-namestring (which was previously used) does. What I am not > sure about is whether the create-folder-dispatcher-and-handler > dispatcher and handler can handle an empty path. What you'd propably > want to see is a directory listing? Can you fetch files if you > specify the complete URL? > > -Hans > > Yes, I can fetch files with a complete URL. And you are right - I'd like to see a directory listing. I just re-ran hunchentoot-test. With this url I get the image file: http://localhost:3001/hunchentoot/code/fz.jpg But with this URL I get a 400: bad request http://localhost:3001/hunchentoot/code/ --Jeff -------------- next part -------------- An HTML attachment was scrubbed... URL: From hans at huebner.org Mon Dec 17 19:14:35 2007 From: hans at huebner.org (=?ISO-8859-1?Q?Hans_H=FCbner?=) Date: Mon, 17 Dec 2007 20:14:35 +0100 Subject: [Fwd: [hunchentoot-devel] Having trouble withcreate-folder-dispatcher-and-handler] In-Reply-To: <4766BCEC.9050004@boeing.com> References: <4766B836.2090300@boeing.com> <4766BCEC.9050004@boeing.com> Message-ID: What I am not sure about is whether the error message is the right thing, but I am pretty sure that hunchentoot does not do directory listings by default. Can you please check out whether http://site.znain.com/dl/lisp/hunchentoot-dir-lister/ helps you out? -Hans On Dec 17, 2007 7:16 PM, Cunningham, Jeff wrote: > > Hans H?bner wrote: > The change to enough-url modifies it so that it behaves the same as > enough-namestring (which was previously used) does. What I am not > sure about is whether the create-folder-dispatcher-and-handler > dispatcher and handler can handle an empty path. What you'd propably > want to see is a directory listing? Can you fetch files if you > specify the complete URL? > > -Hans > > > Yes, I can fetch files with a complete URL. And you are right - I'd like to > see a directory listing. > I just re-ran hunchentoot-test. With this url I get the image file: > > http://localhost:3001/hunchentoot/code/fz.jpg > > But with this URL I get a 400: bad request > > http://localhost:3001/hunchentoot/code/ > > --Jeff > > _______________________________________________ > tbnl-devel site list > tbnl-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/tbnl-devel > From jeffrey.k.cunningham at boeing.com Mon Dec 17 19:50:03 2007 From: jeffrey.k.cunningham at boeing.com (Cunningham, Jeff) Date: Mon, 17 Dec 2007 11:50:03 -0800 Subject: [Fwd: [hunchentoot-devel] Having troublewithcreate-folder-dispatcher-and-handler] In-Reply-To: References: <4766B836.2090300@boeing.com><4766BCEC.9050004@boeing.com> Message-ID: <4766D2EB.2020405@boeing.com> Okay. I'll do that. I've been reading the code more closely and I think I see my misunderstanding. The folder-dispatcher seems to be used to enable file dispatch handlers to access files in directories that differ from the URI-prefix? I notice that if I take it out of the hunchentoot-test then it is unable to "view the code". Thanks for clearing this up for me. --Jeff Hans H?bner wrote: > What I am not sure about is whether the error message is the right > thing, but I am pretty sure that hunchentoot does not do directory > listings by default. Can you please check out whether > http://site.znain.com/dl/lisp/hunchentoot-dir-lister/ helps you out? > > -Hans > > On Dec 17, 2007 7:16 PM, Cunningham, Jeff > wrote: > >> Hans H?bner wrote: >> The change to enough-url modifies it so that it behaves the same as >> enough-namestring (which was previously used) does. What I am not >> sure about is whether the create-folder-dispatcher-and-handler >> dispatcher and handler can handle an empty path. What you'd propably >> want to see is a directory listing? Can you fetch files if you >> specify the complete URL? >> >> -Hans >> >> >> Yes, I can fetch files with a complete URL. And you are right - I'd like to >> see a directory listing. >> I just re-ran hunchentoot-test. With this url I get the image file: >> >> http://localhost:3001/hunchentoot/code/fz.jpg >> >> But with this URL I get a 400: bad request >> >> http://localhost:3001/hunchentoot/code/ >> >> --Jeff >> >> _______________________________________________ >> tbnl-devel site list >> tbnl-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/tbnl-devel >> >> > _______________________________________________ > tbnl-devel site list > tbnl-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/tbnl-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From avodonosov at yandex.ru Fri Dec 21 05:02:31 2007 From: avodonosov at yandex.ru (Anton Vodonosov) Date: Fri, 21 Dec 2007 07:02:31 +0200 Subject: [hunchentoot-devel] clisp port Message-ID: Hello. Attached is the adjustment of the clisp port suggested by Luis Oliveira ( http://common-lisp.net/pipermail/tbnl-devel/2006-November/000815.html) to hunchentoot 0.14.7. I've dropped clisp-serve-event because didn't find it useful. Just use start-server as you do in case of single-threaded sbcl. Edi, is it possible to commit it? I'm using it from about hunchentoot 0.11.1, it works OK for me (not to say i program for hunchentoot too much, but sometimes i do). Clisp port may be useful in at least two cases: - If you want free lisp on windows. SBCL on windows has problems with sockets (for example http://sourceforge.net/mailarchive/message.php?msg_id=33541185225400%40webmail24.yandex.ru, also it doesn't close sockets and after few thousands of connections windows runs out of handles). Although it isn't a problem for development, in fact I switch freely between sbcl and clisp all the time. - When small memory amount is available. Windows task manager reports memory usage by sbcl process of about 60 MB when hunchentoot is loaded. For clisp it's less then 25 MB. I've recently deployed small bulletin board I'm writing to a linux VPN with 64 MB of memory. The test is to serve static 200 KB file and request it using ApacheBench tool: ab -n 1000 -c 10 http://localhost:4242/static/file (-n is the number of requests, -c number of threads). In case of SBCL few such commands leads to furious swapping, whole system almost hangs and finally linux kills SBCL with message "out of memory". But clisp works stable. Best regards, -Anton -------------- next part -------------- A non-text attachment was scrubbed... Name: clisp-port.diff Type: text/x-patch Size: 12356 bytes Desc: clisp-port.diff URL: From edi at agharta.de Fri Dec 21 10:26:53 2007 From: edi at agharta.de (Edi Weitz) Date: Fri, 21 Dec 2007 11:26:53 +0100 Subject: [hunchentoot-devel] clisp port In-Reply-To: (Anton Vodonosov's message of "Fri, 21 Dec 2007 07:02:31 +0200") References: Message-ID: Hi Anton, On Fri, 21 Dec 2007 07:02:31 +0200, Anton Vodonosov wrote: > Attached is the adjustment of the clisp port suggested by Luis > Oliveira [...] Edi, is it possible to commit it? that's fine with me in principle although I won't be able to test new features against CLISP. However, several of the functions in the patch don't have documentation strings, and the HTML documentation should be updated in such a way that it at least mentions the CLISP port. http://weitz.de/patches.html > I've dropped clisp-serve-event because didn't find it useful. Could you elaborate on this? Thanks, Edi. From vseguip at gmail.com Fri Dec 21 15:58:57 2007 From: vseguip at gmail.com (vseguip at gmail.com) Date: Fri, 21 Dec 2007 16:58:57 +0100 Subject: [hunchentoot-devel] Hunchentoot performance Message-ID: Hi all, First, as a disclaimer, I'm a total Lisp noobie and have been playing with it recently poking here and there. I just tried Hunchentoot and was happy to see how easy it is to use. I also tried to make some preformance benchmarks on static files and compared it with lighttpd. Unsurprisingly lighttpd beat Hunchentoot quite badly, but I see there is still a lot of improvement for hunchentoot. I did some profiling and added a specialized handler that would serve static files using the linux sendfile system call which really improved the performance. Anyway I still see plenty room for improvements and I would like to take a stab at it, so I did some profiling. I'm putting the results in the ML in case someone wants to discuss it and help me figure out whether the speed for this use case can be improved. Details about the benchmark setup follow: Intel Dual Core 2 2.0 GHz 1 Gb Ram Ubuntu Gutsy 7.10 SBCL version: Hunchentoot version: Lighttpd version: Benchmark program: ab File: 10 mb bitmap Command: ab -n 1000 -c 20 -k http://127.0.0.1/sendfile/1.jpg Using sendfile system call | consed | calls | sec/call | name ----------------------------------------------------------- 0.644 | 25,410,696 | 1,002 | 0.000643 | HUNCHENTOOT::GET-REQUEST-DATA 0.453 | 197,900,328 | 6,006 | 0.000075 | HUNCHENTOOT::WRITE-HEADER-LINE 0.435 | 19,617,768 | 1,001 | 0.000435 | HUNCHENTOOT::HANDLE-STATIC-SENDFILE 0.367 | 0 | 1,001 | 0.000367 | HUNCHENTOOT::SENDTHEFILE 0.230 | 53,605,640 | 1,001 | 0.000230 | HUNCHENTOOT:LOG-MESSAGE 0.163 | 36,415,056 | 2,002 | 0.000082 | HUNCHENTOOT::START-OUTPUT ... Using hunchentoot static folder dispatcher | consed | calls | sec/call | name ----------------------------------------------------------- 0.678 | 165,618,816 | 5,000 | 0.000136 | HUNCHENTOOT::WRITE-HEADER-LINE 0.556 | 27,725,384 | 1,000 | 0.000556 | HUNCHENTOOT::GET-REQUEST-DATA 0.347 | 45,721,424 | 1,000 | 0.000347 | HUNCHENTOOT::START-OUTPUT 0.266 | 57,833,952 | 1,000 | 0.000266 | HUNCHENTOOT:LOG-MESSAGE 0.247 | 20,982,224 | 1,000 | 0.000247 | HUNCHENTOOT::COMPUTE-LENGTH 0.135 | 12,926,840 | 2,000 | 0.000067 | HUNCHENTOOT::FORMAT-ADDRESS 0.115 | 13,878,936 | 1,000 | 0.000115 | HUNCHENTOOT::ISO-TIME 0.082 | 237,656 | 1,000 | 0.000082 | HUNCHENTOOT::MAKE-SOCKET-STREAM 0.079 | 16,425,808 | 1,000 | 0.000079 | HUNCHENTOOT:RFC-1123-DATE .... This is my very dirty implementation of the dispatcher function which is shameles rip of hunchentoot's with some minor functions cleaned : (declaim (inline sendfile)) (define-alien-routine "sendfile" int (out_fd int :in) (in_fd int :in) (offset unsigned-long :in-out) (count unsigned-long :in)) (in-package :hunchentoot) (defun handle-static-sendfile (path &optional content-type) "A function which acts like a Hunchentoot handler for the file denoted by PATH but uses the sendfile call. Send a content type header corresponding to CONTENT-TYPE or \(if that is NIL) tries to determine the content type via the file's suffix." (unless (or (pathname-name path) (pathname-type path)) ;; not a file (setf (return-code) +http-bad-request+) (throw 'handler-done nil)) (unless (probe-file path) ;; does not exist (setf (return-code) +http-not-found+) (throw 'handler-done nil)) (setf (content-type) (or content-type (mime-type path) "application/octet-stream")) (with-open-file (file path :direction :input :element-type '(unsigned-byte 8) :if-does-not-exist nil) (setf (content-length) (file-length file)) (let ((out (chunga:chunked-stream-stream (flex:flexi-stream-stream *hunchentoot-stream*))) (in (sb-sys:fd-stream-fd file))) (send-headers) (force-output out) (sendthefile (sb-sys:fd-stream-fd out) in 0 (content-length))))) (defun sendthefile (out in off size) (common-lisp-user::sendfile out in off size)) Just for reference, the sendfile dispatcher does some 350 req/s wherea hunchentoots does about 228 req/s on my machine and against 1199.41 req/s that lighttps can serve using the sendfile backend (data for a 993Kb jpg file with 20 concurrency level). Thanks for your time, V. Segu? From eadmund42 at gmail.com Fri Dec 21 20:57:15 2007 From: eadmund42 at gmail.com (Robert Uhl) Date: Fri, 21 Dec 2007 13:57:15 -0700 Subject: [hunchentoot-devel] url-encode doesn't encode apostrophes Message-ID: I'm using url-encode to encode strings that I use for keys (see e.g. ; the brewer and beer name strings are both Unicode/HTML and contain characters meaningful in a URL). While running my pages through the w3c validator, I discovered that url-encode doesn't encode apostrophes ('), so now I run my strings through (cl-who:encode-string (hunchentoot:url-encode string)); is this the appropriate way to do things? It might be: url-encode turns a string into a string suitable for a URL and encode-string turns a string into a string suitable for an HTML attribute value. Still, it seems a bit...complex. -- Robert Uhl I'm proud to be an old-fashioned bigoted unixoid. They'll take my keyboard away from me when they pry it from the shattered skulls of my enemies. --Mark Hughes From ctdean at sokitomi.com Fri Dec 21 21:24:00 2007 From: ctdean at sokitomi.com (Chris Dean) Date: Fri, 21 Dec 2007 13:24:00 -0800 Subject: [hunchentoot-devel] url-encode doesn't encode apostrophes In-Reply-To: (Robert Uhl's message of "Fri, 21 Dec 2007 13:57:15 -0700") References: Message-ID: > While running my pages through the w3c validator, I discovered that > url-encode doesn't encode apostrophes ('), so now I run my strings > through (cl-who:encode-string (hunchentoot:url-encode string)); is this > the appropriate way to do things? I'm trying to follow along here, but I wasn't aware of cl-who had an encode-string function. What version are you using? Second question: why does the apostrophe need to be encoded? Is it that the w3c validator wants it to be? Cheers, Chris Dean From edi at agharta.de Sat Dec 22 02:50:34 2007 From: edi at agharta.de (Edi Weitz) Date: Sat, 22 Dec 2007 03:50:34 +0100 Subject: [hunchentoot-devel] Hunchentoot performance In-Reply-To: (vseguip@gmail.com's message of "Fri, 21 Dec 2007 16:58:57 +0100") References: Message-ID: Hi, On Fri, 21 Dec 2007 16:58:57 +0100, vseguip at gmail.com wrote: > First, as a disclaimer, I'm a total Lisp noobie and have been > playing with it recently poking here and there. I just tried > Hunchentoot and was happy to see how easy it is to use. I also tried > to make some preformance benchmarks on static files and compared it > with lighttpd. Unsurprisingly lighttpd beat Hunchentoot quite > badly, but I see there is still a lot of improvement for > hunchentoot. I did some profiling and added a specialized handler > that would serve static files using the linux sendfile system call > which really improved the performance. Anyway I still see plenty > room for improvements and I would like to take a stab at it, so I > did some profiling. Thanks for the code and the benchmarks, but currently I have no plans to integrate something like this into Hunchentoot. First, the technical reasons: 1. This is a Linux-only feature, so it won't buy you anything on Windows or OS X. I hear FreeBSD has something similar, but you'll almost certainly need different code to cater to FreeBSD as well. Furthermore, from a quick glance at the first Google results for "sendfile" it is not clear to me if this will work with all filesystems supported by Linux and if it will work if you use a very old Linux installation. 2. Adding this to Hunchentoot in a reasonable way would imply adding FFI code for all supported Lisp implementations - or adding another dependency like CFFI. That's a lot of complexity for such a small improvement. But, more importantly, the "philosophical" reason: Do you have a real-world application that you'd like to use Hunchentoot for where you actually expect more than 250 requests per second? If so, it seems that you have a very successful business and I'd be happy to sign a support contract with you and add whatever feature you want to Hunchentoot. If not, I'd rather concentrate on bugfixes, interesting new features, and ease of use. Improving the speed of static file serving on Linux has a very low priority for me and this will only change if someone convinces me that they can't use Hunchentoot for their project because of this particular omission. Cheers, Edi. From edi at agharta.de Sat Dec 22 03:03:53 2007 From: edi at agharta.de (Edi Weitz) Date: Sat, 22 Dec 2007 04:03:53 +0100 Subject: [hunchentoot-devel] url-encode doesn't encode apostrophes In-Reply-To: (Robert Uhl's message of "Fri, 21 Dec 2007 13:57:15 -0700") References: Message-ID: On Fri, 21 Dec 2007 13:57:15 -0700, Robert Uhl wrote: > I'm using url-encode to encode strings that I use for keys (see > e.g. ; > the brewer and beer name strings are both Unicode/HTML and contain > characters meaningful in a URL). While running my pages through the > w3c validator, I discovered that url-encode doesn't encode > apostrophes ('), so now I run my strings through > (cl-who:encode-string (hunchentoot:url-encode string)); is this the > appropriate way to do things? > > It might be: url-encode turns a string into a string suitable for a > URL and encode-string turns a string into a string suitable for an > HTML attribute value. Still, it seems a bit...complex. It seems complex, but it is the right way to do it. FWIW, the CL-WHO function is called ESCAPE-STRING, not ENCODE-STRING, and that shows its intent more clearly. As you said, you URL-encode the string to make it suitable for a URL. You might want to use the result for a header value in an HTTP reply, or as a URL you're giving to a client like Drakma. That's fine. But if you want to put the URL-encoded string into an HTML page, then the HTML rules apply, and you might have to escape the string in order not to create conflicts. That's life... :) Still, I'm surprised to see parts like "r%26rsquo%3Bs" in your example URL. With recent Hunchentoot and CL-WHO I get this: CL-USER 4 > (hunchentoot:url-encode "Fuller's") "Fuller's" CL-USER 5 > (cl-who:escape-string *) "Fuller's" And if I put a link like click me into an HTML file, then Firefox will go to http://weitz.de/foo?a=Fuller's if I click the link. Your example almost looks as if you have it the other way around. Edi. From edi at agharta.de Sat Dec 22 03:29:34 2007 From: edi at agharta.de (Edi Weitz) Date: Sat, 22 Dec 2007 04:29:34 +0100 Subject: [hunchentoot-devel] url-encode doesn't encode apostrophes In-Reply-To: (Edi Weitz's message of "Sat, 22 Dec 2007 04:03:53 +0100") References: Message-ID: On Sat, 22 Dec 2007 04:03:53 +0100, Edi Weitz wrote: > FWIW, the CL-WHO function is called ESCAPE-STRING And, I forgot to mention, you don't need CL-WHO for this: http://weitz.de/hunchentoot/#escape-for-html Edi. From eadmund42 at gmail.com Sat Dec 22 04:30:53 2007 From: eadmund42 at gmail.com (Robert Uhl) Date: Fri, 21 Dec 2007 21:30:53 -0700 Subject: [hunchentoot-devel] url-encode doesn't encode apostrophes References: Message-ID: Edi Weitz writes: > > It seems complex, but it is the right way to do it. FWIW, the CL-WHO > function is called ESCAPE-STRING, not ENCODE-STRING, and that shows > its intent more clearly. Doh! Typing from memory. Because of course with a computer capable of billions of operations per second it's just too painful to look up these minor details *grin* > As you said, you URL-encode the string to make it suitable for a URL. > You might want to use the result for a header value in an HTTP reply, > or as a URL you're giving to a client like Drakma. That's fine. But > if you want to put the URL-encoded string into an HTML page, then the > HTML rules apply, and you might have to escape the string in order not > to create conflicts. That's life... :) Yeah, it makes sense--just a bit surprising I guess. > Still, I'm surprised to see parts like "r%26rsquo%3Bs" in your example > URL. With recent Hunchentoot and CL-WHO I get this: > > CL-USER 4 > (hunchentoot:url-encode "Fuller's") > "Fuller's" > > CL-USER 5 > (cl-who:escape-string *) > "Fuller's" The '’' is part of the original string (for hysterical raisins all strings in the database are HTML strings); it then gets encoded into %26rsquo%3B by URL-ENCODE; OTOH when I have an ASCII apostrophe (_not_ a right single quote) then it gets passed as-is by URL-ENCODE but gets escaped by ESCAPE-STRING. -- Robert Uhl OTOH there are some of us in .UK who would rather like NAFTA to become the North Atlantic Free-Trade Area, and for .UK to sign up to it instead of us piddling around on the periphery of some wannabe second-rate-superpower European bureaucracy that, in all honesty, couldn't come to a consensus on how many primary colours there are. --Tanuki From eadmund42 at gmail.com Sat Dec 22 04:32:43 2007 From: eadmund42 at gmail.com (Robert Uhl) Date: Fri, 21 Dec 2007 21:32:43 -0700 Subject: [hunchentoot-devel] url-encode doesn't encode apostrophes References: Message-ID: Edi Weitz writes: > > And, I forgot to mention, you don't need CL-WHO for this: > > http://weitz.de/hunchentoot/#escape-for-html Ah, thanks. I'm using CL-WHO and Hunchentoot, but in this particular place within my code it's a lot more logical to use Hunchentoot. I hadn't found ESCAPE-FOR-HTML because I was searching for 'encode' rather than 'escape' (silly me). Thanks again for the good work. I couldn't have gotten where I am without it. -- Robert Uhl ...benefits arising from moderate use of liquor have been experienced in all armies and are not to be disputed. --George Washington, writing to John Hancock From edi at agharta.de Sat Dec 22 09:26:57 2007 From: edi at agharta.de (Edi Weitz) Date: Sat, 22 Dec 2007 10:26:57 +0100 Subject: [hunchentoot-devel] url-encode doesn't encode apostrophes In-Reply-To: (Robert Uhl's message of "Fri, 21 Dec 2007 21:30:53 -0700") References: Message-ID: On Fri, 21 Dec 2007 21:30:53 -0700, Robert Uhl wrote: > OTOH when I have an ASCII apostrophe (_not_ a right single quote) > then it gets passed as-is by URL-ENCODE Yes, that's correct behaviour. You can test it here: http://www.albionresearch.com/misc/urlencode.php Edi. From hutch at recursive.ca Sat Dec 22 15:23:19 2007 From: hutch at recursive.ca (Bob Hutchison) Date: Sat, 22 Dec 2007 10:23:19 -0500 Subject: [hunchentoot-devel] Hunchentoot performance In-Reply-To: References: Message-ID: On 21-Dec-07, at 10:58 AM, vseguip at gmail.com wrote: > Hi all, > First, as a disclaimer, I'm a total Lisp noobie and have been playing > with it recently poking here and there. I just tried Hunchentoot and > was happy to see how easy it is to use. I also tried to make some > preformance benchmarks on static files and compared it with lighttpd. > Unsurprisingly lighttpd beat Hunchentoot quite badly, but I see there > is still a lot of improvement for hunchentoot. I did some profiling > and added a specialized handler that would serve static files using > the linux sendfile system call which really improved the performance. > Anyway I still see plenty room for improvements and I would like to > take a stab at it, so I did some profiling. I'm putting the results in > the ML in case someone wants to discuss it and help me figure out > whether the speed for this use case can be improved. Details about the > benchmark setup follow: Have you looked at using the x-sendfile header? To make this work you would run hunchentoot behind lighttpd (or apache). I use apache and a reverse-proxy setup for my lisp and ruby stuff and use this header routinely. Nginx has the same functionality but it is invoked differently (or it used to be). It is easy to use and so a benchmark shouldn't be hard. Cheers, Bob ---- Bob Hutchison -- tumblelog at http://www.recursive.ca/so/ Recursive Design Inc. -- weblog at http://www.recursive.ca/hutch http://www.recursive.ca/ -- works on http://www.raconteur.info/cms-for-static-content/home/ From vseguip at gmail.com Sat Dec 22 17:03:00 2007 From: vseguip at gmail.com (vseguip at gmail.com) Date: Sat, 22 Dec 2007 18:03:00 +0100 Subject: [hunchentoot-devel] Hunchentoot performance In-Reply-To: References: Message-ID: Hi Edi, Thanks for your quick reply. > Thanks for the code and the benchmarks, but currently I have no plans > to integrate something like this into Hunchentoot. That was not my intention (I would have sent it as a patch using your coding conventions) and it wouldn't make any sense either since it doesn't need to be integrated in Hunchentoot. > > But, more importantly, the "philosophical" reason: > > Do you have a real-world application that you'd like to use > Hunchentoot for where you actually expect more than 250 requests per > second? If so, it seems that you have a very successful business and > I'd be happy to sign a support contract with you and add whatever > feature you want to Hunchentoot. If not, I'd rather concentrate on > bugfixes, interesting new features, and ease of use. Well if I had a successful business requiring that load I would just go along with lighttpd for static files :). My intention is just to learn Lisp, and when I try to learn any language I try to read existing code bases and try to see if I can hack them to do something, be it improving performance, adding features etc. I think I didn't express my intentions well in my opening post so it's really my fault. So I will try to make more concrete questions: why are functions like get-request-data or write-header-line more time consuming the sending the whole file (even when not using "sendfile")? I was trying to seek an explanation for that since I doesn't seem to make much sense from just reading the code, and maybe, just maybe, try to improve performance (I am guessig that those functions are also being called in dynamic requests so it could help there too) as a learning exercise. Cheers, V. Segu? @Bob: thanks a lot for your hints, but I was just doing this as an educational exercise in my quest to learn Common Lisp. I do however keep in mind that information in case I ever have the need for it. From sohail at taggedtype.net Sun Dec 23 21:36:43 2007 From: sohail at taggedtype.net (Sohail Somani) Date: Sun, 23 Dec 2007 21:36:43 +0000 (UTC) Subject: [hunchentoot-devel] Sessions not secure? Message-ID: Hi, In the documentation for Hunchentoot it says that start-session either uses a cookie or url rewriting to maintain the session state. Then it goes on to say: Hunchentoot makes some reasonable effort to prevent eavesdroppers from hijacking sessions (see below), but this should not be considered really secure. Don't store sensitive data in sessions and rely solely on the session mechanism as a safeguard against malicious users who want to get at this data! What is it referring to when it says "see below?" If we assume malicious users can get at the data, is this different from session hijacking? I am assuming (maybe incorrectly) that the following sequence is as secure as the user name and password. * User connects via https (no session exists) * Since no session exists, user is redirected to some login page * User enters user name and password and submits form * This information is authenticated and a session is created * (setf hunchentoot:*session* (start-session)) * On subsequent requests, since the user has a session, we can assume they are authenticated.: * (unless hunchentoot:*session* (redirect-to-login-page)) I guess it all hinges on start-session and how it generates the session id's. Thanks. -- Sohail Somani http://uint32t.blogspot.com From sohail at taggedtype.net Sun Dec 23 21:58:53 2007 From: sohail at taggedtype.net (Sohail Somani) Date: Sun, 23 Dec 2007 21:58:53 +0000 (UTC) Subject: [hunchentoot-devel] Sessions not secure? References: Message-ID: [fixing formatting, I hope] Hi, In the documentation for Hunchentoot it says that start-session either uses a cookie or url rewriting to maintain the session state. Then it goes on to say: Hunchentoot makes some reasonable effort to prevent eavesdroppers from hijacking sessions (see below), but this should not be considered really secure. Don't store sensitive data in sessions and rely solely on the session mechanism as a safeguard against malicious users who want to get at this data! What is it referring to when it says "see below?" If we assume malicious users can get at the data, is this different from session hijacking? I am assuming (maybe incorrectly) that the following sequence is as secure as the user name and password. - User connects via https (no session exists) - Since no session exists, user is redirected to some login page - User enters user name and password and submits form - This information is authenticated and a session is created: - (setf hunchentoot:*session* (start-session)) - On subsequent requests, since the user has a session, we can assume they are authenticated.: - (unless hunchentoot:*session* (redirect-to-login-page)) I guess it all hinges on start-session and how it generates the session id's. Thanks. -- Sohail Somani http://uint32t.blogspot.com From edi at agharta.de Sun Dec 23 22:04:20 2007 From: edi at agharta.de (Edi Weitz) Date: Sun, 23 Dec 2007 23:04:20 +0100 Subject: [hunchentoot-devel] Sessions not secure? In-Reply-To: (Sohail Somani's message of "Sun, 23 Dec 2007 21:36:43 +0000 (UTC)") References: Message-ID: On Sun, 23 Dec 2007 21:36:43 +0000 (UTC), Sohail Somani wrote: > In the documentation for Hunchentoot it says that start-session > either uses a cookie or url rewriting to maintain the session > state. Then it goes on to say: > > Hunchentoot makes some reasonable effort to prevent eavesdroppers > from hijacking sessions (see below), but this should not be > considered really secure. Don't store sensitive data in sessions and > rely solely on the session mechanism as a safeguard against > malicious users who want to get at this data! > > What is it referring to when it says "see below?" The explanation of the details of the session mechanism, specifically *USE-REMOTE-ADDR-FOR-SESSIONS* and *USE-USER-AGENT-FOR-SESSIONS*. > If we assume malicious users can get at the data, is this different > from session hijacking? No. > I am assuming (maybe incorrectly) that the following sequence is as > secure as the user name and password. > > * User connects via https (no session exists) > * Since no session exists, user is redirected to some login page > * User enters user name and password and submits form > * This information is authenticated and a session is created > * (setf hunchentoot:*session* (start-session)) > * On subsequent requests, since the user has a session, we can > assume they are authenticated.: > * (unless hunchentoot:*session* > (redirect-to-login-page)) Yes, that's basically how it is usually done. > I guess it all hinges on start-session and how it generates the > session id's. To hijack a session someone has to get at the session ID. That's not Hunchentoot's problem, but if you (see above) already start with https, that's certainly a good thing. The question is what you can do with a foreign session ID once you have it. How long does it last? Will it work with different browsers and/or from different IPs? And so on. Hunchentoot tries to provide a couple of options, but eventually you will have to make a decision. From edi at agharta.de Sun Dec 23 22:15:54 2007 From: edi at agharta.de (Edi Weitz) Date: Sun, 23 Dec 2007 23:15:54 +0100 Subject: [hunchentoot-devel] Hunchentoot performance In-Reply-To: (vseguip@gmail.com's message of "Sat, 22 Dec 2007 18:03:00 +0100") References: Message-ID: On Sat, 22 Dec 2007 18:03:00 +0100, vseguip at gmail.com wrote: > That was not my intention (I would have sent it as a patch using > your coding conventions) OK, that was a misunderstanding on my side then. > Well if I had a successful business requiring that load I would just > go along with lighttpd for static files :) Right, that's what I'd do as well... :) > So I will try to make more concrete questions: why are functions > like get-request-data or write-header-line more time consuming the > sending the whole file (even when not using "sendfile")? Dunno, to be honest. I never profiled them as I never really had performance problems so far. This also implies that these functions are so far in no way optimized for performance. Are they really more time consuming, BTW? I'm not familiar with the output of SBCL's profiler, so maybe you could explain that a bit. Also, I'm wondering if the methodology (sending requests via ApacheBench and then profiling within SBCL) yields correct results. From sohail at taggedtype.net Sun Dec 23 22:22:22 2007 From: sohail at taggedtype.net (Sohail Somani) Date: Sun, 23 Dec 2007 22:22:22 +0000 (UTC) Subject: [hunchentoot-devel] Sessions not secure? References: Message-ID: On Sun, 23 Dec 2007 23:04:20 +0100, Edi Weitz wrote: [snip] > > The explanation of the details of the session mechanism, specifically > *USE-REMOTE-ADDR-FOR-SESSIONS* and *USE-USER-AGENT-FOR-SESSIONS*. > [snip] > To hijack a session someone has to get at the session ID. That's not > Hunchentoot's problem, but if you (see above) already start with https, > that's certainly a good thing. The question is what you can do with a > foreign session ID once you have it. How long does it last? Will it > work with different browsers and/or from different IPs? And so on. > > Hunchentoot tries to provide a couple of options, but eventually you > will have to make a decision. Ok, thank you for your clarifications. I went through the source and noticed that the session id's are generated sequentially. So one other way that sessions can be hijacked is by guessing the session ID. Of course, this is if you do not use the IP address and user agent options that you mentioned above. Otherwise, even if you use SSL, your session can be hijacked. Hypothetically speaking, if I wanted to prevent hijacking by guessing, I could just redefine hunchentoot:get-next-session-id. Does this sound correct? Btw, great set of libraries. I don't know how you do it. My theory is that you are really three people. -- Sohail Somani http://uint32t.blogspot.com From edi at agharta.de Sun Dec 23 22:31:03 2007 From: edi at agharta.de (Edi Weitz) Date: Sun, 23 Dec 2007 23:31:03 +0100 Subject: [hunchentoot-devel] Sessions not secure? In-Reply-To: (Sohail Somani's message of "Sun, 23 Dec 2007 22:22:22 +0000 (UTC)") References: Message-ID: On Sun, 23 Dec 2007 22:22:22 +0000 (UTC), Sohail Somani wrote: > Hypothetically speaking, if I wanted to prevent hijacking by > guessing, I could just redefine hunchentoot:get-next-session-id. > > Does this sound correct? Yes, I think so. > Btw, great set of libraries. I don't know how you do it. My theory > is that you are really three people. Four... :) From avodonosov at yandex.ru Sun Dec 23 23:46:40 2007 From: avodonosov at yandex.ru (Anton Vodonosov) Date: Mon, 24 Dec 2007 01:46:40 +0200 Subject: [hunchentoot-devel] clisp port In-Reply-To: (Edi Weitz's message of "Fri\, 21 Dec 2007 11\:26\:53 +0100") References: Message-ID: Edi Weitz writes: > Hi Anton, > > On Fri, 21 Dec 2007 07:02:31 +0200, Anton Vodonosov wrote: > >> Attached is the adjustment of the clisp port suggested by Luis >> Oliveira [...] Edi, is it possible to commit it? > > that's fine with me in principle although I won't be able to test new > features against CLISP. I've tested it and if you incorporate it into new release I'll retest it on Windows and linux. > However, several of the functions in the > patch don't have documentation strings, and the HTML documentation > should be updated in such a way that it at least mentions the CLISP > port. > > http://weitz.de/patches.html Yes, indeed, I've sent something raw, sorry. Please, check the new version attached. > >> I've dropped clisp-serve-event because didn't find it useful. > > Could you elaborate on this? > Original patch by Luis Oliveira added support for both clisp and single threaded sbcl. Some common places were taken into little framework intended to be a basis for single-threaded lisp ports. SBCL version was based on sbcl's serve-event supprt. For clisp there was an emulation of serve-event - endless loop that checks all registered socket handles and if socket is "ready", handler function associated with socket is called. I.e. clisp serve-event was needed to make sbcl and clisp code compatible. As current sbcl code doesn't use serve-event, it doesn't make sense to keep serve-event emulation for clisp. Best regards -Anton -------------- next part -------------- A non-text attachment was scrubbed... Name: clisp-port.diff Type: text/x-patch Size: 15795 bytes Desc: clisp-port.diff URL: From edi at agharta.de Wed Dec 26 17:18:24 2007 From: edi at agharta.de (Edi Weitz) Date: Wed, 26 Dec 2007 18:18:24 +0100 Subject: [hunchentoot-devel] Sessions not secure? In-Reply-To: (Edi Weitz's message of "Sun, 23 Dec 2007 23:31:03 +0100") References: Message-ID: On Sun, 23 Dec 2007 23:31:03 +0100, Edi Weitz wrote: > On Sun, 23 Dec 2007 22:22:22 +0000 (UTC), Sohail Somani wrote: > >> Hypothetically speaking, if I wanted to prevent hijacking by >> guessing, I could just redefine hunchentoot:get-next-session-id. >> >> Does this sound correct? > > Yes, I think so. Er, no, actually. I've seen this mentioned in your blog http://uint32t.blogspot.com/2007/12/abusing-hunchentoots-dispatch-mechanism.html and thought about it again. So, tell me, if you happen to know for sure that my session ID is 42 and if you also know my user agent string and my IP address, how would you construct a cookie to hijack my session? Edi. From sohail at taggedtype.net Wed Dec 26 21:09:37 2007 From: sohail at taggedtype.net (Sohail Somani) Date: Wed, 26 Dec 2007 21:09:37 +0000 (UTC) Subject: [hunchentoot-devel] Sessions not secure? References: Message-ID: On Wed, 26 Dec 2007 18:18:24 +0100, Edi Weitz wrote: > On Sun, 23 Dec 2007 23:31:03 +0100, Edi Weitz wrote: > >> On Sun, 23 Dec 2007 22:22:22 +0000 (UTC), Sohail Somani >> wrote: >> >>> Hypothetically speaking, if I wanted to prevent hijacking by guessing, >>> I could just redefine hunchentoot:get-next-session-id. >>> >>> Does this sound correct? >> >> Yes, I think so. > > Er, no, actually. I've seen this mentioned in your blog > > http://uint32t.blogspot.com/2007/12/abusing-hunchentoots-dispatch- mechanism.html > > and thought about it again. So, tell me, if you happen to know for sure > that my session ID is 42 and if you also know my user agent string and > my IP address, how would you construct a cookie to hijack my session? I lied. At the time of my post, I hadn't looked at enough of the code to make that judgement. The thing I was worried about is that (I thought) all the information that went into the md5sum'ed ID was deterministic. I thought it looked like: * (concatenate 'string id user-agent ip-address) All that is needed to make this work is to spoof the ip-address and user- agent, which is not trivial, but not hard either. In reality, it looks like this: * (concatenate 'string *session-secret* id user-agent ip-address time-of-session-start) Since the *session-secret* is generated when the server starts and so is presumably random, it is somewhat secure (but I don't know enough about the Lisp random number generators to say.) Hopefully it isn't as pseudo- random as the C rand function which would more accurately be called rand_for_when_you_dont_need_rand. Thanks for your time and thanks for making me clarify. I will update my post to that effect. By the way, since you read the post, do you think it is use or abuse of the dispatch table? -- Sohail Somani http://uint32t.blogspot.com From edi at agharta.de Wed Dec 26 21:21:44 2007 From: edi at agharta.de (Edi Weitz) Date: Wed, 26 Dec 2007 22:21:44 +0100 Subject: [hunchentoot-devel] Sessions not secure? In-Reply-To: (Sohail Somani's message of "Wed, 26 Dec 2007 21:09:37 +0000 (UTC)") References: Message-ID: On Wed, 26 Dec 2007 21:09:37 +0000 (UTC), Sohail Somani wrote: > In reality, it looks like this: > > * (concatenate 'string *session-secret* > id user-agent ip-address time-of-session-start) And don't forget MD5. Even if the random number generator were weak, you'd have a hard time to figure out where in the random sequence you are, right? > but I don't know enough about the Lisp random number generators to > say. This is obviously implementation-dependent. Some Lisp implementations also offer more choices for random number generators, for example: http://www.lispworks.com/documentation/lw50/LWRM/html/lwref-326.htm > By the way, since you read the post, do you think it is use or abuse > of the dispatch table? I think it's a valid use. I've been doing things like that as well. Edi. From sohail at taggedtype.net Wed Dec 26 21:38:46 2007 From: sohail at taggedtype.net (Sohail Somani) Date: Wed, 26 Dec 2007 21:38:46 +0000 (UTC) Subject: [hunchentoot-devel] Sessions not secure? References: Message-ID: On Wed, 26 Dec 2007 22:21:44 +0100, Edi Weitz wrote: > On Wed, 26 Dec 2007 21:09:37 +0000 (UTC), Sohail Somani > wrote: > >> In reality, it looks like this: >> >> * (concatenate 'string *session-secret* >> id user-agent ip-address time-of-session-start) > > And don't forget MD5. Even if the random number generator were weak, > you'd have a hard time to figure out where in the random sequence you > are, right? Practically, yes. I think it still depends on the RNG and how much knowledge I have of your server setup. To me, the documentation makes it seem like there is no randomness involved. I think it should mention that there is some randomness but the quality of the security is dependent on the quality of the RNG. In any case, Hunchentoot has done as close to an optimal job as is economical. In my highly unqualified opinion of course (IMHUQO?) :-) >> but I don't know enough about the Lisp random number generators to say. > > This is obviously implementation-dependent. Some Lisp implementations > also offer more choices for random number generators, for example: > > http://www.lispworks.com/documentation/lw50/LWRM/html/lwref-326.htm Thanks! I've been meaning to try LW but SBCL is very nice to me so far :-) -- Sohail Somani http://uint32t.blogspot.com From boldyrev+nospam at cgitftp.uiggm.nsc.ru Wed Dec 26 16:27:40 2007 From: boldyrev+nospam at cgitftp.uiggm.nsc.ru (Ivan Boldyrev) Date: Wed, 26 Dec 2007 22:27:40 +0600 Subject: [hunchentoot-devel] Hunchentoot performance References: Message-ID: On 10060 day of my life vseguip at gmail.com wrote: > I did some profiling... My profiling shows that on SBCL most "slowness" comes from flexi-streams. For example, such low-level method as flexi-streams::unread-char% calls flexi-streams:translate-char that uses with-output-to-sequence. And last macro generates code that create new instance of flexi-stream, and making instances conses very much on SBCL. You may blame either SBCL or flexi-streams (I tend to blame flexi-streams). unread-char (and unread-char%) is called by chunga quite often. -- Ivan Boldyrev Is 'morning' a gerund? From vseguip at gmail.com Thu Dec 27 22:58:44 2007 From: vseguip at gmail.com (vseguip at gmail.com) Date: Thu, 27 Dec 2007 23:58:44 +0100 Subject: [hunchentoot-devel] Hunchentoot performance In-Reply-To: References: Message-ID: On Dec 26, 2007 5:27 PM, Ivan Boldyrev wrote: > On 10060 day of my life vseguip at gmail.com wrote: > > I did some profiling... > > My profiling shows that on SBCL most "slowness" comes from > flexi-streams. For example, such low-level method as > flexi-streams::unread-char% calls flexi-streams:translate-char that uses > with-output-to-sequence. And last macro generates code that create new > instance of flexi-stream, and making instances conses very much on SBCL. > You may blame either SBCL or flexi-streams (I tend to blame > flexi-streams). Yes, I profiled flexi-streams and chunga and it showed as much, even when using it only for the headers. Is it possible to create a server that doesn't use flexi-streams at all, or can a handler-request actually configure if it want's to use flexi-streams? I know it's possible to bypass it in the handler, but I think it's called anyway in process-connection, get-request-data, etc. In other words, what would I have to mess around in hunchentoot so I can create a flexi-less handler/server? Cheers, V. Segu? (*) Note that even when serving an empty file hunchentoot only reaches ~ 287 requests/second on my system, so dynamic pages will "suffer" from this also. From edi at agharta.de Sat Dec 29 17:58:41 2007 From: edi at agharta.de (Edi Weitz) Date: Sat, 29 Dec 2007 18:58:41 +0100 Subject: [hunchentoot-devel] New release 0.15.0 (Was: clisp port) In-Reply-To: (Anton Vodonosov's message of "Mon, 24 Dec 2007 01:46:40 +0200") References: Message-ID: On Mon, 24 Dec 2007 01:46:40 +0200, Anton Vodonosov wrote: > Please, check the new version attached. Thanks, I've made a new release which incorporates your changes. Edi. From edi at agharta.de Sat Dec 29 23:22:38 2007 From: edi at agharta.de (Edi Weitz) Date: Sun, 30 Dec 2007 00:22:38 +0100 Subject: [hunchentoot-devel] Hunchentoot performance In-Reply-To: (Ivan Boldyrev's message of "Wed, 26 Dec 2007 22:27:40 +0600") References: Message-ID: On Wed, 26 Dec 2007 22:27:40 +0600, Ivan Boldyrev wrote: > For example, such low-level method as flexi-streams::unread-char% > calls flexi-streams:translate-char that uses > with-output-to-sequence. Can you check with the latest release (0.14.0) of FLEXI-STREAMS and see if it makes a difference for you? > And last macro generates code that create new instance of > flexi-stream, and making instances conses very much on SBCL. What "last macro" do you mean? Or are you referring to WITH-OUTPUT-TO-SEQUENCE? Edi. From avodonosov at yandex.ru Sun Dec 30 05:23:48 2007 From: avodonosov at yandex.ru (Anton Vodonosov) Date: Sun, 30 Dec 2007 07:23:48 +0200 Subject: [hunchentoot-devel] New release 0.15.0 (Was: clisp port) In-Reply-To: (Edi Weitz's message of "Sat\, 29 Dec 2007 18\:58\:41 +0100") References: Message-ID: Edi Weitz writes: > Thanks, I've made a new release which incorporates your changes. Tested new release with clisp on windows and linux - works OK. Thanks, -Anton From vseguip at gmail.com Sun Dec 30 13:38:56 2007 From: vseguip at gmail.com (vseguip at gmail.com) Date: Sun, 30 Dec 2007 14:38:56 +0100 Subject: [hunchentoot-devel] Hunchentoot performance In-Reply-To: References: Message-ID: On Dec 30, 2007 12:22 AM, Edi Weitz wrote: > On Wed, 26 Dec 2007 22:27:40 +0600, Ivan Boldyrev wrote: > > > For example, such low-level method as flexi-streams::unread-char% > > calls flexi-streams:translate-char that uses > > with-output-to-sequence. > > Can you check with the latest release (0.14.0) of FLEXI-STREAMS and > see if it makes a difference for you? > Hi Edi, It definitely makes a difference, get-request-data certainly is not on the top list anymore but write-header-line still tops right there (flexi-streams 0.14) using sendfile system call: ab -n 1000 -k -c 1 http://localhost:8000/test2.jpg seconds | consed | calls | sec/call | name ------------------------------------------------------------- 0.475 | 199,787,592 | 6,006 | 0.000079 | WRITE-HEADER-LINE 0.276 | 59,215,472 | 1,001 | 0.000276 | LOG-MESSAGE 0.140 | 0 | 1,001 | 0.000139 | SENDTHEFILE It looks like write-header-line still takes an inordinate amount of time for no good reason so I changed it to first build a string and the output it to the stream: (defmethod write-header-line ((mod-lisp-p (eql nil)) key value) "Accepts strings KEY and VALUE and writes them directly to the client as an HTTP header line." (let ((header-lines (with-output-to-string (str) (write-string key str) (write-string ": " str) ;; remove line breaks (write-string (regex-replace-all "[\\r\\n]" value " ") str) (write-string +crlf+ str)))) (write-sequence header-lines *hunchentoot-stream*))) This seemed to work very well, and now write-header-line is down to tens of millisecond range, (maybe using write-string on the *hunchentoot-stream* was causing a flush on the socket each time?): seconds | consed | calls | sec/call | name ------------------------------------------------------------- 0.256 | 57,464,072 | 1,001 | 0.000256 | LOG-MESSAGE 0.176 | 0 | 1,001 | 0.000175 | SENDTHEFILE 0.089 | 11,123,696 | 277,277 | 0.0000003 | FLEXI-STREAMS::CHAR-TO-OCTETS 0.084 | 16,824,960 | 1,001 | 0.000083 | RFC-1123-DATE 0.080 | 14,609,960 | 1,001 | 0.000079 | ISO-TIME 0.071 | 818,104 | 2,005 | 0.000035 | FLEXI-STREAMS::SET-CLASS 0.047 | 12,264 | 1,002 | 0.000047 | FORCE-OUTPUT* 0.046 | 0 | 277,277 | 0.0000002 | FLEXI-STREAMS::WRITE-BYTE* 0.035 | 0 | 130,130 | 0.0000003 | (SETF FLEXI-STREAMS::FLEXI-STREAM-LAST-CHAR-CODE) 0.032 | 1,514,000 | 1,001 | 0.000031 | SERVER-NAME-HEADER Just for the sake of it, I redefined log-messsage to a "do-nothing" function and this is what I got: seconds | consed | calls | sec/call | name ------------------------------------------------------------- 0.192 | 0 | 1,001 | 0.000191 | SENDTHEFILE 0.139 | 12,404,872 | 1,001 | 0.000139 | HANDLE-STATIC-SENDFILE 0.103 | 4,741,480 | 2,005 | 0.000051 | FLEXI-STREAMS::SET-CLASS 0.100 | 16,327,208 | 1,001 | 0.000099 | RFC-1123-DATE 0.097 | 18,786,488 | 126,127 | 0.000001 | FLEXI-STREAMS::READ-BYTE* 0.074 | 0 | 182,182 | 0.0000004 | FLEXI-STREAMS::WRITE-BYTE* 0.063 | 0 | 130,132 | 0.0000005 | (SETF FLEXI-STREAM-POSITION) 0.055 | 750,888 | 1,002 | 0.000055 | FORCE-OUTPUT* 0.053 | 50,860,872 | 6,006 | 0.000009 | WRITE-HEADER-LINE Now it seems like rfc-1123-date could do with some optimization, but all in all I'm getting 465 req/sec wich is a very nice performance improvement. Cheers, V. Segu? From edi at agharta.de Sun Dec 30 17:24:22 2007 From: edi at agharta.de (Edi Weitz) Date: Sun, 30 Dec 2007 18:24:22 +0100 Subject: [hunchentoot-devel] Hunchentoot performance In-Reply-To: (vseguip@gmail.com's message of "Sun, 30 Dec 2007 14:38:56 +0100") References: Message-ID: On Sun, 30 Dec 2007 14:38:56 +0100, vseguip at gmail.com wrote: > It looks like write-header-line still takes an inordinate amount of > time for no good reason so I changed it to first build a string and > the output it to the stream OK, that seems like an easy fix. I'll integrate that into the next release. Thanks. > Now it seems like rfc-1123-date could do with some optimization You're on SBCL? Try to benchmark RFC-1123-DATE with pretty-printing turned off. > but all in all I'm getting 465 req/sec wich is a very nice > performance improvement. Right. All you need now is a website with 400 visitors per second... :) Edi. From vseguip at gmail.com Sun Dec 30 17:55:22 2007 From: vseguip at gmail.com (vseguip at gmail.com) Date: Sun, 30 Dec 2007 18:55:22 +0100 Subject: [hunchentoot-devel] Hunchentoot performance In-Reply-To: References: Message-ID: Hi Edi, > You're on SBCL? Try to benchmark RFC-1123-DATE with pretty-printing > turned off. That sure helped, I'm now in he 510~550 req/sec range. Anyway, is there a more elegant way to turn logging off than simply redefining the method? I think the fact that log-message is locking when logging to a file makes it take a sever performance hit, maybe the use of a logging framework that solves this would be nice. > Right. All you need now is a website with 400 visitors per > second... :) > That's a piece of cake, just run apache-bench in a forever loop :). On a side note, I wanted to comment some refactoring to hunchentoot that I think may make sense. Right now process-connection is a function that takes care of reading from the client, be it via mod_lisp or directly, binding the chunga/flexi-streams if needed and dispatching it. I think a better design would be to make process-connection a generic method on class connection and then implementing methods for mod_lisp/standalone/fastcgi/whatever. Chunga and flexi-streams wrapping could be implemented as mixins so the user can choose to create a FCGI connection with flexi but not chunga, etc. With this we can allow for custom backends to be developed by users so we gain flexibility (and maintainability) at the cost of some overhead when dispatching. What do you think? Cheers, V Segui From edi at agharta.de Sun Dec 30 18:47:06 2007 From: edi at agharta.de (Edi Weitz) Date: Sun, 30 Dec 2007 19:47:06 +0100 Subject: [hunchentoot-devel] Hunchentoot performance In-Reply-To: (vseguip@gmail.com's message of "Sun, 30 Dec 2007 18:55:22 +0100") References: Message-ID: On Sun, 30 Dec 2007 18:55:22 +0100, vseguip at gmail.com wrote: > Anyway, is there a more elegant way to turn logging off than simply > redefining the method? Sure. There are several ways to control this: http://weitz.de/hunchentoot/#log You could for example turn access log messages off. Of course, once you have your 300-visitors-per-second website, you probably /want/ logging... :) > I think the fact that log-message is locking when logging to a file > makes it take a sever performance hit, maybe the use of a logging > framework that solves this would be nice. Yes, maybe. Is there a mature, stable, well-documented logging framework out there that works with all Lisps supported by Hunchentoot? How would such a framework deal with several threads trying to log at the same time if not with locking? One could also think of a setup where there's one dedicated logging process and other processes write to the log file by sending objects to its mailbox: http://cl-cookbook.sourceforge.net/process.html#mailbox (Or use cl-muproc? Maybe a bit of overkill.) But that would cancel the support for non-threaded Lisps we have now and it would introduce more implementation-dependent code. Anyway - not a big priority for me. If it's important for you, I'd suggest you make a detailed proposal, we discuss it here, then you send a clean patch. > On a side note, I wanted to comment some refactoring to hunchentoot > that I think may make sense. Right now process-connection is a > function that takes care of reading from the client, be it via > mod_lisp or directly, binding the chunga/flexi-streams if needed and > dispatching it. I think a better design would be to make > process-connection a generic method on class connection and then > implementing methods for > mod_lisp/standalone/fastcgi/whatever. Chunga and flexi-streams > wrapping could be implemented as mixins so the user can choose to > create a FCGI connection with flexi but not chunga, etc. With this > we can allow for custom backends to be developed by users so we gain > flexibility (and maintainability) at the cost of some overhead when > dispatching. What do you think? I'm always open for changes which improve flexibility and maintainability and at first sight your proposal sounds fine to me. If you want to implement that, I'd suggest to proceed like I described above. Edi. From avodonosov at yandex.ru Sun Dec 30 20:11:41 2007 From: avodonosov at yandex.ru (Anton Vodonosov) Date: Sun, 30 Dec 2007 22:11:41 +0200 Subject: [hunchentoot-devel] Hunchentoot performance In-Reply-To: (Edi Weitz's message of "Sun\, 30 Dec 2007 18\:24\:22 +0100") References: Message-ID: Edi Weitz writes: > Right. All you need now is a website with 400 visitors per > second... :) > I totally agree with the position that bugfixes, interesting new features, and ease of use is the main priority. On the other hand I glad to hear about this performance improvement. It is not only about 400 visitors per second, but also means less resource consumption and possibility to run web server on smaller machine, for example cheap VPN. Also, if we are serving pages with frames, images etc, even several visitors may produce many requests, and with performance improvement they will have better response time. Several people mentioned that they were surprised by low hunchentoot performance: http://abstractstuff.livejournal.com/2007/06/28/ http://www.newartisans.com/blog_files/lisp.web.servers.php If you remember, I too tested hunchentoot by ApacheBench when running it in SBCL on a VPN with 64 MB of memory and SBCL was killed by linux because of out-of-memory. But now, we see that simple fix significantly improve performance, that proves robust design of hunchentoot. So it's good. Sorry for the long letter. Happy new year, -- Anton From rsynnott at gmail.com Sun Dec 30 23:05:03 2007 From: rsynnott at gmail.com (Robert Synnott) Date: Sun, 30 Dec 2007 23:05:03 +0000 Subject: [hunchentoot-devel] Hunchentoot performance In-Reply-To: References: Message-ID: <24f203480712301505g1a43ef64k591d1ee564eda81d@mail.gmail.com> On Dec 30, 2007 6:47 PM, Edi Weitz wrote: > On Sun, 30 Dec 2007 18:55:22 +0100, vseguip at gmail.com wrote: > > > > I think the fact that log-message is locking when logging to a file > > makes it take a sever performance hit, maybe the use of a logging > > framework that solves this would be nice. > > Yes, maybe. Is there a mature, stable, well-documented logging > framework out there that works with all Lisps supported by > Hunchentoot? How would such a framework deal with several threads > trying to log at the same time if not with locking? > > One could also think of a setup where there's one dedicated logging > process and other processes write to the log file by sending objects > to its mailbox: > > http://cl-cookbook.sourceforge.net/process.html#mailbox > > (Or use cl-muproc? Maybe a bit of overkill.) > > But that would cancel the support for non-threaded Lisps we have now > and it would introduce more implementation-dependent code. > One file per thread might work, though I suppose it would break down if you had many thousands of threads... Of course, if you were running behind lighttpd or similar, you could have it doing request logging. As I (unfortunately :) ) don't see anything like 400 requests per second to my personal projects, this isn't something I'd really be interested in looking into too much myself. Rob From edi at agharta.de Sun Dec 30 23:18:29 2007 From: edi at agharta.de (Edi Weitz) Date: Mon, 31 Dec 2007 00:18:29 +0100 Subject: [hunchentoot-devel] Hunchentoot performance In-Reply-To: <24f203480712301505g1a43ef64k591d1ee564eda81d@mail.gmail.com> (Robert Synnott's message of "Sun, 30 Dec 2007 23:05:03 +0000") References: <24f203480712301505g1a43ef64k591d1ee564eda81d@mail.gmail.com> Message-ID: On Sun, 30 Dec 2007 23:05:03 +0000, "Robert Synnott" wrote: > One file per thread might work Well, who wants to read all these log files then? > Of course, if you were running behind lighttpd or similar, you could > have it doing request logging. That's what already happens (with Apache) if you're using mod_lisp. From edi at agharta.de Sun Dec 30 23:24:04 2007 From: edi at agharta.de (Edi Weitz) Date: Mon, 31 Dec 2007 00:24:04 +0100 Subject: [hunchentoot-devel] Hunchentoot performance In-Reply-To: (Edi Weitz's message of "Sun, 30 Dec 2007 19:47:06 +0100") References: Message-ID: On Sun, 30 Dec 2007 19:47:06 +0100, Edi Weitz wrote: > Yes, maybe. Is there a mature, stable, well-documented logging > framework out there that works with all Lisps supported by > Hunchentoot? How would such a framework deal with several threads > trying to log at the same time if not with locking? > > One could also think of a setup where there's one dedicated logging > process and other processes write to the log file by sending objects > to its mailbox: > > http://cl-cookbook.sourceforge.net/process.html#mailbox > > (Or use cl-muproc? Maybe a bit of overkill.) > > But that would cancel the support for non-threaded Lisps we have now > and it would introduce more implementation-dependent code. Another option would be to make the current way of logging the default but to let the user bypass and/or change this behaviour if he so wishes. For example by making LOG-MESSAGE a generic function. Oh, wait, it /is/ already a generic function. And the documentation says: "LOG-MESSAGE is a generic function, so you can specialize it or bypass it completely with an around method." Well... :) From boldyrev+nospam at cgitftp.uiggm.nsc.ru Mon Dec 31 09:58:09 2007 From: boldyrev+nospam at cgitftp.uiggm.nsc.ru (Ivan Boldyrev) Date: Mon, 31 Dec 2007 15:58:09 +0600 Subject: [hunchentoot-devel] Hunchentoot performance References: Message-ID: On 10069 day of my life Edi Weitz wrote: > Can you check with the latest release (0.14.0) of FLEXI-STREAMS and > see if it makes a difference for you? I see something strange: no difference for unibyte windows-1251, but improvement for utf-8. I will do proper profiling and statistics later, sorry. Happy New Year! :) >> And last macro generates code that create new instance of >> flexi-stream, and making instances conses very much on SBCL. > > What "last macro" do you mean? Or are you referring to > WITH-OUTPUT-TO-SEQUENCE? Yes. -- Ivan Boldyrev Today is the first day of the rest of your life. From edi at agharta.de Mon Dec 31 10:45:58 2007 From: edi at agharta.de (Edi Weitz) Date: Mon, 31 Dec 2007 11:45:58 +0100 Subject: [hunchentoot-devel] Hunchentoot performance In-Reply-To: (Ivan Boldyrev's message of "Mon, 31 Dec 2007 15:58:09 +0600") References: Message-ID: On Mon, 31 Dec 2007 15:58:09 +0600, Ivan Boldyrev wrote: > I see something strange: no difference for unibyte windows-1251, but > improvement for utf-8. Hmm, should've been the other way around. Specifically, I would expect improvements for ASCII and LATIN-1 which is what counts for Chunga's internals. Edi.