From pete-tbnl-dev at kazmier.com Tue Jul 13 20:43:22 2004 From: pete-tbnl-dev at kazmier.com (pete-tbnl-dev at kazmier.com) Date: Tue, 13 Jul 2004 16:43:22 -0400 Subject: [tbnl-devel] Last-Modified / If-Modified-Since Headers Message-ID: <20040713204322.GA32010@kazmier.com> Is there any interest in adding functionality that enables TBNL to automatically send a "304 Not Modified" in response to content that has not changed? For example, in the Java Servlet world, a servlet author can define getLastModified(). What does this buy the servlet author? Automatic client-side caching. For those not familiar with the "Last-Modified/If-Modified-Since" headers, here's a brief overview. The first time client A visits our servlet page, our servlet is called to generate some HTML output. If a servlet author has implemented getLastModified(), the servlet container invokes this method and inserts a "Last-Modified" header in the response to the client. The value of this header is a date specifying when the page was last modified. This value is cached in the client's browser. The next time client A visits our servlet page, in the HTTP request, the client's browser includes a "If-Modified-Since" header with the value that our servlet had previously sent. When our servlet receives the request, it notices the "If-Modified-Since" header, and thus invokes our servlet's getLastModified() method (if defined). When the time returned by this method is newer than the time the client sent, the servlet container proceeds to dispatch the request to our servlet for processing and generation of output. Things are more interesting when the value returned from getLastModified() is the not newer than the value of the "If-Modified-Since" header. In this case, the servlet container bypasses our servlet and simply sends a "304 Not Modified" back to the client which instructs the browser to display the cached content. It is important to note that this servlet container did not dispatch to our code to generate HTML output, it is completely bypassed and all handled by the container. This is the functionality that I would like to add to TBNL. I'm going to do it for myself even if its not part of the core; however, I thought it would be useful for others as well, and thus this post. How would one implement this within TBNL? Would the same approach the Java folks used suffice? Provide a special variable, *last-modified-handler*, that users can set to return a timestamp? From within that method, users would have access to *request* so they could return specific times for various URIs. This is what I need it for, my entire site will be served by HTML-TEMPLATEs and I'll use the last time the template file was modified as my timestamp. Thoughts? Thanks, Pete From edi at agharta.de Tue Jul 13 23:19:45 2004 From: edi at agharta.de (Edi Weitz) Date: Wed, 14 Jul 2004 01:19:45 +0200 Subject: [tbnl-devel] Last-Modified / If-Modified-Since Headers In-Reply-To: <20040713204322.GA32010@kazmier.com> (pete-tbnl-dev@kazmier.com's message of "Tue, 13 Jul 2004 16:43:22 -0400") References: <20040713204322.GA32010@kazmier.com> Message-ID: <87oemjbilq.fsf@bird.agharta.de> On Tue, 13 Jul 2004 16:43:22 -0400, pete-tbnl-dev at kazmier.com wrote: > Provide a special variable, *last-modified-handler*, that users can > set to return a timestamp? From within that method, users would > have access to *request* so they could return specific times for > various URIs. Sounds like a good idea to me. The initial value of this special variable should probably be (CONSTANTLY NIL) where NIL returned by the handler means that it declines to provide a timestamp - which would be equivalent to returning (GET-UNIVERSAL-TIME). The handlers should generally return universal-times, of course. Something like that. Would you like to send a patch? > This is what I need it for, my entire site will be served by > HTML-TEMPLATEs and I'll use the last time the template file was > modified as my timestamp. But even if the template itself hasn't changed you might fill the template with different values each time it is called, isn't that the whole purpose of HTML-TEMPLATE? Or am I missing something? Cheers, Edi. From edi at agharta.de Wed Jul 14 08:11:50 2004 From: edi at agharta.de (Edi Weitz) Date: Wed, 14 Jul 2004 10:11:50 +0200 Subject: [tbnl-devel] Last-Modified / If-Modified-Since Headers In-Reply-To: <20040714051435.GA3309@kazmier.com> (pete-tbnl-dev@kazmier.com's message of "Wed, 14 Jul 2004 01:14:35 -0400") References: <20040713204322.GA32010@kazmier.com> <87oemjbilq.fsf@bird.agharta.de> <20040714051435.GA3309@kazmier.com> Message-ID: <87fz7vovnd.fsf@bird.agharta.de> On Wed, 14 Jul 2004 01:14:35 -0400, pete-tbnl-dev at kazmier.com wrote: > Just to make sure I understand what you are saying: > > - If the user does not assign a handler to the special variable, > then TBNL should not make any assumptions on the intention of the > coder and thus not send any Last-Modified header and never send a > 304 reply in response to a client's If-Modified-Since header. Yes. The default value should be a function which always returns NIL. > - If the user does assign a handler and it returns a non-nil value, > then TBNL should use this universal-time as the value of the > Last-Modified header or the value to compare to a client's > If-Modified-Since header. > > - If the user does assign a handler and it returns a nil value, then > TBNL should not send any Last-Modified header and never send a 304 > reply in response to a client's If-Modified-Since header because the > coder has indicated that the last modified time is not known for > this request. Yes and yes. > I'd be happy to send a patch. I probably won't get to this right > away as I'm busy trying to get other parts of my new site working > and just trying to learn lisp in general. However, at some point > I'll probably need the functionality at which point I'll submit a > patch. That's fine. No need to hurry. > I've built sites that use template engines that are backed by a > "data miner" that refreshes the context used to fill templates on a > periodic basis. Web-based reporting interfaces can take advantage > of this scenario. But why then don't you serve these pages statically directly from Apache? Cheers, Edi. From pete-tbnl-dev at kazmier.com Wed Jul 14 13:39:24 2004 From: pete-tbnl-dev at kazmier.com (pete-tbnl-dev at kazmier.com) Date: Wed, 14 Jul 2004 09:39:24 -0400 Subject: [tbnl-devel] Last-Modified / If-Modified-Since Headers In-Reply-To: <87fz7vovnd.fsf@bird.agharta.de> References: <20040713204322.GA32010@kazmier.com> <87oemjbilq.fsf@bird.agharta.de> <20040714051435.GA3309@kazmier.com> <87fz7vovnd.fsf@bird.agharta.de> Message-ID: <20040714133924.GA6961@kazmier.com> On Wed, Jul 14, 2004 at 10:11:50AM +0200, Edi Weitz wrote: > > I've built sites that use template engines that are backed by a > > "data miner" that refreshes the context used to fill templates on a > > periodic basis. Web-based reporting interfaces can take advantage > > of this scenario. > > But why then don't you serve these pages statically directly from > Apache? In this specific example, the data model can be modified by other external events as well. Furthermore, there are numerous templates that provide visibility into the data model and each is customized to the user. I.e., the template contains the username, last login, etc. It isn't practical to store 'n' copies of output. In addition, each page includes a Refresh header (for 5 minutes) as these reports are meant to be used by level 1 and level 2 support teams that leave them open indefinitely. Thus, when a change occurs to the data model, we can simply use getLastModified to invalidate all of the pages stored in cache on the client side. From pete-tbnl-dev at kazmier.com Thu Jul 15 21:22:52 2004 From: pete-tbnl-dev at kazmier.com (pete-tbnl-dev at kazmier.com) Date: Thu, 15 Jul 2004 17:22:52 -0400 Subject: [tbnl-devel] Odd behavior (socket leak) Message-ID: <20040715212252.GA29894@kazmier.com> I could use a sanity check. I've been trying to track down a problem with either mod_lisp (for apache 2.0) or TBNL. Since I was not able to isolate my problem, I decided to revert back to Apache 1.3 and mod_lisp 2.33 (for Apache 1.x). The problem is that I think I'm seeing some odd behavior and was hoping someone else could confirm. In a nutshell, I added a simple print statement to APACHE-LISTEN in TBNL's modlisp.lisp file so I would know when this function was being invoked. I also added print statements before each significant line in that function so I could see what the heck was going on. Here is my version of that function: (defun apache-listen (*apache-stream* command-processor &rest args) "Listens on *APACHE-STREAM* for commands from mod_lisp. Packages the command using GET-APACHE-COMMAND and passes it to the COMMAND-PROCESSOR function \(which is PROCESS-APACHE-COMMAND). ARGS are ignored. Designed to be called by a KMRCL:LISTENER object." (declare (ignore args)) (format t ">>>>>>> Opening stream~%") (force-output) (let ((*close-apache-stream* t)) (unwind-protect (loop for *apache-socket-usage-counter* from 0 for command = (debug-value *command* (get-apache-command)) while command do (cond ((ignore-errors (format t ">>>>>>> Processing command ~A~%" (subseq command 0 3)) (force-output) (funcall command-processor command)) (ignore-errors (format t ">>>>>>> Flushing stream cuz no errors occurred for ~A~%" (subseq command 0 3)) (force-output) (force-output *apache-stream*))) (t ;; if an error occured during processing of ;; COMMAND we close this particular connection ;; to Apache (ignore-errors (format t ">>>>>>> Closing stream cuz error occurred for ~A~%" (subseq command 0 3)) (force-output) (setq *close-apache-stream* t)))) until *close-apache-stream*) (ignore-errors (format t ">>>>>>> Closing stream - no more commands~%") (force-output) (kmrcl:close-active-socket *apache-stream*))))) Now, back in Apache 1.3 world, I hit a webpage that invokes the mod_lisp handler and thus TBNL. I see the opening, processing, and flushing messages. The next time I hit the page (30 seconds later), I see the same thing again. I never see a closing message, and a look at 'netstat -an | grep 3000' indicates that another socket has been allocated and is in a connected state. I repeated this process for a few minutes and got up to 6 connected sockets without ever seeing a closing message. If a subsequent hit comes close enough, then it will reuse the socket and I'll only see a processing and flusing message (no opening message). I'm using SBCL (with multithreaded support). I'll continue debugging in the meantime and I'm going to start with fresh copies of everything to make sure I have not inadvertently affected things while debugging my original problems with Apache 2.x (which still appear when I use Apache 1.x). I'll summarize those problems as well (as I think they may be related), essentially, I have a web page that has three stylesheets included in it (and are processed by TBNL as well), so basically, TBNL receives 4 requests in a fairly short amount of time. After adding debugging code to the mod-lisp.c Apache module, I've confirmed that mod-lisp sends the 4 requests to TBNL. Unfortunately, mod-lisp never receives any headers from TBNL for the last request although I've seen TBNL send them (I just didn't realize until now that they were probably going over different sockets). In any case, I've been banging my head all day, I need a break, but it would be appreciated if someone could either confirm or deny the opening of multiple sockets while never closing any of the old ones. Its easy to see via a 'netstat -an | grep 3000' (or whatever port you run it on). Thanks, Pete From edi at agharta.de Fri Jul 16 00:40:19 2004 From: edi at agharta.de (Edi Weitz) Date: Fri, 16 Jul 2004 02:40:19 +0200 Subject: [tbnl-devel] Odd behavior (socket leak) In-Reply-To: <20040715212252.GA29894@kazmier.com> (pete-tbnl-dev@kazmier.com's message of "Thu, 15 Jul 2004 17:22:52 -0400") References: <20040715212252.GA29894@kazmier.com> Message-ID: <87wu14wzrg.fsf@bird.agharta.de> I can confirm this behaviour with AllegroCL. This is obviously not what was intended. Unfortunately, I will be out of town until next week so I won't be able to investigate this further until then. Cheers, Edi. From pete-tbnl-dev at kazmier.com Fri Jul 16 02:15:55 2004 From: pete-tbnl-dev at kazmier.com (pete-tbnl-dev at kazmier.com) Date: Thu, 15 Jul 2004 22:15:55 -0400 Subject: [tbnl-devel] Odd behavior (socket leak) In-Reply-To: <87wu14wzrg.fsf@bird.agharta.de> References: <20040715212252.GA29894@kazmier.com> <87wu14wzrg.fsf@bird.agharta.de> Message-ID: <20040716021555.GA9928@kazmier.com> On Fri, Jul 16, 2004 at 02:40:19AM +0200, Edi Weitz wrote: > I can confirm this behaviour with AllegroCL. Good! At least I'm not going crazy then!! > This is obviously not what was intended. Unfortunately, I will be out > of town until next week so I won't be able to investigate this further > until then. I'll continue poking around and see if I can find anything. Its helping me learn how to debug lisp applications! Thanks, Pete From pete-tbnl-dev at kazmier.com Fri Jul 16 14:36:02 2004 From: pete-tbnl-dev at kazmier.com (pete-tbnl-dev at kazmier.com) Date: Fri, 16 Jul 2004 10:36:02 -0400 Subject: [tbnl-devel] Odd behavior (socket leak) In-Reply-To: <50835.212.23.126.4.1089974269.squirrel@212.23.126.4> References: <20040715212252.GA29894@kazmier.com> <50835.212.23.126.4.1089974269.squirrel@212.23.126.4> Message-ID: <20040716143602.GA7050@kazmier.com> On Fri, Jul 16, 2004 at 12:37:49PM +0200, edi at miles wrote: > I'm sitting in a train and trying to send this via my cell > phone. Let's see if it works... :) Nice :-) > I've looked at the source code of mod_lisp.c (version 2.33) and Apache > (1.3.29) and I'm beginning to ask myself whether the "Keep-Socket" > option makes any sense at all. What I see with my limited knowledge of > Apache is the following: You missed one key part of the process. > 2. To get a socket, lisp-handler calls OpenLispSocket. > > 3. OpenLispSocket uses ap_psocket (using the global variable > SocketPool) to get a socket from Apache. OpenLispSocket only fetches a new socket if and only if one has not already established and is safe to use (see here): if (cfg->LispSocket) if (cfg->UnsafeLispSocket) { ap_pclosesocket(SocketPool, cfg->LispSocket); cfg->LispSocket = 0; cfg->UnsafeLispSocket = 0; } else { return cfg->LispSocket; } This is the step that bypasses the creation of a new socket. Thus, Keep-Socket does make sense. I'll continue looking at the TBNL and mod_Lisp interaction to see if I can find anything else. Thanks, Pete From pete-tbnl-dev at kazmier.com Fri Jul 16 16:31:07 2004 From: pete-tbnl-dev at kazmier.com (pete-tbnl-dev at kazmier.com) Date: Fri, 16 Jul 2004 12:31:07 -0400 Subject: [tbnl-devel] Odd behavior (socket leak) In-Reply-To: <20040715212252.GA29894@kazmier.com> References: <20040715212252.GA29894@kazmier.com> Message-ID: <20040716163107.GA7402@kazmier.com> On Thu, Jul 15, 2004 at 05:22:52PM -0400, pete-tbnl-dev at kazmier.com wrote: > In any case, I've been banging my head all day, I need a break, but it > would be appreciated if someone could either confirm or deny the opening > of multiple sockets while never closing any of the old ones. Its easy > to see via a 'netstat -an | grep 3000' (or whatever port you run it on). Upon further thought, isn't this behavior to be expected? I.e. as requests come into Apache, and Apache dispatches them to one or more of its children (processes/threads), doesn't each child via the mod_lisp handler create its own connection to TBNL on port 3000? And since Keep-Socket is "1", this is the reason why I never see any of the sockets closing. Thus, I should expect to see a socket for every child Apache process/thread, right? Or does mod_lisp make sure the same socket is used for all cases? This does not appear to be the case as I do see TBNL accept more than one connection on port 3000. Obviously, I'm not very familiar with the Apache API or mod_lisp. Back to my original problem, I'm still a little confused as to why I can't get a simple page that includes 3 style links (irrelevant I suspect other than it does cause multiple requests to occur in a very short amount of time perhaps on different sockets from the client) in it to load properly under any combination of mod_lisp, TBNL and lisp implementation. I've tried using SBCL, CMUCL, Apache 1.x (w/ modlisp-2.33), Apache 2.x (w/ modlisp for 2.x Apache). The symptom is the same, the web browser just sits there and hangs until the mod_lisp module times out reading from TBNL. Debugging the mod_lisp module seems to indicate that mod_lisp just sits there waiting for a response from TBNL. This seems to only occur for one of the four initial requests that it sent to TBNL (one of the style links). My only guess is that perhaps TBNL is somehow writing its response to the wrong *apache-stream* as mod_lisp handler does open multiple connections to TBNL on port 3000. I'm new to lisp and I'm not quite sure how special variables work in a multithreaded app or one that uses the CMUCL event loop. I look forward to your return Edi for some enlightenment! :-) Thanks, Pete From edi at agharta.de Sat Jul 17 10:48:18 2004 From: edi at agharta.de (Edi Weitz) Date: Sat, 17 Jul 2004 12:48:18 +0200 (CEST) Subject: [tbnl-devel] Odd behavior (socket leak) Message-ID: <2024.217.81.19.199.1090061298.squirrel@217.81.19.199> [Looks like it didn't work the first time with common-lisp.net - maybe due to my misconfiguration of SquirrelMail. I'll try it again. Note that Peter has already pointed out my mistaken assumption.] I'm sitting in a train and trying to send this via my cell phone. Let's see if it works... :) I've looked at the source code of mod_lisp.c (version 2.33) and Apache (1.3.29) and I'm beginning to ask myself whether the "Keep-Socket" option makes any sense at all. What I see with my limited knowledge of Apache is the following: 1. The actual content handler in mod_lisp.c which is called each time a new request comes in (from the client) is the function lisp_handler. 2. To get a socket, lisp-handler calls OpenLispSocket. 3. OpenLispSocket uses ap_psocket (using the global variable SocketPool) to get a socket from Apache. 4. ap_psocket is defined in Apache's src/main/alloc.c and basically just calls ap_psocket_ex. 5. ap_socket_ex always unconditionally acquires a new socket via socket(7). The pool which is its first argument (and which is SocketPool from above) is only used for cleanup purposes. So, as a consequence, I cannot understand if and how the socket used by lisp_handler is ever going to be re-used. I'm sending a copy of this email to mod-lisp-devel because Marc is the authoritative source for this kind of questions... :) Cheers, Edi. From edi at agharta.de Sun Jul 18 15:37:11 2004 From: edi at agharta.de (Edi Weitz) Date: Sun, 18 Jul 2004 17:37:11 +0200 (CEST) Subject: [tbnl-devel] Socket leaks, again Message-ID: <35019.212.23.126.4.1090165031.squirrel@212.23.126.4> [Again, I'm in a train and sending via cell phone and GPRS. Let's see if it works better this time. Also, I'm not sure if SquirrelMail will be able to keep indentations.] I'm prepared to make myself a fool in public again. I've debugged the mod_lisp/Apache interaction (by inserting log statements in strategic places) a bit and here's what I found out (Linux x86, mod_lisp 2.33, Apache 1.3.29, CMUCL 18e): 1. Whenever OpenLispSocket is called, cfg->LispSocket is 0. This can be best seen if you start Apache with the -X option so that you only one child process. This would imply that sockets can never be re-used. 2. Another data point (see also Peter's earlier post) is that netstat -an | grep 3000 reports far more established connections than there are Apache child processes. 3. I guess this is due to the fact that the configuration record is reset by Apache on each request. I observe that the comment in Apache's example module (which is also in mod_lisp) says "Note that while the per-directory and per-server configuration records are available to most of the module handlers, they should be treated as READ-ONLY by all except the command and merge handlers. Sometimes handlers are handed a record that applies to the current location by implication or inheritance, and modifying it will change the rules for other locations." This is obviously violated by mod_lisp which modifies the record in the content handler. 4. My na?ve attempt to fix this is to make LispSocket and UnsafeLispSocket global variables. A patch which reflects my changes is attached at the bottom. This currently works for me. Debugging shows that sockets are in fact re-used this time. Also, if I benchmark with ApacheBench (2000 request, 200 concurrent, 1300 byte static page, client and server on same machine) then the original mod_lisp code is able to handle roughly 140 requests per seconds while my modified code handles more than 460 requests per seconds. 5. Of course, this patch will only work if the Apache children are separate processes spawned by the server process as with Apache 1.3, i.e. another approach must be taken for the new, alternative Apache 2.0 thread implementations. OK, that's all for now. Let me know what you think. Cheers, Edi. --- /home/edi/downloads/Lisp/mod_lisp-2.33.c 2004-07-16 01:52:23.000000000 +0200 +++ mod_lisp-2.33.c 2004-07-18 16:58:23.000000000 +0200 @@ -189,10 +189,11 @@ char LispServerIP[20]; long LispServerPort; char LispServerId[100]; - long LispSocket; - long UnsafeLispSocket; } excfg; +static long LispSocket = 0; +static long UnsafeLispSocket = 0; + pool *SocketPool = NULL; /* @@ -371,13 +372,6 @@ if (lisp_pool == NULL) { lisp_pool = ap_make_sub_pool(NULL); }; - /* - * Likewise for the table of routine/environment pairs we visit outside of - * request context. - */ - if (static_calls_made == NULL) { - static_calls_made = ap_make_table(lisp_pool, 16); - }; } #endif @@ -421,21 +415,21 @@ int ret; #ifndef WIN32 - if (cfg->LispSocket) - if (cfg->UnsafeLispSocket) + if (LispSocket) + if (UnsafeLispSocket) { - ap_pclosesocket(SocketPool, cfg->LispSocket); - cfg->LispSocket = 0; - cfg->UnsafeLispSocket = 0; + ap_pclosesocket(SocketPool, LispSocket); + LispSocket = 0; + UnsafeLispSocket = 0; } else { - return cfg->LispSocket; + return LispSocket; } #endif - cfg->LispSocket = 0; - cfg->UnsafeLispSocket = 0; + LispSocket = 0; + UnsafeLispSocket = 0; addr.sin_addr.s_addr = inet_addr(cfg->LispServerIP); addr.sin_port = htons((unsigned short) cfg->LispServerPort); addr.sin_family = AF_INET; @@ -459,7 +453,7 @@ if (ret == -1) return -1; - cfg->LispSocket = sock; + LispSocket = sock; return sock; } @@ -498,13 +492,13 @@ if (Socket != -1) ap_pclosesocket(SocketPool, Socket); #else - if (!cfg->LispSocket) + if (!LispSocket) return; - ap_pclosesocket(SocketPool, cfg->LispSocket); + ap_pclosesocket(SocketPool, LispSocket); - cfg->LispSocket = 0; - cfg->UnsafeLispSocket = 0; + LispSocket = 0; + UnsafeLispSocket = 0; #endif } @@ -605,7 +599,7 @@ } ap_reset_timeout(r); - dcfg->UnsafeLispSocket = 1; + UnsafeLispSocket = 1; if (r->subprocess_env) { @@ -860,7 +854,7 @@ ap_bpushfd(BuffSocket, -1, -1); /* unlink buffer to keep socket */ BuffSocket->flags &= ~B_SOCKET; } - dcfg->UnsafeLispSocket = 0; + UnsafeLispSocket = 0; return OK; } @@ -985,7 +979,6 @@ strcpy(cfg->LispServerId, "apache"); cfg->LispServerPort = 3000; cfg->DefaultLispServer = 1; - cfg->LispSocket = 0; cfg->cmode = CONFIG_MODE_DIRECTORY; dname = (dname != NULL) ? dname : ""; @@ -1059,7 +1052,6 @@ merged_config->DefaultLispServer = 1; } - merged_config->LispSocket = 0; return (void *) merged_config; } @@ -1087,7 +1079,6 @@ strcpy(cfg->LispServerIP, "127.0.0.1"); strcpy(cfg->LispServerId, "apache"); cfg->LispServerPort = 3000; - cfg->LispSocket = 0; cfg->DefaultLispServer = 1; sname = (sname != NULL) ? sname : ""; cfg->loc = ap_pstrcat(p, "SVR(", sname, ")", NULL); @@ -1148,7 +1139,6 @@ merged_config->DefaultLispServer = 1; } - merged_config->LispSocket = 0; return (void *) merged_config; } @@ -1309,7 +1299,6 @@ cfg->LispServerId[99] = 0; cfg->LispServerPort = atoi(port); cfg->DefaultLispServer = 0; - cfg->LispSocket = 0; return NULL; } From edi at agharta.de Sun Jul 18 18:29:57 2004 From: edi at agharta.de (Edi Weitz) Date: Sun, 18 Jul 2004 20:29:57 +0200 Subject: [tbnl-devel] Re: Socket leaks, again Message-ID: <87d62tp3ru.fsf@bird.agharta.de> Hi Marc! I managed to delete your email I wanted to reply to but I'll try to copy the relevant parts from the c-l.net archive... :) > Heh?? Reporting and fixing problems is not really what I would call > making oneself a fool in public. ;-) Well, I wasn't so sure. I missed spectaclularly with my last one... :) > I think: Good catch! I looked at your patches, they look ok for me > so I commited them to the mod_lisp repository and switched my > servers to mod_lisp 2.35. OK, cool. Why, by the way, is it 2.35 and not 2.34? According to the mod_lisp website the latest release was 2.33 (which is also the one from Debian). > Please get the latest one (2.35) on the mod_lisp repository: > http://www.fractalconcept.com:8000/public/open-source/ I get no reply when I try to connect... :( Cheers, Edi. From edi at agharta.de Sat Jul 17 23:07:09 2004 From: edi at agharta.de (Edi Weitz) Date: Sun, 18 Jul 2004 01:07:09 +0200 Subject: [mod-lisp-devel] Re: [tbnl-devel] Odd behavior (socket leak) In-Reply-To: <042901c46bed$c80ddf30$0a02a8c0@marcxp> (Marc Battyani's message of "Sat, 17 Jul 2004 13:04:06 +0200") References: <2024.217.81.19.199.1090061298.squirrel@217.81.19.199> <042901c46bed$c80ddf30$0a02a8c0@marcxp> Message-ID: <87smbqgrmq.fsf@bird.agharta.de> Hi Marc! On Sat, 17 Jul 2004 13:04:06 +0200, "Marc Battyani" wrote: > in OpenLispSocket: > > If there is a socket in cfg->LispSocket and that socket is not > unsafe (cfg->UnsafeLispSocket) then I return this socket. I don't > create a new one. Re-using the socket is much faster. > > if (cfg->LispSocket) > if (cfg->UnsafeLispSocket) > { > ap_pclosesocket(SocketPool, cfg->LispSocket); > cfg->LispSocket = 0; > cfg->UnsafeLispSocket = 0; > } > else > return cfg->LispSocket; > > Or did I mis-read your question ? No, you didn't, it was just an oversight of mine. Sorry for the noise, Edi. From edi at agharta.de Sun Jul 18 22:57:22 2004 From: edi at agharta.de (Edi Weitz) Date: Mon, 19 Jul 2004 00:57:22 +0200 Subject: [tbnl-devel] Odd behavior (socket leak) In-Reply-To: <20040716163107.GA7402@kazmier.com> (pete-tbnl-dev@kazmier.com's message of "Fri, 16 Jul 2004 12:31:07 -0400") References: <20040715212252.GA29894@kazmier.com> <20040716163107.GA7402@kazmier.com> Message-ID: <87isck3ovh.fsf_-_@bird.agharta.de> Hi Pete! On Fri, 16 Jul 2004 12:31:07 -0400, pete-tbnl-dev at kazmier.com wrote: > Upon further thought, isn't this behavior to be expected? I.e. as > requests come into Apache, and Apache dispatches them to one or more > of its children (processes/threads), doesn't each child via the > mod_lisp handler create its own connection to TBNL on port 3000? > And since Keep-Socket is "1", this is the reason why I never see any > of the sockets closing. Thus, I should expect to see a socket for > every child Apache process/thread, right? Or does mod_lisp make > sure the same socket is used for all cases? This does not appear to > be the case as I do see TBNL accept more than one connection on port > 3000. Obviously, I'm not very familiar with the Apache API or > mod_lisp. See the recent postings by Marc and myself. You've obviously found a bug in mod_lisp which is fixed now. > Back to my original problem, I'm still a little confused as to why I > can't get a simple page that includes 3 style links (irrelevant I > suspect other than it does cause multiple requests to occur in a > very short amount of time perhaps on different sockets from the > client) in it to load properly under any combination of mod_lisp, > TBNL and lisp implementation. I've tried using SBCL, CMUCL, Apache > 1.x (w/ modlisp-2.33), Apache 2.x (w/ modlisp for 2.x Apache). The > symptom is the same, the web browser just sits there and hangs until > the mod_lisp module times out reading from TBNL. > > Debugging the mod_lisp module seems to indicate that mod_lisp just > sits there waiting for a response from TBNL. This seems to only > occur for one of the four initial requests that it sent to TBNL (one > of the style links). My only guess is that perhaps TBNL is somehow > writing its response to the wrong *apache-stream* as mod_lisp > handler does open multiple connections to TBNL on port 3000. I'm > new to lisp and I'm not quite sure how special variables work in a > multithreaded app or one that uses the CMUCL event loop. > > I look forward to your return Edi for some enlightenment! :-) If it's not confidential then maybe you can just send me your code and I'll have a look at it. I currently don't have an idea what's the problem might be. As for special variables: Although there's no standard for that there seems to be an agreement that every process/thread has its own set of bindings for special variables, i.e. if you do (let ((*foo* ...)) ...) within a thread and *FOO* is a special variable then within the body of the LET form the thread is manipulating its "private" version of *FOO* which is distinct from the value other threads have access to. This is what happens in TBNL because of this line: (defun apache-listen (*apache-stream* command-processor &rest args) ...) When APACHE-LISTEN is called, the special variable *APACHE-STREAM* is bound. That would be equivalent to: (defun apache-listen (apache-stream command-processor &rest args) (let ((*apache-stream* apache-stream)) ...)) If you haven't done so you might want to look at which is about LispWorks but is nevertheless a pretty good general introduction to multi-processing in CL. Cheers, Edi. PS: TBNL (via KMRCL) doesn't use CMUCL's event loop but rather CMUCL's multi-processing facilities. It will therefore only work on x86 machines (unless you use other CL implementations, of course). From stesch at no-spoon.de Mon Jul 19 09:54:42 2004 From: stesch at no-spoon.de (Stefan Scholl) Date: Mon, 19 Jul 2004 11:54:42 +0200 Subject: [tbnl-devel] URL rewrite for XHTML content-types, too Message-ID: <20040719095442.GP21996@parsec.no-spoon.de> Hi! The documentation for *rewrite-for-session-urls* says that ""This only happens, though, if the body's content type (see CONTENT-TYPE) starts with "text/html""". XHTML can have other content types, too. See OK, application/xml and text/xml are a bad choice, because you can't say if it's really a kind of XHTML or just another XML application. You have to parse the content itself for that. But application/xhtml+xml is OK and recommended. Regards, Stefan From edi at agharta.de Mon Jul 19 10:47:16 2004 From: edi at agharta.de (Edi Weitz) Date: Mon, 19 Jul 2004 12:47:16 +0200 Subject: [tbnl-devel] Re: [mod-lisp-devel] Re: Socket leaks, again In-Reply-To: <0a4e01c46d03$93d38960$0a02a8c0@marcxp> (Marc Battyani's message of "Sun, 18 Jul 2004 22:12:39 +0200") References: <87d62tp3ru.fsf@bird.agharta.de> <0a4e01c46d03$93d38960$0a02a8c0@marcxp> Message-ID: <87d62sff4b.fsf@bird.agharta.de> On Sun, 18 Jul 2004 22:12:39 +0200, "Marc Battyani" wrote: > Hum, this web site is poorly maintained... :( I change it from time > to time, when I find some time. The trouble is that these days, when > I find some time, I prefer hacking cl-typesetting or something like > that rather than updating the web site... I'll add a link to the SVN repository to the TBNL docs. >> > Please get the latest one (2.35) on the mod_lisp repository: >> > http://www.fractalconcept.com:8000/public/open-source/ >> >> I get no reply when I try to connect... :( > > I restarted Apache2 (used by svn), it's ok for me now. Yep, it worked briefly last night but tomorrow again the connection is refused... :( Cheers, Edi. From edi at agharta.de Mon Jul 19 12:13:23 2004 From: edi at agharta.de (Edi Weitz) Date: Mon, 19 Jul 2004 14:13:23 +0200 Subject: [tbnl-devel] Re: [mod-lisp-devel] Re: Socket leaks, again In-Reply-To: <87d62sff4b.fsf@bird.agharta.de> (Edi Weitz's message of "Mon, 19 Jul 2004 12:47:16 +0200") References: <87d62tp3ru.fsf@bird.agharta.de> <0a4e01c46d03$93d38960$0a02a8c0@marcxp> <87d62sff4b.fsf@bird.agharta.de> Message-ID: <87llhgp53w.fsf@bird.agharta.de> On Mon, 19 Jul 2004 12:47:16 +0200, Edi Weitz wrote: > Yep, it worked briefly last night but tomorrow again the connection > is refused... :( Oh my, I wanted to say "today" of course... :) From edi at agharta.de Mon Jul 19 12:26:52 2004 From: edi at agharta.de (Edi Weitz) Date: Mon, 19 Jul 2004 14:26:52 +0200 Subject: [tbnl-devel] New version 0.2.4 Message-ID: <87d62sp4hf.fsf@bird.agharta.de> Hi! A new release is available from . Here's the relevant part from the changelog: Version 0.2.4 2004-07-19 New variable *CONTENT-TYPES-FOR-URL-REWRITE* (suggested by Stefan Scholl) Updated index.html regarding new version of mod_lisp Have fun, Edi From edi at agharta.de Mon Jul 19 12:27:51 2004 From: edi at agharta.de (Edi Weitz) Date: Mon, 19 Jul 2004 14:27:51 +0200 Subject: [tbnl-devel] URL rewrite for XHTML content-types, too In-Reply-To: <20040719095442.GP21996@parsec.no-spoon.de> (Stefan Scholl's message of "Mon, 19 Jul 2004 11:54:42 +0200") References: <20040719095442.GP21996@parsec.no-spoon.de> Message-ID: <878ydgp4fs.fsf@bird.agharta.de> On Mon, 19 Jul 2004 11:54:42 +0200, Stefan Scholl wrote: > The documentation for *rewrite-for-session-urls* says that ""This > only happens, though, if the body's content type (see CONTENT-TYPE) > starts with "text/html""". > > XHTML can have other content types, too. See > > > OK, application/xml and text/xml are a bad choice, because you can't > say if it's really a kind of XHTML or just another XML > application. You have to parse the content itself for that. > > But application/xhtml+xml is OK and recommended. How about this approach? Cheers, Edi. From stesch at no-spoon.de Mon Jul 19 13:02:26 2004 From: stesch at no-spoon.de (Stefan Scholl) Date: Mon, 19 Jul 2004 15:02:26 +0200 Subject: [tbnl-devel] URL rewrite for XHTML content-types, too In-Reply-To: <878ydgp4fs.fsf@bird.agharta.de> References: <20040719095442.GP21996@parsec.no-spoon.de> <878ydgp4fs.fsf@bird.agharta.de> Message-ID: <20040719130226.GR21996@parsec.no-spoon.de> On 2004-07-19 14:27:51, Edi Weitz wrote: > On Mon, 19 Jul 2004 11:54:42 +0200, Stefan Scholl wrote: > > OK, application/xml and text/xml are a bad choice, because you can't > > say if it's really a kind of XHTML or just another XML > > application. You have to parse the content itself for that. > > > > But application/xhtml+xml is OK and recommended. > > How about this approach? > > Nice. That's perfect. Heaven knows if the W3C invents new content types for HTML. But I could add "text/plain" for debugging purposes. Regards, Stefan (aka stesch @ #lisp, FreeNode) From edi at agharta.de Mon Jul 19 17:17:23 2004 From: edi at agharta.de (Edi Weitz) Date: Mon, 19 Jul 2004 19:17:23 +0200 Subject: [tbnl-devel] Re: [mod-lisp-devel] Re: Socket leaks, again In-Reply-To: <0c8601c46db3$6faf4eb0$0a02a8c0@marcxp> (Marc Battyani's message of "Mon, 19 Jul 2004 19:11:29 +0200") References: <87d62tp3ru.fsf@bird.agharta.de> <0a4e01c46d03$93d38960$0a02a8c0@marcxp> <87d62sff4b.fsf@bird.agharta.de> <0c8601c46db3$6faf4eb0$0a02a8c0@marcxp> Message-ID: <874qo3diho.fsf@bird.agharta.de> On Mon, 19 Jul 2004 19:11:29 +0200, "Marc Battyani" wrote: > I'm checking it now and it's ok for me. Do you still have connection > problems ? No, it works now. Thanks, Edi. From edi at agharta.de Tue Jul 20 10:40:26 2004 From: edi at agharta.de (Edi Weitz) Date: Tue, 20 Jul 2004 12:40:26 +0200 Subject: [tbnl-devel] "Leaked children" Message-ID: <87ekn79d2d.fsf@bird.agharta.de> Hi! Again, a little patch. This time it's not about leaked sockets but rather about "leaked children," so to say. It was Stefan Scholl who actually prompted me to investigate this further and here's what I found out: This piece of code if (ReadLength < ContentLength || r->connection->aborted) { char buffer[HUGE_STRING_LEN]; ContentLength -= ReadLength; do { ReadLength = ForceGets(buffer, (BUFF *) BuffSocket, HUGE_STRING_LEN > ContentLength ? ContentLength : HUGE_STRING_LEN); ContentLength -= ReadLength; } while (ReadLength > 0 && ContentLength > 0); } doesn't work if ContentLength is large enough. What's happening is that the process of emptying BuffSocket (the loop above) always hangs when exactly the last 8192 (HUGE_STRING_LEN) bytes are waiting to be removed. I tried with ap_bread instead of ForceGets but got the same result. (Note that ap_bgets which is used by ForceGets does CR/LF handling which is not needed here.) The result is that this child becomes unusable but is still there, it is never killed by the Apache root process. An easy way to reproduce this (with TBNL) is to do (asdf:oos 'asdf:load-op :tbnl-test) (tbnl:start-test) with a proper Apache configuration (see TBNL docs) and then call ApacheBench with large values like so: ab -n 2000 -c 200 http://localhost/tbnl/test/image.jpg (The important point is that image.jpg is big enough - about 20kB in this case.) After doing this you'll see a large number of Apache processes with ps(1) and the same amount of Lisp processes from within your Lisp image. Call ApacheBench often enough (two or three times) and Apache will completely stop to respond because it has reached its 'MaxClient' limit of 150 (unless you've changed it, of course) but none of the 150 clients is usable. This happens because ApacheBench will abort all pending connections as soons as its finished with its tests. The pattern can obviously be used as a DoS attack on mod_lisp. Now, what to do? One option would probably be to set a timeout before emptying the buffer (not tested). But I think the better (and faster) solution is to get rid of the buffer and the socket as well. The next time the client is used we'll have to open up a new socket to Lisp but this won't need 300 seconds (the default Apache value for 'Timeout'). So... if (ReadLength < ContentLength || r->connection->aborted) { ap_log_error("mod_lisp", 0, APLOG_WARNING|APLOG_NOERRNO, r->server, "Could not send complete body to client, closing socket to Lisp"); ap_bclose(BuffSocket); KeepSocket = 0; LispSocket = 0; } The appended patch also adds a couple of braces to appease gcc and it removes some code that is never used at all. Hope that's OK. Cheers, Edi. -------------- next part -------------- A non-text attachment was scrubbed... Name: mod_lisp.patch Type: text/x-patch Size: 7734 bytes Desc: not available URL: From stesch at no-spoon.de Tue Jul 20 13:30:37 2004 From: stesch at no-spoon.de (Stefan Scholl) Date: Tue, 20 Jul 2004 15:30:37 +0200 Subject: [tbnl-devel] Tip: Don't forget to close your ports! Message-ID: <20040720133037.GD21996@parsec.no-spoon.de> Hi! It's hardcoded into KMRCL to listen on 0.0.0.0. ==> It's possible to reach your lisp server directly from outside your server. Don't forget to close your ports! If you're using Linux, outside interface is eth0 and *apache-port* is 3000 (default), then use the following line: iptables -A INPUT -i eth0 -p tcp --dport 3000 -j REJECT Regards, Stefan From pete-tbnl-dev at kazmier.com Tue Jul 20 14:07:16 2004 From: pete-tbnl-dev at kazmier.com (pete-tbnl-dev at kazmier.com) Date: Tue, 20 Jul 2004 10:07:16 -0400 Subject: [tbnl-devel] Odd behavior (socket leak) In-Reply-To: <87isck3ovh.fsf_-_@bird.agharta.de> References: <20040715212252.GA29894@kazmier.com> <20040716163107.GA7402@kazmier.com> <87isck3ovh.fsf_-_@bird.agharta.de> Message-ID: <20040720140716.GA9334@kazmier.com> On Mon, Jul 19, 2004 at 12:57:22AM +0200, Edi Weitz wrote: > > See the recent postings by Marc and myself. You've obviously found a > bug in mod_lisp which is fixed now. Yes, I can now confirm that all is well when working under Apache 1.3, mod_lisp 2.35, SBCL, and TBNL. My previous problem, which spawned this topic, appears to have been resolved. Pages are now loading correctly without incident. Thanks for your help!! Now I just have to wait for an Apache 2.x module. Perhaps I'll take a stab at it when I finish my TBNL project, in the meantime, I'll just proxy between Apache 2.x and Apache 1.x. > As for special variables: Thank you for the detailed explanation on special variables and multi processing environments. Pete From pete-tbnl-dev at kazmier.com Tue Jul 20 14:13:02 2004 From: pete-tbnl-dev at kazmier.com (pete-tbnl-dev at kazmier.com) Date: Tue, 20 Jul 2004 10:13:02 -0400 Subject: [tbnl-devel] Tip: Don't forget to close your ports! In-Reply-To: <20040720133037.GD21996@parsec.no-spoon.de> References: <20040720133037.GD21996@parsec.no-spoon.de> Message-ID: <20040720141302.GB9334@kazmier.com> Good point. I also noticed that when I first started experimenting with mod_lisp; however, I take the opposite approach to security and deny everything by default and only explicitly permit when necessary. Nonetheless, you bring up a good point, thank you for highlighting it. On Tue, Jul 20, 2004 at 03:30:37PM +0200, Stefan Scholl wrote: > Hi! > > It's hardcoded into KMRCL to listen on 0.0.0.0. ==> It's possible > to reach your lisp server directly from outside your server. > > Don't forget to close your ports! > > If you're using Linux, outside interface is eth0 and > *apache-port* is 3000 (default), then use the following line: > > iptables -A INPUT -i eth0 -p tcp --dport 3000 -j REJECT > > > Regards, > Stefan > > > _______________________________________________ > tbnl-devel site list > tbnl-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/tbnl-devel From pete-tbnl-dev at kazmier.com Tue Jul 20 14:29:41 2004 From: pete-tbnl-dev at kazmier.com (pete-tbnl-dev at kazmier.com) Date: Tue, 20 Jul 2004 10:29:41 -0400 Subject: [tbnl-devel] [PATCH] Typo in modlisp.lisp Message-ID: <20040720142941.GA18484@kazmier.com> This is certainly not worth an actual patch file ... -------------- next part -------------- --- modlisp.lisp.orig 2004-07-15 22:43:54.000000000 -0400 +++ modlisp.lisp 2004-07-20 10:25:18.000000000 -0400 @@ -273,7 +273,7 @@ (declare (ignore args)) (let ((*close-apache-stream* t)) (unwind-protect - (loop for *apache-socket-useage-counter* from 0 + (loop for *apache-socket-usage-counter* from 0 for command = (debug-value *command* (get-apache-command)) while command do (cond ((ignore-errors From stesch at no-spoon.de Tue Jul 20 20:56:21 2004 From: stesch at no-spoon.de (Stefan Scholl) Date: Tue, 20 Jul 2004 22:56:21 +0200 Subject: [tbnl-devel] create-static-file-dispatcher-and-handler Message-ID: <20040720205621.GF21996@parsec.no-spoon.de> Hi! A simple convenience function for serving static files with TBNL. Example: (create-static-file-dispatcher-and-handler "/css/main.css" #P"css/main.css" "text/css") Maybe someone finds a better name and documentation. :-) (defun create-static-file-dispatcher-and-handler (uri path content-type) "Creates a dispatch function which will dispatch to a handler function which emits the file PATH with CONTENT-TYPE if the URI matches the SCRIPT-NAME of the request." (let ((buf (make-array 8192 :element-type #-:tbnl-bivalent-streams 'character #+:tbnl-bivalent-streams '(unsigned-byte 8)))) (let ((handler #'(lambda () (setf (content-type) content-type) (with-output-to-string (str) (with-open-file (file path :direction :input #+:tbnl-bivalent-streams :element-type #+:tbnl-bivalent-streams '(unsigned-byte 8)) (do ((pos (read-sequence buf file) (read-sequence buf file))) ((zerop pos)) (write-sequence buf str :end pos))))))) #'(lambda (request) (when (equal (script-name request) uri) handler))))) Regards, Stefan From edi at agharta.de Tue Jul 20 23:02:13 2004 From: edi at agharta.de (Edi Weitz) Date: Wed, 21 Jul 2004 01:02:13 +0200 Subject: [tbnl-devel] New version 0.2.5 Message-ID: <87briath8q.fsf@bird.agharta.de> Hi! A new release is available from . Here's the relevant part from the changelog: Version 0.2.5 2004-07-21 Added CREATE-STATIC-FILE-DISPATCHER-AND-HANDLER (provided by Stefan Scholl) Improved test suite Enjoy, Edi From edi at agharta.de Tue Jul 20 23:02:51 2004 From: edi at agharta.de (Edi Weitz) Date: Wed, 21 Jul 2004 01:02:51 +0200 Subject: [tbnl-devel] [PATCH] Typo in modlisp.lisp In-Reply-To: <20040720142941.GA18484@kazmier.com> (pete-tbnl-dev@kazmier.com's message of "Tue, 20 Jul 2004 10:29:41 -0400") References: <20040720142941.GA18484@kazmier.com> Message-ID: <877jsyth7o.fsf@bird.agharta.de> On Tue, 20 Jul 2004 10:29:41 -0400, pete-tbnl-dev at kazmier.com wrote: > This is certainly not worth an actual patch file ... Thanks Pete, but that was already fixed in 0.2.4... :) Cheers, Edi. From edi at agharta.de Tue Jul 20 23:03:10 2004 From: edi at agharta.de (Edi Weitz) Date: Wed, 21 Jul 2004 01:03:10 +0200 Subject: [tbnl-devel] create-static-file-dispatcher-and-handler In-Reply-To: <20040720205621.GF21996@parsec.no-spoon.de> (Stefan Scholl's message of "Tue, 20 Jul 2004 22:56:21 +0200") References: <20040720205621.GF21996@parsec.no-spoon.de> Message-ID: <873c3mth75.fsf@bird.agharta.de> On Tue, 20 Jul 2004 22:56:21 +0200, Stefan Scholl wrote: > A simple convenience function for serving static files with TBNL. Thanks, it's in 0.2.5 now. From pete-tbnl-dev at kazmier.com Thu Jul 22 14:55:09 2004 From: pete-tbnl-dev at kazmier.com (pete-tbnl-dev at kazmier.com) Date: Thu, 22 Jul 2004 10:55:09 -0400 Subject: [tbnl-devel] Question about dispatchers and their actions Message-ID: <20040722145509.GA8256@kazmier.com> I've got a question about the *dispatch-table* and the associated actions that are returned from the dispatch functions. Based on the code in modlisp.lisp: (loop for dispatcher in *dispatch-table* for action = (funcall dispatcher *request*) when action return (funcall action) finally (setf (return-code *reply*) +http-not-found+)))) Might it be desirable to check the return of (funcall action), and if nil, continue processing with the next dispatcher rather than returning and sending output at this point? If one truly wanted to return an empty body, one could still throw to the tbnl-handler-done tag which would bypass the rest of the dispatch table. I'm writing a template dispatcher which will process a request using HTML-TEMPLATE. Of course, this is only possible if I can map a request to a template (I may store the templates on the filesystem or in a db, not sure yet). If a template cannot be found for the request, then I'd like to fall back to processing the request with the rest of the dispatch table (which may just serve a static file or may be another template dispatcher thats resolves templates differently). In order to accomplish the above in the today's TBNL, I need to make the template resolution occur in the dispatch function (instead of the action function that the dispatcher returns) because I can return nil from the dispatcher if resolution fails and thus instruct TBNL to move on to the next dispatcher. In addition, I'd also like to make my dispatch decision based on a prefix. So I'd like to use create-prefix-dispatcher to limit the template processing to only part of the URI namespace. Thus, I need two 'tests' to occur in my dispatch function: 1) is there a prefix match (replicating the create-prefix-dispatcher functionality), and 2) does the request map to a template. With that said, I have created: (defun create-template-dispatcher (prefix resolver-function) #'(lambda (request) (when (prefix-match-p prefix (script-name request)) (let ((template (funcall resolver-function request))) (when template #'(lambda () (funcall *default-template-handler* template))))))) prefix-match-p is refactored code from create-prefix-dispatcher, but is essentially the same, does the request start with the given prefix? If it does not, I return nil, and processing continues with the rest of the dispatch table. Assuming the prefix does match, I then go on to call the template resolution function (resolver-function) which returns a template as a string or nil if a template could not be found. Template resolution might query a db or just read a template from the filesystem. Finally, if a template if found, I process the template with the default template handler which points to: (defun template-handler (template) (let ((context (assemble-context))) (with-output-to-string (*default-template-output*) (fill-and-print-template template context)))) Here is a sample template resolver function that maps a request to a template on the filesystem (only works on systems with '/' as a separator, my lisp path manipulations skills are absent at the moment): (defun filesystem-resolver (base &optional (request *request*)) (contents-of-file (parse-namestring (concatenate 'string base (script-name request))))) And here is my sample dispatch table that pulls templates from multiple locations on the filesystem depending on the request URI: (setf *dispatch-table* (list (create-template-dispatcher "/tbnl/space1" #'(lambda (request) (filesystem-resolver "/tmp" request))) (create-template-dispatcher "/tbnl/space2" #'(lambda (request) (filesystem-resolver "/var/www/html" request))) #'default-dispatcher)) Back to the original question and my rationale for asking it, I think I'd prefer if I didn't have to put all of my predicates for testing whether or not the template handler should be run within the dispatch function as it just doesn't seem to be the right spot for this logic. However, I definitely want the template handler executed based on the prefix. If TBNL could continue processing the dispatch table upon the return of nil from the handler (action), my dispatch table would look like: (setq *dispatch-table* (nconc (mapcar (lambda (args) (apply #'create-prefix-dispatcher args)) '(("/tbnl/space1" template-dispatcher) ("/tbnl/space2" template-dispatcher))) (list #'default-dispatcher))) And the template-dispatcher would consult a *resolver-table* and invoke the appropriate resolution function for the request. The nice thing about this approach is that my *dispatch-table* is not cluttered with non-dispatch related stuff such as template resolution. Of course that all requires the ability for an action (the template-dispatcher above) to signify that it cannot handle the request by returning a nil, which could then indicate to TBNL that the next entry in the *dispatch-table* should be tried. Would this functionality appeal to anyone else? Sorry about the length of email, but I thought it would be useful to share what I'm doing with TBNL. This is all just for fun and non-profit and my pursuit of learning lisp so if I've said something stupid or more specifically am not doing things in the traditional "lisp way", please let me know. Thanks, Pete From stesch at no-spoon.de Thu Jul 22 15:51:52 2004 From: stesch at no-spoon.de (Stefan Scholl) Date: Thu, 22 Jul 2004 17:51:52 +0200 Subject: [tbnl-devel] Patch for handling If-Modified-Since Message-ID: <20040722155152.GC2310@parsec.no-spoon.de> Hi! A patch for handling If-Modified-Since is attached to this e-mail. Maybe it could be useful. I've only tested it with Mozilla 1.7.1 (with help from ). So additional tests would be fine. New function rfc1123-date for generating a date string for http headers. Function handle-if-modified-since wants a time and handles almost everything by itself. Checks against If-Modified-Since and returns a 304 return code if necessary. You have to set the Last-Modified header by yourself because you don't know if you can produce some output at this point. See create-static-file-dispatcher-and-handler in html.lisp for an example. I've improved create-static-file-dispatcher-and-handler by adding cache functionality with handle-if-modified-since and setting Last-Modified, and a simple check if the file exists. (==> 404 if not). You see that Last-Modified isn't set when the file doesn't exist. Regards, Stefan -------------- next part -------------- diff -ru --exclude='*.fasl' --exclude='*~' tbnl-0.2.5/html.lisp tbnl-0.2.5-stesch/html.lisp --- tbnl-0.2.5/html.lisp 2004-07-21 00:58:28.000000000 +0200 +++ tbnl-0.2.5-stesch/html.lisp 2004-07-22 17:40:42.000000000 +0200 @@ -141,12 +141,22 @@ (when (equal (script-name request) uri) ;; the handler (lambda () - (setf (content-type) content-type) - (with-output-to-string (out) - (with-open-file (file path - :direction :input - #+:tbnl-bivalent-streams :element-type - #+:tbnl-bivalent-streams '(unsigned-byte 8)) - (loop for pos = (read-sequence buf file) - until (zerop pos) - do (write-sequence buf out :end pos))))))))) \ No newline at end of file + (let ((time (or (file-write-date path) + (get-universal-time)))) + (handle-if-modified-since time) + (setf (content-type) content-type) + (let ((str + (with-output-to-string (out) + (with-open-file (file path + :direction :input + #+:tbnl-bivalent-streams :element-type + #+:tbnl-bivalent-streams '(unsigned-byte 8) + :if-does-not-exist nil) + (unless file ; does not exist + (setf (return-code) +http-not-found+) + (throw 'tbnl-handler-done nil)) + (loop for pos = (read-sequence buf file) + until (zerop pos) + do (write-sequence buf out :end pos)))))) + (setf (header-out "Last-Modified") (rfc1123-date time)) + str))))))) diff -ru --exclude='*.fasl' --exclude='*~' tbnl-0.2.5/request.lisp tbnl-0.2.5-stesch/request.lisp --- tbnl-0.2.5/request.lisp 2004-05-07 16:10:40.000000000 +0200 +++ tbnl-0.2.5-stesch/request.lisp 2004-07-22 17:01:53.000000000 +0200 @@ -261,3 +261,13 @@ returned. Search is case-sensitive." (or (get-parameter name request) (post-parameter name request))) + +(defun handle-if-modified-since (time &optional (request *request*)) + "Handles the If-Modified-Since header of the REQUEST. Date string is +compared to the one generated from the supplied TIME." + (let ((if-modified-since (header-in "If-Modified-Since" request)) + (time-string (rfc1123-date time))) + ;; Simple string compare is sufficient. See RFC 2616 14.25 + (when (and if-modified-since (equal if-modified-since time-string)) + (setf (return-code) +http-not-modified+) + (throw 'tbnl-handler-done nil)))) diff -ru --exclude='*.fasl' --exclude='*~' tbnl-0.2.5/specials.lisp tbnl-0.2.5-stesch/specials.lisp --- tbnl-0.2.5/specials.lisp 2004-07-19 14:22:09.000000000 +0200 +++ tbnl-0.2.5-stesch/specials.lisp 2004-07-22 15:37:29.000000000 +0200 @@ -46,6 +46,7 @@ (defconstant +http-ok+ 200) (defconstant +http-moved-permanently+ 301) (defconstant +http-moved-temporarily+ 302) +(defconstant +http-not-modified+ 304) (defconstant +http-authorization-required+ 401) (defconstant +http-forbidden+ 403) (defconstant +http-not-found+ 404) diff -ru --exclude='*.fasl' --exclude='*~' tbnl-0.2.5/tbnl.asd tbnl-0.2.5-stesch/tbnl.asd --- tbnl-0.2.5/tbnl.asd 2004-05-12 19:05:09.000000000 +0200 +++ tbnl-0.2.5-stesch/tbnl.asd 2004-07-22 16:56:19.000000000 +0200 @@ -41,7 +41,7 @@ (:file "util" :depends-on ("specials")) (:file "log" :depends-on ("util")) (:file "cookie" :depends-on ("util")) - (:file "request" :depends-on ("util")) + (:file "request" :depends-on ("util" "reply" "specials")) (:file "reply" :depends-on ("util")) (:file "session" :depends-on ("cookie" "log")) (:file "html" :depends-on ("session" "request" "util")) diff -ru --exclude='*.fasl' --exclude='*~' tbnl-0.2.5/util.lisp tbnl-0.2.5-stesch/util.lisp --- tbnl-0.2.5/util.lisp 2004-07-19 14:22:09.000000000 +0200 +++ tbnl-0.2.5-stesch/util.lisp 2004-07-22 16:38:42.000000000 +0200 @@ -66,6 +66,7 @@ ((#.+http-ok+) "OK") ((#.+http-moved-permanently+) "Moved Permanently") ((#.+http-moved-temporarily+) "Moved Temporarily") + ((#.+http-not-modified+) "Not Modified") ((#.+http-authorization-required+) "Authorization Required") ((#.+http-forbidden+) "Forbidden") ((#.+http-not-found+) "Not Found") @@ -210,4 +211,18 @@ (defun get-backtrace (error) (declare (ignore error)) (format nil "Output of backtrace currently not implemented for ~A" - (lisp-implementation-type))) \ No newline at end of file + (lisp-implementation-type))) + +(defun rfc1123-date (&optional (time (get-universal-time))) + "Generate time string according to RFC 1123. Default is current time." + (multiple-value-bind + (second minute hour date month year day-of-week) + (decode-universal-time time 0) + (format nil "~A, ~2d ~A ~4d ~2,'0d:~2,'0d:~2,'0d GMT" + (svref +day-names+ day-of-week) + date + (svref +month-names+ (1- month)) + year + hour + minute + second))) From stesch at no-spoon.de Thu Jul 22 16:22:04 2004 From: stesch at no-spoon.de (Stefan Scholl) Date: Thu, 22 Jul 2004 18:22:04 +0200 Subject: [tbnl-devel] Patch for handling If-Modified-Since In-Reply-To: <20040722155152.GC2310@parsec.no-spoon.de> References: <20040722155152.GC2310@parsec.no-spoon.de> Message-ID: <20040722162204.GD2310@parsec.no-spoon.de> Hi! On 2004-07-22 17:51:52, Stefan Scholl wrote: > + ;; Simple string compare is sufficient. See RFC 2616 14.25 > + (when (and if-modified-since (equal if-modified-since time-string)) One word to the above: When you change from a newer to an older version it is still a modified version. So when you have to delete the newest entry from a database you want the visitors to see the remaining entries, even when they are older. Regards, Stefan From stesch at no-spoon.de Fri Jul 23 12:55:38 2004 From: stesch at no-spoon.de (Stefan Scholl) Date: Fri, 23 Jul 2004 14:55:38 +0200 Subject: [tbnl-devel] Patch for handling If-Modified-Since In-Reply-To: <20040722155152.GC2310@parsec.no-spoon.de> References: <20040722155152.GC2310@parsec.no-spoon.de> Message-ID: <20040723125538.GO2310@parsec.no-spoon.de> Hi! On 2004-07-22 17:51:52, Stefan Scholl wrote: > I've only tested it with Mozilla 1.7.1 (with help from > ). So additional tests would > be fine. OK, tested with MS IE 6.0 and Opera 7.51. No problems. Nice 304 in the logfile. The handling of return codes other than 200 in modlisp.lisp could emit less data for 304 (+http-not-modified+). Regards, Stefan From stesch at no-spoon.de Fri Jul 23 13:10:26 2004 From: stesch at no-spoon.de (Stefan Scholl) Date: Fri, 23 Jul 2004 15:10:26 +0200 Subject: [tbnl-devel] Question about dispatchers and their actions In-Reply-To: <20040722145509.GA8256@kazmier.com> References: <20040722145509.GA8256@kazmier.com> Message-ID: <20040723131026.GP2310@parsec.no-spoon.de> Hi! I don't know. A dispatcher is a dispatcher. Its task is to find the right handler for the request. And a handler returning NIL indicates an error. TBNL should emit an internal server error 500. On the other hand: Internal server errors are a bad thing. Better produce a 404 instead of an 500. The handler can't handle the request? Let the handler report this if it's no real problem/error. If no dispatcher is found and no handler "feels fit" to handle the request, then a 404 is the right answer. But I think it should be an exception to the rule. Let the handler return _two_ values if he is sure that the reason for not processing the request is no fatal error and dispatching should continue. Normal handlers just return a string or nil if they have a problem. Handlers which are aware of the possibility that other dispatchers could catch the request when they fail return (values nil t) Regards, Stefan From pete-tbnl-dev at kazmier.com Fri Jul 23 13:36:57 2004 From: pete-tbnl-dev at kazmier.com (Pete Kazmier) Date: Fri, 23 Jul 2004 09:36:57 -0400 Subject: [tbnl-devel] Question about dispatchers and their actions In-Reply-To: <20040723131026.GP2310@parsec.no-spoon.de> References: <20040722145509.GA8256@kazmier.com> <20040723131026.GP2310@parsec.no-spoon.de> Message-ID: <1090589817.1238.12.camel@mayo.ivanet.net> On Fri, 2004-07-23 at 09:10, Stefan Scholl wrote: > But I think it should be an exception to the rule. Let the > handler return _two_ values if he is sure that the reason for not > processing the request is no fatal error and dispatching should > continue. > Normal handlers just return a string or nil if they have a > problem. Handlers which are aware of the possibility that other > dispatchers could catch the request when they fail return > (values nil t) Thats a good idea. I keep forgetting about the usefulness of the ability to return multiple values. With this ability, I would be able to minimize the complexity of my dispatch functions and move some of that logic into the handler itself which could then have the option of opting to not process the request (defer to the next dispatcher). Out of curiousity, what do you guys like to use for generating your HTML? Are you firm believers in the whole "separation of logic and presentation" and prefer the use of template engines? Or do you prefer to output HTML from your TBNL apps and then let folks use CSS to adjust the presentation as they see fit? Or do you use a combination of both? Thanks, Pete From jdcal at yahoo.com Fri Jul 23 22:22:54 2004 From: jdcal at yahoo.com (Jeff Caldwell) Date: Fri, 23 Jul 2004 15:22:54 -0700 (PDT) Subject: [tbnl-devel] 1) Aborted connections 2) create-static-file-dispatcher-and-handler Message-ID: <20040723222254.17640.qmail@web50602.mail.yahoo.com> Hi guys, Edi, I responded to your email last week but, by accident, didn't cc you. I told Pete I had just started my first Debian stable -> unstable upgrade and that my world was torn apart. The upgrade was a bear for me but I finally have the pieces glued back in place, at least enough to do rough work. I now know more than I did about X, BlackBox, KDE, apt-get, dselect, and all that jazz. A couple of things. Edi, your aborted connection patch is a good thing. I don't remember if you noted that the aborted connections never happen until after ab completes its run. It seems to me that the there is one additional request sent to mod_lisp as each httpd process terminates. I used the Apache logging feature and did 'tail -f error_log' to watch what happens as ab runs. Everything was fine until after ab completed. Then and only then did the aborted connection logs start appearing. I didn't measure it exactly but it seemed the messages dribbled along as httpd's terminated after reaching their approx. 150 peak. I put in enough log messages to verify that with ab -n 2000 -c 200, I got about 2199 total messages, of which 199 ended with aborts. With ab -n 2000 -c 1, there were no aborts. With ab -n 2000 -c 2, there was 1 abort. I was unable to find any fields in the request_rec that indicated that the request was a "duplicate". I set flags in the .. from memory, what are the functions, lib_child_init and lib_child_exit? .. but the "extra" call comes before any call to lib_child_exit. The requests are all full ones, and TBNL gets the full request and generates a full response, even through more requests are made than are asked for with ab's -n parameter. I'll do more research, for example looking at other Apache module's source code, and google searches, or writing a custom ab, when I get a chance. Tonight I'm leaving for vacation. I'm taking my laptop but my time and access will be occasional only. I'll be back the second Tuesday after today. In create-static-file-dispatcher-and-handler, I'm pretty sure the make-array is thread unsafe where it sits. Aren't only special variables thread safe? The code as is establishes the closure at best when the dispatch table is constructed. With multiple threads, they each are sharing buf, which isn't good for several reasons. In CMUCL, I got lots of "bad address" errors in a function underneath read-sequence on the stream reading fz.jpg. The errors seemed to go away when I moved the let/make-array inside the handler. Sorry I don't have time for a real test or real patches but I hope to be on the road in an hour. It's great to see an active tbnl-devel. Jeff --- Stefan Scholl wrote: ... > (defun create-static-file-dispatcher-and-handler > (uri path content-type) > "Creates a dispatch function which will dispatch > to a handler > function which emits the file PATH with CONTENT-TYPE > if the URI > matches the SCRIPT-NAME of the request." > (let ((buf (make-array 8192 :element-type > #-:tbnl-bivalent-streams 'character > #+:tbnl-bivalent-streams > '(unsigned-byte 8)))) > (let ((handler #'(lambda () > (setf (content-type) > content-type) > (with-output-to-string (str) > (with-open-file (file path > :direction :input > > #+:tbnl-bivalent-streams :element-type > > #+:tbnl-bivalent-streams '(unsigned-byte 8)) > (do ((pos (read-sequence > buf file) (read-sequence buf file))) > ((zerop pos)) > (write-sequence buf str > :end pos))))))) > #'(lambda (request) > (when (equal (script-name request) uri) > handler))))) ... __________________________________ Do you Yahoo!? Vote for the stars of Yahoo!'s next ad campaign! http://advision.webevents.yahoo.com/yahoo/votelifeengine/ From edi at agharta.de Sat Jul 24 00:07:49 2004 From: edi at agharta.de (Edi Weitz) Date: Sat, 24 Jul 2004 02:07:49 +0200 Subject: [tbnl-devel] New version 0.2.6 Message-ID: <87u0vys1wq.fsf@bird.agharta.de> Hi! A new release is available from . Here's the relevant part from the changelog: Version 0.2.6 2004-07-24 Make CREATE-STATIC-FILE-DISPATCHER-AND-HANDLER thread-safe (caught by Jeff Caldwell) Added support for 'If-Modified-Since' request headers (provided by Stefan Scholl) Enjoy, Edi From edi at agharta.de Sat Jul 24 00:09:47 2004 From: edi at agharta.de (Edi Weitz) Date: Sat, 24 Jul 2004 02:09:47 +0200 Subject: [tbnl-devel] Patch for handling If-Modified-Since In-Reply-To: <20040722155152.GC2310@parsec.no-spoon.de> (Stefan Scholl's message of "Thu, 22 Jul 2004 17:51:52 +0200") References: <20040722155152.GC2310@parsec.no-spoon.de> Message-ID: <87pt6ms1tg.fsf@bird.agharta.de> On Thu, 22 Jul 2004 17:51:52 +0200, Stefan Scholl wrote: > A patch for handling If-Modified-Since is attached to this > e-mail. Maybe it could be useful. Thanks, I've added it to 0.2.6. (I wouldn't mind if you'd also send patches for the docs... :) Cheers, Edi. From edi at agharta.de Sat Jul 24 00:10:11 2004 From: edi at agharta.de (Edi Weitz) Date: Sat, 24 Jul 2004 02:10:11 +0200 Subject: [tbnl-devel] Patch for handling If-Modified-Since In-Reply-To: <20040723125538.GO2310@parsec.no-spoon.de> (Stefan Scholl's message of "Fri, 23 Jul 2004 14:55:38 +0200") References: <20040722155152.GC2310@parsec.no-spoon.de> <20040723125538.GO2310@parsec.no-spoon.de> Message-ID: <87llhas1ss.fsf@bird.agharta.de> On Fri, 23 Jul 2004 14:55:38 +0200, Stefan Scholl wrote: > The handling of return codes other than 200 in modlisp.lisp could > emit less data for 304 (+http-not-modified+). I've added that. Cheers, Edi. From edi at agharta.de Sat Jul 24 00:17:33 2004 From: edi at agharta.de (Edi Weitz) Date: Sat, 24 Jul 2004 02:17:33 +0200 Subject: [tbnl-devel] 1) Aborted connections 2) create-static-file-dispatcher-and-handler In-Reply-To: <20040723222254.17640.qmail@web50602.mail.yahoo.com> (Jeff Caldwell's message of "Fri, 23 Jul 2004 15:22:54 -0700 (PDT)") References: <20040723222254.17640.qmail@web50602.mail.yahoo.com> Message-ID: <87hdrys1gi.fsf@bird.agharta.de> Hi Jeff! On Fri, 23 Jul 2004 15:22:54 -0700 (PDT), Jeff Caldwell wrote: > Edi, your aborted connection patch is a good thing. I don't remember > if you noted that the aborted connections never happen until after > ab completes its run. Yes, I noticed that. As I said, I think that ab simply aborts all pending connections as soon as the tests are finished. > The requests are all full ones, and TBNL gets the full request and > generates a full response, even through more requests are made than > are asked for with ab's -n parameter. Yes, looks so. I think one ab thread is firing up new requests as fast as possible while another one is counting them and stops all as soon as enough are there. Something like that. > I'll do more research, for example looking at other Apache module's > source code, and google searches, or writing a custom ab, when I get > a chance. Tonight I'm leaving for vacation. I'm taking my laptop but > my time and access will be occasional only. I'll be back the second > Tuesday after today. Have a nice vacation! I don't think it is necessary to investigate this further unless you still see problems with mod_lisp/TBNL. But I won't stop you from doing it... :) > In create-static-file-dispatcher-and-handler, I'm pretty sure the > make-array is thread unsafe where it sits. Good catch. I've changed that. Cheers, Edi. From edi at agharta.de Sat Jul 24 00:33:35 2004 From: edi at agharta.de (Edi Weitz) Date: Sat, 24 Jul 2004 02:33:35 +0200 Subject: [tbnl-devel] Question about dispatchers and their actions In-Reply-To: <1090589817.1238.12.camel@mayo.ivanet.net> (Pete Kazmier's message of "Fri, 23 Jul 2004 09:36:57 -0400") References: <20040722145509.GA8256@kazmier.com> <20040723131026.GP2310@parsec.no-spoon.de> <1090589817.1238.12.camel@mayo.ivanet.net> Message-ID: <873c3iz1k0.fsf@bird.agharta.de> On Fri, 23 Jul 2004 09:36:57 -0400, Pete Kazmier wrote: > On Fri, 2004-07-23 at 09:10, Stefan Scholl wrote: >> But I think it should be an exception to the rule. Let the >> handler return _two_ values if he is sure that the reason for not >> processing the request is no fatal error and dispatching should >> continue. >> Normal handlers just return a string or nil if they have a >> problem. Handlers which are aware of the possibility that other >> dispatchers could catch the request when they fail return >> (values nil t) > > Thats a good idea. I keep forgetting about the usefulness of the > ability to return multiple values. With this ability, I would be > able to minimize the complexity of my dispatch functions and move > some of that logic into the handler itself which could then have the > option of opting to not process the request (defer to the next > dispatcher). I'm not sure what would be the right way. At the moment it is not specified what it means if a handler returns NIL so we could as well implement Pete's initial idea to let the next dispatcher take over in this case. Using a second return value would also be a possibility but it would require more work in modlisp.lisp because the HANDLER-BIND code already generates a second return value to denote errors. > Out of curiousity, what do you guys like to use for generating your > HTML? Are you firm believers in the whole "separation of logic and > presentation" and prefer the use of template engines? Or do you > prefer to output HTML from your TBNL apps and then let folks use CSS > to adjust the presentation as they see fit? Or do you use a > combination of both? I've written HTML-TEMPLATE for separation of logic and presentation. Not because I think it's "the way to go" but mostly because I sometimes have to work with web designers (graphical artists) who I can't bother to modify my Lisp code directly. It turned out that I use it very rarely - I think it's clumsy. I usually use CL-WHO. I personally think that "separation of logic and presentation" is a good idea in theory but it rarely works in practice, unfortunately. Cheers, Edi. From stesch at no-spoon.de Sat Jul 24 00:39:03 2004 From: stesch at no-spoon.de (Stefan Scholl) Date: Sat, 24 Jul 2004 02:39:03 +0200 Subject: [tbnl-devel] Patch for handling If-Modified-Since In-Reply-To: <87pt6ms1tg.fsf@bird.agharta.de> References: <20040722155152.GC2310@parsec.no-spoon.de> <87pt6ms1tg.fsf@bird.agharta.de> Message-ID: <20040724003903.GU2310@parsec.no-spoon.de> On 2004-07-24 02:09:47, Edi Weitz wrote: > On Thu, 22 Jul 2004 17:51:52 +0200, Stefan Scholl wrote: > > A patch for handling If-Modified-Since is attached to this > > e-mail. Maybe it could be useful. > > Thanks, I've added it to 0.2.6. > > (I wouldn't mind if you'd also send patches for the docs... :) Uaaah! And I've forgotten to add the stuff to the exports in packages.lisp. handle-if-modified-since is missing there. And rfc1123-date needs to be exported, too. People want so set their own Last-Modified. Sorry, i've just tested it inside create-static-file-dispatcher-and-handler. Oops, Stefan From edi at agharta.de Sat Jul 24 00:57:59 2004 From: edi at agharta.de (Edi Weitz) Date: Sat, 24 Jul 2004 02:57:59 +0200 Subject: [tbnl-devel] New version 0.2.7 In-Reply-To: <87u0vys1wq.fsf@bird.agharta.de> (Edi Weitz's message of "Sat, 24 Jul 2004 02:07:49 +0200") References: <87u0vys1wq.fsf@bird.agharta.de> Message-ID: <87iscexluw.fsf@bird.agharta.de> And now we have 0.2.7... Version 0.2.7 2004-07-24 Add missing exports and docs Sorry for the noise. Edi. From edi at agharta.de Sat Jul 24 21:15:44 2004 From: edi at agharta.de (Edi Weitz) Date: Sat, 24 Jul 2004 23:15:44 +0200 Subject: [tbnl-devel] New version 0.2.8 In-Reply-To: <87iscexluw.fsf@bird.agharta.de> (Edi Weitz's message of "Sat, 24 Jul 2004 02:57:59 +0200") References: <87u0vys1wq.fsf@bird.agharta.de> <87iscexluw.fsf@bird.agharta.de> Message-ID: <87658dt8cf.fsf_-_@bird.agharta.de> So it looks I was a bit sloppy yesterday... :( Version 0.2.8 2004-07-24 Fixed typo in html.lisp and improved docs (both caught by Stefan Scholl) Edi. From stesch at no-spoon.de Tue Jul 27 12:01:48 2004 From: stesch at no-spoon.de (Stefan Scholl) Date: Tue, 27 Jul 2004 14:01:48 +0200 Subject: [tbnl-devel] create-host-dispatch-proxy Message-ID: <20040727120148.GI2311@parsec.no-spoon.de> Hi! (defun create-host-dispatch-proxy (host dispatcher) "Creates a proxy dispatcher which calls the given DISPATCHER when the HOST matches." (lambda (request) (when (and (host request) (string-equal host (host request))) (funcall dispatcher request)))) [Function] create-host-dispatch-proxy host dispatcher => dispatch-fn A convenience function which will return a dispatcher which calls the given dispatcher when the host matches. Purpose: Use the known functions for creating a dispatcher but add the feature to match the host first. Example: (create-host-dispatch-proxy "www.zappa.test" (create-static-file-dispatcher-and-handler "/tbnl/test/image.jpg" (make-pathname :name "fz" :type "jpg" :version nil :defaults (load-time-value *load-pathname*)) "image/jpeg")) (create-host-dispatch-proxy "www.king.test" (create-static-file-dispatcher-and-handler "/tbnl/test/image.jpg" (make-pathname :name "elvis" :type "jpg" :version nil :defaults (load-time-value *load-pathname*)) "image/jpeg")) This delivers fz.jpg on http://www.zappa.test/tbnl/test/image.jpg and elvis.jpg on http://www.king.test/tbnl/test/image.jpg Name has the word "proxy" in it. It's the right name, but it's no web-proxy as the user would expect. Other name for the function? The documentation uses the word "dispatcher" two times but one time it's the one created and the second time it's the one given in the lambda list. Hard to understand? Not easy to use in test/test.lisp. I don't know if this code should be included into TBNL. The list of convenience functions grows on and could become a bit complex. I see it more as an example on how to solve a certain problem when you want to run one lisp image with different (virtual) webserver addresses. But feel free to add it anyway. Regards, Stefan From stesch at no-spoon.de Tue Jul 27 12:02:11 2004 From: stesch at no-spoon.de (Stefan Scholl) Date: Tue, 27 Jul 2004 14:02:11 +0200 Subject: [tbnl-devel] Tip: You can use , too Message-ID: <20040727120211.GJ2311@parsec.no-spoon.de> Hi! Looking for a right place for your "SetHandler lisp-handler"? It isn't restricted to . See http://httpd.apache.org/docs/sections.html Think about mod_lisp-pages with the extension .mlp: SetHandler lisp-handler Path must exist. When you access http://some.server.test/foo/bar.mlp directory "foo" must exist. Regards, Stefan From edi at agharta.de Tue Jul 27 12:53:40 2004 From: edi at agharta.de (Edi Weitz) Date: Tue, 27 Jul 2004 14:53:40 +0200 Subject: [tbnl-devel] Tip: You can use , too In-Reply-To: <20040727120211.GJ2311@parsec.no-spoon.de> (Stefan Scholl's message of "Tue, 27 Jul 2004 14:02:11 +0200") References: <20040727120211.GJ2311@parsec.no-spoon.de> Message-ID: <87n01lzk4r.fsf@bird.agharta.de> On Tue, 27 Jul 2004 14:02:11 +0200, Stefan Scholl wrote: > Looking for a right place for your "SetHandler lisp-handler"? > > It isn't restricted to . See > http://httpd.apache.org/docs/sections.html > > Think about mod_lisp-pages with the extension .mlp: > > > SetHandler lisp-handler > > > Path must exist. When you access http://some.server.test/foo/bar.mlp > directory "foo" must exist. As mentioned in the docs you can also use "LocationMatch." If you do that the directory "foo" doesn't have to be there. SetHandler lisp-handler (Not tested.) Cheers, Edi. From edi at agharta.de Tue Jul 27 12:59:02 2004 From: edi at agharta.de (Edi Weitz) Date: Tue, 27 Jul 2004 14:59:02 +0200 Subject: [tbnl-devel] create-host-dispatch-proxy In-Reply-To: <20040727120148.GI2311@parsec.no-spoon.de> (Stefan Scholl's message of "Tue, 27 Jul 2004 14:01:48 +0200") References: <20040727120148.GI2311@parsec.no-spoon.de> Message-ID: <87isc9zjvt.fsf@bird.agharta.de> On Tue, 27 Jul 2004 14:01:48 +0200, Stefan Scholl wrote: > I don't know if this code should be included into TBNL. The list of > convenience functions grows on and could become a bit complex. I > see it more as an example on how to solve a certain problem when you > want to run one lisp image with different (virtual) webserver > addresses. I don't think it should be added because a) it's rather easy to do it yourself and b) I think it'll rarely be used. I guess that if two virtual hosts use the same TBNL image than they're usually handled as if they were equal - otherwise you'd use Apache's virtual hosts to dispatch to different Lisp images. Cheers, Edi. From pete-tbnl-dev at kazmier.com Tue Jul 27 18:28:00 2004 From: pete-tbnl-dev at kazmier.com (pete-tbnl-dev at kazmier.com) Date: Tue, 27 Jul 2004 14:28:00 -0400 Subject: [tbnl-devel] Tip: You can use , too In-Reply-To: <87n01lzk4r.fsf@bird.agharta.de> References: <20040727120211.GJ2311@parsec.no-spoon.de> <87n01lzk4r.fsf@bird.agharta.de> Message-ID: <20040727182800.GA700@kazmier.com> On Tue, Jul 27, 2004 at 02:53:40PM +0200, Edi Weitz wrote: > As mentioned in the docs you can also use "LocationMatch." If you do > that the directory "foo" doesn't have to be there. > > > SetHandler lisp-handler > In addition to SetHandler, do not forget about AddHandler. I use it as follows: AddHandler lisp-handler .html .css This sends only .css and .html files under /tbnl to the lisp handler. From stesch at no-spoon.de Sat Jul 31 00:10:11 2004 From: stesch at no-spoon.de (Stefan Scholl) Date: Sat, 31 Jul 2004 02:10:11 +0200 Subject: [tbnl-devel] Handling HTTP errors Message-ID: <20040731001011.GH4076@parsec.no-spoon.de> Hi! Function send-output in modlisp.lisp handles the response for different return codes. E. g. there's a "The requested URL ~A was not found on this server." for the 404 (+http-not-found+). How about letting the user change this behavior? One could explain the error to the user. Maybe in another language. Or think of all the funny 404 games. Remember the SGI babies? There could be different approaches: * Simple hash A hash with error code as key and handler as value. + Easy to implement + Can handle all errors (thrown by handlers _and_ when no dispatcher is found) - Not very flexible. Just checks for the code and not for any request data. * Error dispatcher Like the normal dispatcher, but checking for the right error output handler. + Can handle all errors (thrown by handlers _and_ when no dispatcher is found) + Flexible. Can check the code _and_ request data. - A bit big. The same effort like handling normal requests. * Handler outputs itself and signals that A handler wants to return a 404. It constructs the output string and signals the fact that there's no need for TBNL to build the HTML for the output. + Easy to implement(? clean?) + Flexible - Can't handle all errors. (No handler found? Then the code won't be executed.) Regards, Stefan