[hunchentoot-devel] serious H'toot performance problem on CCL OSX ?

Thu Aug 19 10:27:25 UTC 2010

On Aug 18, 2010, at 10:11 PM, Hans Hübner wrote:

> John,
> 
> what you are seeing is probably related to the fact that the default
> multi threaded task master of Hunchentoot is not clever about creating
> threads.  It creates a new thread for every incoming connection, so it
> is easy to force it into situations where too many threads are
> created.
> 
> Personally, I am not using multi threaded Hunchentoot.  Instead, I run
> it single threaded and behind a HTTP caching proxy (squid) that is
> configured to use if-modified-since headers to negotiate reloading
> stuff from the Hunchentoot backend.  That strategy works well and
> copes with all load patterns that I have tried (and I have tried a
> few, using both ab and Tsung).
> 
> If you absolutely need threads, try the threading patch that has been
> submitted by Scott McKay a few weeks ago.  It prevents Hunchentoot
> from creating threads without bounds, and might solve the problem for
> you.  We will be incorporating that patch once it has been separated
> from the xcvb changes that won't go into the Hunchentoot mainline.
> Any reports on how it helps with artificial loads tests would be
> appreciated.

Here's a simple webserver that avoids H'toot altogether.  It has a thread manager
(make-safe-process) that avoids making too many threads at once.  The threshold *max-procs* is very low
and it never seems to hit the limit (because it prints warnings when it does).

If you run it as (do-dumb-server-loop :port NNNN)  it runs great with just one request at 
a time, but fails with more than one thread (just like H'toot).
Moreover, even in single-connection mode, the server performance deteriorates
as one does more runs.

So it almost looks to me that CCL threads on the the mac might be broken somehow.

;; ===========================================
;; simple threaded webserver to exercise threads and sockets

(eval-when (load eval compile)
 (require 'bordeaux-threads)
 (require 'usocket))

(defparameter *lock* (bt:make-recursive-lock "proc-lock"))
(defparameter *max-procs* 10) ;; max number of threads we allow

(let ((nprocs 0)) ;; counter for how many threads are running
  (defun safe-make-process (func)
    "A function that launches FUNC in a new thread only if 
there are not more than *MAX-PROCS* running; otherwise it sleeps
for a bit"
    (bt:with-lock-held (*lock*)
      (format t "Making proc with ~A already existing~%" nprocs)
      (force-output))
    ;; sleep until some processes die
    (loop
       while
	 (bt:with-lock-held (*lock*)
	   (> nprocs *max-procs*))
       do
	 (format t "Warning - nprocs=~A~%" nprocs) (force-output)
	 (sleep 0.1))
    ;;
    (bt:with-lock-held (*lock*)
      (incf nprocs))
    (bt:make-thread 
     (lambda ()
       (unwind-protect ;; protect the decrement of the counter
	    (funcall func)
	 (bt:with-lock-held (*lock*)
	   (decf nprocs)))))))

(defun do-dumb-server-loop (&key (port 9001))
  (loop
     with listener = (usocket:socket-listen "127.0.0.1" port)
     for sock = (usocket:socket-accept listener)
     when sock
     do
       (safe-make-process
	(lambda ()
	  ;; I think we need to close over sock here or this thread's
	  ;; loop might change what REPLY-FUNC sees
	  (let ((this-sock sock)) 
	    (reply-func this-sock)))
       )))

(defparameter *response*
"HTTP/1.1 200 OK
Server: dumb
Connection: Close
Content-Type: text/plain

This is the content.
")

(defun reply-func (sock) ;; read headers and send content
  (let ((s (usocket:socket-stream sock)))
    (ignore-errors
      ;; eat client headers
      (loop for line = (read-line s nil nil) 
	 until (< (length line) 3))
      ;; and send response
      (write-string *response* s))
    ;; the socket close is protected by ignore-errors
    (usocket:socket-close sock)))

;; ===========================================

> 
> For the SBCL crash:  This should be reported to the SBCL folks. I
> never had a lot of faith in SBCL's thread implementation and it also
> is possible that the locking patterns inside of Hunchentoot trigger
> the misbehavior, but we need to have detailed advise how to fix that,
> if we need to.
> 
> -Hans
> 
> On Thu, Aug 19, 2010 at 01:53, JTK <jetmonk at gmail.com> wrote:
>> 
>> Hello,
>> 
>> I'm finding that the speed of hunchentoot falls drastically for more than 1 simultaneous connection,
>> at least for CCL on OS X.  And more than a few connections just fails.
>> 
>> I downloaded the latest svn version from http://bknr.net/html/
>> 
>> To reduce any complicating factors, I loaded just h'toot and ran ccl on the shell command line, not in slime.
>> 
>> I tried a simple page:
>> 
>> CL-USER> (hunchentoot:start (make-instance 'hunchentoot:acceptor :port 4242))
>> #<ACCEPTOR (host *, port 4242)>
>> CL-USER> (hunchentoot:define-easy-handler (say-yo :uri "/yo") ()
>>  (setf (hunchentoot:content-type*) "text/plain")
>>  "Yo! This. Is. Content")
>> SAY-YO
>> 
>> Then I tried the ab (apache bench) benchmark with c=1 (1 connection), and 500
>> iterations:
>> 
>> $ ab -n 500 -c 1   http://127.0.0.1:4242/yo
>> 
>>  HTML transferred:       10500 bytes
>>  Requests per second:    138.26 [#/sec] (mean)   <- ##### note speed ######
>>  Time per request:       7.233 [ms] (mean)
>>  Time per request:       7.233 [ms] (mean, across all concurrent requests)
>>  Transfer rate:          22.82 [Kbytes/sec] received
>> 
>> 
>> Then I tried it with TWO concurrent connections:
>> 
>> $ ab -n 500 -c 2   http://127.0.0.1:4242/yo
>> 
>>  HTML transferred:       10500 bytes
>>  Requests per second:    29.10 [#/sec] (mean)    <- ##### note speed ######
>>  Time per request:       68.722 [ms] (mean)
>>  Time per request:       34.361 [ms] (mean, across all concurrent requests)
>>  Transfer rate:          4.80 [Kbytes/sec] received
>> 
>> It's nearly 5x slower with just 2 simultaneous connections, going from 138 to 34 requests/sec.
>> 
>> 10 connections gives the same speed as 2, and 30 fails because connection was reset by peer (maybe my fault for not having enough open files available, but it should never open more than 30 filehandles at once, no?).
>> 
>> I'm running on a Mac OS X 10.6 with CCL "Version 1.4-r13119  (DarwinX8664)"
>> 
>> Can anyone else reproduce this?  Is it some threading problem?
>> 
>> John
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> _______________________________________________
>> tbnl-devel site list
>> tbnl-devel at common-lisp.net
>> http://common-lisp.net/mailman/listinfo/tbnl-devel
>> 
> 
> _______________________________________________
> tbnl-devel site list
> tbnl-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/tbnl-devel