[hunchentoot-devel] Hunchentoot and threads

[Hans Hübner]

Hi Cyrus,

On Mon, Nov 17, 2008 at 16:17, Cyrus Harmon <ch-tbnl at bobobeach.com> wrote:
> On Nov 16, 2008, at 11:08 AM, Hans Hübner wrote:

>> when you see worker threads accumulate, do you also see that there is
>> a large number of connections to Hunchentoot that are not being
>> closed?  Or are there just threads, but no connections?  Are you
>> running Hunchentoot behind a proxy/frontend, or standalone?  How many
>> dead workers did you see?
>
> So, how do I see if the connections are still open?

On the shell, use "netstat -nf inet | grep [port]", replacing [port]
by the port number that the worker thread is serving (35892 and 34650
are the ports for the two workers in the thread list below).

> It's only two so far, but here's what it looks like:

So this is the situation in which Hunchentoot does not accept more requests?

>
> CL-USER> (sb-thread:list-all-threads)
> (#<SB-THREAD:THREAD "repl-thread" RUNNING {5CBD0411}>
>  #<SB-THREAD:THREAD "reader-thread" RUNNING {5CBD02C1}>
>  #<SB-THREAD:THREAD "control-thread" RUNNING {5CBD0169}>
>  #<SB-THREAD:THREAD "Hunchentoot worker (client:
> 66.255.53.123:35892)" RUNNING {5CDF8E91}>
>  #<SB-THREAD:THREAD "Hunchentoot worker (client:
> 66.255.53.123:34650)" RUNNING {5CD329A1}>
>  #<SB-THREAD:THREAD "auto-flush-thread" RUNNING {5CBBFF79}>
>  #<SB-THREAD:THREAD "Hunchentoot acceptor (*:443)" RUNNING {5A5C7019}>
>  #<SB-THREAD:THREAD "Hunchentoot acceptor (*:80)" RUNNING {5A3D80B9}>
>  #<SB-THREAD:THREAD "Swank 4005" RUNNING {5A2304A1}>
>  #<SB-THREAD:THREAD "initial thread" RUNNING {598D7BE1}>)
>
> Here's a backtrace from thread 4:

I may be missing something, but to me, it seems as if this backtrace
is coming straight from a Slime REPL threads.  Are you sure that it is
the backtrace of one of the workers?

>
> 0: (SB-DEBUG::MAP-BACKTRACE #<CLOSURE (LAMBDA #) {5A4E49D5}>)[:EXTERNAL]
> 1: (BACKTRACE 536870911 #<SYNONYM-STREAM :SYMBOL *TERMINAL-IO*
> {5812FA99}>)
> 2: ((FLET SB-UNIX::INTERRUPTION))
> 3: ((FLET #:WITHOUT-INTERRUPTS-BODY-[INVOKE-INTERRUPTION]10))
> 4: (SB-SYS:INVOKE-INTERRUPTION
>     #<FUNCTION (FLET SB-UNIX::INTERRUPTION) {58049035}>)
> 5: ("foreign function: call_into_lisp")
> 6: ("foreign function: post_signal_tramp")
> 7: ("foreign function: nanosleep")
> 8: ((LAMBDA ()))
> 9: (SB-C::INVOKE-WITH-SAVED-FP-AND-PC #<CLOSURE (LAMBDA #) {5A4D929D}>)
> 10: (SB-UNIX:NANOSLEEP 0 150000016)
> 11: (SLEEP 0.15)
> 12: (SWANK-BACKEND::FLUSH-STREAMS)
> 13: (SWANK-BACKEND::FLUSH-STREAMS)[:EXTERNAL]
> 14: ((FLET SB-THREAD::WITH-MUTEX-THUNK))
> 15: ((FLET #:WITHOUT-INTERRUPTS-BODY-[CALL-WITH-MUTEX]477))
> 16: (SB-THREAD::CALL-WITH-MUTEX
>      #<CLOSURE (FLET SB-THREAD::WITH-MUTEX-THUNK) {29E3DDE5}>
>      #S(SB-THREAD:MUTEX
>         :NAME "thread result lock"
>         :%OWNER #<SB-THREAD:THREAD "auto-flush-thread" RUNNING
> {5CBBFF79}>
>         :STATE 1)
>      #<SB-THREAD:THREAD "auto-flush-thread" RUNNING {5CBBFF79}>
>      T)
> 17: ((LAMBDA ()))
> 18: ("foreign function: call_into_lisp")
> 19: ("foreign function: funcall0")
> 20: ("foreign function: new_thread_trampoline")
> 21: ("foreign function: _pthread_create")
>
> Also, I don't know if it's related, but I see lots of these messages
> in the log:
>
> [2008-11-17 06:59:47 [ERROR]] Error while processing connection: I/O
> timeout reading #<SB-SYS:FD-STREAM for "a socket" {5A465A29}>.

This is normal, although it needs fixing, two.  Usually, clients leave
HTTP connections to servers open and servers close them whenever they
deem it be neccessary as a part of their resource management strategy.
 Hunchentoot does this, too, by setting a timeout on all TCP sockets
which results in a "I/O timeout" error being generated if any
connection is idle for more than a configurable number of seconds.
This timeout error must really be caught and muted, as it is no
exceptional situation, but rather the simple form of resource
management that Hunchentoot implements.

-Hans