From franks-muc at web.de  Thu Apr  3 20:34:59 2008
From: franks-muc at web.de (Frank Schorr)
Date: Thu, 03 Apr 2008 22:34:59 +0200
Subject: [drakma-devel] eof error
Message-ID: <1052748668@web.de>

This is on win32/5.0.2

I receive an eof error when accessing a web page. Backtrace below.

This happens occasionally, every second or third call to HTTP-REQUEST, 
without regular pattern, when the call is made from a muproc thread. 
My code works fine when called from the ide.

I suspect that the proper initial bindings should be made when the thread is spawned,
but I was not able to get this done.

Any help would be very welcome.

Frank



Error: End of file while reading stream #<FLEXI-STREAMS::FLEXI-BINARY-LATIN-1-IO-STREAM 2393B4BF>.
  1 (abort) Quit process.

Type :b for backtrace, :c <option number> to proceed,  or :? for other options

GPTO 1 : 1 > :<
Call to INVOKE-DEBUGGER

GPTO 2 : 1 > :b
Call to INVOKE-DEBUGGER
Call to SIGNAL
Call to CHUNGA:READ-LINE*
Call to DRAKMA::READ-STATUS-LINE
Call to (HARLEQUIN-COMMON-LISP:SUBFUNCTION (LABELS DRAKMA::FINISH-REQUEST) DRAKMA:HTTP-REQUEST)
Call to DRAKMA:HTTP-REQUEST
Call to GET-PAGE-2
Call to CLIENT-SUBMIT-FORM
Call to LET*
Interpreted call to #:REMOTE-FUNCTION223724
Call to UTIL::WORKER
Call to (HARLEQUIN-COMMON-LISP:SUBFUNCTION 1 UTIL::MAKE-PROCESS)
Call to CL-MUPROC::MUPROC-PROCESS-FUNCTION
Call to (HARLEQUIN-COMMON-LISP:SUBFUNCTION MP::PROCESS-SG-FUNCTION MP::INITIALIZE-PROCESS-STACK)

_______________________________________________________________
Schon geh?rt? Der neue WEB.DE MultiMessenger kann`s mit allen: 
http://www.produkte.web.de/messenger/?did=3016



From edi at agharta.de  Thu Apr  3 20:49:04 2008
From: edi at agharta.de (Edi Weitz)
Date: Thu, 03 Apr 2008 16:49:04 -0400
Subject: [drakma-devel] eof error
In-Reply-To: <1052748668@web.de> (Frank Schorr's message of "Thu, 03 Apr
	2008 22:34:59 +0200")
References: <1052748668@web.de>
Message-ID: <u1w5m4qbj.fsf@agharta.de>

On Thu, 03 Apr 2008 22:34:59 +0200, Frank Schorr <franks-muc at web.de> wrote:

> This is on win32/5.0.2

LispWorks, I assume.

> I receive an eof error when accessing a web page. Backtrace below.
>
> This happens occasionally, every second or third call to
> HTTP-REQUEST, without regular pattern, when the call is made from a
> muproc thread.  My code works fine when called from the ide.

Hmm, hard to say without a reproducible test case.  Does it happen
only with specific URLs or with arbitrary ones?

> I suspect that the proper initial bindings should be made when the
> thread is spawned, but I was not able to get this done.

Which initial bindings are you thinking of?


From ctdean at sokitomi.com  Thu Apr  3 21:04:05 2008
From: ctdean at sokitomi.com (Chris Dean)
Date: Thu, 03 Apr 2008 14:04:05 -0700
Subject: [drakma-devel] eof error
In-Reply-To: <1052748668@web.de> (Frank Schorr's message of "Thu,
	03 Apr 2008 22:34:59 +0200")
References: <1052748668@web.de>
Message-ID: <m23aq2wsze.fsf@jacaranda.local>


Frank Schorr <franks-muc at web.de> writes:
> This is on win32/5.0.2
>
> I receive an eof error when accessing a web page. Backtrace below.
>
> This happens occasionally, every second or third call to HTTP-REQUEST, 
> without regular pattern, when the call is made from a muproc thread. 
> My code works fine when called from the ide.

What urls are you trying to fetch?  If they are local ones that are
known to be good that's one thing, but if your downloading from the
general internet EOF and other errors are likely.  To debug this we'll
need more information.

FWIW, I have used drakma to download millions of pages of data and
have not seen the problem you where every few fetches we get an EOF.
We do often see errors but those are just errors in the environment
(lost network connection, badly behaving server, etc).

Cheers,
Chris Dean



From franks-muc at web.de  Thu Apr  3 21:10:53 2008
From: franks-muc at web.de (Frank Schorr)
Date: Thu, 03 Apr 2008 23:10:53 +0200
Subject: [drakma-devel] eof error
Message-ID: <1052776676@web.de>



> 
> On Thu, 03 Apr 2008 22:34:59 +0200, Frank Schorr <franks-muc at web.de> wrote:
> 
> > This is on win32/5.0.2
> 
> LispWorks, I assume.

Yes (version 5.0.2)

> 
> > I receive an eof error when accessing a web page. Backtrace below.
> >
> > This happens occasionally, every second or third call to
> > HTTP-REQUEST, without regular pattern, when the call is made from a
> > muproc thread.  My code works fine when called from the ide.
> 
> Hmm, hard to say without a reproducible test case.  Does it happen
> only with specific URLs or with arbitrary ones?
> 

My code tries only one, parallel threads access the same web page, 
but different URLs, because sessions are tracked. 
But it works fine from the ide thread. 

I will try to find a simple test case.

> > I suspect that the proper initial bindings should be made when the
> > thread is spawned, but I was not able to get this done.
> 
> Which initial bindings are you thinking of?

see 

http://www.mu.dk/cl-muproc#muproc-spawn

used to set up and initialize special variables in the new thread.
I thought that every thread might want its own variable for 
things like a current stream or something similar to be independent from 
other threads. It was only guessing.



_________________________________________________________________________
In 5 Schritten zur eigenen Homepage. Jetzt Domain sichern und gestalten! 
Nur 3,99 EUR/Monat! http://www.maildomain.web.de/?mc=021114



From edi at agharta.de  Thu Apr  3 21:43:25 2008
From: edi at agharta.de (Edi Weitz)
Date: Thu, 03 Apr 2008 17:43:25 -0400
Subject: [drakma-devel] eof error
In-Reply-To: <1052776676@web.de> (Frank Schorr's message of "Thu, 03 Apr
	2008 23:10:53 +0200")
References: <1052776676@web.de>
Message-ID: <uwsne398i.fsf@agharta.de>

On Thu, 03 Apr 2008 23:10:53 +0200, Frank Schorr <franks-muc at web.de> wrote:

> My code tries only one, parallel threads access the same web page,
> but different URLs, because sessions are tracked.  But it works fine
> from the ide thread.

Is that a local server?  Are you sure it can cope with parallel
requests?  Are the different threads using the same session?  Could
that be the problem?

> I will try to find a simple test case.

That'd be the best.

> I thought that every thread might want its own variable for things
> like a current stream or something similar to be independent from
> other threads. It was only guessing.

That shouldn't be a problem if you only use "pure" Drakma calls.


From franks-muc at web.de  Thu Apr  3 22:33:02 2008
From: franks-muc at web.de (Frank Schorr)
Date: Fri, 04 Apr 2008 00:33:02 +0200
Subject: [drakma-devel] eof error
Message-ID: <1052868990@web.de>



> 
> Is that a local server?  Are you sure it can cope with parallel
> requests?  Are the different threads using the same session?  Could
> that be the problem?
> 

It is a public server in the www which has many simultaneous users.
I believe my session handling is correct, but it is not heavily tested.
What makes me confident is that my program is 
sometimes successful. I'm always doing the same requests 
(apart from the session information) with 3 threads and about 20 requests in total.
The eofs occur at random.

My program has done some 10.000 requests from this site when run from the ide in the past.

Do you believe the server could generate the error ? Wouldn't drakma handle it then ?

May be this is a way to kick out a robot ? (They are using captchas at some places (which are not
accessed by me)). 

I'm also not aware of using destructive operations...

> > I will try to find a simple test case.
> 
> That'd be the best.
> 

I noticed that this is not easy.  It appears that I will need some more time for this.

______________________________________________________
Bis 50 MB Dateianh?nge? Kein Problem!
http://freemail.web.de/club/landingpage.htm/?mc=025556



From ctdean at sokitomi.com  Thu Apr  3 23:03:01 2008
From: ctdean at sokitomi.com (Chris Dean)
Date: Thu, 03 Apr 2008 16:03:01 -0700
Subject: [drakma-devel] eof error
In-Reply-To: <1052868990@web.de> (Frank Schorr's message of "Fri,
	04 Apr 2008 00:33:02 +0200")
References: <1052868990@web.de>
Message-ID: <m2fxu2v8wq.fsf@jacaranda.local>


Frank Schorr <franks-muc at web.de> writes:
> Do you believe the server could generate the error ? Wouldn't drakma
> handle it then ?

Roughly, the right way to handle an error between the client and the
server is to pass it back to the caller.  It seems like that's what
drakma is doing.

> May be this is a way to kick out a robot ? (They are using captchas
> at some places (which are not accessed by me)).

All robots (including yours) should obey their robots.txt.  If you are
accessing pages they don't want you to access (I'm not saying you are,
just making a comment) you might see unexpected behavior.

Cheers,
Chris Dean


From edi at agharta.de  Thu Apr  3 23:14:51 2008
From: edi at agharta.de (Edi Weitz)
Date: Thu, 03 Apr 2008 19:14:51 -0400
Subject: [drakma-devel] eof error
In-Reply-To: <1052868990@web.de> (Frank Schorr's message of "Fri, 04 Apr
	2008 00:33:02 +0200")
References: <1052868990@web.de>
Message-ID: <usky23504.fsf@agharta.de>

On Fri, 04 Apr 2008 00:33:02 +0200, Frank Schorr <franks-muc at web.de> wrote:

> The eofs occur at random.

Errors that occur in multi-threaded programs often seem to be random
although they actually aren't... :)

> Do you believe the server could generate the error ?

Depends on how the server is implemented.  /Maybe/ it somehow can't
cope with lots of simultaneous requests.  But if it's a public server
used by many other people, this is very unlikely.

> Wouldn't drakma handle it then ?

If it's really an EOF, then there's no way to handle it, of course.

> May be this is a way to kick out a robot ? (They are using captchas
> at some places (which are not accessed by me)).

That'd be a /very/ strange way of doing it.  I don't think so.

> I'm also not aware of using destructive operations...

Do you use global variables?  What about the cookies, for example?
(Not that I'd have an idea why this should result in an apparent EOF,
but hey...)

Another way of debugging this would be to try from another Lisp and/or
another OS.  There's always a chance that what you're seeing is a bug
in your Lisp.  Or, OMG, a bug in Drakma....

(And I seem to remember there's some limit on simultaneous outgoing
connections on never versions of Windows.  But, again, that shouldn't
result in EOFs.)


From franks-muc at web.de  Fri Apr  4 10:25:07 2008
From: franks-muc at web.de (Frank Schorr)
Date: Fri, 04 Apr 2008 12:25:07 +0200
Subject: [drakma-devel] eof error
Message-ID: <1053567184@web.de>


> Another way of debugging this would be to try from another Lisp and/or
> another OS.  There's always a chance that what you're seeing is a bug
> in your Lisp.  Or, OMG, a bug in Drakma....

ok.
I'm using 'pure' Drakma. I do not have access to other lisps and OSs.
However, I replaced Drakma's http-request with that of aserve and received no errors,
but an empty page sometimes and randomly. Now I'm sure the problem is outside of 
lisp and more on the side of the server, or somewhere in between.

Now I handle the eof error and return an empty page, and drakma and aserve behave the same. 
If I receive an empty page, I repeat the request, what is ok for me. 

What I learned now is, that multi-threaded access to this particular server 
(Oracle application server) takes the same time as 
single threaded serial access. And this is disappointing.

Thank you for your help and suggestions.


_________________________________________________________________________
In 5 Schritten zur eigenen Homepage. Jetzt Domain sichern und gestalten! 
Nur 3,99 EUR/Monat! http://www.maildomain.web.de/?mc=021114



From mk234 at o2.pl  Wed Apr  9 21:22:37 2008
From: mk234 at o2.pl (Marek Kowerski)
Date: Wed, 09 Apr 2008 23:22:37 +0200
Subject: [drakma-devel] Drakma asdf-install problem
Message-ID: <1207776157.7297.14.camel@localhost>


I,m getting this error while installing drakma with asdf-install on 
gentoo-amd64 with sbcl 1.0.15: 


Downloading 42624 bytes from http://weitz.de/files/drakma.tar.gz ... 
Installing /root/DRAKMA.asdf-install-tmp in /usr/lib64/sbcl/site/,/usr/ 
lib64/sbcl/site-systems/ 
drakma-0.11.5/ 
drakma-0.11.5/CHANGELOG.txt 
drakma-0.11.5/cookies.lisp 
drakma-0.11.5/doc/ 
drakma-0.11.5/doc/index.html 
drakma-0.11.5/drakma.asd 
drakma-0.11.5/packages.lisp 
drakma-0.11.5/read.lisp 
drakma-0.11.5/request.lisp 
drakma-0.11.5/specials.lisp 
drakma-0.11.5/util.lisp 


; compiling file "/usr/lib/sbcl/site/drakma-0.11.5/ 
request.lisp" (written 21 MAR 2008 11:40:04 PM): 
; compiling (IN-PACKAGE :DRAKMA) 
; compiling (DEFUN DETERMINE-BODY-FORMAT ...) 
; compiling (DEFUN SEND-CONTENT ...) 
; compiling (DEFUN MAKE-FORM-DATA-FUNCTION ...) 
; compiling (DEFUN READ-BODY ...); compilation aborted because of 
fatal error: 
                                 ;   READ failure in COMPILE-FILE: 
                                 ;     SB-INT:SIMPLE-READER-PACKAGE- 
ERROR at 24225 (line 460, column 42) on #<SB-SYS:FD-STREAM for "file / 
usr/lib/sbcl/site/drakma-0.11.5/request.lisp" {1003824541}>: 
                                 ;       Symbol "STREAM-FD" not found 
in the CL+SSL package. 
; compilation aborted after 0:00:01 


I need drakma because it's dependency of cl-selenium. Please help. 







From edi at agharta.de  Wed Apr  9 21:38:06 2008
From: edi at agharta.de (Edi Weitz)
Date: Wed, 09 Apr 2008 23:38:06 +0200
Subject: [drakma-devel] Drakma asdf-install problem
In-Reply-To: <1207776157.7297.14.camel@localhost> (Marek Kowerski's message
	of "Wed, 09 Apr 2008 23:22:37 +0200")
References: <1207776157.7297.14.camel@localhost>
Message-ID: <uzls2k8u9.fsf@agharta.de>

On Wed, 09 Apr 2008 23:22:37 +0200, Marek Kowerski <mk234 at o2.pl> wrote:

> fatal error: 
>                                  ;   READ failure in COMPILE-FILE: 
>                                  ;     SB-INT:SIMPLE-READER-PACKAGE- 
> ERROR at 24225 (line 460, column 42) on #<SB-SYS:FD-STREAM for "file / 
> usr/lib/sbcl/site/drakma-0.11.5/request.lisp" {1003824541}>: 
>                                  ;       Symbol "STREAM-FD" not found 
> in the CL+SSL package. 

Do you have the latest version of CL-SSL?


From ivan at toshkov.org  Thu Apr 10 07:51:09 2008
From: ivan at toshkov.org (Ivan Toshkov)
Date: Thu, 10 Apr 2008 10:51:09 +0300
Subject: [drakma-devel] Drakma asdf-install problem
In-Reply-To: <1207776157.7297.14.camel@localhost>
References: <1207776157.7297.14.camel@localhost>
Message-ID: <7c23adaa0804100051u34d636c1xe106e4bd519dd983@mail.gmail.com>

On Thu, Apr 10, 2008 at 12:22 AM, Marek Kowerski <mk234 at o2.pl> wrote:
>
>  I,m getting this error while installing drakma with asdf-install on
>  gentoo-amd64 with sbcl 1.0.15:
>

BTW, do you know there is a gentoo lisp overlay[1], which has latest
versions of drakma and many more packages?

[1] http://repo.or.cz/w/gentoo-lisp-overlay.git

-- 
"...Please don't assume Lisp is only useful for Animation and Graphics,
AI, Bioinformatics, B2B and E-Commerce, Data Mining, EDA/Semiconductor
applications, Expert Systems, Finance, Intelligent Agents, Knowledge
Management, Mechanical CAD, Modeling and Simulation, Natural Language,
Optimization, Research, Risk Analysis, Scheduling, Telecom, and Web
Authoring just because these are the only things they happened to
list."
 -- Kent Pitman


From mk234 at o2.pl  Thu Apr 10 19:05:56 2008
From: mk234 at o2.pl (Marek Kowerski)
Date: Thu, 10 Apr 2008 21:05:56 +0200
Subject: [drakma-devel] Re  asdf install problem
Message-ID: <1207854356.6557.42.camel@localhost>

Thank you for quick response!

I have managed to install drakma using Lispy, and then cl-selenium using
asdf-install. 
I know about gentoo overlay, but I have decided to use asdf-install,
because this way I can always have latest versions, and all packages are
installed in my home directory. And I don't need to use
packages.keywords and remember to setup "~amd64" keywords for every
possible lisp package.
Because cl+ssl is not asdf-installable, I did 
(asdf:oos 'asdf:load-op 'cl+ssl). 
I have cl+ssl-2007-07-07, but it is from tarballs not cvs.
Lispy have installed drakma-0.11.3 not 0.11.5.
I think asdf-install is a great advantage of lisp, very convenient, just
jak CPAN for perl. All packages should be asdf-insallabe and repository
must be consistent if the CL wants to be more popular. 
Thank you again for your kind response, and for Hunchentoot, which the
more I try the more I like.

MK




From daniel at sentivision.com  Fri Apr 11 06:27:46 2008
From: daniel at sentivision.com (Daniel Janus)
Date: Fri, 11 Apr 2008 08:27:46 +0200
Subject: [drakma-devel] Re  asdf install problem
In-Reply-To: <1207854356.6557.42.camel@localhost>
References: <1207854356.6557.42.camel@localhost>
Message-ID: <1207895266.5779.2.camel@daniel>

Dnia 2008-04-10, czw o godzinie 21:05 +0200, Marek Kowerski pisze:

> Because cl+ssl is not asdf-installable, I did 
> (asdf:oos 'asdf:load-op 'cl+ssl). 

Actually, cl+ssl is asdf-installable -- I tried it only a while ago and
it does work.  However, the version it installs is cl+ssl-2007-07-07, so
it shouldn't make any difference as far as your setup is concerned.  :)

Best regards,
Daniel Janus




From mk234 at o2.pl  Fri Apr 11 20:38:15 2008
From: mk234 at o2.pl (Marek Kowerski)
Date: Fri, 11 Apr 2008 22:38:15 +0200
Subject: [drakma-devel] Re  asdf install problem
In-Reply-To: <1207895266.5779.2.camel@daniel>
References: <1207854356.6557.42.camel@localhost>
	<1207895266.5779.2.camel@daniel>
Message-ID: <1207946295.6563.10.camel@localhost>


Dnia 11-04-2008, Pt o godzinie 08:27 +0200, Daniel Janus pisze:
> Dnia 2008-04-10, czw o godzinie 21:05 +0200, Marek Kowerski pisze:
> 
> > Because cl+ssl is not asdf-installable, I did 
> > (asdf:oos 'asdf:load-op 'cl+ssl). 
> 
> Actually, cl+ssl is asdf-installable -- I tried it only a while ago and
> it does work.  However, the version it installs is cl+ssl-2007-07-07, so
> it shouldn't make any difference as far as your setup is concerned.  :)
> 
> Best regards,
> Daniel Janus

Right, it is. My mistake. And rather unfortunately it is  2007-07-07.
I will try cvs version during the weekend.

Best regards
MK