From edi at agharta.de  Thu Jan  5 01:09:31 2006
From: edi at agharta.de (Edi Weitz)
Date: Thu, 05 Jan 2006 02:09:31 +0100
Subject: [rdnzl-devel] attaching handler / Tablet PC ink/ stroke/ not
	working
In-Reply-To: <uu0d0ddol.fsf@agharta.de> (Edi Weitz's message of "Thu, 22 Dec
	2005 22:28:26 +0100")
References: <43AAE1CE.80000@cs.berkeley.edu> <umzitrppa.fsf@agharta.de>
	<ufyolrpl0.fsf@agharta.de> <uu0d0ddol.fsf@agharta.de>
Message-ID: <upsn78oqc.fsf@agharta.de>

On Thu, 22 Dec 2005 22:28:26 +0100, Edi Weitz <edi at agharta.de> wrote:

> On Thu, 22 Dec 2005 18:47:39 +0100, Edi Weitz <edi at agharta.de> wrote:
>
>> My guess is that this doesn't work because of threading issues -
>> .NET is trying to call into Lisp from a thread which wasn't created
>> by Lisp.
>
> And I was right.  Here's a solution that avoids these problems and
> works for me.

Still talking to myself... :)

Here's a solution that'll enable you to keep your interactive
development environment:

  <http://weitz.de/lw-callbacks/>

With a LispWorks built like that I can run the demo fine directly from
the IDE.

Cheers,
Edi.


From lisp at spikeisland.com  Fri Jan 13 00:29:08 2006
From: lisp at spikeisland.com (lisp)
Date: Fri, 13 Jan 2006 00:29:08 +0000
Subject: [rdnzl-devel] RDNZL - bug in delegate adapter release mechanism
Message-ID: <1137112148.7503.251742658@webmail.messagingengine.com>

Hi Edi

I've tracked down what looks to be a bug in the mechanism for releasing
delegate adapters.

The foreign callable ReleaseDelegateAdapter is declared with a return
type of ffi-void-pointer but tries to return the result of remhash - a
boolean. On LWW (at least) this results in:

Condition: T cannot be converted to foreign type :POINTER. Call to 
SYSTEM::FOREIGN-CALLABLE-ENTRY-POINT-NEW-PROCESS-AUX (offset 21)  
  SYSTEM::FUNC : RDNZL::%FOREIGN-CALLABLE/RELEASEDELEGATEADAPTER

I would suggest changing it (and the corresponding initialisation and
DLL code) to have a void return type:

(ffi-define-callable (ReleaseDelegateAdapter ffi-void) 
 ((index ffi-integer)) ;; remove entry from hash table if CLR is done
 with it
 (remhash index *callback-hash*))

Also, in DelegateAdapter.cpp you have a comment on the destructor that
it doesn't appear to be working:

// the destructor notifies Lisp that this instance is no longer used -
// at least in theory, this doesn't seem to work right now
DelegateAdapter::~DelegateAdapter() {  release(indexIntoLisp); }

This may have beeen the caused by the callback/threading issues
discussed recently.  At least it has certainly worked for me here (hence
this bug report) when using a DLL based LWW image (many thanks for your
recent example of this).  Perhaps the .NET GC is running in a different
thread, so the callback was failing in a normal .EXE image?

The full traceback from the bug above seems to support this - it shows 
LWW creating a new process to handle the callback:

CL-USER 4 : 1 > :bb
#<PACKAGE COMMON-LISP-USER>

Condition: T cannot be converted to foreign type :POINTER. Call to 
SYSTEM::FOREIGN-CALLABLE-ENTRY-POINT-NEW-PROCESS-AUX (offset 21)
  SYSTEM::FUNC : RDNZL::%FOREIGN-CALLABLE/RELEASEDELEGATEADAPTER

Catch frame: MP::PROCESS-TAG

Catch frame: (NIL)

Binding frame:
  MP:*CURRENT-PROCESS* : NIL

Binding frame:  WIN32::*DDE-THREAD-INSTANCE* : #<unbound>

Binding frame:
  MQ::*ACTIONS-AFTER-SLEEPING* : NIL

Binding frame:
  MR::*ACTIONS-BEFORE-SLEEPING* : NIL

Binding frame:  SYSTEM::*READER-STATE* : #<unbound>

Binding frame:  *PACKAGE* : #<PACKAGE COMMON-LISP-USER>

Binding frame:  EDITOR::*CURRENT-BUFFER* : NIL

Binding frame:  EDITOR::*CURRENT-WINDOW* : NIL

Binding frame:  EDITOR::*EDITOR-STATE* : NIL

Binding frame:  EDITOR::*EDITOR-INPUT-STYLE* : #S(EDITOR:EDITOR-INPUT-
STYLE :KEY-BINDINGS #<EDITOR::KEY-TABLE #<EQUALP Hash Table{332}
20BCA90C> 1 331> :DELETE-SELECTION-MODE NIL :LOGICAL-CHARACTERS
#<EQUALP Hash Table{21} 20C3D364> :EXECUTE-MODE #<EDITOR::MODE-OBJECT
"Execute" 20CE37FC> :ECHO-MODE #<EDITOR::MODE-OBJECT "Echo Area"
20CE34A4> :PLIST NIL :INTERRUPT-KEYS (#S(SYSTEM::GESTURE-SPEC :DATA 103
:MODIFIERS 2) #S(SYSTEM::GESTURE-SPEC :DATA 71 :MODIFIERS 2)) :STYLE
:EMACS :POINT-ALWAYS-VISIBLE T :USE-FACE-TO-FULL-WIDTH-P T)

Binding frame:  CAPI::*USE-ACCELERATORS* : T

Binding frame:  CAPI-WIN32-LIB::*ALT-IS-META-KEY* : T

Catch frame: #:CATCHER100675

Call to (SUBFUNCTION MP::PROCESS-SG-FUNCTION MP::INITIALIZE-PROCESS-
STACK) (offset 347)
  MS::IGNORED : :DONT-KNOW

Catch frame: SYSTEM::EXIT-FOREIGN-CALLABLE

 Call to SYSTEM::FOREIGN-CALLABLE-ENTRY-POINT-NEW-PROCESS (offset 133)
     SYSTEM::FUNC : RDNZL::%FOREIGN-CALLABLE/RELEASEDELEGATEADAPTER

Call to SYSTEM::%FUNCALL1-ON-LISP-STACK (offset 34)


SYSTEM::%FUNCALL1-ON-LISP-STACK T



- Dominic Robinson


From edi at agharta.de  Fri Jan 13 07:37:30 2006
From: edi at agharta.de (Edi Weitz)
Date: Fri, 13 Jan 2006 08:37:30 +0100
Subject: [rdnzl-devel] New versions 0.8.0/0.5.0 (was: bug in delegate
 adapter release mechanism)
In-Reply-To: <1137112148.7503.251742658@webmail.messagingengine.com>
	(lisp@spikeisland.com's
	message of "Fri, 13 Jan 2006 00:29:08 +0000")
References: <1137112148.7503.251742658@webmail.messagingengine.com>
Message-ID: <ubqygmvd1.fsf@agharta.de>

Hi Dominic!

On Fri, 13 Jan 2006 00:29:08 +0000, "lisp" <lisp at spikeisland.com> wrote:

> I've tracked down what looks to be a bug in the mechanism for
> releasing delegate adapters.
>
> The foreign callable ReleaseDelegateAdapter is declared with a
> return type of ffi-void-pointer but tries to return the result of
> remhash - a boolean. On LWW (at least) this results in:
>
> Condition: T cannot be converted to foreign type :POINTER. Call to 
> SYSTEM::FOREIGN-CALLABLE-ENTRY-POINT-NEW-PROCESS-AUX (offset 21)  
>   SYSTEM::FUNC : RDNZL::%FOREIGN-CALLABLE/RELEASEDELEGATEADAPTER
>
> I would suggest changing it (and the corresponding initialisation
> and DLL code) to have a void return type:
>
> (ffi-define-callable (ReleaseDelegateAdapter ffi-void) 
>  ((index ffi-integer)) ;; remove entry from hash table if CLR is done with it
>  (remhash index *callback-hash*))
>
> Also, in DelegateAdapter.cpp you have a comment on the destructor
> that it doesn't appear to be working:
>
> // the destructor notifies Lisp that this instance is no longer used -
> // at least in theory, this doesn't seem to work right now
> DelegateAdapter::~DelegateAdapter() {  release(indexIntoLisp); }
>
> This may have beeen the caused by the callback/threading issues
> discussed recently.  At least it has certainly worked for me here
> (hence this bug report) when using a DLL based LWW image (many
> thanks for your recent example of this).  Perhaps the .NET GC is
> running in a different thread, so the callback was failing in a
> normal .EXE image?

Makes perfect sense to me.  I've updated both the Lisp and the C++
code according to your recommendations - see ChangeLogs below.

However, I'm a bit in a hurry and I couldn't figure out an easy
example which shows that ReleaseDelegateAdapter is actually called -
at least not from a DLL-based LWW image.  Could you please check if it
works and report back to the list if possible?

Thanks a lot,
Edi.



  Lisp code:
  ----------

  Version 0.8.0
  2006-01-13
  Fix mechanism which releases delegate adapters (thanks to Dominic Robinson)
  Updated to DLL version 0.5.0

  <http://weitz.de/files/RDNZL.tar.gz>


  C++ code (optional):
  --------------------

  Version 0.5.0
  2006-01-13
  Fix mechanism which releases delegate adapters (thanks to Dominic Robinson)

  <http://weitz.de/files/RDNZL_cpp.tar.gz>


From edi at agharta.de  Mon Jan 16 01:50:16 2006
From: edi at agharta.de (Edi Weitz)
Date: Mon, 16 Jan 2006 02:50:16 +0100
Subject: [rdnzl-devel] Re: New versions 0.8.0/0.5.0
In-Reply-To: <1137373139.2451.251914884@webmail.messagingengine.com>
	(lisp@spikeisland.com's
	message of "Mon, 16 Jan 2006 00:58:59 +0000")
References: <1137112148.7503.251742658@webmail.messagingengine.com>
	<ubqygmvd1.fsf@agharta.de>
	<1137373139.2451.251914884@webmail.messagingengine.com>
Message-ID: <upsmtrlev.fsf@agharta.de>

On Mon, 16 Jan 2006 00:58:59 +0000, "lisp" <lisp at spikeisland.com> wrote:

> I am happy to report that this works!

Great!

> If you run the function gc-test below in a LWW DLL image, you can
> see the delegate adapter for the Click event being reclaimed and
> releasing the closure from the callback hash table.
>
> If you open a process browser window *before* running this, you will
> see the process created by LWW to handle the callback from .NET's gc
> thread.  Inspecting this process's arguments reveals
> (rdnzl:%foreign-callable/releasedelegateadapter).  This process is
> used for all subsequent callbacks from the gc thread.  If you kill
> it, a new one will be created for the next gc callback.
>
> If you run this code in a conventional .exe LWW image you do not get
> the callback from .NET's gc thread.

Thanks for the detailed info.  I'll try this tomorrow.

> I have run into a further problem - with LWW this time I think.  As
> far as I can tell, LWW will occasionally run a foreign callback on
> the wrong thread resulting in a stack overflow exception condition
> being raised.  With a form running in a new process and generating
> event callbacks, the listener process occasionally reports a stack
> overflow and shows the callback happening in *its* backtrace -
> presumably confusing the stack check.  I've not yet been able to
> verify this or reproduce it reliably.
>
> Any thoughts or suggestions would be welcome...

No idea.  But it would be /very/ helpful if you could reproduce this
and submit a bug report to LispWorks support.  As you'll know they're
currently working on the 5.0 release which will have some changes in
the threading code as well.  It'd be nice if they could fix that
before they release the new version.

Thanks again,
Edi.


From fateman at cs.berkeley.edu  Tue Jan 24 20:17:31 2006
From: fateman at cs.berkeley.edu (Richard Fateman)
Date: Tue, 24 Jan 2006 12:17:31 -0800
Subject: [rdnzl-devel] new rdnzl 0.8.0 doesn't compile for me, Allegro
Message-ID: <43D68B5B.4020900@cs.berkeley.edu>

This on Allegro 7.0...

;;; Compiling file h:\lisp\RDNZL-0.8.0\port-acl.lisp
;         Fast loading from bundle code\efmacs.fasl.
;;; Writing fasl file h:\lisp\RDNZL-0.8.0\port-acl.fasl
Error: Object free-dot-net-container cannot be written to a compiled
       file unless an applicable method is defined on make-load-form.
  [condition type: program-error]

Any help would be appreciated.




From edi at agharta.de  Tue Jan 24 22:37:59 2006
From: edi at agharta.de (Edi Weitz)
Date: Tue, 24 Jan 2006 23:37:59 +0100
Subject: [rdnzl-devel] new rdnzl 0.8.0 doesn't compile for me, Allegro
In-Reply-To: <43D68B5B.4020900@cs.berkeley.edu> (Richard Fateman's message
	of "Tue, 24 Jan 2006 12:17:31 -0800")
References: <43D68B5B.4020900@cs.berkeley.edu>
Message-ID: <umzhlxneg.fsf@agharta.de>

On Tue, 24 Jan 2006 12:17:31 -0800, Richard Fateman <fateman at cs.berkeley.edu> wrote:

> This on Allegro 7.0...
>
> ;;; Compiling file h:\lisp\RDNZL-0.8.0\port-acl.lisp
> ;         Fast loading from bundle code\efmacs.fasl.
> ;;; Writing fasl file h:\lisp\RDNZL-0.8.0\port-acl.fasl
> Error: Object free-dot-net-container cannot be written to a compiled
>        file unless an applicable method is defined on make-load-form.
>   [condition type: program-error]
>
> Any help would be appreciated.

Hmm, I can't reproduce that here, I also have Allegro 7.0.  Do you
have loaded all available updates from Franz?  Could you try again
after deleting all FASL files from RDNZL?

No idea what's happening there...

Cheers,
Edi.

PS: Any news from Franz on the callback issue from December?


From edi at agharta.de  Thu Jan 26 00:06:54 2006
From: edi at agharta.de (Edi Weitz)
Date: Thu, 26 Jan 2006 01:06:54 +0100
Subject: [rdnzl-devel] attaching handler / Tablet PC ink/ stroke/ not
	working
In-Reply-To: <ufyolrpl0.fsf@agharta.de> (Edi Weitz's message of "Thu, 22 Dec
	2005 18:47:39 +0100")
References: <43AAE1CE.80000@cs.berkeley.edu> <umzitrppa.fsf@agharta.de>
	<ufyolrpl0.fsf@agharta.de>
Message-ID: <ud5ifc0o1.fsf@agharta.de>

On Thu, 22 Dec 2005 18:47:39 +0100, Edi Weitz <edi at agharta.de> wrote:

> I forgot to mention: My guess is that this doesn't work because of
> threading issues - .NET is trying to call into Lisp from a thread
> which wasn't created by Lisp. [...] I'm not familiar enough with
> AllegroCL but maybe it's similar there.

Hmm, seems that it's slightly more complicated in this case.  I got a
copy of AllegroCL 8.0 for Windows today and played around with it a
bit.  Still no solution, unfortunately, but here are some more data
points:

1. RDNZL seems to work fine with AllegroCL 8.0.  I had already checked
   with the beta, so no surprise.

2. The problem with the "Microsoft.Ink" library remains the same,
   though.

3. I rummaged around in the AllegroCL docs and found the following:

     "A note on calling into Lisp from threads started by foreign code

      [...]

      In an :os-threads Allegro CL, however, it is legitimate for a
      thread started outside Lisp to call into Lisp via any
      foreign-callable function. Some extra work has to be done to
      create a Lisp process to represent that thread within the lisp
      world. That extra work is performed by a 'customs agent' process
      that is started by a call to START-CUSTOMS. See the
      documentation for that function for details."

   Aha!  So we just have to call START-CUSTOMS and everything works?
   No, unfortunately not.

4. The reference entry for START-CUSTOMS says that the function is
   deprecated, so the paragraph from above is bogus.  Nowadays,
   START-CUSTOM just calls START-SCHEDULER and (again, according to
   the docs) this is actually never necessary because starting a
   process will call this function implicitely.

5. So, back to square one.  But I thought, hey, what will happen if I
   start the function DEMO[1] not from the REPL but from a thread?

     (mp:process-run-function "demo" #'demo)

   The problem is still the same but MP:PROCESS-RUN-FUNCTION does
   /not/ return until I close the .NET window.  So, /something/ is
   blocking there because the function usually returns immediately.

As I said, still no cigar.  But maybe someone more familiar with
AllegroCL's MP implementation knows what to make of this.

Cheers,
Edi.


[1] see <http://common-lisp.net/pipermail/rdnzl-devel/2005-December/000044.html>