From Dr.Alain.Picard at gmail.com  Fri Oct  2 02:05:32 2009
From: Dr.Alain.Picard at gmail.com (Alain Picard)
Date: Fri, 02 Oct 2009 12:05:32 +1000
Subject: [elephant-devel] Bug, omission, or desired behaviour?
Message-ID: <873a62eej7.fsf@gmail.com>


Hello all,

I've noticed that elephant blithely ignores type declarations on slots;
e.g. something like this:

(deftype position ()
  `(integer 1 99))


(defclass foo-object ()
  ((name          :type string
                  :index t
		  :reader  foo-name
                  :initarg :name)
   (position      :type     position
                  :accessor foo-position
                  :initform 1))
  (:metaclass persistent-metaclass))

Elephant will happily let me store 100 in the foo-object position slot.  

Cheers,
                Alain Picard

-- 
Please read about why Top Posting
is evil at: http://en.wikipedia.org/wiki/Top-posting
and http://www.dickalba.demon.co.uk/usenet/guide/faq_topp.html
Please read about why HTML in email is evil at: http://www.birdhouse.org/etc/evilmail.html




From sky at viridian-project.de  Fri Oct  2 07:43:10 2009
From: sky at viridian-project.de (Leslie P. Polzer)
Date: Fri, 2 Oct 2009 09:43:10 +0200 (CEST)
Subject: [elephant-devel] Bug, omission, or desired behaviour?
In-Reply-To: <873a62eej7.fsf@gmail.com>
References: <873a62eej7.fsf@gmail.com>
Message-ID: <964e4ce52ffe042b1f371ce3e750a14c.squirrel@mail.stardawn.org>


Alain Picard wrote:
>
> Hello all,
>
> I've noticed that elephant blithely ignores type declarations on slots;
> e.g. something like this:
>
> (deftype position ()
>   `(integer 1 99))
>
>
> (defclass foo-object ()
>   ((name          :type string
>                   :index t
> 		  :reader  foo-name
>                   :initarg :name)
>    (position      :type     position
>                   :accessor foo-position
>                   :initform 1))
>   (:metaclass persistent-metaclass))
>
> Elephant will happily let me store 100 in the foo-object position slot.

Yes, the implementation is responsible for checking this.

If you're using SBCL, read this:

  http://www.sbcl.org/manual/Declarations-as-Assertions.html

Notably,

  CLOS slot types form a notable exception. Types declared
  using the :type slot option in defclass are asserted if and
  only if the class was defined in safe code and the slot
  access location is in safe code as well.

Would you propose that Elephant check the type as well
when serializing slots?

  Leslie

-- 
http://www.linkedin.com/in/polzer




From Dr.Alain.Picard at gmail.com  Fri Oct  2 22:21:11 2009
From: Dr.Alain.Picard at gmail.com (Alain Picard)
Date: Sat, 03 Oct 2009 08:21:11 +1000
Subject: [elephant-devel] Bug, omission, or desired behaviour?
References: <873a62eej7.fsf@gmail.com>
	<964e4ce52ffe042b1f371ce3e750a14c.squirrel@mail.stardawn.org>
Message-ID: <87r5tlcu94.fsf@gmail.com>

"Leslie P. Polzer"
<sky at viridian-project.de> writes:

> Would you propose that Elephant check the type as well
> when serializing slots?

I'm a bit of two minds about it.  I might be jaundiced in my
view because the systems I've used before was an O-R mapping
system, and there it made sense to assert in lisp---if you
didn't you eventually failed to store you data because of
a mismatch between the lisp type and the SQL column type.

Hyperspec 7.5.3 states: 
> * The contents of a slot will always be of type (and T1 ... Tn) where
> T1 ...Tn are the values of the :type slot options contained in all of
> the slot specifiers. If no slot specifier contains the :type slot
> option, the contents of the slot will always be of type t. The
> consequences of attempting to store in a slot a value that does not
> satisfy the type of the slot are undefined.

So, the current "undefined" behaviour of Elephant is to "do the right
thing".  Another valid undefined behaviour might be to assert.

I think I hadn't analyzed this through before reporting the "bug".
Now that I realize it's a lisp semantics issue (rather than a 
DB type safety issue) I'm inclined to think maybe the way to handle
it is as SBCL does (which I didn't know about - thanks for the tip!)
and let the user choose the correct behaviour.  So we could have
a flag *ELEPHANT-ENFORCE-SLOT-TYPE-SAFETY-P* (defaulting to nil)
to catch such violations and turn them into asserts, on both
serialization and deserialization.

This is, obviously, a very low priority matter, and truthfully, I'm
not sure I'd use it, at least not until my application was fully
debugged.  Also, I think it would make migration issues when classes
are redefined even hairier than they currently are.

So, no --- I'm not proposing it.  :-)

    --Alain





From Dr.Alain.Picard at gmail.com  Sun Oct  4 00:28:24 2009
From: Dr.Alain.Picard at gmail.com (Alain Picard)
Date: Sun, 04 Oct 2009 11:28:24 +1100
Subject: [elephant-devel] How should I handle a controller-lost-error?
Message-ID: <87k4zcc89j.fsf@gmail.com>


Dear Elephants,

I'm trying to understand how one is supposed to recover from
store-controller errors, and what the anticipated usage pattern
for when to open controllers is.  Consider this example:

   (with-open-store (*elephant-connection-spec*)
     (setq u (last-room-update :room nil)))

   => #<WAITING-ROOM-UPDATE oid:86987>

If you then attempt to access a slot on u, you get
a CONTROLLER-LOST-ERROR signalled, with a continue
restart saying "do you want to reopen":

  (timestamp u)
   ==>
   Condition #<CONTROLLER-LOST-ERROR #x300045D5CD5D>
      [Condition of type CONTROLLER-LOST-ERROR]

   Restarts:
    0: [CONTINUE] Open a new instance and continue?
    1: [RETRY] Retry SLIME REPL evaluation request.
    2: [ABORT] Return to SLIME's top level.
    3: [ABORT-BREAK] Reset this thread
    4: [ABORT] Kill this thread

This occurs because WITH-OPEN-STORE has now closed the controller
which is referred to in the instance U.  Fine.
Now, I want to find out how to handle lost controllers,
and automatically restart them.  I attempted something like this:

(handler-bind ((controller-lost-error #'(lambda (c)
					  (describe c)
					    (let ((r (find-restart 'continue c)))
					      (when r
						(print 'continuing)
						(invoke-restart r))))))
  (timestamp u))

However, that doesn't work because you then get

    There is no applicable method for the generic function:
      #<STANDARD-GENERIC-FUNCTION ELEPHANT::PERSISTENT-SLOT-READER #x30004284B1EF>
    when called with arguments:
      (NIL #<WAITING-ROOM-UPDATE oid:86987> TIMESTAMP)
       [Condition of type SIMPLE-ERROR]

    Restarts:
     0: [CONTINUE] Try calling it again
     1: [RETRY] Retry SLIME REPL evaluation request.
     2: [ABORT] Return to SLIME's top level.
     3: [ABORT-BREAK] Reset this thread

Trying the RETRY restart at this point _does_ succeed, however.

I think it's a bug to encounter this second condition; i.e. I think
after the first CONTINUE, when the controller gets successfully re-opened,
GET-CON should be able to return the new controller immediately,
so that the retried SLOT-VALUE accessor should then succeed.

Secondly, and, less important, I think using CERROR/CONTINUE is
a bit too generic for this class of error; I think it'd be much
friendlier to be able to invoke a specific restart, like REOPEN-LOST-CONTROLLER,
so I could write some sort of REOPENING-STORE macro.


Maybe I'm missing something basic, here, but the reason I delved into
this is that I noticed that opening a new store controller is a 
HUGELY expensive operation; so an idiom like

    (defun some-web-responding-method (...)
      (with-open-store (spec)
	(with-transaction ()
	  (do-stuff))))

ends up being really, really, really slow; almost 1000 times slower
than just

    (open-store spec)

    (defun some-web-responding-method (...)
      (with-transaction ()
	(do-stuff)))

That's fine - I understand why that is, but the above seems very unsafe.
What am I supposed to do or catch if there are store related errors?
So I was trying to write something sort of like

    (defun some-web-responding-method (...)
      (reopening-store
	(with-transaction ()
	  (do-stuff))))

if you know what I mean.  Is this misguided?  There a section in
the manual about "Design Patterns" and "Multithreaded Web Applications",
but it seems very incomplete and I think it should discuss at length
these sorts of issues.

I guess I'm also unsure of what _other_ sorts of errors might get
thrown and how I'm supposed to handle them during "normal" elephant work.
A section documenting the responsibilities of the programmer in this
area would be invaluable, IMHO.

Thanks for any pointers!

       	   --Alain Picard

-- 
Please read about why Top Posting
is evil at: http://en.wikipedia.org/wiki/Top-posting
and http://www.dickalba.demon.co.uk/usenet/guide/faq_topp.html
Please read about why HTML in email is evil at: http://www.birdhouse.org/etc/evilmail.html




From eslick at media.mit.edu  Sun Oct  4 01:30:56 2009
From: eslick at media.mit.edu (Ian Eslick)
Date: Sat, 3 Oct 2009 18:30:56 -0700
Subject: [elephant-devel] How should I handle a controller-lost-error?
In-Reply-To: <87k4zcc89j.fsf@gmail.com>
References: <87k4zcc89j.fsf@gmail.com>
Message-ID: <4445A2F4-9E69-48EF-BBC5-16B2DFE7F794@media.mit.edu>

I can say more later, but given controller opening costs, we should  
deprecate with-open-store.  The usage model I use with multiple stores  
is (with-transaction (:store-controller *my-store*)

I open the controller just once.  If there is only one store you can  
rely on the default controller argument *store-controller*

Ian

Sent from my iPhone

On Oct 3, 2009, at 5:28 PM, Alain Picard <Dr.Alain.Picard at gmail.com>  
wrote:

>
> Dear Elephants,
>
> I'm trying to understand how one is supposed to recover from
> store-controller errors, and what the anticipated usage pattern
> for when to open controllers is.  Consider this example:
>
>   (with-open-store (*elephant-connection-spec*)
>     (setq u (last-room-update :room nil)))
>
>   => #<WAITING-ROOM-UPDATE oid:86987>
>
> If you then attempt to access a slot on u, you get
> a CONTROLLER-LOST-ERROR signalled, with a continue
> restart saying "do you want to reopen":
>
>  (timestamp u)
>   ==>
>   Condition #<CONTROLLER-LOST-ERROR #x300045D5CD5D>
>      [Condition of type CONTROLLER-LOST-ERROR]
>
>   Restarts:
>    0: [CONTINUE] Open a new instance and continue?
>    1: [RETRY] Retry SLIME REPL evaluation request.
>    2: [ABORT] Return to SLIME's top level.
>    3: [ABORT-BREAK] Reset this thread
>    4: [ABORT] Kill this thread
>
> This occurs because WITH-OPEN-STORE has now closed the controller
> which is referred to in the instance U.  Fine.
> Now, I want to find out how to handle lost controllers,
> and automatically restart them.  I attempted something like this:
>
> (handler-bind ((controller-lost-error #'(lambda (c)
>                      (describe c)
>                        (let ((r (find-restart 'continue c)))
>                          (when r
>                        (print 'continuing)
>                        (invoke-restart r))))))
>  (timestamp u))
>
> However, that doesn't work because you then get
>
>    There is no applicable method for the generic function:
>      #<STANDARD-GENERIC-FUNCTION ELEPHANT::PERSISTENT-SLOT-READER  
> #x30004284B1EF>
>    when called with arguments:
>      (NIL #<WAITING-ROOM-UPDATE oid:86987> TIMESTAMP)
>       [Condition of type SIMPLE-ERROR]
>
>    Restarts:
>     0: [CONTINUE] Try calling it again
>     1: [RETRY] Retry SLIME REPL evaluation request.
>     2: [ABORT] Return to SLIME's top level.
>     3: [ABORT-BREAK] Reset this thread
>
> Trying the RETRY restart at this point _does_ succeed, however.
>
> I think it's a bug to encounter this second condition; i.e. I think
> after the first CONTINUE, when the controller gets successfully re- 
> opened,
> GET-CON should be able to return the new controller immediately,
> so that the retried SLOT-VALUE accessor should then succeed.
>
> Secondly, and, less important, I think using CERROR/CONTINUE is
> a bit too generic for this class of error; I think it'd be much
> friendlier to be able to invoke a specific restart, like REOPEN-LOST- 
> CONTROLLER,
> so I could write some sort of REOPENING-STORE macro.
>
>
> Maybe I'm missing something basic, here, but the reason I delved into
> this is that I noticed that opening a new store controller is a
> HUGELY expensive operation; so an idiom like
>
>    (defun some-web-responding-method (...)
>      (with-open-store (spec)
>    (with-transaction ()
>      (do-stuff))))
>
> ends up being really, really, really slow; almost 1000 times slower
> than just
>
>    (open-store spec)
>
>    (defun some-web-responding-method (...)
>      (with-transaction ()
>    (do-stuff)))
>
> That's fine - I understand why that is, but the above seems very  
> unsafe.
> What am I supposed to do or catch if there are store related errors?
> So I was trying to write something sort of like
>
>    (defun some-web-responding-method (...)
>      (reopening-store
>    (with-transaction ()
>      (do-stuff))))
>
> if you know what I mean.  Is this misguided?  There a section in
> the manual about "Design Patterns" and "Multithreaded Web  
> Applications",
> but it seems very incomplete and I think it should discuss at length
> these sorts of issues.
>
> I guess I'm also unsure of what _other_ sorts of errors might get
> thrown and how I'm supposed to handle them during "normal" elephant  
> work.
> A section documenting the responsibilities of the programmer in this
> area would be invaluable, IMHO.
>
> Thanks for any pointers!
>
>              --Alain Picard
>
> -- 
> Please read about why Top Posting
> is evil at: http://en.wikipedia.org/wiki/Top-posting
> and http://www.dickalba.demon.co.uk/usenet/guide/faq_topp.html
> Please read about why HTML in email is evil at: http://www.birdhouse.org/etc/evilmail.html
>
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel



From Dr.Alain.Picard at gmail.com  Sun Oct  4 05:02:06 2009
From: Dr.Alain.Picard at gmail.com (Alain Picard)
Date: Sun, 04 Oct 2009 16:02:06 +1100
Subject: [elephant-devel] How should I handle a controller-lost-error?
References: <87k4zcc89j.fsf@gmail.com>
	<4445A2F4-9E69-48EF-BBC5-16B2DFE7F794@media.mit.edu>
Message-ID: <87fx9zda5t.fsf@gmail.com>

Ian Eslick <eslick at media.mit.edu> writes:

> I can say more later, but given controller opening costs, we should  

I will definitely be looking forward to your further exposition.  :-)

But as...

> Sent from my iPhone

I can definitely understand why you'd want to be brief. 

Cheers,
                --ap




From eslick at media.mit.edu  Mon Oct  5 01:20:48 2009
From: eslick at media.mit.edu (Ian Eslick)
Date: Sun, 4 Oct 2009 21:20:48 -0400
Subject: [elephant-devel] Bug, omission, or desired behaviour?
In-Reply-To: <87r5tlcu94.fsf@gmail.com>
References: <873a62eej7.fsf@gmail.com>
	<964e4ce52ffe042b1f371ce3e750a14c.squirrel@mail.stardawn.org>
	<87r5tlcu94.fsf@gmail.com>
Message-ID: <E6E1BD6D-2037-48AC-9EF9-AA53FED26C53@media.mit.edu>

My only concern about the variable is adding addition checks to  
performance.  We have significant overhead going on during slot access  
already.  One thought would be to make this a compilation option for  
high safety settings.  I've recorded a Trac ticket for this issue.

Ian

On Oct 2, 2009, at 3:21 PM, Alain Picard wrote:

> "Leslie P. Polzer"
> <sky at viridian-project.de> writes:
>
>> Would you propose that Elephant check the type as well
>> when serializing slots?
>
> I'm a bit of two minds about it.  I might be jaundiced in my
> view because the systems I've used before was an O-R mapping
> system, and there it made sense to assert in lisp---if you
> didn't you eventually failed to store you data because of
> a mismatch between the lisp type and the SQL column type.
>
> Hyperspec 7.5.3 states:
>> * The contents of a slot will always be of type (and T1 ... Tn) where
>> T1 ...Tn are the values of the :type slot options contained in all of
>> the slot specifiers. If no slot specifier contains the :type slot
>> option, the contents of the slot will always be of type t. The
>> consequences of attempting to store in a slot a value that does not
>> satisfy the type of the slot are undefined.
>
> So, the current "undefined" behaviour of Elephant is to "do the right
> thing".  Another valid undefined behaviour might be to assert.
>
> I think I hadn't analyzed this through before reporting the "bug".
> Now that I realize it's a lisp semantics issue (rather than a
> DB type safety issue) I'm inclined to think maybe the way to handle
> it is as SBCL does (which I didn't know about - thanks for the tip!)
> and let the user choose the correct behaviour.  So we could have
> a flag *ELEPHANT-ENFORCE-SLOT-TYPE-SAFETY-P* (defaulting to nil)
> to catch such violations and turn them into asserts, on both
> serialization and deserialization.
>
> This is, obviously, a very low priority matter, and truthfully, I'm
> not sure I'd use it, at least not until my application was fully
> debugged.  Also, I think it would make migration issues when classes
> are redefined even hairier than they currently are.
>
> So, no --- I'm not proposing it.  :-)
>
>    --Alain
>
>
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel




From eslick at media.mit.edu  Mon Oct  5 01:45:44 2009
From: eslick at media.mit.edu (Ian Eslick)
Date: Sun, 4 Oct 2009 21:45:44 -0400
Subject: [elephant-devel] How should I handle a controller-lost-error?
In-Reply-To: <87k4zcc89j.fsf@gmail.com>
References: <87k4zcc89j.fsf@gmail.com>
Message-ID: <0D7904A4-918C-45AA-8F9B-A4AFDCF614CE@media.mit.edu>

There was a bug with the get-con function that returns nil after the  
restart.  I just checked in a fix for that and went ahead and  
implemented your suggestion for a saner error + restart model.  An  
interactive error is asserted which provides a 'reopen-controller  
restart.  Keep in mind that re-opening the store does not bind *store- 
controller* so you have to be careful with your uses of with- 
transaction in a multi-store environment.

A few notes.  New objects are created in the store provided in the :sc  
argument to make-instance.  The default is to use the current dynamic  
value of *store-controller*.  Once created, objects know what their  
home store is and get-con is used to retrieve it.  Within the body of  
a transaction, *store-controller* is bound to the store on which that  
transaction is run so any new objects that don't have an explicit :sc  
argument will be created in that store.  You'll get a nice error if  
you try to create an object, provide no :sc designation and *store- 
controller* is nil.

For the web responding function you describe, with-transaction will  
handle most cases of error as it aborts a transaction on an error and  
the web page should respond with a 500 error.   If the store has  
closed, this is basically a global and probably catastrophic server- 
level rather than request-level problem.  There is no reason you  
couldn't reopen the store as part of a recovery procedure however.

Frankly the code paths that are exercised when *store-controller* is  
nil or a store is closed have not been heavily used or terribly well  
thought through.  All the usage models for Elephant that I'm aware of  
have been single-store settings where *store-controller* is bound once  
globally.  The only real reason I can see to use with-open-store is in  
a multiple store environment and the issues involved with this are not  
well documented, as you rightly point out.

We have been adding more signals (thanks mostly to Leslie) for  
serialization and other errors since the last revision of the manual,  
but   I'm not sure when one of us will have the time to update the  
manual.

Is this helpful?

Ian

On Oct 3, 2009, at 5:28 PM, Alain Picard wrote:

>
> Dear Elephants,
>
> I'm trying to understand how one is supposed to recover from
> store-controller errors, and what the anticipated usage pattern
> for when to open controllers is.  Consider this example:
>
>   (with-open-store (*elephant-connection-spec*)
>     (setq u (last-room-update :room nil)))
>
>   => #<WAITING-ROOM-UPDATE oid:86987>
>
> If you then attempt to access a slot on u, you get
> a CONTROLLER-LOST-ERROR signalled, with a continue
> restart saying "do you want to reopen":
>
>  (timestamp u)
>   ==>
>   Condition #<CONTROLLER-LOST-ERROR #x300045D5CD5D>
>      [Condition of type CONTROLLER-LOST-ERROR]
>
>   Restarts:
>    0: [CONTINUE] Open a new instance and continue?
>    1: [RETRY] Retry SLIME REPL evaluation request.
>    2: [ABORT] Return to SLIME's top level.
>    3: [ABORT-BREAK] Reset this thread
>    4: [ABORT] Kill this thread
>
> This occurs because WITH-OPEN-STORE has now closed the controller
> which is referred to in the instance U.  Fine.
> Now, I want to find out how to handle lost controllers,
> and automatically restart them.  I attempted something like this:
>
> (handler-bind ((controller-lost-error #'(lambda (c)
> 					  (describe c)
> 					    (let ((r (find-restart 'continue c)))
> 					      (when r
> 						(print 'continuing)
> 						(invoke-restart r))))))
>  (timestamp u))
>
> However, that doesn't work because you then get

>    There is no applicable method for the generic function:
>      #<STANDARD-GENERIC-FUNCTION ELEPHANT::PERSISTENT-SLOT-READER  
> #x30004284B1EF>
>    when called with arguments:
>      (NIL #<WAITING-ROOM-UPDATE oid:86987> TIMESTAMP)
>       [Condition of type SIMPLE-ERROR]
>
>    Restarts:
>     0: [CONTINUE] Try calling it again
>     1: [RETRY] Retry SLIME REPL evaluation request.
>     2: [ABORT] Return to SLIME's top level.
>     3: [ABORT-BREAK] Reset this thread
>
> Trying the RETRY restart at this point _does_ succeed, however.
>
> I think it's a bug to encounter this second condition; i.e. I think
> after the first CONTINUE, when the controller gets successfully re- 
> opened,
> GET-CON should be able to return the new controller immediately,
> so that the retried SLOT-VALUE accessor should then succeed.
>
> Secondly, and, less important, I think using CERROR/CONTINUE is
> a bit too generic for this class of error; I think it'd be much
> friendlier to be able to invoke a specific restart, like REOPEN-LOST- 
> CONTROLLER,
> so I could write some sort of REOPENING-STORE macro.
>
>
> Maybe I'm missing something basic, here, but the reason I delved into
> this is that I noticed that opening a new store controller is a
> HUGELY expensive operation; so an idiom like
>
>    (defun some-web-responding-method (...)
>      (with-open-store (spec)
> 	(with-transaction ()
> 	  (do-stuff))))
>
> ends up being really, really, really slow; almost 1000 times slower
> than just
>
>    (open-store spec)
>
>    (defun some-web-responding-method (...)
>      (with-transaction ()
> 	(do-stuff)))
>
> That's fine - I understand why that is, but the above seems very  
> unsafe.
> What am I supposed to do or catch if there are store related errors?
> So I was trying to write something sort of like
>
>    (defun some-web-responding-method (...)
>      (reopening-store
> 	(with-transaction ()
> 	  (do-stuff))))
>
> if you know what I mean.  Is this misguided?  There a section in
> the manual about "Design Patterns" and "Multithreaded Web  
> Applications",
> but it seems very incomplete and I think it should discuss at length
> these sorts of issues.
>
> I guess I'm also unsure of what _other_ sorts of errors might get
> thrown and how I'm supposed to handle them during "normal" elephant  
> work.
> A section documenting the responsibilities of the programmer in this
> area would be invaluable, IMHO.
>
> Thanks for any pointers!
>
>       	   --Alain Picard
>
> -- 
> Please read about why Top Posting
> is evil at: http://en.wikipedia.org/wiki/Top-posting
> and http://www.dickalba.demon.co.uk/usenet/guide/faq_topp.html
> Please read about why HTML in email is evil at: http://www.birdhouse.org/etc/evilmail.html
>
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel




From Dr.Alain.Picard at gmail.com  Mon Oct  5 11:38:49 2009
From: Dr.Alain.Picard at gmail.com (Alain Picard)
Date: Mon, 05 Oct 2009 22:38:49 +1100
Subject: [elephant-devel] How should I handle a controller-lost-error?
References: <87k4zcc89j.fsf@gmail.com>
	<0D7904A4-918C-45AA-8F9B-A4AFDCF614CE@media.mit.edu>
Message-ID: <87bpkmcbp2.fsf@gmail.com>

Ian Eslick <eslick at media.mit.edu> writes:

> There was a bug with the get-con function that returns nil after the  
> restart.  I just checked in a fix for that and went ahead and  
> implemented your suggestion for a saner error + restart model.  

That's great.  I'll do a checkout and start playing with that
tomorrow.

> For the web responding function you describe, with-transaction will  
> handle most cases of error as it aborts a transaction on an error and  
> the web page should respond with a 500 error.   If the store has  
> closed, this is basically a global and probably catastrophic server- 
> level rather than request-level problem.  

OK - that's the sort of info I was after  --- i.e. that the
store getting closed "behind your back" is a very serious, "can't
happen" type of error.  Something to do with deleting libdb.so
while the application runs, or the disk filling up, or something
equally severe.  So if I ever catch something like that, it's time
to stop the application, raise a loud alarm somewhere, and wait for
a human to fix things up.

> Frankly the code paths that are exercised when *store-controller* is  
> nil or a store is closed have not been heavily used or terribly well  
> thought through.  All the usage models for Elephant that I'm aware of  
> have been single-store settings where *store-controller* is bound once  
> globally.

Great!  That's what I was hoping to get away with.  so, START-APP
opens the store; sets *store-controller* globally, and everone
just uses WITH-TRANSACTION to isolate threads against each other.
And you get excellent performance, as a side benefit.

> Is this helpful?

Incredibly so.  Thank you very much for your time.

One last question, while I have your attention: I think I intend
to have more than 1 process on the machine access the store
concurrently; but I assume that's no more dangerous than having
multiple threads doing so; and Berkeley DB should be able to
take care of itself and protect the integrity of the store against
multiple simultaneous writers.  I may have to set some magic
locking flags when opening the environment, but that should be
about it.  Is that your understanding also?

Anybody else do this?  Anyone seen any mysterious crashes?  

I greatly (greatly!) appreciate your answers, not to mention
the wonderful elephant package.  It feels a lot more solid than
when I last looked at it a couple of years ago.  Thanks!

       --Alain




From sky at viridian-project.de  Mon Oct  5 11:55:08 2009
From: sky at viridian-project.de (Leslie P. Polzer)
Date: Mon, 5 Oct 2009 13:55:08 +0200 (CEST)
Subject: [elephant-devel] How should I handle a controller-lost-error?
In-Reply-To: <87bpkmcbp2.fsf@gmail.com>
References: <87k4zcc89j.fsf@gmail.com>
	<0D7904A4-918C-45AA-8F9B-A4AFDCF614CE@media.mit.edu>
	<87bpkmcbp2.fsf@gmail.com>
Message-ID: <0033f68178f6820d43021b5ee6985a7d.squirrel@mail.stardawn.org>


Alain Picard wrote:

> One last question, while I have your attention: I think I intend
> to have more than 1 process on the machine access the store
> concurrently; but I assume that's no more dangerous than having
> multiple threads doing so; and Berkeley DB should be able to
> take care of itself and protect the integrity of the store against
> multiple simultaneous writers.  I may have to set some magic
> locking flags when opening the environment, but that should be
> about it.  Is that your understanding also?
>
> Anybody else do this?  Anyone seen any mysterious crashes?

I do this, it's working fine. Be sure to specify :register t when
opening the store.

  Leslie

-- 
http://www.linkedin.com/in/polzer




From eslick at media.mit.edu  Mon Oct  5 14:47:59 2009
From: eslick at media.mit.edu (Ian Eslick)
Date: Mon, 5 Oct 2009 10:47:59 -0400
Subject: [elephant-devel] How should I handle a controller-lost-error?
In-Reply-To: <87bpkmcbp2.fsf@gmail.com>
References: <87k4zcc89j.fsf@gmail.com>
	<0D7904A4-918C-45AA-8F9B-A4AFDCF614CE@media.mit.edu>
	<87bpkmcbp2.fsf@gmail.com>
Message-ID: <EA9E326A-A26D-4068-AB98-06A0B18729BE@media.mit.edu>

> One last question, while I have your attention: I think I intend
> to have more than 1 process on the machine access the store
> concurrently; but I assume that's no more dangerous than having
> multiple threads doing so; and Berkeley DB should be able to
> take care of itself and protect the integrity of the store against
> multiple simultaneous writers.  I may have to set some magic
> locking flags when opening the environment, but that should be
> about it.  Is that your understanding also?

So long as you follow Leslie's suggestion, multiple process locking is  
fully supported by BDB (via a shared memory region managed by the  
library).  If you want to do something like this across multiple  
machines, you'll need to use the slower, but more scalable Postmodern  
store.

> Anybody else do this?  Anyone seen any mysterious crashes?
>
> I greatly (greatly!) appreciate your answers, not to mention
> the wonderful elephant package.  It feels a lot more solid than
> when I last looked at it a couple of years ago.  Thanks!

 From a casual user's standpoint, I think it's in pretty good shape; I  
use it for customer facing projects quite reliably.  From a software  
engineering standpoint, the last 10% of feature tuning, code cleanup,  
and documentation is going to be a bear to get done for 1.0!

>       --Alain
>
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel




From binghe.lisp at gmail.com  Sat Oct 10 01:39:32 2009
From: binghe.lisp at gmail.com (Chun Tian)
Date: Sat, 10 Oct 2009 09:39:32 +0800
Subject: [elephant-devel] libberkeley-db.dylib build error on Mac OS X 10.6
Message-ID: <F3224E38-DB3D-4385-B2E4-6C00AEF811E3@gmail.com>

Hi, Elephant

It seems that "gcc" on Mac OS X 10.6 compiles to 64-bit executes by  
default. And if I load Elephant-1.0 in a 32-bit CL, libberkeley-db.c  
cannot compile correctly: (I'm under 32-bit LispWorks)

; $ gcc -L/sw64/lib/ -I/sw64/include/db4/ -bundle -fPIC -Wall -g -O2 - 
g /Users/binghe/Lisp/packages/elephant-1.0/src/db-bdb/libberkeley-db.c  
-o /Users/binghe/Lisp/packages/elephant-1.0/src/db-bdb/libberkeley- 
db.dylib -lm -ldb-4.7
/Users/binghe/Lisp/packages/elephant-1.0/src/db-bdb/libberkeley-db.c:  
In function 'lisp_compare2':
/Users/binghe/Lisp/packages/elephant-1.0/src/db-bdb/libberkeley-db.c: 
1061: warning: unused variable 'i'
/Users/binghe/Lisp/packages/elephant-1.0/src/db-bdb/libberkeley-db.c:  
In function 'lisp_compare_key2':
/Users/binghe/Lisp/packages/elephant-1.0/src/db-bdb/libberkeley-db.c: 
1155: warning: unused variable 'i'
ld: warning: in /sw64/lib//libdb-4.7.dylib, file is not of required  
architecture
Undefined symbols:
   "_db_sequence_create", referenced from:
       _db_sequence_create2 in ccpBICdb.o
   "_db_env_create", referenced from:
       _db_env_cr in ccpBICdb.o
   "_db_create", referenced from:
       _db_cr in ccpBICdb.o
   "_db_strerror", referenced from:
       _db_strerr in ccpBICdb.o
ld: symbol(s) not found
collect2: ld returned 1 exit status

Error: erred while invoking #<ASDF:COMPILE-OP NIL 2259272B> on
        #<ELE-BDB-SYSTEM::BDB-C-SOURCE "libberkeley-db" 200DF123>
   1 (continue) Retry performing #<ASDF:COMPILE-OP NIL 2259272B> on
                #<ELE-BDB-SYSTEM::BDB-C-SOURCE "libberkeley-db"  
200DF123>.
   2 Continue, treating #<ASDF:COMPILE-OP NIL 2259272B> on
     #<ELE-BDB-SYSTEM::BDB-C-SOURCE "libberkeley-db" 200DF123> as  
having been successful.
   3 (abort) Return to level 0.
   4 Return to top loop level 0.

One solution may be explicit add a "-m32" compiler option for all 32- 
bit CL platforms under Mac OS X. A small patch as in attach could fix  
this, also paste here:

diff -rN -u old-elephant-1.0/elephant.asd new-elephant-1.0/elephant.asd
--- old-elephant-1.0/elephant.asd	2009-10-10 09:31:52.000000000 +0800
+++ new-elephant-1.0/elephant.asd	2009-10-10 09:31:52.000000000 +0800
@@ -225,6 +225,7 @@
     #-(or darwin macosx darwin-host) "-shared"
     #+(or darwin macosx darwin-host) "-bundle"
     #+(and X86-64 (or macosx darwin darwin-host)) "-arch x86_64"
+   #+(and X86 (or macosx darwin darwin-host)) "-m32" ; Snow Leopard
     #+(and X86-64 linux) "-march=x86-64"
     "-fPIC"
     "-Wall"

Please confirm and merge it.

Regards,

Chun Tian (binghe)

-------------- next part --------------
A non-text attachment was scrubbed...
Name: compiler-options.diff
Type: application/octet-stream
Size: 526 bytes
Desc: not available
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20091010/d93fe91c/attachment.obj>

From sky at viridian-project.de  Sat Oct 10 08:26:48 2009
From: sky at viridian-project.de (Leslie P. Polzer)
Date: Sat, 10 Oct 2009 10:26:48 +0200 (CEST)
Subject: [elephant-devel] libberkeley-db.dylib build error on Mac OS X
 10.6
In-Reply-To: <F3224E38-DB3D-4385-B2E4-6C00AEF811E3@gmail.com>
References: <F3224E38-DB3D-4385-B2E4-6C00AEF811E3@gmail.com>
Message-ID: <f4c40ca03f68b1719620ab8a0497d013.squirrel@mail.stardawn.org>


Hi Chun,

I can't test your patch but since it looks sane enough I will
apply it if no one objects within the next few days.

  Leslie

-- 
http://www.linkedin.com/in/polzer




From sky at viridian-project.de  Sun Oct 11 08:12:36 2009
From: sky at viridian-project.de (Leslie P. Polzer)
Date: Sun, 11 Oct 2009 10:12:36 +0200 (CEST)
Subject: [elephant-devel] libberkeley-db.dylib build error on Mac OS X
 10.6
In-Reply-To: <F3224E38-DB3D-4385-B2E4-6C00AEF811E3@gmail.com>
References: <F3224E38-DB3D-4385-B2E4-6C00AEF811E3@gmail.com>
Message-ID: <8aecbfacac1e0fbda8ff02eac1c9649e.squirrel@mail.stardawn.org>


Chun Tian wrote:

> +   #+(and X86 (or macosx darwin darwin-host)) "-m32" ; Snow Leopard
>
> Please confirm and merge it.

It's in.

  Leslie

-- 
http://www.linkedin.com/in/polzer




From sky at viridian-project.de  Wed Oct 14 12:37:43 2009
From: sky at viridian-project.de (Leslie P. Polzer)
Date: Wed, 14 Oct 2009 14:37:43 +0200 (CEST)
Subject: [elephant-devel] More work on BDB interrupt-proofing
Message-ID: <c8226f6e985b2e0c6f6819f75981b10d.squirrel@mail.stardawn.org>

The attached patch makes BDB more robust against interrupts in various
places. It achieves this mainly by wrapping critical paths in
WITHOUT-INTERRUPTS and by registering cursors on the Lisp level
(and cleaning up any dangling ones when the txn ends).

This is work in progress, although it passes the test suite.

  Leslie

-- 
http://www.linkedin.com/in/polzer
-------------- next part --------------
A non-text attachment was scrubbed...
Name: interrupt-proofing.diff
Type: text/x-patch
Size: 17956 bytes
Desc: not available
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20091014/ee6c3c1e/attachment.bin>

From Dr.Alain.Picard at gmail.com  Thu Oct 15 04:32:49 2009
From: Dr.Alain.Picard at gmail.com (Alain Picard)
Date: Thu, 15 Oct 2009 15:32:49 +1100
Subject: [elephant-devel] Bug in GET-CON
Message-ID: <87vdihi8em.fsf@gmail.com>


The current version of GET-CON is broken.  The code reads:

    (defmethod get-con ((instance persistent))
      (let ((con (fast-lookup-con-spec (db-spec instance))))
        (cond ((not con)
               (aif (slow-lookup-con-spec (db-spec instance))
                 (progn
                   (setf (db-spec instance)
                         (car (find it *dbconnection-spec* :key #'cdr)))
                   (get-con instance))
                 (restart-case
                     (signal-controller-lost-error instance)
                   (reopen-controller ()
                     :report "Re-open the store controller"
                     (open-controller
                      (get-controller (db-spec instance)))))))
              ;; If it's valid and open
              ((and con (connection-is-indeed-open con))
               con)
              ;; If the controller object exists but is closed, reopen
              (t (open-controller con)))))

    (defun open-store (spec &rest args)
      "Conveniently open a store controller.  Set *store-controller* to the new controller
       unless it is already set (opening a second controller means you must keep track of
       controllers yourself.  *store-controller* is a convenience variable for single-store
       applications or single-store per thread apps.  Multi-store apps should either confine
       their *store-controller* to a given dynamic context or wrap each store-specific op in
       a transaction using with or ensure transaction.  Returns the opened store controller."
      (assert (consp spec))
      ;; Ensure that parameters are set
      (initialize-user-parameters)
      (let ((controller (get-controller spec)))
        (apply #'open-controller controller args)
        (if *store-controller*
            (progn
    ;;	  (warn "Store controller already set so was not updated") ;; this was annoying me
              controller)
            (setq *store-controller* controller))))

Unfortunately OPEN-CONTROLLER has some arguments which can be
very important to set properly; e.g. in

    (defmethod open-controller ((sc bdb-store-controller) &key (recover t)
                                (recover-fatal nil) (thread t) (register nil)
                                (deadlock-detect t)
                                (cache-size elephant::*berkeley-db-cachesize*)
                                (max-locks elephant::*berkeley-db-max-locks*)
                                (max-objects elephant::*berkeley-db-max-objects*)
                                (max-transactions elephant::*berkeley-db-max-transactions*)
                                (mvcc elephant::*default-mvcc*)
                                error-log)

It is imperative that :REGISTER T be passed if one is to properly access
a DB environment from two separate processes (otherwise, each process
gets a DB_RUNRECOVERY error if the other one uses the store.

However you can see from the code that if the controller gets reopened
by GET-CON, it does not apply all of the arguments which are passed to
OPEN-STORE.

I'm currently changing the default value of :REGISTER in OPEN-CONTROLLER
to T in my code, but that's hardly a fix; the bug exists for all other
arguments.

The correct fix involved capturing the ARGS which are required to
properly reopen a store controller.  I'm not sure where those should
be captured (Perhaps as a slot on the store-controller?)  I'm happy to
attempt a fix if someone would confirm that the approach I propose is
likely not to break something else.

   --Alain



--
Please read about why Top Posting
is evil at: http://en.wikipedia.org/wiki/Top-posting
and http://www.dickalba.demon.co.uk/usenet/guide/faq_topp.html
Please read about why HTML in email is evil at: http://www.birdhouse.org/etc/evilmail.html




From sky at viridian-project.de  Thu Oct 15 08:26:01 2009
From: sky at viridian-project.de (Leslie P. Polzer)
Date: Thu, 15 Oct 2009 10:26:01 +0200 (CEST)
Subject: [elephant-devel] Bug in GET-CON
In-Reply-To: <87vdihi8em.fsf@gmail.com>
References: <87vdihi8em.fsf@gmail.com>
Message-ID: <a3f5d0f66a3a0a5b0b59878ec8b4ba85.squirrel@mail.stardawn.org>


Alain Picard wrote:

> The correct fix involved capturing the ARGS which are required to
> properly reopen a store controller.  I'm not sure where those should
> be captured (Perhaps as a slot on the store-controller?)  I'm happy to
> attempt a fix if someone would confirm that the approach I propose is
> likely not to break something else.

Keeping the args in a slot of the store controller sounds appropriate.
I don't think we need to worry about breaking other users' code since
that code would be relying on very broken code in the first place.

I'd be happy to review and apply your patch. It should contain a test
case if that's possible.

  Leslie

-- 
http://www.linkedin.com/in/polzer




From Dr.Alain.Picard at gmail.com  Fri Oct 16 04:12:05 2009
From: Dr.Alain.Picard at gmail.com (Alain Picard)
Date: Fri, 16 Oct 2009 15:12:05 +1100
Subject: [elephant-devel] Bug in GET-CON
References: <87vdihi8em.fsf@gmail.com>
	<a3f5d0f66a3a0a5b0b59878ec8b4ba85.squirrel@mail.stardawn.org>
Message-ID: <87oco8ht9m.fsf@gmail.com>

"Leslie P. Polzer"
<sky at viridian-project.de> writes:

> I'd be happy to review and apply your patch. It should contain a test
> case if that's possible.


OK, here it is.  Sorry, I don't have a test case, but I did verify
that it now works properly from two separate processes when I
now pass :register t to OPEN-STORE.

This patch works against what was tip in darcs 3 days ago.
If this helps
   darcs show repo
             Type: darcs
           Format: darcs-1.0
             Root: /home/ap/Subway/subway-elephant/src/systems/elephant-1.0
         Pristine: PlainPristine "_darcs/pristine"
            Cache: thisrepo:/home/ap/Subway/subway-elephant/src/systems/elephant-1.0, cache:/home/ap/.darcs/cache
   Default Remote: http://www.common-lisp.net/project/elephant/darcs/elephant-1.0
      Num Patches: 416

(Sorry, my darcs-fu is extremely weak, I'm a mercurial-head myself.)

-------------- next part --------------
A non-text attachment was scrubbed...
Name: patch2
Type: text/x-diff
Size: 2486 bytes
Desc: Open store patch
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20091016/68191c2b/attachment.diff>

From sky at viridian-project.de  Fri Oct 16 09:18:16 2009
From: sky at viridian-project.de (Leslie P. Polzer)
Date: Fri, 16 Oct 2009 11:18:16 +0200 (CEST)
Subject: [elephant-devel] Bug in GET-CON
In-Reply-To: <87oco8ht9m.fsf@gmail.com>
References: <87vdihi8em.fsf@gmail.com>
	<a3f5d0f66a3a0a5b0b59878ec8b4ba85.squirrel@mail.stardawn.org>
	<87oco8ht9m.fsf@gmail.com>
Message-ID: <b8d251e06b99e3e38654cf0ad06e1fce.squirrel@mail.stardawn.org>


Alain Picard wrote:

> This patch works against what was tip in darcs 3 days ago.
> If this helps
>    darcs show repo
>              Type: darcs
>            Format: darcs-1.0
>              Root: /home/ap/Subway/subway-elephant/src/systems/elephant-1.0
>          Pristine: PlainPristine "_darcs/pristine"
>             Cache: thisrepo:/home/ap/Subway/subway-elephant/src/systems/elephant-1.0,
> cache:/home/ap/.darcs/cache
>    Default Remote: http://www.common-lisp.net/project/elephant/darcs/elephant-1.0
>       Num Patches: 416
>
> (Sorry, my darcs-fu is extremely weak, I'm a mercurial-head myself.)

Plain diffs as generated by "darcs diff -u" are fine.

But this one doesn't apply cleanly here:

% patch -p4 < ~/Downloads/patch2
patching file src/elephant/controller.lisp
Hunk #1 FAILED at 111.
1 out of 3 hunks FAILED -- saving rejects to file src/elephant/controller.lisp.rej
patching file src/elephant/gc.lisp

Larger fuzz doesn't help either. Could you check this again?

You could also make a Darcs patch using "darcs record" followed by
"darcs send -o mypatch", although in the general case I prefer plain
diffs.

  Leslie

-- 
http://www.linkedin.com/in/polzer




From Dr.Alain.Picard at gmail.com  Fri Oct 16 23:36:42 2009
From: Dr.Alain.Picard at gmail.com (Alain Picard)
Date: Sat, 17 Oct 2009 10:36:42 +1100
Subject: [elephant-devel] Bug in GET-CON
References: <87vdihi8em.fsf@gmail.com>
	<a3f5d0f66a3a0a5b0b59878ec8b4ba85.squirrel@mail.stardawn.org>
	<87oco8ht9m.fsf@gmail.com>
	<b8d251e06b99e3e38654cf0ad06e1fce.squirrel@mail.stardawn.org>
Message-ID: <87k4yuj4hh.fsf@gmail.com>

"Leslie P. Polzer"
<sky at viridian-project.de> writes:

> But this one doesn't apply cleanly here:

> You could also make a Darcs patch using "darcs record" followed by
> "darcs send -o mypatch", although in the general case I prefer plain
> diffs.

OK - here's a patch produced as above.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: mypatch
Type: text/x-diff
Size: 41834 bytes
Desc: patch
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20091017/d2587430/attachment.diff>

From sky at viridian-project.de  Sat Oct 17 07:57:53 2009
From: sky at viridian-project.de (Leslie P. Polzer)
Date: Sat, 17 Oct 2009 09:57:53 +0200 (CEST)
Subject: [elephant-devel] Bug in GET-CON
In-Reply-To: <87k4yuj4hh.fsf@gmail.com>
References: <87vdihi8em.fsf@gmail.com>
	<a3f5d0f66a3a0a5b0b59878ec8b4ba85.squirrel@mail.stardawn.org>
	<87oco8ht9m.fsf@gmail.com>
	<b8d251e06b99e3e38654cf0ad06e1fce.squirrel@mail.stardawn.org>
	<87k4yuj4hh.fsf@gmail.com>
Message-ID: <d27943fbe44805c8a7c88063b4d27195.squirrel@mail.stardawn.org>


Alain Picard wrote:

>> You could also make a Darcs patch using "darcs record" followed by
>> "darcs send -o mypatch", although in the general case I prefer plain
>> diffs.
>
> OK - here's a patch produced as above.

Thank you, this one applies.

But the patch is very cluttered since it's full of places where you changed
the whitespace. Please try picking only the pieces that are really part of
this patch when you do the "darcs record" step, then resubmit it and I
will put it in.

  Leslie

-- 
http://www.linkedin.com/in/polzer




From Dr.Alain.Picard at gmail.com  Sat Oct 17 11:57:17 2009
From: Dr.Alain.Picard at gmail.com (Alain Picard)
Date: Sat, 17 Oct 2009 22:57:17 +1100
Subject: [elephant-devel] Bug in GET-CON
References: <87vdihi8em.fsf@gmail.com>
	<a3f5d0f66a3a0a5b0b59878ec8b4ba85.squirrel@mail.stardawn.org>
	<87oco8ht9m.fsf@gmail.com>
	<b8d251e06b99e3e38654cf0ad06e1fce.squirrel@mail.stardawn.org>
	<87k4yuj4hh.fsf@gmail.com>
	<d27943fbe44805c8a7c88063b4d27195.squirrel@mail.stardawn.org>
Message-ID: <87hbtyjkrm.fsf@gmail.com>

"Leslie P. Polzer"
<sky at viridian-project.de> writes:

> Alain Picard wrote:
>
>>> You could also make a Darcs patch using "darcs record" followed by
>>> "darcs send -o mypatch", although in the general case I prefer plain
>>> diffs.
>>
>> OK - here's a patch produced as above.
>
> Thank you, this one applies.
>
> But the patch is very cluttered since it's full of places where you changed
> the whitespace. Please try picking only the pieces that are really part of
> this patch when you do the "darcs record" step, then resubmit it and I
> will put it in.

That's because my emacs automatically cleans up the whitespace
as a file saving hook --- something I would heartily recommend
to any developer using emacs.  But since this isn't my project,
I will again try to make a simpler patch.





From Dr.Alain.Picard at gmail.com  Sat Oct 17 12:28:48 2009
From: Dr.Alain.Picard at gmail.com (Alain Picard)
Date: Sat, 17 Oct 2009 23:28:48 +1100
Subject: [elephant-devel] Bug in GET-CON
References: <87vdihi8em.fsf@gmail.com>
	<a3f5d0f66a3a0a5b0b59878ec8b4ba85.squirrel@mail.stardawn.org>
	<87oco8ht9m.fsf@gmail.com>
	<b8d251e06b99e3e38654cf0ad06e1fce.squirrel@mail.stardawn.org>
	<87k4yuj4hh.fsf@gmail.com>
	<d27943fbe44805c8a7c88063b4d27195.squirrel@mail.stardawn.org>
	<87hbtyjkrm.fsf@gmail.com>
Message-ID: <87bpk6jjb3.fsf@gmail.com>

Alain Picard <Dr.Alain.Picard at gmail.com>
writes:

OK - here is a MINIMAL patch.  It applies with
patch -p1 < /tmp/patch.

Hope this one works!  

  --ap

-------------- next part --------------
A non-text attachment was scrubbed...
Name: patch
Type: text/x-diff
Size: 2562 bytes
Desc: patch
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20091017/6e382484/attachment.diff>
-------------- next part --------------



-- 
Please read about why Top Posting
is evil at: http://en.wikipedia.org/wiki/Top-posting
and http://www.dickalba.demon.co.uk/usenet/guide/faq_topp.html
Please read about why HTML in email is evil at: http://www.birdhouse.org/etc/evilmail.html

From sky at viridian-project.de  Sat Oct 17 15:23:21 2009
From: sky at viridian-project.de (Leslie P. Polzer)
Date: Sat, 17 Oct 2009 17:23:21 +0200 (CEST)
Subject: [elephant-devel] Bug in GET-CON
In-Reply-To: <87hbtyjkrm.fsf@gmail.com>
References: <87vdihi8em.fsf@gmail.com>
	<a3f5d0f66a3a0a5b0b59878ec8b4ba85.squirrel@mail.stardawn.org>
	<87oco8ht9m.fsf@gmail.com>
	<b8d251e06b99e3e38654cf0ad06e1fce.squirrel@mail.stardawn.org>
	<87k4yuj4hh.fsf@gmail.com>
	<d27943fbe44805c8a7c88063b4d27195.squirrel@mail.stardawn.org>
	<87hbtyjkrm.fsf@gmail.com>
Message-ID: <b0ce920153dec96430db43f9a46c1d81.squirrel@mail.stardawn.org>


Alain Picard wrote:

> That's because my emacs automatically cleans up the whitespace
> as a file saving hook --- something I would heartily recommend
> to any developer using emacs.  But since this isn't my project,
> I will again try to make a simpler patch.

FWIW I guess most projects wouldn't accept such a patch. It's
fine to adapt whitespace to your needs in a non-destructive way,
and most patches where you change the whitespace of lines that
are changed anyway are fine too, but unrelated whitespace changes
are usually a no-go.


> OK - here is a MINIMAL patch.  It applies with
> patch -p1 < /tmp/patch.

> Hope this one works!

It does. I've applied it and will push it later after I ran
the test suite.

  Leslie

-- 
http://www.linkedin.com/in/polzer




From Dr.Alain.Picard at gmail.com  Sat Oct 17 23:42:37 2009
From: Dr.Alain.Picard at gmail.com (Alain Picard)
Date: Sun, 18 Oct 2009 10:42:37 +1100
Subject: [elephant-devel] Bug in GET-CON
References: <87vdihi8em.fsf@gmail.com>
	<a3f5d0f66a3a0a5b0b59878ec8b4ba85.squirrel@mail.stardawn.org>
	<87oco8ht9m.fsf@gmail.com>
	<b8d251e06b99e3e38654cf0ad06e1fce.squirrel@mail.stardawn.org>
	<87k4yuj4hh.fsf@gmail.com>
	<d27943fbe44805c8a7c88063b4d27195.squirrel@mail.stardawn.org>
	<87hbtyjkrm.fsf@gmail.com>
	<b0ce920153dec96430db43f9a46c1d81.squirrel@mail.stardawn.org>
Message-ID: <877hutk2oi.fsf@gmail.com>

"Leslie P. Polzer"
<sky at viridian-project.de> writes:

> FWIW I guess most projects wouldn't accept such a patch. It's

I understand that perfectly, which is why I was happy to provide
a new patch.  No harm done.  It's just that I've forced everone
on my team for 10 years to use the same .emacs file, so there's
never any "empty" or "meaningless" white space in any file (precisely
to minimize these sorts of fake conflicts) so I forgot that out
here in the "real world" not everybody will be using my
"conventions".  :-)

It's your project --- I'm happy to play by your rules.

> It does. I've applied it and will push it later after I ran
> the test suite.

Cool.  Lastly, it occured to me -- there's probably no reason
to store that new slot (OPENING-ARGS) permanently --- if you
wish to change that to a transient slot, I think it would work
fine and probably be better.

  --ap




From sky at viridian-project.de  Sun Oct 18 08:49:35 2009
From: sky at viridian-project.de (Leslie P. Polzer)
Date: Sun, 18 Oct 2009 10:49:35 +0200 (CEST)
Subject: [elephant-devel] Bug in GET-CON
In-Reply-To: <877hutk2oi.fsf@gmail.com>
References: <87vdihi8em.fsf@gmail.com>
	<a3f5d0f66a3a0a5b0b59878ec8b4ba85.squirrel@mail.stardawn.org>
	<87oco8ht9m.fsf@gmail.com>
	<b8d251e06b99e3e38654cf0ad06e1fce.squirrel@mail.stardawn.org>
	<87k4yuj4hh.fsf@gmail.com>
	<d27943fbe44805c8a7c88063b4d27195.squirrel@mail.stardawn.org>
	<87hbtyjkrm.fsf@gmail.com>
	<b0ce920153dec96430db43f9a46c1d81.squirrel@mail.stardawn.org>
	<877hutk2oi.fsf@gmail.com>
Message-ID: <e80314211771b3b33e22fb6e94ac1008.squirrel@mail.stardawn.org>


Hi Alain,

I just pushed your patch.

You wrote:

> I understand that perfectly, which is why I was happy to provide
> a new patch.  No harm done.  It's just that I've forced everone
> on my team for 10 years to use the same .emacs file, so there's
> never any "empty" or "meaningless" white space in any file (precisely
> to minimize these sorts of fake conflicts) so I forgot that out
> here in the "real world" not everybody will be using my
> "conventions".  :-)

I'm glad we agree on this. Thanks for your patience and understanding.


> It's your project --- I'm happy to play by your rules.

Well it's a community project, actually. I happen to have the commit
bit but that shouldn't really make my opinion worth much more than
yours.


> Cool.  Lastly, it occured to me -- there's probably no reason
> to store that new slot (OPENING-ARGS) permanently --- if you
> wish to change that to a transient slot, I think it would work
> fine and probably be better.

But STORE-CONTROLLER is not a persistent class, so that's neither
needed nor possible.

  Leslie

-- 
http://www.linkedin.com/in/polzer




From Dr.Alain.Picard at gmail.com  Mon Oct 19 01:26:19 2009
From: Dr.Alain.Picard at gmail.com (Alain Picard)
Date: Mon, 19 Oct 2009 12:26:19 +1100
Subject: [elephant-devel] Bug in GET-CON
References: <87vdihi8em.fsf@gmail.com>
	<a3f5d0f66a3a0a5b0b59878ec8b4ba85.squirrel@mail.stardawn.org>
	<87oco8ht9m.fsf@gmail.com>
	<b8d251e06b99e3e38654cf0ad06e1fce.squirrel@mail.stardawn.org>
	<87k4yuj4hh.fsf@gmail.com>
	<d27943fbe44805c8a7c88063b4d27195.squirrel@mail.stardawn.org>
	<87hbtyjkrm.fsf@gmail.com>
	<b0ce920153dec96430db43f9a46c1d81.squirrel@mail.stardawn.org>
	<877hutk2oi.fsf@gmail.com>
	<e80314211771b3b33e22fb6e94ac1008.squirrel@mail.stardawn.org>
Message-ID: <87hbtwmax0.fsf@gmail.com>

"Leslie P. Polzer"
<sky at viridian-project.de> writes:


> But STORE-CONTROLLER is not a persistent class, so that's neither
> needed nor possible.

Doh!!  :-)

-- 
Please read about why Top Posting
is evil at: http://en.wikipedia.org/wiki/Top-posting
and http://www.dickalba.demon.co.uk/usenet/guide/faq_topp.html
Please read about why HTML in email is evil at: http://www.birdhouse.org/etc/evilmail.html




From Dr.Alain.Picard at gmail.com  Mon Oct 19 04:33:45 2009
From: Dr.Alain.Picard at gmail.com (Alain Picard)
Date: Mon, 19 Oct 2009 15:33:45 +1100
Subject: [elephant-devel] What to use, what not to use...
Message-ID: <877husm28m.fsf@gmail.com>


Dear Pachyderm lovers,

I'm noticing that the manual describes "derived slot indices",
however I can find no tests or example usage of these.

Conversely, I can find a test for what appears to be a very
cool feature: ":associate" type slots (in testassociations.lisp)
but can find no mention of them anywhere in the docs.

Are either of these usages OK for prime time, or are they
considered experimental and subject to change/removal?


[Sorry to be pestering the group, but I imagine this is the sort
of puzzling out that has to occur when someone starts to use a
library "in anger".  Thanks for your patience.]

        --ap

-- 
Please read about why Top Posting
is evil at: http://en.wikipedia.org/wiki/Top-posting
and http://www.dickalba.demon.co.uk/usenet/guide/faq_topp.html
Please read about why HTML in email is evil at: http://www.birdhouse.org/etc/evilmail.html




From eslick at media.mit.edu  Mon Oct 19 06:17:58 2009
From: eslick at media.mit.edu (Ian Eslick)
Date: Sun, 18 Oct 2009 23:17:58 -0700
Subject: [elephant-devel] What to use, what not to use...
In-Reply-To: <877husm28m.fsf@gmail.com>
References: <877husm28m.fsf@gmail.com>
Message-ID: <08922372-757A-4466-AC75-A528E494B676@media.mit.edu>

Derived index slots have been around for quite some time (before 0.5  
and 0.6 at least).  I'm surprised coverage of them isn't in the  
manual.  The interface changed a few years ago (they are now defined  
in the slot defs and not subject to dynamic addition/removal).   
However they were broken a few months ago; I think Leslie was looking  
at it but I can't recall.

Association slots were added recently and have not been fully tested.   
They should be considered in beta.

If I send you a short description in the next few days will you  
volunteer to update the docs?  :)

Ian

On Oct 18, 2009, at 9:33 PM, Alain Picard wrote:

>
> Dear Pachyderm lovers,
>
> I'm noticing that the manual describes "derived slot indices",
> however I can find no tests or example usage of these.
>
> Conversely, I can find a test for what appears to be a very
> cool feature: ":associate" type slots (in testassociations.lisp)
> but can find no mention of them anywhere in the docs.
>
> Are either of these usages OK for prime time, or are they
> considered experimental and subject to change/removal?
>
>
> [Sorry to be pestering the group, but I imagine this is the sort
> of puzzling out that has to occur when someone starts to use a
> library "in anger".  Thanks for your patience.]
>
>        --ap
>
> -- 
> Please read about why Top Posting
> is evil at: http://en.wikipedia.org/wiki/Top-posting
> and http://www.dickalba.demon.co.uk/usenet/guide/faq_topp.html
> Please read about why HTML in email is evil at: http://www.birdhouse.org/etc/evilmail.html
>
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel




From sky at viridian-project.de  Mon Oct 19 07:16:54 2009
From: sky at viridian-project.de (Leslie P. Polzer)
Date: Mon, 19 Oct 2009 09:16:54 +0200 (CEST)
Subject: [elephant-devel] What to use, what not to use...
In-Reply-To: <08922372-757A-4466-AC75-A528E494B676@media.mit.edu>
References: <877husm28m.fsf@gmail.com>
	<08922372-757A-4466-AC75-A528E494B676@media.mit.edu>
Message-ID: <d0969a43c9ae058ab216271d796f4738.squirrel@mail.stardawn.org>


Ian Eslick wrote:
> Derived index slots have been around for quite some time (before 0.5
> and 0.6 at least).  I'm surprised coverage of them isn't in the
> manual.  The interface changed a few years ago (they are now defined
> in the slot defs and not subject to dynamic addition/removal).
> However they were broken a few months ago; I think Leslie was looking
> at it but I can't recall.

Someone volunteered to do this with my help but I haven't heard from
him since, unfortunately.

IIRC it wasn't that hard to fix.

  Leslie

-- 
http://www.linkedin.com/in/polzer




From sky at viridian-project.de  Mon Oct 19 07:17:38 2009
From: sky at viridian-project.de (Leslie P. Polzer)
Date: Mon, 19 Oct 2009 09:17:38 +0200 (CEST)
Subject: [elephant-devel] What to use, what not to use...
In-Reply-To: <877husm28m.fsf@gmail.com>
References: <877husm28m.fsf@gmail.com>
Message-ID: <316616a6a8c33057cdfb35556c3446bb.squirrel@mail.stardawn.org>


Alain Picard wrote:

> [Sorry to be pestering the group, but I imagine this is the sort
> of puzzling out that has to occur when someone starts to use a
> library "in anger".  Thanks for your patience.]

Feel free to ask more, right now it's quiet around here anyway. ;)

  Leslie

-- 
http://www.linkedin.com/in/polzer




From Dr.Alain.Picard at gmail.com  Mon Oct 19 23:20:42 2009
From: Dr.Alain.Picard at gmail.com (Alain Picard)
Date: Tue, 20 Oct 2009 10:20:42 +1100
Subject: [elephant-devel] What to use, what not to use...
References: <877husm28m.fsf@gmail.com>
	<08922372-757A-4466-AC75-A528E494B676@media.mit.edu>
Message-ID: <873a5fm0mt.fsf@gmail.com>

Ian Eslick <eslick at media.mit.edu> writes:

> Association slots were added recently and have not been fully tested.   
> They should be considered in beta.
>
> If I send you a short description in the next few days will you  
> volunteer to update the docs?  :)

Possibly, but I can't make promises.  I might have a couple
of hours late next weekend, if you can get me the necessary
info by Friday.

Cheers!

        --ap




From eslick at media.mit.edu  Tue Oct 20 00:36:03 2009
From: eslick at media.mit.edu (Ian Eslick)
Date: Mon, 19 Oct 2009 17:36:03 -0700
Subject: [elephant-devel] What to use, what not to use...
In-Reply-To: <873a5fm0mt.fsf@gmail.com>
References: <877husm28m.fsf@gmail.com>
	<08922372-757A-4466-AC75-A528E494B676@media.mit.edu>
	<873a5fm0mt.fsf@gmail.com>
Message-ID: <AEBB7E76-43E1-4155-93FE-D7DACF191E58@media.mit.edu>

This will need some cleaning up, and I'm sure you'll have questions.

Derived index:
=============================

To compute an index value derived from the current values of an  
instance, simply define a special slot with the argument :derived-fn.   
The derived-fn is a function name or form that will compute the  
derived value.  An index is defined on this slot and the computed  
value is stored in the instance (not recomputed upon read).

:derived-fn - the name or form of a function which computes the  
derived value and returns two values: the value and a boolean  
indicating whether it should be indexed or not

:slot-deps  - is a hint that restricts derived index updates to writes  
to the named slots.

(defpclass foo ()
   ((slot1 :accessor slot1 ...)
    (slot2 :accessor slot2 ...)
    (slot3 :accessor slot3 ...)
    (sum1-2 :reader sum :derived-fn my-derived :slot-deps (slot1  
slot2))))

(defun my-derived (instance)
	(values  (+ (slot1 instance) (slot2 instance)))
	               t))

:index t is not necessary - in fact it is ignored.  :slot-deps are  
also not required, but the derived index is updated on any slot write  
if that slot is not transient, set-valued or an association.  We can  
add those last three slot types into the mix if necessary, but I'm  
trying to avoid too much complex computation taking place during slot  
writes (self-deadlock, etc) for the time being.


Associations:
=======================

One-to-Many associations:

If class B has a slot that stores a single instance of A, then class A  
would like to have a method that returns the set of instances of B  
that refer to the instance of A.

If A is person and B is job, then (slot-value job-inst 'owner is a  
reference to A and A->jobs is a method (no slot storage) that refers  
to an index on job->owner.  The index keys are A oids.

(defpclass person ()
   ((jobs :accessor jobs :associate (job person))))

(defpclass job ()
   ((person :accessor person :associate person)
    (title :accessor title ...)
    (company :accessor company ...)))

The slot 'person in job behaves like an indexed slot.  However, when  
you evaluate (jobs person-instance) you get the list of jobs that have  
person-instance as their person slot value.

reading:  Reads using slot-value or the accessor will return the  
appropriate associations.
writing:  (setf (jobs person-instance) job-instance) will set the slot- 
value of 'person for job-instance to the person-instance, removing any  
prior association.
boundp:  The slot 'jobs' is virtual, and therefore permanently slot- 
unboundp.
remove: To remove an association, you can set the slot in the job  
class to nil or make it unbound.

(get-associations instance slotname) => same as calling (slot-accessor  
instance)
(add-association instance slotname associated) => same as calling  
(setf (slot-accessor instance) associated)
(remove-association instance slotname associated) => removes the  
association from a virtual association slot only


Many-to-Many:

To add many-to-many relations, any A->B relation also requires that we  
can find the relation from the other direction B->A.  Typically this  
requires a special table to keep track of pairs of related objects.   
You can then query on either column to get the set of related  
objects.  This interface simply causes both sides to behave like the  
jobs slot of the person class above.

(defpclass person ()
   ((jobs :accessor jobs :associate (job person))

(defpclass job ()
   ((persons :accessor persons :associate (person jobs))
    (title :accessor title ...)
    (company :accessor company ...)))

reading: (jobs pinst)  (persons jinst) => returns the associated  
persons/jobs
writing: (setf (jobs pinst) jinst) => will add jinst to the pinst jobs  
slot and pinst to the jinst persons slot. Writes to either slot  
results in a bi-directional association
slot-boundp: both slots will return nil for slot-boundp

The only way to remove a created association is to call (remove- 
association instance 'slot associated) on either associated instance.

Note: schema changes between association slot types may be buggy and/ 
or cause data loss.  Be careful!

On Oct 19, 2009, at 4:20 PM, Alain Picard wrote:

> Ian Eslick <eslick at media.mit.edu> writes:
>
>> Association slots were added recently and have not been fully tested.
>> They should be considered in beta.
>>
>> If I send you a short description in the next few days will you
>> volunteer to update the docs?  :)
>
> Possibly, but I can't make promises.  I might have a couple
> of hours late next weekend, if you can get me the necessary
> info by Friday.
>
> Cheers!
>
>        --ap
>
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel