From ihuquan at gmail.com  Thu Jan  1 13:31:30 2009
From: ihuquan at gmail.com (quan hu)
Date: Thu, 1 Jan 2009 21:31:30 +0800
Subject: [elephant-devel] DB migration issue
In-Reply-To: <286EADCF-6B83-4A2F-981E-C085311892FB@media.mit.edu>
References: <1cc8deb80812290040k7720ca03hdbca03dbd8375135@mail.gmail.com>
	<D5A0560C-AFCB-44F1-9B32-F18DAB149960@media.mit.edu>
	<49255C15-AE3C-4B83-A73C-940670519C51@media.mit.edu>
	<ecb948b30812291013y7dc5b01eka090e18d8c4e9fea@mail.gmail.com>
	<1cc8deb80812292147i5d39c76dq612c2d0b2e5dc70b@mail.gmail.com>
	<1cc8deb80812300634n57d81ca4iaf770dbc3c7b0dfd@mail.gmail.com>
	<F0B94D5B-6F33-4BC6-AEB4-A8352974AD93@media.mit.edu>
	<1cc8deb80812310537n7048dbd5w14f200d990dd2439@mail.gmail.com>
	<286EADCF-6B83-4A2F-981E-C085311892FB@media.mit.edu>
Message-ID: <1cc8deb80901010531mefc0f88t123b359c57a5147b@mail.gmail.com>

Hi Ian,

     For the pathname fix, I attached the patch.

     For the 0.9.1 upgrade fix,  I got one failure and one successful
result.
     I need to double check if the failure is caused by my test execution
step or other issue.
     I'll send the result later.

Thanks
Quan


2008/12/31 Ian Eslick <eslick at media.mit.edu>

> I don't think we've tried the upgrade procedure in awhile, so it's
> possible an incompatibility slipped in.  It may be that all you need
> to do is make sure that defaults are properly created for slot values
> that are unbound during the store controller opening phase.
>
> Adding a case for pathname is easy enough.  I take it one of the slots
> of the pathname structure object is a function; migrating this is
> causing the error.
>
> With a little work you should be able to do both of these fixes,
> certainly the pathname fix.  I'd be happy to review and commit them to
> the repository, but unfortunately I won't have any more time to work
> on elephant for a little while.
>
> Ian
>
> On Dec 31, 2008, at 8:37 AM, quan hu wrote:
>
> > Hi Ian,
> >
> >      I downloaded the patches and have more testing today. The lock
> > issue is gone!
> >      Thanks a lot.
> >
> >      1. But, another issue is found. migrate pathname fails. I test
> > it under SBCL 1.0.21
> >       e.g.
> >        >  (migrate *store-controller* #p"/tmp) => Error: Cannot
> > migrate function objects (i.e. #<FUNCTION SB-IMPL::UNPARSE-UNIX-
> > ENOUGH>)
> >
> >       It is caused by that (defmethod migrate ((dst store-
> > controller) (src structure-object)  is called for this case, as
> > pathname is taken as subtype of structure-object.
> >       > (subtypep 'pathname 'structure-object) => t t
> >       Maybe, one more migrate method is needed for pathname case.
> >
> >       2. I also want to know if upgrading database file from 0.9.1
> > to unstable release is supported.
> >       When opening the DB file generated in 0.9.1 in unstable
> > release, I get error message
> >        " The slot DB-BDB::INDICES is unbound in the object #<BDB-
> > INDEXED-BTREE oid:-3>.
> >           [Condition of type UNBOUND-SLOT]"
> >
> > Thanks
> > Quan
> >
> >
> >
> >
> > 2008/12/31 Ian Eslick <eslick at media.mit.edu>
> > Hi Quan,
> >
> > Thank you again for the information on this bug.  The short story is
> > that the way we are keeping track of transactions assumed that there
> > were only two stores in the nested transaction stack (outer and
> > inner).  Migrating a nested object inside a with-transaction would not
> > pass the parent transaction to the backend so it wouldn't know a
> > transaction in progress and attempted to start a new one, which would
> > block until the old one was completed.
> >
> > I just checked in a patch that made transaction tracking a bit more
> > intelligent (it depends on the backends taking care of handling nested
> > transactions - which I believe they all do fine).  I also checked in a
> > test which implements Quan's test case.
> >
> > This patch and the prior patches of the past few days pass all tests
> > on SBCL/Mac/BDB.
> >
> > Ian
> >
> > PS - I'd love to know the current status of the elephant-unstable tree
> > with the postmodern backend.
> >
> >
> > On Dec 30, 2008, at 9:34 AM, quan hu wrote:
> >
> > > Hello,
> > >
> > >       After more investigation, here are some suspects:
> > >
> > >       In the current elephant codes, the macro my-current-
> > > transaction will check if the current transaction belongs to the sc
> > > parameter.   If not, just return +NULL-CHAR+. That's exactly the
> > > case in the following nested transaction where the store-controller
> > > is different.
> > >       The situation becomes:
> > >              Transaction 1 begins
> > >                     One data is put into DB with Transaction 1
> > >                     ...
> > >                     Another data is put into the same DB with NULL
> > > transaction parameter(i.e. no transaction??)
> > >
> > >        It looks like this scenario is not supported by BDB.
> > >        I simulate above case with a C program to access BDB and get
> > > the similar failure symptom.
> > > ...
> > >   txn1 = NULL;
> > >   ret = dbenvp1->txn_begin(dbenvp1, NULL, &txn1, 0);
> > >   if (ret != 0) {
> > >     printf ("Transaction begin failed.\n");
> > >     return 1;
> > >   }
> > >   put_record(FOO,FOO,txn1, dbp1);
> > >   printf("foo\n");
> > >   put_record(FOO,FOO,NULL, dbp1);
> > >   printf("bar\n");
> > >   txn1->commit(txn1,0);
> > >   printf("baz\n");
> > > ...
> > >  After "foo" is printed, the C program is blocked.  If I change the
> > > NULL to txn1, everything is fine.
> > >
> > >
> > >  Regarding to the solution, my gut feeling is that it would be
> > > difficult to avoid this kind of nested transaction in migrate
> > > process, because of the recursive function call. Maybe,
> > > *transaction-stack*  can be added to track the active transaction
> > > stack. Get it pushed and poped in right place of execute-
> > > transaction. Then, in my-current-transaction, check *transaction-
> > > stack* to see if there is an active transaction belongs to the same
> > > store-controller. If it is found, use it.   My simple test shows it
> > > works. If this is the right direction, I can do more test.
> > >
> > > Thanks for your help.
> > > Quan
> > >
> > >
> > >
> > >
> > >
> > > 2008/12/30 quan hu <ihuquan at gmail.com>
> > > Hi Ian, Yarek,
> > >
> > >     Thanks for providing the information.
> > >      I downloaded the elephant-unstable and test it with BDB4.7.
> > >      The failure symptom is the same: target store controller get
> > > locked.
> > >
> > >      After I make following changes in migrate.lisp; The test
> > passed.
> > >
> > >       (defmethod copy-btree-contents ((sc store-controller) dst src)
> > >             (let (to-be-migrated)
> > >                   (map-btree (lambda (key value)
> > >                                      (let ((newval (migrate sc
> > value))
> > >                                             (newkey (migrate sc
> > key)))
> > >                                            (push (list newkey
> > > newval) to-be-migrated)))
> > >                                            ;;;;    (setf (get-value
> > > newkey dst) newval)))  ;;
> > >                                            src)
> > >                   (loop for (k v) in to-be-migrated
> > >                     do
> > >                     (setf (get-value k dst) v))))
> > >
> > >      Above debug change just avoids the nested transaction of
> > > different store controller for the specific test case.
> > >      However, it is not the solution, as the nested transaction
> > > still could appear in other data pattern.
> > >
> > >     What confuses me is that the nested transaction of the same
> > > store controller seem not bring any block.
> > >      Following test passed:
> > >         (with-transaction (:store-controller *migrate-dst*)
> > >
> > >              (add-to-root 'foo 'foo :sc *migrate-dst*)
> > >              (with-transaction (:store-controller *migrate-dst*)
> > >
> > >                 (add-to-root 'bar 'bar :sc *migrate-dst*)))
> > >
> > >     But, the nested transaction of different store controller always
> > > results in lock.
> > >
> > >         (with-transaction (:store-controller *migrate-dst*)
> > >
> > >              (add-to-root 'foo 'foo :sc *migrate-dst*)
> > >              (with-transaction (:store-controller *migrate-
> > > src*)         ;; trigger database lock
> > >                  (add-to-root 'bar 'bar :sc *migrate-dst*)))
> > >
> > > Thanks
> > > Quan
> > >
> > >
> > >
> > > 2008/12/30 Yarek Kowalik <yarek.kowalik at gmail.com>
> > >
> > > Quan,
> > >
> > > Unstable is here:
> > >
> > >  http://www.common-lisp.net/project/elephant/darcs/elephant-unstable
> > >
> > > Yarek
> > >
> > >
> > > On Mon, Dec 29, 2008 at 5:37 AM, Ian Eslick <eslick at media.mit.edu>
> > > wrote:
> > > That would be elephant-unstable
> > >
> > > Sent from my iPhone
> > >
> > > On Dec 29, 2008, at 7:36 AM, Ian Eslick <eslick at media.mit.edu>
> > wrote:
> > >
> > > > Hi Quan,
> > > >
> > > > Can you try the latest darcs with BDB 4.6 or 4.7?  I made some
> > > changes
> > > > to migration, but I can't recall if they were in the latest
> > darcs or
> > > > not - I think the latest darcs doesn't play well with BDB 4.5 so
> > > > please make sure you can reproduce under the above configuration
> > if
> > > > you can.
> > > >
> > > > Thank you,
> > > > Ian
> > > >
> > > > On Dec 29, 2008, at 3:40 AM, quan hu wrote:
> > > >
> > > >> Hello,
> > > >>
> > > >>     I run into a problem when doing the garbage collection via
> > data
> > > >> migration.
> > > >>     The environment is elephant 0.9.1 and BDB 4.5.  I also tried
> > > >> the latest elephant in darc and get the same result.
> > > >>
> > > >>     1.  Test case to reproduce the problem.
> > > >>
> > > >>           (defpclass user-profile()
> > > >>             ((id :initform nil :index t)
> > > >>             (sms-inbox  :initform (make-pset))))
> > > >>
> > > >>            ;; Create a new db for test purpose
> > > >>           (setf *migrate-src* (open-store '(:bdb "/tmp/db/
> > src/")))
> > > >>           (setf *test-obj* (make-instance 'user-profile :sc
> > > >> *migrate-src*))
> > > >>            (insert-item 'foo (slot-value *test-obj* 'sms-inbox))
> > > >>            (setf *migrate-dst* (open-store '(:bdb "/tmp/db/
> > dst/")))
> > > >>
> > > >>          TEST> (migrate *migrate-dst* *migrate-src*)
> > > >>              Migrating class indexes for: USER-PROFILE
> > > >>           ; => Input get blocked, not response anymore
> > > >>
> > > >>       2.  Using "db_stat -C A" get following output:
> > > >>             1    Lock requests not available due to conflicts,
> > for
> > > >> which we waited
> > > >>
> > > >>       3.  I do some debug work and found the problem may be
> > caused
> > > >> by nested transaction of different store controller.
> > > >>            I can reproduce the issue with following code segment
> > > >>
> > > >>          TEST> (ensure-transaction (:store-controller *migrate-
> > > dst*)
> > > >>                            (add-to-root 'foo 'foo :sc *migrate-
> > > dst*)
> > > >>                            (ensure-transaction (:store-controller
> > > >> *migrate-src*)
> > > >>                                 (add-to-root 'bar 'bar :sc
> > > *migrate-
> > > >> dst*)))
> > > >>
> > > >>             This kind of nested transaction scenario can appear
> > in
> > > >> the migrate process, because it uses source/destination store
> > > >> controller at the same time.
> > > >>
> > > >>            I do not understand why different store controller's
> > > >> transaction can result in the lock conflict and want to know
> > how to
> > > >> fix it.
> > > >>
> > > >>            Is there anyone meet the similar problem before?
> > > >>
> > > >> Thanks
> > > >> Quan
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> _______________________________________________
> > > >> elephant-devel site list
> > > >> elephant-devel at common-lisp.net
> > > >> http://common-lisp.net/mailman/listinfo/elephant-devel
> > > >
> > > >
> > > > _______________________________________________
> > > > elephant-devel site list
> > > > elephant-devel at common-lisp.net
> > > > http://common-lisp.net/mailman/listinfo/elephant-devel
> > >
> > > _______________________________________________
> > > elephant-devel site list
> > > elephant-devel at common-lisp.net
> > > http://common-lisp.net/mailman/listinfo/elephant-devel
> > >
> > >
> > > _______________________________________________
> > > elephant-devel site list
> > > elephant-devel at common-lisp.net
> > > http://common-lisp.net/mailman/listinfo/elephant-devel
> > >
> > >
> > > _______________________________________________
> > > elephant-devel site list
> > > elephant-devel at common-lisp.net
> > > http://common-lisp.net/mailman/listinfo/elephant-devel
> >
> >
> > _______________________________________________
> > elephant-devel site list
> > elephant-devel at common-lisp.net
> > http://common-lisp.net/mailman/listinfo/elephant-devel
> >
> > _______________________________________________
> > elephant-devel site list
> > elephant-devel at common-lisp.net
> > http://common-lisp.net/mailman/listinfo/elephant-devel
>
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20090101/7c78736b/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: migrate-pathname-patch.gz
Type: application/x-gzip
Size: 7356 bytes
Desc: not available
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20090101/7c78736b/attachment.bin>

From eslick at media.mit.edu  Thu Jan  1 15:42:56 2009
From: eslick at media.mit.edu (Ian Eslick)
Date: Thu, 1 Jan 2009 10:42:56 -0500
Subject: [elephant-devel] DB migration issue
In-Reply-To: <1cc8deb80901010531mefc0f88t123b359c57a5147b@mail.gmail.com>
References: <1cc8deb80812290040k7720ca03hdbca03dbd8375135@mail.gmail.com>
	<D5A0560C-AFCB-44F1-9B32-F18DAB149960@media.mit.edu>
	<49255C15-AE3C-4B83-A73C-940670519C51@media.mit.edu>
	<ecb948b30812291013y7dc5b01eka090e18d8c4e9fea@mail.gmail.com>
	<1cc8deb80812292147i5d39c76dq612c2d0b2e5dc70b@mail.gmail.com>
	<1cc8deb80812300634n57d81ca4iaf770dbc3c7b0dfd@mail.gmail.com>
	<F0B94D5B-6F33-4BC6-AEB4-A8352974AD93@media.mit.edu>
	<1cc8deb80812310537n7048dbd5w14f200d990dd2439@mail.gmail.com>
	<286EADCF-6B83-4A2F-981E-C085311892FB@media.mit.edu>
	<1cc8deb80901010531mefc0f88t123b359c57a5147b@mail.gmail.com>
Message-ID: <8F7B3CFC-3958-477C-B7F3-D6A540D3EDB6@media.mit.edu>

Great, thank you!  I'll check this in later today.

Ian

Sent from my iPhone

On Jan 1, 2009, at 8:31 AM, "quan hu" <ihuquan at gmail.com> wrote:

> Hi Ian,
>
>      For the pathname fix, I attached the patch.
>
>      For the 0.9.1 upgrade fix,  I got one failure and one  
> successful result.
>      I need to double check if the failure is caused by my test  
> execution step or other issue.
>      I'll send the result later.
>
> Thanks
> Quan
>
>
> 2008/12/31 Ian Eslick <eslick at media.mit.edu>
> I don't think we've tried the upgrade procedure in awhile, so it's
> possible an incompatibility slipped in.  It may be that all you need
> to do is make sure that defaults are properly created for slot values
> that are unbound during the store controller opening phase.
>
> Adding a case for pathname is easy enough.  I take it one of the slots
> of the pathname structure object is a function; migrating this is
> causing the error.
>
> With a little work you should be able to do both of these fixes,
> certainly the pathname fix.  I'd be happy to review and commit them to
> the repository, but unfortunately I won't have any more time to work
> on elephant for a little while.
>
> Ian
>
> On Dec 31, 2008, at 8:37 AM, quan hu wrote:
>
> > Hi Ian,
> >
> >      I downloaded the patches and have more testing today. The lock
> > issue is gone!
> >      Thanks a lot.
> >
> >      1. But, another issue is found. migrate pathname fails. I test
> > it under SBCL 1.0.21
> >       e.g.
> >        >  (migrate *store-controller* #p"/tmp) => Error: Cannot
> > migrate function objects (i.e. #<FUNCTION SB-IMPL::UNPARSE-UNIX-
> > ENOUGH>)
> >
> >       It is caused by that (defmethod migrate ((dst store-
> > controller) (src structure-object)  is called for this case, as
> > pathname is taken as subtype of structure-object.
> >       > (subtypep 'pathname 'structure-object) => t t
> >       Maybe, one more migrate method is needed for pathname case.
> >
> >       2. I also want to know if upgrading database file from 0.9.1
> > to unstable release is supported.
> >       When opening the DB file generated in 0.9.1 in unstable
> > release, I get error message
> >        " The slot DB-BDB::INDICES is unbound in the object #<BDB-
> > INDEXED-BTREE oid:-3>.
> >           [Condition of type UNBOUND-SLOT]"
> >
> > Thanks
> > Quan
> >
> >
> >
> >
> > 2008/12/31 Ian Eslick <eslick at media.mit.edu>
> > Hi Quan,
> >
> > Thank you again for the information on this bug.  The short story is
> > that the way we are keeping track of transactions assumed that there
> > were only two stores in the nested transaction stack (outer and
> > inner).  Migrating a nested object inside a with-transaction would  
> not
> > pass the parent transaction to the backend so it wouldn't know a
> > transaction in progress and attempted to start a new one, which  
> would
> > block until the old one was completed.
> >
> > I just checked in a patch that made transaction tracking a bit more
> > intelligent (it depends on the backends taking care of handling  
> nested
> > transactions - which I believe they all do fine).  I also checked  
> in a
> > test which implements Quan's test case.
> >
> > This patch and the prior patches of the past few days pass all tests
> > on SBCL/Mac/BDB.
> >
> > Ian
> >
> > PS - I'd love to know the current status of the elephant-unstable  
> tree
> > with the postmodern backend.
> >
> >
> > On Dec 30, 2008, at 9:34 AM, quan hu wrote:
> >
> > > Hello,
> > >
> > >       After more investigation, here are some suspects:
> > >
> > >       In the current elephant codes, the macro my-current-
> > > transaction will check if the current transaction belongs to the  
> sc
> > > parameter.   If not, just return +NULL-CHAR+. That's exactly the
> > > case in the following nested transaction where the store- 
> controller
> > > is different.
> > >       The situation becomes:
> > >              Transaction 1 begins
> > >                     One data is put into DB with Transaction 1
> > >                     ...
> > >                     Another data is put into the same DB with NULL
> > > transaction parameter(i.e. no transaction??)
> > >
> > >        It looks like this scenario is not supported by BDB.
> > >        I simulate above case with a C program to access BDB and  
> get
> > > the similar failure symptom.
> > > ...
> > >   txn1 = NULL;
> > >   ret = dbenvp1->txn_begin(dbenvp1, NULL, &txn1, 0);
> > >   if (ret != 0) {
> > >     printf ("Transaction begin failed.\n");
> > >     return 1;
> > >   }
> > >   put_record(FOO,FOO,txn1, dbp1);
> > >   printf("foo\n");
> > >   put_record(FOO,FOO,NULL, dbp1);
> > >   printf("bar\n");
> > >   txn1->commit(txn1,0);
> > >   printf("baz\n");
> > > ...
> > >  After "foo" is printed, the C program is blocked.  If I change  
> the
> > > NULL to txn1, everything is fine.
> > >
> > >
> > >  Regarding to the solution, my gut feeling is that it would be
> > > difficult to avoid this kind of nested transaction in migrate
> > > process, because of the recursive function call. Maybe,
> > > *transaction-stack*  can be added to track the active transaction
> > > stack. Get it pushed and poped in right place of execute-
> > > transaction. Then, in my-current-transaction, check *transaction-
> > > stack* to see if there is an active transaction belongs to the  
> same
> > > store-controller. If it is found, use it.   My simple test shows  
> it
> > > works. If this is the right direction, I can do more test.
> > >
> > > Thanks for your help.
> > > Quan
> > >
> > >
> > >
> > >
> > >
> > > 2008/12/30 quan hu <ihuquan at gmail.com>
> > > Hi Ian, Yarek,
> > >
> > >     Thanks for providing the information.
> > >      I downloaded the elephant-unstable and test it with BDB4.7.
> > >      The failure symptom is the same: target store controller get
> > > locked.
> > >
> > >      After I make following changes in migrate.lisp; The test
> > passed.
> > >
> > >       (defmethod copy-btree-contents ((sc store-controller) dst  
> src)
> > >             (let (to-be-migrated)
> > >                   (map-btree (lambda (key value)
> > >                                      (let ((newval (migrate sc
> > value))
> > >                                             (newkey (migrate sc
> > key)))
> > >                                            (push (list newkey
> > > newval) to-be-migrated)))
> > >                                            ;;;;    (setf (get- 
> value
> > > newkey dst) newval)))  ;;
> > >                                            src)
> > >                   (loop for (k v) in to-be-migrated
> > >                     do
> > >                     (setf (get-value k dst) v))))
> > >
> > >      Above debug change just avoids the nested transaction of
> > > different store controller for the specific test case.
> > >      However, it is not the solution, as the nested transaction
> > > still could appear in other data pattern.
> > >
> > >     What confuses me is that the nested transaction of the same
> > > store controller seem not bring any block.
> > >      Following test passed:
> > >         (with-transaction (:store-controller *migrate-dst*)
> > >
> > >              (add-to-root 'foo 'foo :sc *migrate-dst*)
> > >              (with-transaction (:store-controller *migrate-dst*)
> > >
> > >                 (add-to-root 'bar 'bar :sc *migrate-dst*)))
> > >
> > >     But, the nested transaction of different store controller  
> always
> > > results in lock.
> > >
> > >         (with-transaction (:store-controller *migrate-dst*)
> > >
> > >              (add-to-root 'foo 'foo :sc *migrate-dst*)
> > >              (with-transaction (:store-controller *migrate-
> > > src*)         ;; trigger database lock
> > >                  (add-to-root 'bar 'bar :sc *migrate-dst*)))
> > >
> > > Thanks
> > > Quan
> > >
> > >
> > >
> > > 2008/12/30 Yarek Kowalik <yarek.kowalik at gmail.com>
> > >
> > > Quan,
> > >
> > > Unstable is here:
> > >
> > >  http://www.common-lisp.net/project/elephant/darcs/elephant-unstable
> > >
> > > Yarek
> > >
> > >
> > > On Mon, Dec 29, 2008 at 5:37 AM, Ian Eslick <eslick at media.mit.edu>
> > > wrote:
> > > That would be elephant-unstable
> > >
> > > Sent from my iPhone
> > >
> > > On Dec 29, 2008, at 7:36 AM, Ian Eslick <eslick at media.mit.edu>
> > wrote:
> > >
> > > > Hi Quan,
> > > >
> > > > Can you try the latest darcs with BDB 4.6 or 4.7?  I made some
> > > changes
> > > > to migration, but I can't recall if they were in the latest
> > darcs or
> > > > not - I think the latest darcs doesn't play well with BDB 4.5 so
> > > > please make sure you can reproduce under the above configuration
> > if
> > > > you can.
> > > >
> > > > Thank you,
> > > > Ian
> > > >
> > > > On Dec 29, 2008, at 3:40 AM, quan hu wrote:
> > > >
> > > >> Hello,
> > > >>
> > > >>     I run into a problem when doing the garbage collection via
> > data
> > > >> migration.
> > > >>     The environment is elephant 0.9.1 and BDB 4.5.  I also  
> tried
> > > >> the latest elephant in darc and get the same result.
> > > >>
> > > >>     1.  Test case to reproduce the problem.
> > > >>
> > > >>           (defpclass user-profile()
> > > >>             ((id :initform nil :index t)
> > > >>             (sms-inbox  :initform (make-pset))))
> > > >>
> > > >>            ;; Create a new db for test purpose
> > > >>           (setf *migrate-src* (open-store '(:bdb "/tmp/db/
> > src/")))
> > > >>           (setf *test-obj* (make-instance 'user-profile :sc
> > > >> *migrate-src*))
> > > >>            (insert-item 'foo (slot-value *test-obj* 'sms- 
> inbox))
> > > >>            (setf *migrate-dst* (open-store '(:bdb "/tmp/db/
> > dst/")))
> > > >>
> > > >>          TEST> (migrate *migrate-dst* *migrate-src*)
> > > >>              Migrating class indexes for: USER-PROFILE
> > > >>           ; => Input get blocked, not response anymore
> > > >>
> > > >>       2.  Using "db_stat -C A" get following output:
> > > >>             1    Lock requests not available due to conflicts,
> > for
> > > >> which we waited
> > > >>
> > > >>       3.  I do some debug work and found the problem may be
> > caused
> > > >> by nested transaction of different store controller.
> > > >>            I can reproduce the issue with following code  
> segment
> > > >>
> > > >>          TEST> (ensure-transaction (:store-controller *migrate-
> > > dst*)
> > > >>                            (add-to-root 'foo 'foo :sc *migrate-
> > > dst*)
> > > >>                            (ensure-transaction (:store- 
> controller
> > > >> *migrate-src*)
> > > >>                                 (add-to-root 'bar 'bar :sc
> > > *migrate-
> > > >> dst*)))
> > > >>
> > > >>             This kind of nested transaction scenario can appear
> > in
> > > >> the migrate process, because it uses source/destination store
> > > >> controller at the same time.
> > > >>
> > > >>            I do not understand why different store controller's
> > > >> transaction can result in the lock conflict and want to know
> > how to
> > > >> fix it.
> > > >>
> > > >>            Is there anyone meet the similar problem before?
> > > >>
> > > >> Thanks
> > > >> Quan
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> _______________________________________________
> > > >> elephant-devel site list
> > > >> elephant-devel at common-lisp.net
> > > >> http://common-lisp.net/mailman/listinfo/elephant-devel
> > > >
> > > >
> > > > _______________________________________________
> > > > elephant-devel site list
> > > > elephant-devel at common-lisp.net
> > > > http://common-lisp.net/mailman/listinfo/elephant-devel
> > >
> > > _______________________________________________
> > > elephant-devel site list
> > > elephant-devel at common-lisp.net
> > > http://common-lisp.net/mailman/listinfo/elephant-devel
> > >
> > >
> > > _______________________________________________
> > > elephant-devel site list
> > > elephant-devel at common-lisp.net
> > > http://common-lisp.net/mailman/listinfo/elephant-devel
> > >
> > >
> > > _______________________________________________
> > > elephant-devel site list
> > > elephant-devel at common-lisp.net
> > > http://common-lisp.net/mailman/listinfo/elephant-devel
> >
> >
> > _______________________________________________
> > elephant-devel site list
> > elephant-devel at common-lisp.net
> > http://common-lisp.net/mailman/listinfo/elephant-devel
> >
> > _______________________________________________
> > elephant-devel site list
> > elephant-devel at common-lisp.net
> > http://common-lisp.net/mailman/listinfo/elephant-devel
>
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel
>
> _________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel
>
> br>
> <migrate-pathname-patch.gz>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20090101/7587f959/attachment.html>

From eslick at media.mit.edu  Thu Jan  1 20:39:17 2009
From: eslick at media.mit.edu (Ian Eslick)
Date: Thu, 1 Jan 2009 15:39:17 -0500
Subject: [elephant-devel] DB migration issue
In-Reply-To: <1cc8deb80901010531mefc0f88t123b359c57a5147b@mail.gmail.com>
References: <1cc8deb80812290040k7720ca03hdbca03dbd8375135@mail.gmail.com>
	<D5A0560C-AFCB-44F1-9B32-F18DAB149960@media.mit.edu>
	<49255C15-AE3C-4B83-A73C-940670519C51@media.mit.edu>
	<ecb948b30812291013y7dc5b01eka090e18d8c4e9fea@mail.gmail.com>
	<1cc8deb80812292147i5d39c76dq612c2d0b2e5dc70b@mail.gmail.com>
	<1cc8deb80812300634n57d81ca4iaf770dbc3c7b0dfd@mail.gmail.com>
	<F0B94D5B-6F33-4BC6-AEB4-A8352974AD93@media.mit.edu>
	<1cc8deb80812310537n7048dbd5w14f200d990dd2439@mail.gmail.com>
	<286EADCF-6B83-4A2F-981E-C085311892FB@media.mit.edu>
	<1cc8deb80901010531mefc0f88t123b359c57a5147b@mail.gmail.com>
Message-ID: <402A3877-6C95-44F3-9CA6-FF1B80DDAA79@media.mit.edu>

pushed.  -Ian

On Jan 1, 2009, at 8:31 AM, quan hu wrote:

> Hi Ian,
>
>      For the pathname fix, I attached the patch.
>
>      For the 0.9.1 upgrade fix,  I got one failure and one  
> successful result.
>      I need to double check if the failure is caused by my test  
> execution step or other issue.
>      I'll send the result later.
>
> Thanks
> Quan
>
>
> 2008/12/31 Ian Eslick <eslick at media.mit.edu>
> I don't think we've tried the upgrade procedure in awhile, so it's
> possible an incompatibility slipped in.  It may be that all you need
> to do is make sure that defaults are properly created for slot values
> that are unbound during the store controller opening phase.
>
> Adding a case for pathname is easy enough.  I take it one of the slots
> of the pathname structure object is a function; migrating this is
> causing the error.
>
> With a little work you should be able to do both of these fixes,
> certainly the pathname fix.  I'd be happy to review and commit them to
> the repository, but unfortunately I won't have any more time to work
> on elephant for a little while.
>
> Ian
>
> On Dec 31, 2008, at 8:37 AM, quan hu wrote:
>
> > Hi Ian,
> >
> >      I downloaded the patches and have more testing today. The lock
> > issue is gone!
> >      Thanks a lot.
> >
> >      1. But, another issue is found. migrate pathname fails. I test
> > it under SBCL 1.0.21
> >       e.g.
> >        >  (migrate *store-controller* #p"/tmp) => Error: Cannot
> > migrate function objects (i.e. #<FUNCTION SB-IMPL::UNPARSE-UNIX-
> > ENOUGH>)
> >
> >       It is caused by that (defmethod migrate ((dst store-
> > controller) (src structure-object)  is called for this case, as
> > pathname is taken as subtype of structure-object.
> >       > (subtypep 'pathname 'structure-object) => t t
> >       Maybe, one more migrate method is needed for pathname case.
> >
> >       2. I also want to know if upgrading database file from 0.9.1
> > to unstable release is supported.
> >       When opening the DB file generated in 0.9.1 in unstable
> > release, I get error message
> >        " The slot DB-BDB::INDICES is unbound in the object #<BDB-
> > INDEXED-BTREE oid:-3>.
> >           [Condition of type UNBOUND-SLOT]"
> >
> > Thanks
> > Quan
> >
> >
> >
> >
> > 2008/12/31 Ian Eslick <eslick at media.mit.edu>
> > Hi Quan,
> >
> > Thank you again for the information on this bug.  The short story is
> > that the way we are keeping track of transactions assumed that there
> > were only two stores in the nested transaction stack (outer and
> > inner).  Migrating a nested object inside a with-transaction would  
> not
> > pass the parent transaction to the backend so it wouldn't know a
> > transaction in progress and attempted to start a new one, which  
> would
> > block until the old one was completed.
> >
> > I just checked in a patch that made transaction tracking a bit more
> > intelligent (it depends on the backends taking care of handling  
> nested
> > transactions - which I believe they all do fine).  I also checked  
> in a
> > test which implements Quan's test case.
> >
> > This patch and the prior patches of the past few days pass all tests
> > on SBCL/Mac/BDB.
> >
> > Ian
> >
> > PS - I'd love to know the current status of the elephant-unstable  
> tree
> > with the postmodern backend.
> >
> >
> > On Dec 30, 2008, at 9:34 AM, quan hu wrote:
> >
> > > Hello,
> > >
> > >       After more investigation, here are some suspects:
> > >
> > >       In the current elephant codes, the macro my-current-
> > > transaction will check if the current transaction belongs to the  
> sc
> > > parameter.   If not, just return +NULL-CHAR+. That's exactly the
> > > case in the following nested transaction where the store- 
> controller
> > > is different.
> > >       The situation becomes:
> > >              Transaction 1 begins
> > >                     One data is put into DB with Transaction 1
> > >                     ...
> > >                     Another data is put into the same DB with NULL
> > > transaction parameter(i.e. no transaction??)
> > >
> > >        It looks like this scenario is not supported by BDB.
> > >        I simulate above case with a C program to access BDB and  
> get
> > > the similar failure symptom.
> > > ...
> > >   txn1 = NULL;
> > >   ret = dbenvp1->txn_begin(dbenvp1, NULL, &txn1, 0);
> > >   if (ret != 0) {
> > >     printf ("Transaction begin failed.\n");
> > >     return 1;
> > >   }
> > >   put_record(FOO,FOO,txn1, dbp1);
> > >   printf("foo\n");
> > >   put_record(FOO,FOO,NULL, dbp1);
> > >   printf("bar\n");
> > >   txn1->commit(txn1,0);
> > >   printf("baz\n");
> > > ...
> > >  After "foo" is printed, the C program is blocked.  If I change  
> the
> > > NULL to txn1, everything is fine.
> > >
> > >
> > >  Regarding to the solution, my gut feeling is that it would be
> > > difficult to avoid this kind of nested transaction in migrate
> > > process, because of the recursive function call. Maybe,
> > > *transaction-stack*  can be added to track the active transaction
> > > stack. Get it pushed and poped in right place of execute-
> > > transaction. Then, in my-current-transaction, check *transaction-
> > > stack* to see if there is an active transaction belongs to the  
> same
> > > store-controller. If it is found, use it.   My simple test shows  
> it
> > > works. If this is the right direction, I can do more test.
> > >
> > > Thanks for your help.
> > > Quan
> > >
> > >
> > >
> > >
> > >
> > > 2008/12/30 quan hu <ihuquan at gmail.com>
> > > Hi Ian, Yarek,
> > >
> > >     Thanks for providing the information.
> > >      I downloaded the elephant-unstable and test it with BDB4.7.
> > >      The failure symptom is the same: target store controller get
> > > locked.
> > >
> > >      After I make following changes in migrate.lisp; The test
> > passed.
> > >
> > >       (defmethod copy-btree-contents ((sc store-controller) dst  
> src)
> > >             (let (to-be-migrated)
> > >                   (map-btree (lambda (key value)
> > >                                      (let ((newval (migrate sc
> > value))
> > >                                             (newkey (migrate sc
> > key)))
> > >                                            (push (list newkey
> > > newval) to-be-migrated)))
> > >                                            ;;;;    (setf (get- 
> value
> > > newkey dst) newval)))  ;;
> > >                                            src)
> > >                   (loop for (k v) in to-be-migrated
> > >                     do
> > >                     (setf (get-value k dst) v))))
> > >
> > >      Above debug change just avoids the nested transaction of
> > > different store controller for the specific test case.
> > >      However, it is not the solution, as the nested transaction
> > > still could appear in other data pattern.
> > >
> > >     What confuses me is that the nested transaction of the same
> > > store controller seem not bring any block.
> > >      Following test passed:
> > >         (with-transaction (:store-controller *migrate-dst*)
> > >
> > >              (add-to-root 'foo 'foo :sc *migrate-dst*)
> > >              (with-transaction (:store-controller *migrate-dst*)
> > >
> > >                 (add-to-root 'bar 'bar :sc *migrate-dst*)))
> > >
> > >     But, the nested transaction of different store controller  
> always
> > > results in lock.
> > >
> > >         (with-transaction (:store-controller *migrate-dst*)
> > >
> > >              (add-to-root 'foo 'foo :sc *migrate-dst*)
> > >              (with-transaction (:store-controller *migrate-
> > > src*)         ;; trigger database lock
> > >                  (add-to-root 'bar 'bar :sc *migrate-dst*)))
> > >
> > > Thanks
> > > Quan
> > >
> > >
> > >
> > > 2008/12/30 Yarek Kowalik <yarek.kowalik at gmail.com>
> > >
> > > Quan,
> > >
> > > Unstable is here:
> > >
> > >  http://www.common-lisp.net/project/elephant/darcs/elephant-unstable
> > >
> > > Yarek
> > >
> > >
> > > On Mon, Dec 29, 2008 at 5:37 AM, Ian Eslick <eslick at media.mit.edu>
> > > wrote:
> > > That would be elephant-unstable
> > >
> > > Sent from my iPhone
> > >
> > > On Dec 29, 2008, at 7:36 AM, Ian Eslick <eslick at media.mit.edu>
> > wrote:
> > >
> > > > Hi Quan,
> > > >
> > > > Can you try the latest darcs with BDB 4.6 or 4.7?  I made some
> > > changes
> > > > to migration, but I can't recall if they were in the latest
> > darcs or
> > > > not - I think the latest darcs doesn't play well with BDB 4.5 so
> > > > please make sure you can reproduce under the above configuration
> > if
> > > > you can.
> > > >
> > > > Thank you,
> > > > Ian
> > > >
> > > > On Dec 29, 2008, at 3:40 AM, quan hu wrote:
> > > >
> > > >> Hello,
> > > >>
> > > >>     I run into a problem when doing the garbage collection via
> > data
> > > >> migration.
> > > >>     The environment is elephant 0.9.1 and BDB 4.5.  I also  
> tried
> > > >> the latest elephant in darc and get the same result.
> > > >>
> > > >>     1.  Test case to reproduce the problem.
> > > >>
> > > >>           (defpclass user-profile()
> > > >>             ((id :initform nil :index t)
> > > >>             (sms-inbox  :initform (make-pset))))
> > > >>
> > > >>            ;; Create a new db for test purpose
> > > >>           (setf *migrate-src* (open-store '(:bdb "/tmp/db/
> > src/")))
> > > >>           (setf *test-obj* (make-instance 'user-profile :sc
> > > >> *migrate-src*))
> > > >>            (insert-item 'foo (slot-value *test-obj* 'sms- 
> inbox))
> > > >>            (setf *migrate-dst* (open-store '(:bdb "/tmp/db/
> > dst/")))
> > > >>
> > > >>          TEST> (migrate *migrate-dst* *migrate-src*)
> > > >>              Migrating class indexes for: USER-PROFILE
> > > >>           ; => Input get blocked, not response anymore
> > > >>
> > > >>       2.  Using "db_stat -C A" get following output:
> > > >>             1    Lock requests not available due to conflicts,
> > for
> > > >> which we waited
> > > >>
> > > >>       3.  I do some debug work and found the problem may be
> > caused
> > > >> by nested transaction of different store controller.
> > > >>            I can reproduce the issue with following code  
> segment
> > > >>
> > > >>          TEST> (ensure-transaction (:store-controller *migrate-
> > > dst*)
> > > >>                            (add-to-root 'foo 'foo :sc *migrate-
> > > dst*)
> > > >>                            (ensure-transaction (:store- 
> controller
> > > >> *migrate-src*)
> > > >>                                 (add-to-root 'bar 'bar :sc
> > > *migrate-
> > > >> dst*)))
> > > >>
> > > >>             This kind of nested transaction scenario can appear
> > in
> > > >> the migrate process, because it uses source/destination store
> > > >> controller at the same time.
> > > >>
> > > >>            I do not understand why different store controller's
> > > >> transaction can result in the lock conflict and want to know
> > how to
> > > >> fix it.
> > > >>
> > > >>            Is there anyone meet the similar problem before?
> > > >>
> > > >> Thanks
> > > >> Quan
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> _______________________________________________
> > > >> elephant-devel site list
> > > >> elephant-devel at common-lisp.net
> > > >> http://common-lisp.net/mailman/listinfo/elephant-devel
> > > >
> > > >
> > > > _______________________________________________
> > > > elephant-devel site list
> > > > elephant-devel at common-lisp.net
> > > > http://common-lisp.net/mailman/listinfo/elephant-devel
> > >
> > > _______________________________________________
> > > elephant-devel site list
> > > elephant-devel at common-lisp.net
> > > http://common-lisp.net/mailman/listinfo/elephant-devel
> > >
> > >
> > > _______________________________________________
> > > elephant-devel site list
> > > elephant-devel at common-lisp.net
> > > http://common-lisp.net/mailman/listinfo/elephant-devel
> > >
> > >
> > > _______________________________________________
> > > elephant-devel site list
> > > elephant-devel at common-lisp.net
> > > http://common-lisp.net/mailman/listinfo/elephant-devel
> >
> >
> > _______________________________________________
> > elephant-devel site list
> > elephant-devel at common-lisp.net
> > http://common-lisp.net/mailman/listinfo/elephant-devel
> >
> > _______________________________________________
> > elephant-devel site list
> > elephant-devel at common-lisp.net
> > http://common-lisp.net/mailman/listinfo/elephant-devel
>
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel
>
> <migrate-pathname- 
> patch.gz>_______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel




From hugo_duncan at yahoo.com  Thu Jan  1 21:54:14 2009
From: hugo_duncan at yahoo.com (Hugo Duncan)
Date: Thu, 01 Jan 2009 16:54:14 -0500
Subject: [elephant-devel] postmodern btree problem
Message-ID: <op.um3lsoalpcoyut@cyclops>

Hi,

I have some test code that uses btree's as persistent slot values of a  
persistent class, initialised with an :initform (make-btree).  Running the  
code frequently, but not always, gives the error:

   "The slot DB-POSTMODERN::KEY-TYPE is unbound in the object  
#<DB-POSTMODERN::PM-BTREE db-table:NIL {12F27649}>."

in DB-POSTMODERN::INITIALIZED-P.

The error doesn't always happen at the same place - sometimes on btree  
read, sometimes on btree insert.  Against a BDB 4.7 backend, the code  
works reliably.  This is using elephant-unstable with a postgresql 8.3  
database on OS X.

I am having trouble reducing the code to a sensible test case that  
exhibits the same behaviour.  Any suggestions on how I can track this down?

Hugo
__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 




From killerstorm at newmail.ru  Fri Jan  2 11:44:10 2009
From: killerstorm at newmail.ru (Alex Mizrahi)
Date: Fri, 2 Jan 2009 13:44:10 +0200
Subject: [elephant-devel] postmodern btree problem
References: <op.um3lsoalpcoyut@cyclops>
Message-ID: <gjkumh$6bg$1@ger.gmane.org>

 HD> I am having trouble reducing the code to a sensible test case that
 HD> exhibits the same behaviour.  Any suggestions on how I can track this
 HD> down?

what kind of objects do you put in this btree? i think i've seen a problem 
with
upgrading btree object type, it might be related.

other questions:
 what's the structure of your code? do you just get slot's value and 
read/write
 to btree, or something else?
 do you have transactions that are aborted?
 do you have multiple threads? 






From ihuquan at gmail.com  Fri Jan  2 13:15:27 2009
From: ihuquan at gmail.com (quan hu)
Date: Fri, 2 Jan 2009 21:15:27 +0800
Subject: [elephant-devel] DB migration issue
In-Reply-To: <402A3877-6C95-44F3-9CA6-FF1B80DDAA79@media.mit.edu>
References: <1cc8deb80812290040k7720ca03hdbca03dbd8375135@mail.gmail.com>
	<49255C15-AE3C-4B83-A73C-940670519C51@media.mit.edu>
	<ecb948b30812291013y7dc5b01eka090e18d8c4e9fea@mail.gmail.com>
	<1cc8deb80812292147i5d39c76dq612c2d0b2e5dc70b@mail.gmail.com>
	<1cc8deb80812300634n57d81ca4iaf770dbc3c7b0dfd@mail.gmail.com>
	<F0B94D5B-6F33-4BC6-AEB4-A8352974AD93@media.mit.edu>
	<1cc8deb80812310537n7048dbd5w14f200d990dd2439@mail.gmail.com>
	<286EADCF-6B83-4A2F-981E-C085311892FB@media.mit.edu>
	<1cc8deb80901010531mefc0f88t123b359c57a5147b@mail.gmail.com>
	<402A3877-6C95-44F3-9CA6-FF1B80DDAA79@media.mit.edu>
Message-ID: <1cc8deb80901020515t3588d29fle9c461f644e7d1da@mail.gmail.com>

Hi Ian,

   The 0.9.1 upgrade fix patch is attached.
   I continuously run the upgrade test 3 times. All passed.

   In the fix,  if the DB version is <= 0.9.1, (make-instance
'bdb-indexed-btree :from-oid -3 :sc sc :indices (make-hash-table) is always
called, regardless of the indices exist before.
    In my previous fix, I only call it when the indices does not exist in DB
file, but find that approach does not work. A unserialized error is
triggered when try to open the source 0.9.1 DB file for the second time.
This is caused by that the oid->schema-id has an assumption on that
instance-table is only created on a brand new DB file.

    If this is not a suitable solution, please let me know.

Thanks
Quan





2009/1/2 Ian Eslick <eslick at media.mit.edu>

> pushed.  -Ian
>
> On Jan 1, 2009, at 8:31 AM, quan hu wrote:
>
> > Hi Ian,
> >
> >      For the pathname fix, I attached the patch.
> >
> >      For the 0.9.1 upgrade fix,  I got one failure and one
> > successful result.
> >      I need to double check if the failure is caused by my test
> > execution step or other issue.
> >      I'll send the result later.
> >
> > Thanks
> > Quan
> >
> >
> > 2008/12/31 Ian Eslick <eslick at media.mit.edu>
> > I don't think we've tried the upgrade procedure in awhile, so it's
> > possible an incompatibility slipped in.  It may be that all you need
> > to do is make sure that defaults are properly created for slot values
> > that are unbound during the store controller opening phase.
> >
> > Adding a case for pathname is easy enough.  I take it one of the slots
> > of the pathname structure object is a function; migrating this is
> > causing the error.
> >
> > With a little work you should be able to do both of these fixes,
> > certainly the pathname fix.  I'd be happy to review and commit them to
> > the repository, but unfortunately I won't have any more time to work
> > on elephant for a little while.
> >
> > Ian
> >
> > On Dec 31, 2008, at 8:37 AM, quan hu wrote:
> >
> > > Hi Ian,
> > >
> > >      I downloaded the patches and have more testing today. The lock
> > > issue is gone!
> > >      Thanks a lot.
> > >
> > >      1. But, another issue is found. migrate pathname fails. I test
> > > it under SBCL 1.0.21
> > >       e.g.
> > >        >  (migrate *store-controller* #p"/tmp) => Error: Cannot
> > > migrate function objects (i.e. #<FUNCTION SB-IMPL::UNPARSE-UNIX-
> > > ENOUGH>)
> > >
> > >       It is caused by that (defmethod migrate ((dst store-
> > > controller) (src structure-object)  is called for this case, as
> > > pathname is taken as subtype of structure-object.
> > >       > (subtypep 'pathname 'structure-object) => t t
> > >       Maybe, one more migrate method is needed for pathname case.
> > >
> > >       2. I also want to know if upgrading database file from 0.9.1
> > > to unstable release is supported.
> > >       When opening the DB file generated in 0.9.1 in unstable
> > > release, I get error message
> > >        " The slot DB-BDB::INDICES is unbound in the object #<BDB-
> > > INDEXED-BTREE oid:-3>.
> > >           [Condition of type UNBOUND-SLOT]"
> > >
> > > Thanks
> > > Quan
> > >
> > >
> > >
> > >
> > > 2008/12/31 Ian Eslick <eslick at media.mit.edu>
> > > Hi Quan,
> > >
> > > Thank you again for the information on this bug.  The short story is
> > > that the way we are keeping track of transactions assumed that there
> > > were only two stores in the nested transaction stack (outer and
> > > inner).  Migrating a nested object inside a with-transaction would
> > not
> > > pass the parent transaction to the backend so it wouldn't know a
> > > transaction in progress and attempted to start a new one, which
> > would
> > > block until the old one was completed.
> > >
> > > I just checked in a patch that made transaction tracking a bit more
> > > intelligent (it depends on the backends taking care of handling
> > nested
> > > transactions - which I believe they all do fine).  I also checked
> > in a
> > > test which implements Quan's test case.
> > >
> > > This patch and the prior patches of the past few days pass all tests
> > > on SBCL/Mac/BDB.
> > >
> > > Ian
> > >
> > > PS - I'd love to know the current status of the elephant-unstable
> > tree
> > > with the postmodern backend.
> > >
> > >
> > > On Dec 30, 2008, at 9:34 AM, quan hu wrote:
> > >
> > > > Hello,
> > > >
> > > >       After more investigation, here are some suspects:
> > > >
> > > >       In the current elephant codes, the macro my-current-
> > > > transaction will check if the current transaction belongs to the
> > sc
> > > > parameter.   If not, just return +NULL-CHAR+. That's exactly the
> > > > case in the following nested transaction where the store-
> > controller
> > > > is different.
> > > >       The situation becomes:
> > > >              Transaction 1 begins
> > > >                     One data is put into DB with Transaction 1
> > > >                     ...
> > > >                     Another data is put into the same DB with NULL
> > > > transaction parameter(i.e. no transaction??)
> > > >
> > > >        It looks like this scenario is not supported by BDB.
> > > >        I simulate above case with a C program to access BDB and
> > get
> > > > the similar failure symptom.
> > > > ...
> > > >   txn1 = NULL;
> > > >   ret = dbenvp1->txn_begin(dbenvp1, NULL, &txn1, 0);
> > > >   if (ret != 0) {
> > > >     printf ("Transaction begin failed.\n");
> > > >     return 1;
> > > >   }
> > > >   put_record(FOO,FOO,txn1, dbp1);
> > > >   printf("foo\n");
> > > >   put_record(FOO,FOO,NULL, dbp1);
> > > >   printf("bar\n");
> > > >   txn1->commit(txn1,0);
> > > >   printf("baz\n");
> > > > ...
> > > >  After "foo" is printed, the C program is blocked.  If I change
> > the
> > > > NULL to txn1, everything is fine.
> > > >
> > > >
> > > >  Regarding to the solution, my gut feeling is that it would be
> > > > difficult to avoid this kind of nested transaction in migrate
> > > > process, because of the recursive function call. Maybe,
> > > > *transaction-stack*  can be added to track the active transaction
> > > > stack. Get it pushed and poped in right place of execute-
> > > > transaction. Then, in my-current-transaction, check *transaction-
> > > > stack* to see if there is an active transaction belongs to the
> > same
> > > > store-controller. If it is found, use it.   My simple test shows
> > it
> > > > works. If this is the right direction, I can do more test.
> > > >
> > > > Thanks for your help.
> > > > Quan
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > 2008/12/30 quan hu <ihuquan at gmail.com>
> > > > Hi Ian, Yarek,
> > > >
> > > >     Thanks for providing the information.
> > > >      I downloaded the elephant-unstable and test it with BDB4.7.
> > > >      The failure symptom is the same: target store controller get
> > > > locked.
> > > >
> > > >      After I make following changes in migrate.lisp; The test
> > > passed.
> > > >
> > > >       (defmethod copy-btree-contents ((sc store-controller) dst
> > src)
> > > >             (let (to-be-migrated)
> > > >                   (map-btree (lambda (key value)
> > > >                                      (let ((newval (migrate sc
> > > value))
> > > >                                             (newkey (migrate sc
> > > key)))
> > > >                                            (push (list newkey
> > > > newval) to-be-migrated)))
> > > >                                            ;;;;    (setf (get-
> > value
> > > > newkey dst) newval)))  ;;
> > > >                                            src)
> > > >                   (loop for (k v) in to-be-migrated
> > > >                     do
> > > >                     (setf (get-value k dst) v))))
> > > >
> > > >      Above debug change just avoids the nested transaction of
> > > > different store controller for the specific test case.
> > > >      However, it is not the solution, as the nested transaction
> > > > still could appear in other data pattern.
> > > >
> > > >     What confuses me is that the nested transaction of the same
> > > > store controller seem not bring any block.
> > > >      Following test passed:
> > > >         (with-transaction (:store-controller *migrate-dst*)
> > > >
> > > >              (add-to-root 'foo 'foo :sc *migrate-dst*)
> > > >              (with-transaction (:store-controller *migrate-dst*)
> > > >
> > > >                 (add-to-root 'bar 'bar :sc *migrate-dst*)))
> > > >
> > > >     But, the nested transaction of different store controller
> > always
> > > > results in lock.
> > > >
> > > >         (with-transaction (:store-controller *migrate-dst*)
> > > >
> > > >              (add-to-root 'foo 'foo :sc *migrate-dst*)
> > > >              (with-transaction (:store-controller *migrate-
> > > > src*)         ;; trigger database lock
> > > >                  (add-to-root 'bar 'bar :sc *migrate-dst*)))
> > > >
> > > > Thanks
> > > > Quan
> > > >
> > > >
> > > >
> > > > 2008/12/30 Yarek Kowalik <yarek.kowalik at gmail.com>
> > > >
> > > > Quan,
> > > >
> > > > Unstable is here:
> > > >
> > > >  http://www.common-lisp.net/project/elephant/darcs/elephant-unstable
> > > >
> > > > Yarek
> > > >
> > > >
> > > > On Mon, Dec 29, 2008 at 5:37 AM, Ian Eslick <eslick at media.mit.edu>
> > > > wrote:
> > > > That would be elephant-unstable
> > > >
> > > > Sent from my iPhone
> > > >
> > > > On Dec 29, 2008, at 7:36 AM, Ian Eslick <eslick at media.mit.edu>
> > > wrote:
> > > >
> > > > > Hi Quan,
> > > > >
> > > > > Can you try the latest darcs with BDB 4.6 or 4.7?  I made some
> > > > changes
> > > > > to migration, but I can't recall if they were in the latest
> > > darcs or
> > > > > not - I think the latest darcs doesn't play well with BDB 4.5 so
> > > > > please make sure you can reproduce under the above configuration
> > > if
> > > > > you can.
> > > > >
> > > > > Thank you,
> > > > > Ian
> > > > >
> > > > > On Dec 29, 2008, at 3:40 AM, quan hu wrote:
> > > > >
> > > > >> Hello,
> > > > >>
> > > > >>     I run into a problem when doing the garbage collection via
> > > data
> > > > >> migration.
> > > > >>     The environment is elephant 0.9.1 and BDB 4.5.  I also
> > tried
> > > > >> the latest elephant in darc and get the same result.
> > > > >>
> > > > >>     1.  Test case to reproduce the problem.
> > > > >>
> > > > >>           (defpclass user-profile()
> > > > >>             ((id :initform nil :index t)
> > > > >>             (sms-inbox  :initform (make-pset))))
> > > > >>
> > > > >>            ;; Create a new db for test purpose
> > > > >>           (setf *migrate-src* (open-store '(:bdb "/tmp/db/
> > > src/")))
> > > > >>           (setf *test-obj* (make-instance 'user-profile :sc
> > > > >> *migrate-src*))
> > > > >>            (insert-item 'foo (slot-value *test-obj* 'sms-
> > inbox))
> > > > >>            (setf *migrate-dst* (open-store '(:bdb "/tmp/db/
> > > dst/")))
> > > > >>
> > > > >>          TEST> (migrate *migrate-dst* *migrate-src*)
> > > > >>              Migrating class indexes for: USER-PROFILE
> > > > >>           ; => Input get blocked, not response anymore
> > > > >>
> > > > >>       2.  Using "db_stat -C A" get following output:
> > > > >>             1    Lock requests not available due to conflicts,
> > > for
> > > > >> which we waited
> > > > >>
> > > > >>       3.  I do some debug work and found the problem may be
> > > caused
> > > > >> by nested transaction of different store controller.
> > > > >>            I can reproduce the issue with following code
> > segment
> > > > >>
> > > > >>          TEST> (ensure-transaction (:store-controller *migrate-
> > > > dst*)
> > > > >>                            (add-to-root 'foo 'foo :sc *migrate-
> > > > dst*)
> > > > >>                            (ensure-transaction (:store-
> > controller
> > > > >> *migrate-src*)
> > > > >>                                 (add-to-root 'bar 'bar :sc
> > > > *migrate-
> > > > >> dst*)))
> > > > >>
> > > > >>             This kind of nested transaction scenario can appear
> > > in
> > > > >> the migrate process, because it uses source/destination store
> > > > >> controller at the same time.
> > > > >>
> > > > >>            I do not understand why different store controller's
> > > > >> transaction can result in the lock conflict and want to know
> > > how to
> > > > >> fix it.
> > > > >>
> > > > >>            Is there anyone meet the similar problem before?
> > > > >>
> > > > >> Thanks
> > > > >> Quan
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >> _______________________________________________
> > > > >> elephant-devel site list
> > > > >> elephant-devel at common-lisp.net
> > > > >> http://common-lisp.net/mailman/listinfo/elephant-devel
> > > > >
> > > > >
> > > > > _______________________________________________
> > > > > elephant-devel site list
> > > > > elephant-devel at common-lisp.net
> > > > > http://common-lisp.net/mailman/listinfo/elephant-devel
> > > >
> > > > _______________________________________________
> > > > elephant-devel site list
> > > > elephant-devel at common-lisp.net
> > > > http://common-lisp.net/mailman/listinfo/elephant-devel
> > > >
> > > >
> > > > _______________________________________________
> > > > elephant-devel site list
> > > > elephant-devel at common-lisp.net
> > > > http://common-lisp.net/mailman/listinfo/elephant-devel
> > > >
> > > >
> > > > _______________________________________________
> > > > elephant-devel site list
> > > > elephant-devel at common-lisp.net
> > > > http://common-lisp.net/mailman/listinfo/elephant-devel
> > >
> > >
> > > _______________________________________________
> > > elephant-devel site list
> > > elephant-devel at common-lisp.net
> > > http://common-lisp.net/mailman/listinfo/elephant-devel
> > >
> > > _______________________________________________
> > > elephant-devel site list
> > > elephant-devel at common-lisp.net
> > > http://common-lisp.net/mailman/listinfo/elephant-devel
> >
> >
> > _______________________________________________
> > elephant-devel site list
> > elephant-devel at common-lisp.net
> > http://common-lisp.net/mailman/listinfo/elephant-devel
> >
> > <migrate-pathname-
> > patch.gz>_______________________________________________
> > elephant-devel site list
> > elephant-devel at common-lisp.net
> > http://common-lisp.net/mailman/listinfo/elephant-devel
>
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20090102/82bd0736/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: upgrade-0.9.1-fix-patch
Type: application/octet-stream
Size: 20599 bytes
Desc: not available
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20090102/82bd0736/attachment.obj>

From eslick at media.mit.edu  Fri Jan  2 16:00:03 2009
From: eslick at media.mit.edu (Ian Eslick)
Date: Fri, 2 Jan 2009 11:00:03 -0500
Subject: [elephant-devel] DB migration issue
In-Reply-To: <1cc8deb80901020515t3588d29fle9c461f644e7d1da@mail.gmail.com>
References: <1cc8deb80812290040k7720ca03hdbca03dbd8375135@mail.gmail.com>
	<49255C15-AE3C-4B83-A73C-940670519C51@media.mit.edu>
	<ecb948b30812291013y7dc5b01eka090e18d8c4e9fea@mail.gmail.com>
	<1cc8deb80812292147i5d39c76dq612c2d0b2e5dc70b@mail.gmail.com>
	<1cc8deb80812300634n57d81ca4iaf770dbc3c7b0dfd@mail.gmail.com>
	<F0B94D5B-6F33-4BC6-AEB4-A8352974AD93@media.mit.edu>
	<1cc8deb80812310537n7048dbd5w14f200d990dd2439@mail.gmail.com>
	<286EADCF-6B83-4A2F-981E-C085311892FB@media.mit.edu>
	<1cc8deb80901010531mefc0f88t123b359c57a5147b@mail.gmail.com>
	<402A3877-6C95-44F3-9CA6-FF1B80DDAA79@media.mit.edu>
	<1cc8deb80901020515t3588d29fle9c461f644e7d1da@mail.gmail.com>
Message-ID: <E60BA029-74AF-4591-B677-425CFFCEFD74@media.mit.edu>

This looks good.  I pushed it to elephant-unstable.

I wonder if the same problem will occur with the other backends and  
whether there is a common patch we might apply?  Has anyone tried  
upgrading from 0.9.1 or an earlier release to elephant-unstable?

Thanks,
Ian

On Jan 2, 2009, at 8:15 AM, quan hu wrote:

> Hi Ian,
>
>    The 0.9.1 upgrade fix patch is attached.
>    I continuously run the upgrade test 3 times. All passed.
>
>    In the fix,  if the DB version is <= 0.9.1, (make-instance 'bdb- 
> indexed-btree :from-oid -3 :sc sc :indices (make-hash-table) is  
> always called, regardless of the indices exist before.
>     In my previous fix, I only call it when the indices does not  
> exist in DB file, but find that approach does not work. A  
> unserialized error is triggered when try to open the source 0.9.1 DB  
> file for the second time. This is caused by that the oid->schema-id  
> has an assumption on that instance-table is only created on a brand  
> new DB file.
>
>     If this is not a suitable solution, please let me know.
>
> Thanks
> Quan
>
>
>
>
>
> 2009/1/2 Ian Eslick <eslick at media.mit.edu>
> pushed.  -Ian
>
> On Jan 1, 2009, at 8:31 AM, quan hu wrote:
>
> > Hi Ian,
> >
> >      For the pathname fix, I attached the patch.
> >
> >      For the 0.9.1 upgrade fix,  I got one failure and one
> > successful result.
> >      I need to double check if the failure is caused by my test
> > execution step or other issue.
> >      I'll send the result later.
> >
> > Thanks
> > Quan
> >
> >
> > 2008/12/31 Ian Eslick <eslick at media.mit.edu>
> > I don't think we've tried the upgrade procedure in awhile, so it's
> > possible an incompatibility slipped in.  It may be that all you need
> > to do is make sure that defaults are properly created for slot  
> values
> > that are unbound during the store controller opening phase.
> >
> > Adding a case for pathname is easy enough.  I take it one of the  
> slots
> > of the pathname structure object is a function; migrating this is
> > causing the error.
> >
> > With a little work you should be able to do both of these fixes,
> > certainly the pathname fix.  I'd be happy to review and commit  
> them to
> > the repository, but unfortunately I won't have any more time to work
> > on elephant for a little while.
> >
> > Ian
> >
> > On Dec 31, 2008, at 8:37 AM, quan hu wrote:
> >
> > > Hi Ian,
> > >
> > >      I downloaded the patches and have more testing today. The  
> lock
> > > issue is gone!
> > >      Thanks a lot.
> > >
> > >      1. But, another issue is found. migrate pathname fails. I  
> test
> > > it under SBCL 1.0.21
> > >       e.g.
> > >        >  (migrate *store-controller* #p"/tmp) => Error: Cannot
> > > migrate function objects (i.e. #<FUNCTION SB-IMPL::UNPARSE-UNIX-
> > > ENOUGH>)
> > >
> > >       It is caused by that (defmethod migrate ((dst store-
> > > controller) (src structure-object)  is called for this case, as
> > > pathname is taken as subtype of structure-object.
> > >       > (subtypep 'pathname 'structure-object) => t t
> > >       Maybe, one more migrate method is needed for pathname case.
> > >
> > >       2. I also want to know if upgrading database file from 0.9.1
> > > to unstable release is supported.
> > >       When opening the DB file generated in 0.9.1 in unstable
> > > release, I get error message
> > >        " The slot DB-BDB::INDICES is unbound in the object #<BDB-
> > > INDEXED-BTREE oid:-3>.
> > >           [Condition of type UNBOUND-SLOT]"
> > >
> > > Thanks
> > > Quan
> > >
> > >
> > >
> > >
> > > 2008/12/31 Ian Eslick <eslick at media.mit.edu>
> > > Hi Quan,
> > >
> > > Thank you again for the information on this bug.  The short  
> story is
> > > that the way we are keeping track of transactions assumed that  
> there
> > > were only two stores in the nested transaction stack (outer and
> > > inner).  Migrating a nested object inside a with-transaction would
> > not
> > > pass the parent transaction to the backend so it wouldn't know a
> > > transaction in progress and attempted to start a new one, which
> > would
> > > block until the old one was completed.
> > >
> > > I just checked in a patch that made transaction tracking a bit  
> more
> > > intelligent (it depends on the backends taking care of handling
> > nested
> > > transactions - which I believe they all do fine).  I also checked
> > in a
> > > test which implements Quan's test case.
> > >
> > > This patch and the prior patches of the past few days pass all  
> tests
> > > on SBCL/Mac/BDB.
> > >
> > > Ian
> > >
> > > PS - I'd love to know the current status of the elephant-unstable
> > tree
> > > with the postmodern backend.
> > >
> > >
> > > On Dec 30, 2008, at 9:34 AM, quan hu wrote:
> > >
> > > > Hello,
> > > >
> > > >       After more investigation, here are some suspects:
> > > >
> > > >       In the current elephant codes, the macro my-current-
> > > > transaction will check if the current transaction belongs to the
> > sc
> > > > parameter.   If not, just return +NULL-CHAR+. That's exactly the
> > > > case in the following nested transaction where the store-
> > controller
> > > > is different.
> > > >       The situation becomes:
> > > >              Transaction 1 begins
> > > >                     One data is put into DB with Transaction 1
> > > >                     ...
> > > >                     Another data is put into the same DB with  
> NULL
> > > > transaction parameter(i.e. no transaction??)
> > > >
> > > >        It looks like this scenario is not supported by BDB.
> > > >        I simulate above case with a C program to access BDB and
> > get
> > > > the similar failure symptom.
> > > > ...
> > > >   txn1 = NULL;
> > > >   ret = dbenvp1->txn_begin(dbenvp1, NULL, &txn1, 0);
> > > >   if (ret != 0) {
> > > >     printf ("Transaction begin failed.\n");
> > > >     return 1;
> > > >   }
> > > >   put_record(FOO,FOO,txn1, dbp1);
> > > >   printf("foo\n");
> > > >   put_record(FOO,FOO,NULL, dbp1);
> > > >   printf("bar\n");
> > > >   txn1->commit(txn1,0);
> > > >   printf("baz\n");
> > > > ...
> > > >  After "foo" is printed, the C program is blocked.  If I change
> > the
> > > > NULL to txn1, everything is fine.
> > > >
> > > >
> > > >  Regarding to the solution, my gut feeling is that it would be
> > > > difficult to avoid this kind of nested transaction in migrate
> > > > process, because of the recursive function call. Maybe,
> > > > *transaction-stack*  can be added to track the active  
> transaction
> > > > stack. Get it pushed and poped in right place of execute-
> > > > transaction. Then, in my-current-transaction, check  
> *transaction-
> > > > stack* to see if there is an active transaction belongs to the
> > same
> > > > store-controller. If it is found, use it.   My simple test shows
> > it
> > > > works. If this is the right direction, I can do more test.
> > > >
> > > > Thanks for your help.
> > > > Quan
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > 2008/12/30 quan hu <ihuquan at gmail.com>
> > > > Hi Ian, Yarek,
> > > >
> > > >     Thanks for providing the information.
> > > >      I downloaded the elephant-unstable and test it with BDB4.7.
> > > >      The failure symptom is the same: target store controller  
> get
> > > > locked.
> > > >
> > > >      After I make following changes in migrate.lisp; The test
> > > passed.
> > > >
> > > >       (defmethod copy-btree-contents ((sc store-controller) dst
> > src)
> > > >             (let (to-be-migrated)
> > > >                   (map-btree (lambda (key value)
> > > >                                      (let ((newval (migrate sc
> > > value))
> > > >                                             (newkey (migrate sc
> > > key)))
> > > >                                            (push (list newkey
> > > > newval) to-be-migrated)))
> > > >                                            ;;;;    (setf (get-
> > value
> > > > newkey dst) newval)))  ;;
> > > >                                            src)
> > > >                   (loop for (k v) in to-be-migrated
> > > >                     do
> > > >                     (setf (get-value k dst) v))))
> > > >
> > > >      Above debug change just avoids the nested transaction of
> > > > different store controller for the specific test case.
> > > >      However, it is not the solution, as the nested transaction
> > > > still could appear in other data pattern.
> > > >
> > > >     What confuses me is that the nested transaction of the same
> > > > store controller seem not bring any block.
> > > >      Following test passed:
> > > >         (with-transaction (:store-controller *migrate-dst*)
> > > >
> > > >              (add-to-root 'foo 'foo :sc *migrate-dst*)
> > > >              (with-transaction (:store-controller *migrate-dst*)
> > > >
> > > >                 (add-to-root 'bar 'bar :sc *migrate-dst*)))
> > > >
> > > >     But, the nested transaction of different store controller
> > always
> > > > results in lock.
> > > >
> > > >         (with-transaction (:store-controller *migrate-dst*)
> > > >
> > > >              (add-to-root 'foo 'foo :sc *migrate-dst*)
> > > >              (with-transaction (:store-controller *migrate-
> > > > src*)         ;; trigger database lock
> > > >                  (add-to-root 'bar 'bar :sc *migrate-dst*)))
> > > >
> > > > Thanks
> > > > Quan
> > > >
> > > >
> > > >
> > > > 2008/12/30 Yarek Kowalik <yarek.kowalik at gmail.com>
> > > >
> > > > Quan,
> > > >
> > > > Unstable is here:
> > > >
> > > >  http://www.common-lisp.net/project/elephant/darcs/elephant-unstable
> > > >
> > > > Yarek
> > > >
> > > >
> > > > On Mon, Dec 29, 2008 at 5:37 AM, Ian Eslick <eslick at media.mit.edu 
> >
> > > > wrote:
> > > > That would be elephant-unstable
> > > >
> > > > Sent from my iPhone
> > > >
> > > > On Dec 29, 2008, at 7:36 AM, Ian Eslick <eslick at media.mit.edu>
> > > wrote:
> > > >
> > > > > Hi Quan,
> > > > >
> > > > > Can you try the latest darcs with BDB 4.6 or 4.7?  I made some
> > > > changes
> > > > > to migration, but I can't recall if they were in the latest
> > > darcs or
> > > > > not - I think the latest darcs doesn't play well with BDB  
> 4.5 so
> > > > > please make sure you can reproduce under the above  
> configuration
> > > if
> > > > > you can.
> > > > >
> > > > > Thank you,
> > > > > Ian
> > > > >
> > > > > On Dec 29, 2008, at 3:40 AM, quan hu wrote:
> > > > >
> > > > >> Hello,
> > > > >>
> > > > >>     I run into a problem when doing the garbage collection  
> via
> > > data
> > > > >> migration.
> > > > >>     The environment is elephant 0.9.1 and BDB 4.5.  I also
> > tried
> > > > >> the latest elephant in darc and get the same result.
> > > > >>
> > > > >>     1.  Test case to reproduce the problem.
> > > > >>
> > > > >>           (defpclass user-profile()
> > > > >>             ((id :initform nil :index t)
> > > > >>             (sms-inbox  :initform (make-pset))))
> > > > >>
> > > > >>            ;; Create a new db for test purpose
> > > > >>           (setf *migrate-src* (open-store '(:bdb "/tmp/db/
> > > src/")))
> > > > >>           (setf *test-obj* (make-instance 'user-profile :sc
> > > > >> *migrate-src*))
> > > > >>            (insert-item 'foo (slot-value *test-obj* 'sms-
> > inbox))
> > > > >>            (setf *migrate-dst* (open-store '(:bdb "/tmp/db/
> > > dst/")))
> > > > >>
> > > > >>          TEST> (migrate *migrate-dst* *migrate-src*)
> > > > >>              Migrating class indexes for: USER-PROFILE
> > > > >>           ; => Input get blocked, not response anymore
> > > > >>
> > > > >>       2.  Using "db_stat -C A" get following output:
> > > > >>             1    Lock requests not available due to  
> conflicts,
> > > for
> > > > >> which we waited
> > > > >>
> > > > >>       3.  I do some debug work and found the problem may be
> > > caused
> > > > >> by nested transaction of different store controller.
> > > > >>            I can reproduce the issue with following code
> > segment
> > > > >>
> > > > >>          TEST> (ensure-transaction (:store-controller  
> *migrate-
> > > > dst*)
> > > > >>                            (add-to-root 'foo 'foo :sc  
> *migrate-
> > > > dst*)
> > > > >>                            (ensure-transaction (:store-
> > controller
> > > > >> *migrate-src*)
> > > > >>                                 (add-to-root 'bar 'bar :sc
> > > > *migrate-
> > > > >> dst*)))
> > > > >>
> > > > >>             This kind of nested transaction scenario can  
> appear
> > > in
> > > > >> the migrate process, because it uses source/destination store
> > > > >> controller at the same time.
> > > > >>
> > > > >>            I do not understand why different store  
> controller's
> > > > >> transaction can result in the lock conflict and want to know
> > > how to
> > > > >> fix it.
> > > > >>
> > > > >>            Is there anyone meet the similar problem before?
> > > > >>
> > > > >> Thanks
> > > > >> Quan
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >> _______________________________________________
> > > > >> elephant-devel site list
> > > > >> elephant-devel at common-lisp.net
> > > > >> http://common-lisp.net/mailman/listinfo/elephant-devel
> > > > >
> > > > >
> > > > > _______________________________________________
> > > > > elephant-devel site list
> > > > > elephant-devel at common-lisp.net
> > > > > http://common-lisp.net/mailman/listinfo/elephant-devel
> > > >
> > > > _______________________________________________
> > > > elephant-devel site list
> > > > elephant-devel at common-lisp.net
> > > > http://common-lisp.net/mailman/listinfo/elephant-devel
> > > >
> > > >
> > > > _______________________________________________
> > > > elephant-devel site list
> > > > elephant-devel at common-lisp.net
> > > > http://common-lisp.net/mailman/listinfo/elephant-devel
> > > >
> > > >
> > > > _______________________________________________
> > > > elephant-devel site list
> > > > elephant-devel at common-lisp.net
> > > > http://common-lisp.net/mailman/listinfo/elephant-devel
> > >
> > >
> > > _______________________________________________
> > > elephant-devel site list
> > > elephant-devel at common-lisp.net
> > > http://common-lisp.net/mailman/listinfo/elephant-devel
> > >
> > > _______________________________________________
> > > elephant-devel site list
> > > elephant-devel at common-lisp.net
> > > http://common-lisp.net/mailman/listinfo/elephant-devel
> >
> >
> > _______________________________________________
> > elephant-devel site list
> > elephant-devel at common-lisp.net
> > http://common-lisp.net/mailman/listinfo/elephant-devel
> >
> > <migrate-pathname-
> > patch.gz>_______________________________________________
> > elephant-devel site list
> > elephant-devel at common-lisp.net
> > http://common-lisp.net/mailman/listinfo/elephant-devel
>
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel
>
> <upgrade-0.9.1-fix- 
> patch>_______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel




From hugo_duncan at yahoo.com  Fri Jan  2 18:52:34 2009
From: hugo_duncan at yahoo.com (Hugo Duncan)
Date: Fri, 02 Jan 2009 13:52:34 -0500
Subject: [elephant-devel] postmodern btree problem
In-Reply-To: <gjkumh$6bg$1@ger.gmane.org>
References: <op.um3lsoalpcoyut@cyclops> <gjkumh$6bg$1@ger.gmane.org>
Message-ID: <op.um471w2ppcoyut@cyclops>


On Fri, 02 Jan 2009, Alex Mizrahi wrote:
>  HD> I am having trouble reducing the code to a sensible test case that
>  HD> exhibits the same behaviour.  Any suggestions on how I can track  
> this
>  HD> down?

> what kind of objects do you put in this btree? i think i've seen a  
> problem with upgrading btree object type, it might be related.

I am storing objects of a single persistant class, with 7 slots containing  
integers or floats, into the btree.  The keys are integers.  Changing the  
class to be a standard class does not cange the problem.

>  what's the structure of your code? do you just get slot's value and  
> read/write
>  to btree, or something else?

The btree is created on object creation, and only has key/values inserted  
and read.

>  do you have transactions that are aborted?

No, I have a single transaction (WITH-TRANSACTION) around my test case,  
and I start with an empty (freshly created) database.  I notice however,  
that a EXECUTE-TRANSACTION is being run for each persistant object I  
create within the WITH-TRANSACTION body.  ELEPHANT:*CURRENT-TRANSACTION*  
doesn't seem to be bound, so the ENSURE-TRANSACTION wrapping the object  
creation is calling EXECUTE-TRANSACTION.  Attached is a small test case  
and its output when I run it.

I'm not sure that is causing my problems though, as other code exhibits  
the same behaviour and runs OK.

>  do you have multiple threads?

No, single thread.

Thank you, Alex, for taking the time to help with this.

Hugo
-------------- next part --------------
A non-text attachment was scrubbed...
Name: elephant-transaction.lisp
Type: application/octet-stream
Size: 668 bytes
Desc: not available
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20090102/10eb8d92/attachment.obj>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: elephant-transaction.txt
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20090102/10eb8d92/attachment.txt>

From yarek.kowalik at gmail.com  Fri Jan  2 19:17:53 2009
From: yarek.kowalik at gmail.com (Yarek Kowalik)
Date: Fri, 2 Jan 2009 11:17:53 -0800
Subject: [elephant-devel] Choice of back-end store
Message-ID: <ecb948b30901021117t4733cbf0m83306c66a863c36d@mail.gmail.com>

Could someone point me to any discussions WRT to choice of back-end?  I'm
looking at BDB and Postmodern.   What are advantages/disadvantages of either
one?

I'm guessing that Postmodern offers an opportunity of using and SQL querying
against PostgreSQL backend for out-of-process querying and such (does
Elephant model make this really possible/practical?), whereas BDB is perhaps
faster and you can build/use BDB replication tools.

Thanks,

Yarek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20090102/077e96ff/attachment.html>

From killerstorm at newmail.ru  Sat Jan  3 01:19:25 2009
From: killerstorm at newmail.ru (Alex Mizrahi)
Date: Sat, 3 Jan 2009 03:19:25 +0200
Subject: [elephant-devel] Choice of back-end store
References: <ecb948b30901021117t4733cbf0m83306c66a863c36d@mail.gmail.com>
Message-ID: <gjmef5$l64$1@ger.gmane.org>

> Could someone point me to any discussions WRT to choice of back-end?
>   I'm looking at BDB and Postmodern. What are advantages/disadvantages of 
> either one?

think like BDB is a baseline implementation, and Postmodern is an option for
special cases.

Postmodern is designed to work absolutely transparently with multiple 
threads,
processes and machines (and work absolutely transparently in general).
we think it is good for web sites -- it should scale pretty well if needed.

while this is also theoretically possible with BDB as well, as i understand
(i'm definitely not a BDB expert) it requires substantial amount of 
tweaking,
 or simply won't work in some cases..

the price we pay for flexibility is communication overhead -- each request
travels through process boundaries, and this takes time.

besides that, Elephant's semantics is define in BDB terms, so Postmodern
backend is sort of emulation, and it has substantial amount of limitations
and weirdnesses. thus, if not sure, use BDB :)

stores are more-or-less compatible, so you don't need to choose before
you start development -- you can start development with BDB, for example,
and then try out postmodern if you think it might suite you better.

> I'm guessing that Postmodern offers an opportunity of using and SQL 
> querying
> against PostgreSQL backend for out-of-process querying and such
> (does Elephant model make this really possible/practical?),

it is possible, but not practical. might be useful for debugging and
stuff like that.

> whereas BDB is perhaps faster

it depends how you measure. if you do lots of queries on small tables, than 
indeed BDB
will be much faster. if you do relatively small count of queries on larger 
tables, then
it depends..

> and you can build/use BDB replication tools.

yep, with BDB you can use BDB tools, with PostgreSQL you can use PostgreSQL 
tools :)
what fits you better depends on nature of your project and your background.

another significant difference is concurrency model -- in PostgreSQL it is
MVCC and it works almost transparently (except that conflicting transactions 
get restarted),
BDB's model is lock based (by default) and it works in quite nasty way -- if 
you touch too
many objects, you'll be out of locks. so, for example, if you want to 
serialize whole database,
with PostgreSQL it is trivial, but with BDB it is not. (you can enable MVCC 
in BDB too, but
i'm not sure it works as good as in PostgreSQL.) 






From killerstorm at newmail.ru  Sat Jan  3 02:48:44 2009
From: killerstorm at newmail.ru (Alex Mizrahi)
Date: Sat, 3 Jan 2009 04:48:44 +0200
Subject: [elephant-devel] postmodern btree problem
References: <op.um3lsoalpcoyut@cyclops> <gjkumh$6bg$1@ger.gmane.org>
	<op.um471w2ppcoyut@cyclops>
Message-ID: <gjmjmk$uuv$1@ger.gmane.org>

Looks like that is not one of "usual suspects".. And it is more than 
weird -- key-type slot in pm-btree
has initform, so it is unbound only when one explicitly makes it unbound, 
and I have no idea where
that could be (in theory, we do not do that at all). To debug this you can 
add checks like

(defun check-pm-btree-sane (bt)
  (and (db-postmodern::table-of bt) (slot-boundp bt 
db-postmodern::key-type)))

in places where you read btree from object slots and where you use btree, so 
at
least we'd know if it goes crippled straight from deserialization, or gets 
corrupted somewhere in process.

also it makes sense to investigate nature of this object -- get its oid 
(maybe just descibe it to see if
there is something unusal) and check if there exists table called (format 
nil "tree~a" oid) and
also a record in metatree for this table name.

also it might make sense to instrument function get-cached-instance a bit to 
know whether
crippled pm-btree is recreated or just retrieved from cache. something like:

(defmethod get-cached-instance ((sc store-controller) oid class-name)
  "Get a cached instance, or instantiate!"
  (declare (type store-controller sc)
    (type fixnum oid))
  (let ((obj
  (ele-with-fast-lock ((instance-cache-lock sc))
    (get-cache oid (instance-cache sc)))))

     ;; TRACE
    (when (equal class-name "DB-POSTMODERN::PM-BTREE")
      (if obj (format t "get-cached-instance pm-btree ~a from cache~%" oid)
                 (format t "get-cached-instance pm-btree ~a  recreated~%" 
oid)))

    (if obj obj
 ;; Should get cached since make-instance calls cache-instance
 (recreate-instance-using-class (find-class class-name) :from-oid oid :sc 
sc))))


 ??>>  do you have transactions that are aborted?

 HD> No, I have a single transaction (WITH-TRANSACTION) around my test case,
 HD> and I start with an empty (freshly created) database.  I notice
 HD> however,  that a EXECUTE-TRANSACTION is being run for each persistant
 HD> object I  create within the WITH-TRANSACTION body. 
ELEPHANT:*CURRENT-TRANSACTION*
 HD> doesn't seem to be bound, so the ENSURE-TRANSACTION wrapping the object
 HD> creation is calling EXECUTE-TRANSACTION.

that's ok, actually -- db-postmodern uses its own mechanism to track 
transactions,
so *current-transaction* won't be set, indeed. this ain't harmful and 
shouldn't add significant overhead.
(if you ask me, it is not really good, i'd say it does it this way for 
"historic" reasons) 






From eslick at media.mit.edu  Sat Jan  3 12:55:09 2009
From: eslick at media.mit.edu (Ian Eslick)
Date: Sat, 3 Jan 2009 07:55:09 -0500
Subject: [elephant-devel] Choice of back-end store
In-Reply-To: <ecb948b30901021117t4733cbf0m83306c66a863c36d@mail.gmail.com>
References: <ecb948b30901021117t4733cbf0m83306c66a863c36d@mail.gmail.com>
Message-ID: <D97C6466-8651-4A82-B333-5024C258F0B2@media.mit.edu>

Hi Yarek,

Others may add to this, but the big issues I'm aware of are:

BDB
+ fast
+ can be shared among multiple processes on a single machine
- commercial use requires license

Postmodern
- 5x slower than BDB (last I checked)
++ can be shared among multiple machines
+ open source

Neither system allows you to easily bypass the elephant  
representational framework.  I'll let others comment on this vis-a-vis  
postmodern.

You can use both system's tools for archive, recover, duplication, etc.

Ian

On Jan 2, 2009, at 2:17 PM, Yarek Kowalik wrote:

> Could someone point me to any discussions WRT to choice of back- 
> end?  I'm looking at BDB and Postmodern.   What are advantages/ 
> disadvantages of either one?
>
> I'm guessing that Postmodern offers an opportunity of using and SQL  
> querying against PostgreSQL backend for out-of-process querying and  
> such (does Elephant model make this really possible/practical?),  
> whereas BDB is perhaps faster and you can build/use BDB replication  
> tools.
>
> Thanks,
>
> Yarek
>
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel




From eslick at media.mit.edu  Sat Jan  3 12:59:17 2009
From: eslick at media.mit.edu (Ian Eslick)
Date: Sat, 3 Jan 2009 07:59:17 -0500
Subject: [elephant-devel] postmodern btree problem
In-Reply-To: <op.um471w2ppcoyut@cyclops>
References: <op.um3lsoalpcoyut@cyclops> <gjkumh$6bg$1@ger.gmane.org>
	<op.um471w2ppcoyut@cyclops>
Message-ID: <A66F984F-C292-4C32-B9EB-64E8F591CE41@media.mit.edu>

Do you have the latest version of elephant-unstable as of a day or two  
ago?  The *current-transaction* behavior you mention is suspicious and  
I recently made a change to that part of the code which may have  
inadvertently made a change that effected the postmodern policy.   
Ensure transaction simply calls its body if *current-transaction* is  
bound, otherwise it behaves like with-transaction.

Ian

On Jan 2, 2009, at 1:52 PM, Hugo Duncan wrote:

>
> On Fri, 02 Jan 2009, Alex Mizrahi wrote:
>> HD> I am having trouble reducing the code to a sensible test case  
>> that
>> HD> exhibits the same behaviour.  Any suggestions on how I can  
>> track this
>> HD> down?
>
>> what kind of objects do you put in this btree? i think i've seen a  
>> problem with upgrading btree object type, it might be related.
>
> I am storing objects of a single persistant class, with 7 slots  
> containing integers or floats, into the btree.  The keys are  
> integers.  Changing the class to be a standard class does not cange  
> the problem.
>
>> what's the structure of your code? do you just get slot's value and  
>> read/write
>> to btree, or something else?
>
> The btree is created on object creation, and only has key/values  
> inserted and read.
>
>> do you have transactions that are aborted?
>
> No, I have a single transaction (WITH-TRANSACTION) around my test  
> case, and I start with an empty (freshly created) database.  I  
> notice however, that a EXECUTE-TRANSACTION is being run for each  
> persistant object I create within the WITH-TRANSACTION body.   
> ELEPHANT:*CURRENT-TRANSACTION* doesn't seem to be bound, so the  
> ENSURE-TRANSACTION wrapping the object creation is calling EXECUTE- 
> TRANSACTION.  Attached is a small test case and its output when I  
> run it.
>
> I'm not sure that is causing my problems though, as other code  
> exhibits the same behaviour and runs OK.
>
>> do you have multiple threads?
>
> No, single thread.
>
> Thank you, Alex, for taking the time to help with this.
>
> Hugo<elephant-transaction.lisp><elephant- 
> transaction.txt>_______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel




From eslick at media.mit.edu  Sat Jan  3 13:27:04 2009
From: eslick at media.mit.edu (Ian Eslick)
Date: Sat, 3 Jan 2009 08:27:04 -0500
Subject: [elephant-devel] CLSQL Store
In-Reply-To: <gjmef5$l64$1@ger.gmane.org>
References: <ecb948b30901021117t4733cbf0m83306c66a863c36d@mail.gmail.com>
	<gjmef5$l64$1@ger.gmane.org>
Message-ID: <1F1F136C-D922-46F8-BAC3-B252D34BF8BC@media.mit.edu>

Is anyone still using the CL-SQL store?  Robert and I have discussed  
deprecating it in this or a future release, although SQLite support is  
the only way to use Elephant with a cheap, free, easy to install/ 
distribute backend.

RE: BDB locks - you can increase the lock count pretty easily in the  
my-config.sexp file.  Locks required are proportional to # processes *  
btree access depth within a given transaction.  I've used BDB under  
very heavy single-user loads (web mining) and moderate multi-user  
loads (website) without any locking issues.  Still, if you are bulk  
loading alot of data you can exhaust the transaction buffer since it's  
allocated in a shared region...perhaps MVCC would fix this.

Re: Licensing.  The BDB license allows you to use BDB for any purpose,  
but if you want to 'deploy' your a BDB-powered application in source  
or binary form you have to either get a commercial license or  
distribute your application as open source (such as Subversion).

For awhile it was unclear if powering a website counted as  
'distribution'.  My current understanding is that if you use BDB as  
the store for a website application then you do not need a commercial  
license nor make your application source available.  Right now the  
only easy deployment option for Elephant in a distributable  
application without a commercial license is using SQLite and the CL- 
SQL backend which is _very_ slow.  The long term solution for the  
single machine application scenario is a lisp-only backend.

Given that Berkeley DB is faster and is used mostly for websites, it  
would probably be worth it to explore using the distributed  
transaction support and MVCC capabilities built into BDB and making  
multi-machine use a turnkey option for elephant.

Any takers?  :)

Ian


On Jan 2, 2009, at 8:19 PM, Alex Mizrahi wrote:

>> Could someone point me to any discussions WRT to choice of back-end?
>>  I'm looking at BDB and Postmodern. What are advantages/ 
>> disadvantages of
>> either one?
>
> think like BDB is a baseline implementation, and Postmodern is an  
> option for
> special cases.
>
> Postmodern is designed to work absolutely transparently with multiple
> threads,
> processes and machines (and work absolutely transparently in general).
> we think it is good for web sites -- it should scale pretty well if  
> needed.
>
> while this is also theoretically possible with BDB as well, as i  
> understand
> (i'm definitely not a BDB expert) it requires substantial amount of
> tweaking,
> or simply won't work in some cases..
>
> the price we pay for flexibility is communication overhead -- each  
> request
> travels through process boundaries, and this takes time.
>
> besides that, Elephant's semantics is define in BDB terms, so  
> Postmodern
> backend is sort of emulation, and it has substantial amount of  
> limitations
> and weirdnesses. thus, if not sure, use BDB :)
>
> stores are more-or-less compatible, so you don't need to choose before
> you start development -- you can start development with BDB, for  
> example,
> and then try out postmodern if you think it might suite you better.
>
>> I'm guessing that Postmodern offers an opportunity of using and SQL
>> querying
>> against PostgreSQL backend for out-of-process querying and such
>> (does Elephant model make this really possible/practical?),
>
> it is possible, but not practical. might be useful for debugging and
> stuff like that.
>
>> whereas BDB is perhaps faster
>
> it depends how you measure. if you do lots of queries on small  
> tables, than
> indeed BDB
> will be much faster. if you do relatively small count of queries on  
> larger
> tables, then
> it depends..
>
>> and you can build/use BDB replication tools.
>
> yep, with BDB you can use BDB tools, with PostgreSQL you can use  
> PostgreSQL
> tools :)
> what fits you better depends on nature of your project and your  
> background.
>
> another significant difference is concurrency model -- in PostgreSQL  
> it is
> MVCC and it works almost transparently (except that conflicting  
> transactions
> get restarted),
> BDB's model is lock based (by default) and it works in quite nasty  
> way -- if
> you touch too
> many objects, you'll be out of locks. so, for example, if you want to
> serialize whole database,
> with PostgreSQL it is trivial, but with BDB it is not. (you can  
> enable MVCC
> in BDB too, but
> i'm not sure it works as good as in PostgreSQL.)
>
>
>
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel




From hugo_duncan at yahoo.com  Sat Jan  3 16:51:45 2009
From: hugo_duncan at yahoo.com (Hugo Duncan)
Date: Sat, 03 Jan 2009 11:51:45 -0500
Subject: [elephant-devel] postmodern btree problem
In-Reply-To: <A66F984F-C292-4C32-B9EB-64E8F591CE41@media.mit.edu>
References: <op.um3lsoalpcoyut@cyclops> <gjkumh$6bg$1@ger.gmane.org>
	<op.um471w2ppcoyut@cyclops>
	<A66F984F-C292-4C32-B9EB-64E8F591CE41@media.mit.edu>
Message-ID: <op.um6w4jezpcoyut@cyclops>

On Sat, 03 Jan 2009, Ian Eslick wrote:

> Do you have the latest version of elephant-unstable as of a day or two
> ago?

Indeed I do.  See my reply to Alex.

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 




From hugo_duncan at yahoo.com  Sat Jan  3 16:50:09 2009
From: hugo_duncan at yahoo.com (Hugo Duncan)
Date: Sat, 03 Jan 2009 11:50:09 -0500
Subject: [elephant-devel] postmodern btree problem
In-Reply-To: <gjmjmk$uuv$1@ger.gmane.org>
References: <op.um3lsoalpcoyut@cyclops> <gjkumh$6bg$1@ger.gmane.org>
	<op.um471w2ppcoyut@cyclops> <gjmjmk$uuv$1@ger.gmane.org>
Message-ID: <op.um6w1vorpcoyut@cyclops>


Changing to use elephant stable gets rid of the problem.  Not sure why I  
was using unstable - I know I switched back and forth several times when  
trying to get something to work when I started working with elephant.   
Which of stable/unstable do you recommend using?

BTW, to use unstable I had remove the OIDS-ONLY argument from  
PERSISTENT-SLOT-READER in pm-controller.lisp.

> also it might make sense to instrument function get-cached-instance a  
> bit to know whether crippled pm-btree is recreated or just retrieved  
> from cache. something like:

The code in unstable is different (and has a redundant AWHEN), but  
instrumenting it shows that when the problem occurs, the crippled pm-btree  
is being recreated (not in cache).

If there are any other tests you would like me to run, I would be happy to  
do so.  In the meantime, I'll just use stable.

Many thanks for your help, and apologies for not having tried stable  
before.

Hugo
__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 




From eslick at media.mit.edu  Sat Jan  3 18:01:49 2009
From: eslick at media.mit.edu (Ian Eslick)
Date: Sat, 3 Jan 2009 13:01:49 -0500
Subject: [elephant-devel] Issues adding to root
In-Reply-To: <7A955DAE-E3EB-47E7-93D0-DECDFDEDAF1E@umich.edu>
References: <594f78210811031229l2bb81a80q759759688c807b68@mail.gmail.com>
	<62312.88.73.210.243.1225746698.squirrel@mail.stardawn.org>
	<594f78210811031715r227c9915q55240e99ca13d42@mail.gmail.com>
	<62022.88.73.251.160.1225787942.squirrel@mail.stardawn.org>
	<7A955DAE-E3EB-47E7-93D0-DECDFDEDAF1E@umich.edu>
Message-ID: <C191BCC6-9C98-4BB0-B4B1-7747894670DB@media.mit.edu>

Hi Glen,

How are things going on the skiplist hack and CFFI?  I'm going to  
release an Alpha of 1.0 shortly and am starting to think about what  
goes into a beta release...  Are you likely to have any time to  
contribute in the next few months?  Thanks again for the documentation  
help last year.

Thanks!
Ian

On Nov 4, 2008, at 8:13 AM, Glenn Tarcea wrote:

> Just to follow up on this. I've been working with Leslie in the back
> ground (he's graciously taken the time to do some mentoring and
> reviews of code as well as to put up with some of my not so well
> thought out ideas :-) ). I have large pieces of Elephant switched over
> to CFFI, but largely due to my limited time have temporarily placed
> this on the back burner while working through a skiplist
> implementation with Leslie. I hope to get back to this project in the
> next couple of weeks. While it has not been quite as straight forward
> as I had anticipated (for example deftype assumptions) there isn't any
> reason (beyond spending some time on it) that I shouldn't be able to
> complete it.
>
> Thanks,
>
> Glenn
>
> V. Glenn Tarcea
> gtarcea at umich.edu
>
> On Nov 4, 2008, at 3:39 AM, Leslie P. Polzer wrote:
>
>>
>>> The following is something of a workaround and I'm not really sure
>>> why
>>> it works, but here goes.
>>>
>>> Starting with a fresh SBCL image I loaded "cffi-uffi-compat" (*not*
>>> "cffi" or "uffi") , then "cl-base64" and finally "elephant". After
>>> issuing some warnings about package name clashes between UFFI and
>>> CFFI-UFFI-COMPAT, adding and retrieving from a store seems to  
>>> work. I
>>> have yet to run the tests.
>>
>> Glenn has nearly migrated Elephant and ELE-BDB to CFFI, so
>> this probably won't be an issue in the mid-term.
>>
>> I'm surprised that CFFI-UFFI works for you with Elephant.
>> Perhaps the CFFI guys fixed some bugs since I last tried it.
>>
>>
>> _______________________________________________
>> elephant-devel site list
>> elephant-devel at common-lisp.net
>> http://common-lisp.net/mailman/listinfo/elephant-devel
>>
>>
>
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel




From eslick at media.mit.edu  Sat Jan  3 19:25:27 2009
From: eslick at media.mit.edu (Ian Eslick)
Date: Sat, 3 Jan 2009 14:25:27 -0500
Subject: [elephant-devel] Elephant 1.0 Alpha 1 (elephant-unstable, elephant,
	and elephant-1.0 repos)
In-Reply-To: <op.um6w1vorpcoyut@cyclops>
References: <op.um3lsoalpcoyut@cyclops> <gjkumh$6bg$1@ger.gmane.org>
	<op.um471w2ppcoyut@cyclops> <gjmjmk$uuv$1@ger.gmane.org>
	<op.um6w1vorpcoyut@cyclops>
Message-ID: <ADA6884E-4B4E-46D8-9765-34405F9F9C3C@media.mit.edu>

Hi All,

I'm in the process of cleaning things up and announcing an Alpha of  
release 1.0 with the current state of the unstable tree.  Since we  
recommend just about everyone move to unstable when they have a  
problem, I think that clarifying the current recommendation and its  
current status is in order.

I've cloned elephant-unstable and created a new darcs repository  
'elephant-1.0' that is currently synced to unstable.  I'd ask that  
active developers switch over to this at your earliest convenience.  I  
will tag this in the next day or so and send the announcement to  
elephant-announce.  I will eventually remove elephant and replace it  
with elephant-0.9.1 to avoid confusion.

I'm also in the process of bringing the website up to date.

Please send me comments online or offline regarding this unilateral  
decision.  I know everyone is busy and we've been putting off this  
release for almost nine months now so it's way past time we moved on  
with this.

Status:
BDB 4.7 is clean for me on Linux/SBCL
CL-SQL/SQLite3 is clean for me on Linux/SBCL sans one issue with  
association slots
Postmodern is clean for Alex sans 1 outstanding problem

The upgrade process hasn't been fully verified, but has been shown to  
work for BDB.  Caveats will be clearly spelled out in the  
announcement.  Mostly it's in relation to new features that didn't  
exist in 0.9.1.

Regards,
Ian



From eslick at media.mit.edu  Sat Jan  3 19:27:18 2009
From: eslick at media.mit.edu (Ian Eslick)
Date: Sat, 3 Jan 2009 14:27:18 -0500
Subject: [elephant-devel] postmodern btree problem
In-Reply-To: <op.um6w1vorpcoyut@cyclops>
References: <op.um3lsoalpcoyut@cyclops> <gjkumh$6bg$1@ger.gmane.org>
	<op.um471w2ppcoyut@cyclops> <gjmjmk$uuv$1@ger.gmane.org>
	<op.um6w1vorpcoyut@cyclops>
Message-ID: <07851CE7-63FF-4110-812B-4F023F6C7C59@media.mit.edu>

(I fixed that oids-only problem.  It's a tweak I'm adding to improve  
join efficiency in the new query planner.  Sorry that crept into the  
main code base).

I highly recommend you switch to the new repository, elephant-1.0.  We  
should be able to figure out any problems you have shortly.

Regards,
Ian

On Jan 3, 2009, at 11:50 AM, Hugo Duncan wrote:

>
> Changing to use elephant stable gets rid of the problem.  Not sure  
> why I
> was using unstable - I know I switched back and forth several times  
> when
> trying to get something to work when I started working with elephant.
> Which of stable/unstable do you recommend using?
>
> BTW, to use unstable I had remove the OIDS-ONLY argument from
> PERSISTENT-SLOT-READER in pm-controller.lisp.
>
>> also it might make sense to instrument function get-cached-instance a
>> bit to know whether crippled pm-btree is recreated or just retrieved
>> from cache. something like:
>
> The code in unstable is different (and has a redundant AWHEN), but
> instrumenting it shows that when the problem occurs, the crippled pm- 
> btree
> is being recreated (not in cache).
>
> If there are any other tests you would like me to run, I would be  
> happy to
> do so.  In the meantime, I'll just use stable.
>
> Many thanks for your help, and apologies for not having tried stable
> before.
>
> Hugo
> __________________________________________________
> Do You Yahoo!?
> Tired of spam?  Yahoo! Mail has the best spam protection around
> http://mail.yahoo.com
>
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel




From eslick at media.mit.edu  Sat Jan  3 20:38:40 2009
From: eslick at media.mit.edu (Ian Eslick)
Date: Sat, 3 Jan 2009 15:38:40 -0500
Subject: [elephant-devel] Elephant 1.0 Alpha 1 (elephant-unstable,
	elephant, and elephant-1.0 repos)
In-Reply-To: <ADA6884E-4B4E-46D8-9765-34405F9F9C3C@media.mit.edu>
References: <op.um3lsoalpcoyut@cyclops> <gjkumh$6bg$1@ger.gmane.org>
	<op.um471w2ppcoyut@cyclops> <gjmjmk$uuv$1@ger.gmane.org>
	<op.um6w1vorpcoyut@cyclops>
	<ADA6884E-4B4E-46D8-9765-34405F9F9C3C@media.mit.edu>
Message-ID: <F21D0CEE-1689-4870-8760-862AA2B5E5C4@media.mit.edu>

To clarify a typo regarding the new darcs repository naming scheme

elephant (the old 0.9.1 tree + a few patches, will rename to  
elephant-0.9.1 eventually)
elephant-unstable (will be removed after Alpha 1.0 is announced)
elephant-1.0 (the new development tree, will be tagged with each  
release)

Questions?

Ian

On Jan 3, 2009, at 2:25 PM, Ian Eslick wrote:

> I've cloned elephant-unstable and created a new darcs repository
> 'elephant-1.0' that is currently synced to unstable.  I'd ask that
> active developers switch over to this at your earliest convenience.  I
> will tag this in the next day or so and send the announcement to
> elephant-announce.  I will eventually remove elephant and replace it
> with elephant-0.9.1 to avoid confusion.




From eslick at media.mit.edu  Sat Jan  3 21:49:59 2009
From: eslick at media.mit.edu (Ian Eslick)
Date: Sat, 3 Jan 2009 16:49:59 -0500
Subject: [elephant-devel] MVCC for BDB
Message-ID: <7FAA5CAC-BCA1-44E8-8432-3986929AC8EF@media.mit.edu>

MVCC support is now available for the Elephant BDB backend (Thank you  
Alex for the prompting on this - I finished adding support for it  
months ago but never actually tested it).  It appears to work,  
according to db_stat.

To enable it, simply put (:berkeley-db-mvcc . t) in your my- 
config.sexp file or (setf elephant::*default-mvcc* t) before you open  
the database.

You can also add :mvcc t to open-store and :snapshot t to with- 
transaction to do this manually.  You must open the DB with :mvcc to  
use :snapshot.  *default-mvcc* defaults both of these parameters to  
true.

Ian



From killerstorm at newmail.ru  Sat Jan  3 22:18:38 2009
From: killerstorm at newmail.ru (Alex Mizrahi)
Date: Sun, 4 Jan 2009 00:18:38 +0200
Subject: [elephant-devel] postmodern btree problem
References: <op.um3lsoalpcoyut@cyclops>
	<gjkumh$6bg$1@ger.gmane.org><op.um471w2ppcoyut@cyclops>
	<gjmjmk$uuv$1@ger.gmane.org> <op.um6w1vorpcoyut@cyclops>
Message-ID: <gjoo7v$edn$1@ger.gmane.org>

 HD> Which of stable/unstable do you recommend using?

i think we'll eventually switch to "unstable", so if you're planning
for the future, this would be a better option. also the more people
use it, faster we will find all bugs in a new version. OTOH in a short
term perspective "stable" is more tested.

 HD> instrumenting it shows that when the problem occurs, the crippled
 HD> pm-btree  is being recreated (not in cache).

aha, thanks for finding this out -- it appears that recreation code
in unstable is severely broken.

i do not understand the idea behind these changes, i hope Ian can clarify 
this
so we'll find a way to fix this.

in old version recreate-instance-using class for standard-class objects was 
just called make-instance,
so it went through the normal initialization procedure.

in a new version it just calls allocate-instance, so pm-btree has absolutely 
no chance to
initialize itself -- neither initialize-instance is called to fill slots 
with iniforms, nor shared-initialize is called,
and we end up with a blank object.

here's a piece of documentation:

(defmethod recreate-instance-using-class ((class standard-class) &rest 
initargs &key &allow-other-keys)
  (declare (ignore initargs))
  "Simply allocate store, the state of the slots will be filled by the data 
from the
   database.  We do not want to call initialize-instance and re-evaluate the 
initforms;
   we are just fetching the object & values from the store"
  (allocate-instance class))

but i fail to understand the point -- how exactly are we "feching the object 
& values from the store" for standard-class
classes? is that some new object serialization mechanism or just an error?







From eslick at media.mit.edu  Sat Jan  3 22:53:07 2009
From: eslick at media.mit.edu (Ian Eslick)
Date: Sat, 3 Jan 2009 17:53:07 -0500
Subject: [elephant-devel] postmodern btree problem
In-Reply-To: <gjoo7v$edn$1@ger.gmane.org>
References: <op.um3lsoalpcoyut@cyclops>
	<gjkumh$6bg$1@ger.gmane.org><op.um471w2ppcoyut@cyclops>
	<gjmjmk$uuv$1@ger.gmane.org> <op.um6w1vorpcoyut@cyclops>
	<gjoo7v$edn$1@ger.gmane.org>
Message-ID: <E1BCB6C1-8EFC-46A1-A1B0-313B31673BAF@media.mit.edu>

Hmmmm....

Off the top of my head, I'm not sure what's going on.  I'll look into  
this tomorrow.

Cheers,
Ian

On Jan 3, 2009, at 5:18 PM, Alex Mizrahi wrote:

> HD> Which of stable/unstable do you recommend using?
>
> i think we'll eventually switch to "unstable", so if you're planning
> for the future, this would be a better option. also the more people
> use it, faster we will find all bugs in a new version. OTOH in a short
> term perspective "stable" is more tested.
>
> HD> instrumenting it shows that when the problem occurs, the crippled
> HD> pm-btree  is being recreated (not in cache).
>
> aha, thanks for finding this out -- it appears that recreation code
> in unstable is severely broken.
>
> i do not understand the idea behind these changes, i hope Ian can  
> clarify
> this
> so we'll find a way to fix this.
>
> in old version recreate-instance-using class for standard-class  
> objects was
> just called make-instance,
> so it went through the normal initialization procedure.
>
> in a new version it just calls allocate-instance, so pm-btree has  
> absolutely
> no chance to
> initialize itself -- neither initialize-instance is called to fill  
> slots
> with iniforms, nor shared-initialize is called,
> and we end up with a blank object.
>
> here's a piece of documentation:
>
> (defmethod recreate-instance-using-class ((class standard-class) &rest
> initargs &key &allow-other-keys)
>  (declare (ignore initargs))
>  "Simply allocate store, the state of the slots will be filled by  
> the data
> from the
>   database.  We do not want to call initialize-instance and re- 
> evaluate the
> initforms;
>   we are just fetching the object & values from the store"
>  (allocate-instance class))
>
> but i fail to understand the point -- how exactly are we "feching  
> the object
> & values from the store" for standard-class
> classes? is that some new object serialization mechanism or just an  
> error?
>
>
>
>
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel




From rsynnott at gmail.com  Sat Jan  3 22:56:35 2009
From: rsynnott at gmail.com (Robert Synnott)
Date: Sat, 3 Jan 2009 22:56:35 +0000
Subject: [elephant-devel] Choice of back-end store
In-Reply-To: <D97C6466-8651-4A82-B333-5024C258F0B2@media.mit.edu>
References: <ecb948b30901021117t4733cbf0m83306c66a863c36d@mail.gmail.com>
	<D97C6466-8651-4A82-B333-5024C258F0B2@media.mit.edu>
Message-ID: <24f203480901031456p189d955g9c3bc361f4e0cd1f@mail.gmail.com>

2009/1/3 Ian Eslick <eslick at media.mit.edu>:
> Hi Yarek,
>
> Others may add to this, but the big issues I'm aware of are:
>
> BDB
> + fast
> + can be shared among multiple processes on a single machine
> - commercial use requires license
>
> Postmodern
> - 5x slower than BDB (last I checked)
> ++ can be shared among multiple machines
> + open source
>
> Neither system allows you to easily bypass the elephant
> representational framework.  I'll let others comment on this vis-a-vis
> postmodern.
>
> You can use both system's tools for archive, recover, duplication, etc.
>
> Ian

Messing around with postmodern backend's underlying database is
POSSIBLE (I did it recently to implement a crude form of online
garbage collection) but unpleasant.
Rob



From killerstorm at newmail.ru  Sat Jan  3 23:02:21 2009
From: killerstorm at newmail.ru (Alex Mizrahi)
Date: Sun, 4 Jan 2009 01:02:21 +0200
Subject: [elephant-devel] CLSQL Store
References: <ecb948b30901021117t4733cbf0m83306c66a863c36d@mail.gmail.com><gjmef5$l64$1@ger.gmane.org>
	<1F1F136C-D922-46F8-BAC3-B252D34BF8BC@media.mit.edu>
Message-ID: <461C0B66B0E04C1EBCFFF01B7E1C7D18@killer>

 IE> Is anyone still using the CL-SQL store?

the poll says no one, except one vote in "testing" section

 IE> release, although SQLite support is the only way to use Elephant with a
 IE> cheap, free, easy to install/ distribute backend.

i think a backend using SQLite directly would make a lot of sense

 IE> but if you want to 'deploy' your a BDB-powered application in source
 IE> or binary form you have to either get a commercial license or
 IE> distribute your application as open source (such as Subversion).

explanation on BDB's site says that if you, say, use Elephant rather than
BDB directly, and Elephant is open source, that is enough to be eligible
for a "free" BDB version. it is up to a lawyers to decide if it is such.
i betcha if you ask Oracle directly they'll say "of course you need a 
commercial version" :)




From eslick at media.mit.edu  Sun Jan  4 12:59:27 2009
From: eslick at media.mit.edu (Ian Eslick)
Date: Sun, 4 Jan 2009 07:59:27 -0500
Subject: [elephant-devel] CLSQL Store
In-Reply-To: <461C0B66B0E04C1EBCFFF01B7E1C7D18@killer>
References: <ecb948b30901021117t4733cbf0m83306c66a863c36d@mail.gmail.com><gjmef5$l64$1@ger.gmane.org>
	<1F1F136C-D922-46F8-BAC3-B252D34BF8BC@media.mit.edu>
	<461C0B66B0E04C1EBCFFF01B7E1C7D18@killer>
Message-ID: <8E6475D8-3F69-43BF-9767-116856BBC95B@media.mit.edu>

I found this same discussion several places on the web and both  
Sleepcat and Oracle seem pretty supportive of the more generous  
interpretation, which is nice!

Ian

On Jan 3, 2009, at 6:02 PM, Alex Mizrahi wrote:

> IE> Is anyone still using the CL-SQL store?
>
> the poll says no one, except one vote in "testing" section
>
> IE> release, although SQLite support is the only way to use Elephant  
> with a
> IE> cheap, free, easy to install/ distribute backend.
>
> i think a backend using SQLite directly would make a lot of sense
>
> IE> but if you want to 'deploy' your a BDB-powered application in  
> source
> IE> or binary form you have to either get a commercial license or
> IE> distribute your application as open source (such as Subversion).
>
> explanation on BDB's site says that if you, say, use Elephant rather  
> than
> BDB directly, and Elephant is open source, that is enough to be  
> eligible
> for a "free" BDB version. it is up to a lawyers to decide if it is  
> such.
> i betcha if you ask Oracle directly they'll say "of course you need a
> commercial version" :)
>
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel




From eslick at media.mit.edu  Sun Jan  4 20:18:38 2009
From: eslick at media.mit.edu (Ian Eslick)
Date: Sun, 4 Jan 2009 15:18:38 -0500
Subject: [elephant-devel] postmodern btree problem
In-Reply-To: <gjoo7v$edn$1@ger.gmane.org>
References: <op.um3lsoalpcoyut@cyclops>
	<gjkumh$6bg$1@ger.gmane.org><op.um471w2ppcoyut@cyclops>
	<gjmjmk$uuv$1@ger.gmane.org> <op.um6w1vorpcoyut@cyclops>
	<gjoo7v$edn$1@ger.gmane.org>
Message-ID: <709CC003-3D84-4930-AFDD-723E8CF5FE9F@media.mit.edu>

There are two problems with calling the full initialization protocol  
of an object for all standard classes.

First, you are de-serializing an existing instance not creating a new  
one so the semantics of rerunning the initialization procedures when  
recreating a serialized object is questionable.  There was a long  
discussion on this about a year ago started by Sean Ross which led to  
the recreation protocol.

Two, under the current serializer if you store a standard class with  
an unbound slot but it has an initform, then the recreated object will  
end up with the initform value in the previously unbound slot.  This  
motivated the recent change that caused this bug.

Persistent collections are a special case as they inherit from  
persistent (so they are stored uniquely as OID references in the  
database) but are not of persistent-metaclass.  Thus they end up using  
the new standard-class recreation protocol.  In the BDB backend I  
persist data for indexed-btrees so I make that a persistent-metaclass  
and thus shared-initialize always gets called.  pm-btree is a unique  
case that wasn't handled.

My solution is to clean things up and call recreate-instance on all  
class types (standard-class and persistent-metaclass).

recreate-instance (standard-object) - does nothing
recreate-instance (persistent-object) - caching, schema upgrades,  
shared-initialize
recreate-instance (persistent-collection) - caching, shared-initialize  
(no schema support for bootstrapping purposes)

Please try out the patch that implements this, it should solve the  
special case of pm-btree recreation.  It's in the new elephant-1.0  
repository as well as unstable.

http://www.common-lisp.net/project/elephant/darcs/elephant-1.0

An additional comment below...

Thank you,
Ian

PS - Anyone know why we chose to use recreate-instance instead of  
hooking into reinitialize-instance?

On Jan 3, 2009, at 5:18 PM, Alex Mizrahi wrote:

> (defmethod recreate-instance-using-class ((class standard-class) &rest
> initargs &key &allow-other-keys)
>  (declare (ignore initargs))
>  "Simply allocate store, the state of the slots will be filled by  
> the data
> from the
>   database.  We do not want to call initialize-instance and re- 
> evaluate the
> initforms;
>   we are just fetching the object & values from the store"
>  (allocate-instance class))
>
> but i fail to understand the point -- how exactly are we "feching  
> the object
> & values from the store" for standard-class
> classes? is that some new object serialization mechanism or just an  
> error?

This is part of the deserialiation of standard-class objects; all slot  
values of standard classes that don't inherit from persistent are  
stored and restored on each deserialization.  No caching is performed.





From hugo_duncan at yahoo.com  Sun Jan  4 21:18:47 2009
From: hugo_duncan at yahoo.com (Hugo Duncan)
Date: Sun, 04 Jan 2009 16:18:47 -0500
Subject: [elephant-devel] postmodern btree problem
In-Reply-To: <709CC003-3D84-4930-AFDD-723E8CF5FE9F@media.mit.edu>
References: <op.um3lsoalpcoyut@cyclops> <gjkumh$6bg$1@ger.gmane.org>
	<op.um471w2ppcoyut@cyclops> <gjmjmk$uuv$1@ger.gmane.org>
	<op.um6w1vorpcoyut@cyclops> <gjoo7v$edn$1@ger.gmane.org>
	<709CC003-3D84-4930-AFDD-723E8CF5FE9F@media.mit.edu>
Message-ID: <op.um835lhipcoyut@cyclops>

Ian,

> Please try out the patch that implements this, it should solve the
> special case of pm-btree recreation.  It's in the new elephant-1.0
> repository as well as unstable.
>
> http://www.common-lisp.net/project/elephant/darcs/elephant-1.0

Just to confirm that this works for me.  Many thanks.

Hugo
__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 




From yarek.kowalik at gmail.com  Mon Jan  5 04:43:57 2009
From: yarek.kowalik at gmail.com (Yarek Kowalik)
Date: Sun, 4 Jan 2009 20:43:57 -0800
Subject: [elephant-devel] CLSQL Store
In-Reply-To: <8E6475D8-3F69-43BF-9767-116856BBC95B@media.mit.edu>
References: <ecb948b30901021117t4733cbf0m83306c66a863c36d@mail.gmail.com>
	<gjmef5$l64$1@ger.gmane.org>
	<1F1F136C-D922-46F8-BAC3-B252D34BF8BC@media.mit.edu>
	<461C0B66B0E04C1EBCFFF01B7E1C7D18@killer>
	<8E6475D8-3F69-43BF-9767-116856BBC95B@media.mit.edu>
Message-ID: <ecb948b30901042043s1faf2777gc0448ab5c7fe95b0@mail.gmail.com>

Great! Thanks for sharing this info.

Yarek

On Sun, Jan 4, 2009 at 4:59 AM, Ian Eslick <eslick at media.mit.edu> wrote:

> I found this same discussion several places on the web and both
> Sleepcat and Oracle seem pretty supportive of the more generous
> interpretation, which is nice!
>
> Ian
>
> On Jan 3, 2009, at 6:02 PM, Alex Mizrahi wrote:
>
> > IE> Is anyone still using the CL-SQL store?
> >
> > the poll says no one, except one vote in "testing" section
> >
> > IE> release, although SQLite support is the only way to use Elephant
> > with a
> > IE> cheap, free, easy to install/ distribute backend.
> >
> > i think a backend using SQLite directly would make a lot of sense
> >
> > IE> but if you want to 'deploy' your a BDB-powered application in
> > source
> > IE> or binary form you have to either get a commercial license or
> > IE> distribute your application as open source (such as Subversion).
> >
> > explanation on BDB's site says that if you, say, use Elephant rather
> > than
> > BDB directly, and Elephant is open source, that is enough to be
> > eligible
> > for a "free" BDB version. it is up to a lawyers to decide if it is
> > such.
> > i betcha if you ask Oracle directly they'll say "of course you need a
> > commercial version" :)
> >
> >
> > _______________________________________________
> > elephant-devel site list
> > elephant-devel at common-lisp.net
> > http://common-lisp.net/mailman/listinfo/elephant-devel
>
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20090104/423860e4/attachment.html>

From alex.mizrahi at gmail.com  Mon Jan  5 10:30:11 2009
From: alex.mizrahi at gmail.com (Alex Mizrahi)
Date: Mon, 05 Jan 2009 12:30:11 +0200
Subject: [elephant-devel] postmodern btree problem
References: <op.um3lsoalpcoyut@cyclops> <gjkumh$6bg$1@ger.gmane.org>
	<op.um471w2ppcoyut@cyclops> <gjmjmk$uuv$1@ger.gmane.org>
	<op.um6w1vorpcoyut@cyclops> <gjoo7v$edn$1@ger.gmane.org>
	<709CC003-3D84-4930-AFDD-723E8CF5FE9F@media.mit.edu>
Message-ID: <op.um94slygldr535@your-a80c79e8b3>

>  In the BDB backend I
> persist data for indexed-btrees so I make that a persistent-metaclass
> and thus shared-initialize always gets called.  pm-btree is a unique
> case that wasn't handled.

pm-indexed-btree is of persistent-metaclass, pm-btree is not, for  
bootstrapping
reasons.

> My solution is to clean things up and call recreate-instance on all
> class types (standard-class and persistent-metaclass).

> recreate-instance (standard-object) - does nothing

this way it has no chance to initialize its "transient" slots, that is not  
fair..
well, people who need something advanced could just use  
persistent-objects..

> recreate-instance (persistent-collection) - caching, shared-initialize
> (no schema support for bootstrapping purposes)

> PS - Anyone know why we chose to use recreate-instance instead of
> hooking into reinitialize-instance?

because we need to allocate-instance first, i guess.

but i think that perhaps we could eliminate three branches there, just  
calling reinitialize-instance (or shared-initialize with slot-names being  
NIL directly) after allocate instance -- in this case, as i understand, we  
won't have problems with unbound slots being filled with initforms, and  
objects will have a chance to do initialization in uniform way.

as for persistent-specific initialization, i guess we can place it into  
shared-initilize :before, or into reinitialize-instance.




From eslick at media.mit.edu  Mon Jan  5 14:19:25 2009
From: eslick at media.mit.edu (Ian Eslick)
Date: Mon, 5 Jan 2009 09:19:25 -0500
Subject: [elephant-devel] BDB Replication
Message-ID: <A883D610-6FCB-4C54-8A67-BA0019387B4C@media.mit.edu>

For kicks I spent a little bit of time thinking about what it would  
take to make Elephant work with BDB replication.  There is one thing I  
don't understand, if a client wants to initiate a write transaction on  
the master, does it have to be a completely application level process  
(for example, you have the execute-transaction function call run  
remotely) or can you do this at a lower level where you somehow  
forward the low level read/write log information from client to server  
to be committed?

Ian



From yarek.kowalik at gmail.com  Mon Jan  5 21:05:14 2009
From: yarek.kowalik at gmail.com (Yarek Kowalik)
Date: Mon, 5 Jan 2009 13:05:14 -0800
Subject: [elephant-devel] BDB Replication
In-Reply-To: <A883D610-6FCB-4C54-8A67-BA0019387B4C@media.mit.edu>
References: <A883D610-6FCB-4C54-8A67-BA0019387B4C@media.mit.edu>
Message-ID: <ecb948b30901051305l180dfdfh4edecbc479d93e09@mail.gmail.com>

Ian,

I think the later is what's supported/implemented by BDB:

"Berkeley DB supports replication over multiple systems, enabling
applications to scale massively with low latency and provide fault tolerance
for high availability solutions. This technique works by having all updates
go to a designated master, which distributes changes automatically to a set
of replicas ..."

http://www.oracle.com/technology/products/berkeley-db/db/index.html

I'm not sure at this stage if there are any changes required within Elephant
itself as this may just be an "off-the-shelf" capability of the BDB backend.

Yarek


On Mon, Jan 5, 2009 at 6:19 AM, Ian Eslick <eslick at media.mit.edu> wrote:

> For kicks I spent a little bit of time thinking about what it would
> take to make Elephant work with BDB replication.  There is one thing I
> don't understand, if a client wants to initiate a write transaction on
> the master, does it have to be a completely application level process
> (for example, you have the execute-transaction function call run
> remotely) or can you do this at a lower level where you somehow
> forward the low level read/write log information from client to server
> to be committed?
>
> Ian
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20090105/8d5892b8/attachment.html>

From eslick at media.mit.edu  Mon Jan  5 21:52:38 2009
From: eslick at media.mit.edu (Ian Eslick)
Date: Mon, 5 Jan 2009 16:52:38 -0500
Subject: [elephant-devel] BDB Replication
In-Reply-To: <ecb948b30901051305l180dfdfh4edecbc479d93e09@mail.gmail.com>
References: <A883D610-6FCB-4C54-8A67-BA0019387B4C@media.mit.edu>
	<ecb948b30901051305l180dfdfh4edecbc479d93e09@mail.gmail.com>
Message-ID: <3FD03533-CFDC-4965-A44F-50289A167C33@media.mit.edu>

What confused me is that the documentation FAQ says that it doesn't  
forward updates to the master and that this is the responsibility of  
the application.  However, it doesn't indicate where update message  
are produced...

If it is simply a matter of forwarding low level messages, then this  
is actually somewhat reasonable to implement.  The communications  
consists of registering a callback function in lisp from the BDB  
library that sends and receives packets to/from the master.   
Alternatively, we could have C functions implement the callbacks and  
socket IO that BDB needs a callback into and thus avoid the chain of  
conversions foreign bytes->lisp bytes->lisp stream->foreign bytes- 
 >network.

The steps would be:
- Change the way DBs are opened
- Add some code to start or respond to election events
- Add the communications support
   - Socket IO
   - BDB API callbacks and configuration

I won't be doing this anytime soon, but if someone wants to figure out  
the BDB side of this and particularly the callbacks/comm layer I can  
help integrate it into elephant.

Ian

On Jan 5, 2009, at 4:05 PM, Yarek Kowalik wrote:

> Ian,
>
> I think the later is what's supported/implemented by BDB:
>
> "Berkeley DB supports replication over multiple systems, enabling  
> applications to scale massively with low latency and provide fault  
> tolerance for high availability solutions. This technique works by  
> having all updates go to a designated master, which distributes  
> changes automatically to a set of replicas ..."
>
> http://www.oracle.com/technology/products/berkeley-db/db/index.html
>
> I'm not sure at this stage if there are any changes required within  
> Elephant itself as this may just be an "off-the-shelf" capability of  
> the BDB backend.
>
> Yarek
>
>
> On Mon, Jan 5, 2009 at 6:19 AM, Ian Eslick <eslick at media.mit.edu>  
> wrote:
> For kicks I spent a little bit of time thinking about what it would
> take to make Elephant work with BDB replication.  There is one thing I
> don't understand, if a client wants to initiate a write transaction on
> the master, does it have to be a completely application level process
> (for example, you have the execute-transaction function call run
> remotely) or can you do this at a lower level where you somehow
> forward the low level read/write log information from client to server
> to be committed?
>
> Ian
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel




From eslick at media.mit.edu  Mon Jan  5 23:43:25 2009
From: eslick at media.mit.edu (Ian Eslick)
Date: Mon, 5 Jan 2009 18:43:25 -0500
Subject: [elephant-devel] Online garbage collector
Message-ID: <8DF593C4-82C3-47B1-84BE-8BE0A1BB8D4B@media.mit.edu>

I've just checked in a long-awaited online garbage collection scheme.   
It's not heavily tested or optimized, but you can experiment with it  
right away.  Simply start the following function in a thread from your  
application, or from the repl, and see what happens.

(ele::mark-and-sweep-gc *store-controller* &optional test)

WARNING: this is a prototype and cannot be guaranteed not to lose  
important data.  Use only on a database you don't care about!

When test is true, the collector only prints out all objects that it  
will reclaim, leaving a btree in ele::*mark-table* with the list of  
marked objects.

LIVENESS

The collector is not really very useful right now for objects other  
than btrees and psets as all persistent classes are written to a class  
index by default.  If you pass :index nil as a class initarg you will  
lose the ability to (get-instances-by-class), but any non-live objects  
in the store will be reclaimed.  You can either redefine the class to  
inhibit indexing or do it explicitly via: (setf (ele::%class-indexing  
class) nil).

Liveness is identical to the notion of reachability used by migration  
with one exception, the store's instance cache is also treated as a  
root.  Thus all persistent objects reachable from the memory cache,  
root btree, and class indexes are considered live.  The mark pass  
recurses on both persistent aggregates (e.g. btree, pset) as well as  
standard aggregates (e.g. structures, standard classes, arrays, etc).

You may want to run in-memory GC prior to running persistent GC to  
improve reclamation.  One potential issue: I have noticed that btrees  
that are stored in in-memory variables are dropped from the cache (a  
weak hash table) under SBCL so it's possible that persistent objects  
that are live in memory are dropped from the cache and can be  
reclaimed, leaving dangling references.  This may be a bug in weak  
pointers in the version of SBCL I'm using.

PERFORMANCE / CONTENTION

The mark pass runs all at once, but is predominantly doing reads which  
shouldn't result in much contention.  A fresh btree is used to keep  
track of marks.  For large DBs, this strategy involves long-lived  
transactions and may exhaust locks or cache memory and so a future  
revision should do the mark pass in a discrete series of steps (this  
means doing bookkeeping and checkpointing the recursive descent of the  
tree from the root which is somewhat odious).

The sweep pass is done with one transaction per class to minimize  
contention; reclaiming storage can touch quite a few database pages.

TURNKEY

Later, when I have some time, I'll add an optional bordeaux-threads  
model so you can simply start the collector as part of open-store and  
put frequency and other parameters to control behavior into my- 
config.sexp.  For now you can simply manually start a thread that runs  
the mark-and-sweep-gc function to completion on a given store.

Cheers,
Ian

PS - As a bonus for BDB users, you can call (optimize-layout store)  
after the gc which will attempt to defragment and optionally return  
free pages to the disk.





From eslick at media.mit.edu  Mon Jan  5 23:53:35 2009
From: eslick at media.mit.edu (Ian Eslick)
Date: Mon, 5 Jan 2009 18:53:35 -0500
Subject: [elephant-devel] Announcing Elephant 1.0 Alpha 1
Message-ID: <B81AA4F3-F8A3-4D74-890B-574A75DBA62C@media.mit.edu>

We have just tagged the new elephant-1.0 repository with Elephant 1.0  
Alpha 1 (ELEPHANT-1-0-A1.  There are some small known issues, but the  
core code base has become fairly stable.  Please test this release out  
on as many lisp/OS combination as you can, especially non-SBCL users.   
We're sure to turn up some problems on untested configurations and  
move to an Alpha 2 fairly quickly.

This release is a little less stable than 0.9.1 so unless you are  
engaged in active development and don't mind a little churn, you may  
want to wait for the first beta release before switching.  Many  
problems in 0.9.1 have been fixed in the current 1.0 tree as well as  
many new features.  Some advanced features are not yet reliable, but  
will become so before the final release (associations, garbage  
collection, query language, etc.)

Details of what is contained in this release can be found here:

http://common-lisp.net/project/elephant/news.html

Details of where to pull this release from can be found here:

http://common-lisp.net/project/elephant/downloads.html

Regards,
Ian



From yarek.kowalik at gmail.com  Tue Jan  6 00:14:42 2009
From: yarek.kowalik at gmail.com (Yarek Kowalik)
Date: Mon, 5 Jan 2009 16:14:42 -0800
Subject: [elephant-devel] BDB Replication
In-Reply-To: <3FD03533-CFDC-4965-A44F-50289A167C33@media.mit.edu>
References: <A883D610-6FCB-4C54-8A67-BA0019387B4C@media.mit.edu>
	<ecb948b30901051305l180dfdfh4edecbc479d93e09@mail.gmail.com>
	<3FD03533-CFDC-4965-A44F-50289A167C33@media.mit.edu>
Message-ID: <ecb948b30901051614v623b96ccmfe2bd7d98691558c@mail.gmail.com>

I will be investigating this at some point in the next few months, but it's
not on the top of my priority list at the moment.

Yarek

On Mon, Jan 5, 2009 at 1:52 PM, Ian Eslick <eslick at media.mit.edu> wrote:

> What confused me is that the documentation FAQ says that it doesn't
> forward updates to the master and that this is the responsibility of
> the application.  However, it doesn't indicate where update message
> are produced...
>
> If it is simply a matter of forwarding low level messages, then this
> is actually somewhat reasonable to implement.  The communications
> consists of registering a callback function in lisp from the BDB
> library that sends and receives packets to/from the master.
> Alternatively, we could have C functions implement the callbacks and
> socket IO that BDB needs a callback into and thus avoid the chain of
> conversions foreign bytes->lisp bytes->lisp stream->foreign bytes-
>  >network.
>
> The steps would be:
> - Change the way DBs are opened
> - Add some code to start or respond to election events
> - Add the communications support
>   - Socket IO
>   - BDB API callbacks and configuration
>
> I won't be doing this anytime soon, but if someone wants to figure out
> the BDB side of this and particularly the callbacks/comm layer I can
> help integrate it into elephant.
>
> Ian
>
> On Jan 5, 2009, at 4:05 PM, Yarek Kowalik wrote:
>
> > Ian,
> >
> > I think the later is what's supported/implemented by BDB:
> >
> > "Berkeley DB supports replication over multiple systems, enabling
> > applications to scale massively with low latency and provide fault
> > tolerance for high availability solutions. This technique works by
> > having all updates go to a designated master, which distributes
> > changes automatically to a set of replicas ..."
> >
> > http://www.oracle.com/technology/products/berkeley-db/db/index.html
> >
> > I'm not sure at this stage if there are any changes required within
> > Elephant itself as this may just be an "off-the-shelf" capability of
> > the BDB backend.
> >
> > Yarek
> >
> >
> > On Mon, Jan 5, 2009 at 6:19 AM, Ian Eslick <eslick at media.mit.edu>
> > wrote:
> > For kicks I spent a little bit of time thinking about what it would
> > take to make Elephant work with BDB replication.  There is one thing I
> > don't understand, if a client wants to initiate a write transaction on
> > the master, does it have to be a completely application level process
> > (for example, you have the execute-transaction function call run
> > remotely) or can you do this at a lower level where you somehow
> > forward the low level read/write log information from client to server
> > to be committed?
> >
> > Ian
> >
> > _______________________________________________
> > elephant-devel site list
> > elephant-devel at common-lisp.net
> > http://common-lisp.net/mailman/listinfo/elephant-devel
> >
> > _______________________________________________
> > elephant-devel site list
> > elephant-devel at common-lisp.net
> > http://common-lisp.net/mailman/listinfo/elephant-devel
>
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20090105/bba0cc5e/attachment.html>

From yarek.kowalik at gmail.com  Tue Jan  6 06:36:29 2009
From: yarek.kowalik at gmail.com (Yarek Kowalik)
Date: Mon, 5 Jan 2009 22:36:29 -0800
Subject: [elephant-devel] Querying for objects on two slots
Message-ID: <ecb948b30901052236t4762d164p492ffac56e040152@mail.gmail.com>

I've run into several situations where I want to query for objects that
match values two or more slots.  I currently do something like this:

(defclass my-class ()
  ((a :accessor slot-a)
   (b :accessor slot-b
      :index b))
  (:documentation "my test class"))

(find-if (lambda (obj)
           (equal (slot-a obj) val-a))
         (elephant:get-instances-by-value 'my-class 'slot-b val-b))


Is this a reasonable way of finding an object of type 'my-class that matches
on values val-a and val-b for slots a and b?

Yarek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20090105/48bb352b/attachment.html>

From sky at viridian-project.de  Tue Jan  6 08:35:42 2009
From: sky at viridian-project.de (Leslie P. Polzer)
Date: Tue, 6 Jan 2009 09:35:42 +0100
Subject: [elephant-devel] Querying for objects on two slots
In-Reply-To: <ecb948b30901052236t4762d164p492ffac56e040152@mail.gmail.com>
References: <ecb948b30901052236t4762d164p492ffac56e040152@mail.gmail.com>
Message-ID: <20090106083542.GC858@viridian-project.de>

On Mon, Jan 05, 2009 at 10:36:29PM -0800, Yarek Kowalik wrote:
> I've run into several situations where I want to query for objects that
> match values two or more slots.  I currently do something like this:
> 
> (defclass my-class ()
>   ((a :accessor slot-a)
>    (b :accessor slot-b
>       :index b))
>   (:documentation "my test class"))
> 
> (find-if (lambda (obj)
>            (equal (slot-a obj) val-a))
>          (elephant:get-instances-by-value 'my-class 'slot-b val-b))
> 
> 
> Is this a reasonable way of finding an object of type 'my-class that matches
> on values val-a and val-b for slots a and b?

Use MAP-CLASS, this will considerably speed up the query.



From eslick at media.mit.edu  Tue Jan  6 13:06:05 2009
From: eslick at media.mit.edu (Ian Eslick)
Date: Tue, 6 Jan 2009 08:06:05 -0500
Subject: [elephant-devel] Querying for objects on two slots
In-Reply-To: <ecb948b30901052236t4762d164p492ffac56e040152@mail.gmail.com>
References: <ecb948b30901052236t4762d164p492ffac56e040152@mail.gmail.com>
Message-ID: <2E02593C-1DA9-452E-BAEE-D39748E47C35@media.mit.edu>

Generally using get-instances-xxx is expensive (you have to cons up  
all objects).  A more efficient solution would be to map-inverted- 
index over slot-b value and filter by the slot-a value.  If slot-b  
returns a small set, this is pretty darned efficient.

The little query interpreter we have now should do this  
automatically.  It's something like this (I don't have time to look up  
the syntax just now)

(map-query
  (select ((?c my-class))
    (= (slot-a ?c) val-a)
    (= (slot-b ?c) val-b))

However, if you don't have a small set in either a or b, you can  
create a virtual slot that is some function of a and b and  
automatically updated when slot a and b are updated.  For now the best  
way to do this is to put them into a string that can be totally  
ordered.  We need a good solution for sorting on short sequences of  
primitive values so we can have pre-computed join indices.0

However, later the query system will allow you to do reasonably  
efficient equality joins on the value/ranges in each slot (assuming  
both are indexed).  In fact the new system in query2.lisp might  
actually handle the simple join query you're trying to do above - it  
would be worth a quick try.

Ian

On Jan 6, 2009, at 1:36 AM, Yarek Kowalik wrote:

> I've run into several situations where I want to query for objects  
> that match values two or more slots.  I currently do something like  
> this:
>
> (defclass my-class ()
>   ((a :accessor slot-a)
>    (b :accessor slot-b
>       :index b))
>   (:documentation "my test class"))
>
> (find-if (lambda (obj)
>            (equal (slot-a obj) val-a))
>          (elephant:get-instances-by-value 'my-class 'slot-b val-b))
>
>
> Is this a reasonable way of finding an object of type 'my-class that  
> matches on values val-a and val-b for slots a and b?
>
> Yarek
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel




From eslick at media.mit.edu  Tue Jan  6 13:15:36 2009
From: eslick at media.mit.edu (Ian Eslick)
Date: Tue, 6 Jan 2009 08:15:36 -0500
Subject: [elephant-devel] Online garbage collector
In-Reply-To: <8DF593C4-82C3-47B1-84BE-8BE0A1BB8D4B@media.mit.edu>
References: <8DF593C4-82C3-47B1-84BE-8BE0A1BB8D4B@media.mit.edu>
Message-ID: <49D2E6A5-8EE7-4287-A22B-E4F5836A4EC9@media.mit.edu>

As expected there are a few issues with the current collector in  
guaranteeing that all live objects are marked without having stopping  
the world.  Any new object are automatically and conservatively  
marked, but any operations that move a reference from in front of the  
mark process to behind it in parallel will cause a live variable to be  
unmarked.  The controller can keep track of any OIDs that were written  
and do a quick re-mark phase on those objects that does stop the world.

For very large heaps keeping track of written objects becomes too  
expensive and we'll need to look into generational approaches (which  
adds quite a bit of complexity).  Still, this is a quicker and simpler  
way to clean up your heap than migration.

So two additions needed for a full online solution:
- A way to stop the world
- A way to track written oids

Ian

On Jan 5, 2009, at 6:43 PM, Ian Eslick wrote:

> I've just checked in a long-awaited online garbage collection scheme.
> It's not heavily tested or optimized, but you can experiment with it
> right away.  Simply start the following function in a thread from your
> application, or from the repl, and see what happens.
>
> (ele::mark-and-sweep-gc *store-controller* &optional test)
>
> WARNING: this is a prototype and cannot be guaranteed not to lose
> important data.  Use only on a database you don't care about!
>
> When test is true, the collector only prints out all objects that it
> will reclaim, leaving a btree in ele::*mark-table* with the list of
> marked objects.
>
> LIVENESS
>
> The collector is not really very useful right now for objects other
> than btrees and psets as all persistent classes are written to a class
> index by default.  If you pass :index nil as a class initarg you will
> lose the ability to (get-instances-by-class), but any non-live objects
> in the store will be reclaimed.  You can either redefine the class to
> inhibit indexing or do it explicitly via: (setf (ele::%class-indexing
> class) nil).
>
> Liveness is identical to the notion of reachability used by migration
> with one exception, the store's instance cache is also treated as a
> root.  Thus all persistent objects reachable from the memory cache,
> root btree, and class indexes are considered live.  The mark pass
> recurses on both persistent aggregates (e.g. btree, pset) as well as
> standard aggregates (e.g. structures, standard classes, arrays, etc).
>
> You may want to run in-memory GC prior to running persistent GC to
> improve reclamation.  One potential issue: I have noticed that btrees
> that are stored in in-memory variables are dropped from the cache (a
> weak hash table) under SBCL so it's possible that persistent objects
> that are live in memory are dropped from the cache and can be
> reclaimed, leaving dangling references.  This may be a bug in weak
> pointers in the version of SBCL I'm using.
>
> PERFORMANCE / CONTENTION
>
> The mark pass runs all at once, but is predominantly doing reads which
> shouldn't result in much contention.  A fresh btree is used to keep
> track of marks.  For large DBs, this strategy involves long-lived
> transactions and may exhaust locks or cache memory and so a future
> revision should do the mark pass in a discrete series of steps (this
> means doing bookkeeping and checkpointing the recursive descent of the
> tree from the root which is somewhat odious).
>
> The sweep pass is done with one transaction per class to minimize
> contention; reclaiming storage can touch quite a few database pages.
>
> TURNKEY
>
> Later, when I have some time, I'll add an optional bordeaux-threads
> model so you can simply start the collector as part of open-store and
> put frequency and other parameters to control behavior into my-
> config.sexp.  For now you can simply manually start a thread that runs
> the mark-and-sweep-gc function to completion on a given store.
>
> Cheers,
> Ian
>
> PS - As a bonus for BDB users, you can call (optimize-layout store)
> after the gc which will attempt to defragment and optionally return
> free pages to the disk.
>
>
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel




From killerstorm at newmail.ru  Tue Jan  6 13:17:13 2009
From: killerstorm at newmail.ru (Alex Mizrahi)
Date: Tue, 6 Jan 2009 15:17:13 +0200
Subject: [elephant-devel] Querying for objects on two slots
References: <ecb948b30901052236t4762d164p492ffac56e040152@mail.gmail.com>
Message-ID: <gjvlkt$8hh$1@ger.gmane.org>

YK> Is this a reasonable way of finding an object of type
YK> 'my-class that matches on values val-a and val-b for slots a and b?

yep, it is reasonable if you have relatively low number of objects
in returned by (get-instances-by-value 'my-class 'slot-b val-b) query.
if number of objects is significant and you get a slowdown because
of this, you might want to optimize this. a trivial thing is to try it
symmertrically with slot-a -- whatever returns less objects is better.
less trivial optimizations would be to work on lower-level -- via
map-inverted-index (to avoid allocating whole list but instead test objects
one by one) or even cursor API (this way you can retrieve oids rather
than objects, which should be faster, and also iterating both indices at
once might be a significant benefit if values are not uniformly 
distributed).

but the most optimal way doing this in case of high number of objects
in both slot-a and slot-b queries would be building and using multi-column
index. unfortunately, Elephant does not help you with it -- either you'll 
have
to serialize slot tuples into a string (e.g (format nil "~a_~a" slot-a 
slot-b)),
or reorganize your data to use a custom index structure (like btree of 
btrees).

there are also backend-specific considerations. for postmodern

 (intersection (get-instances-by-value 'my-class 'slot-b val-b)
                    (get-instances-by-value 'my-class 'slot-a val-a))

would be much faster then testing objects one by one, for BDB -- i doubt so.

LP> Use MAP-CLASS, this will considerably speed up the query.

how is that? using at least one index is much better than using no index at 
all.






From sky at viridian-project.de  Tue Jan  6 19:27:21 2009
From: sky at viridian-project.de (Leslie P. Polzer)
Date: Tue, 6 Jan 2009 20:27:21 +0100
Subject: [elephant-devel] Querying for objects on two slots
In-Reply-To: <gjvlkt$8hh$1@ger.gmane.org>
References: <ecb948b30901052236t4762d164p492ffac56e040152@mail.gmail.com>
	<gjvlkt$8hh$1@ger.gmane.org>
Message-ID: <20090106192721.GA9455@viridian-project.de>

On Tue, Jan 06, 2009 at 03:17:13PM +0200, Alex Mizrahi wrote:
>
> LP> Use MAP-CLASS, this will considerably speed up the query.
> 
> how is that? using at least one index is much better than using no index at 
> all.

Indeed. I've been think about this a bit today after I wrote it and
MAP-INVERTED-INDEX is what I really meant.



From rsynnott at gmail.com  Wed Jan  7 06:23:26 2009
From: rsynnott at gmail.com (Robert Synnott)
Date: Wed, 7 Jan 2009 06:23:26 +0000
Subject: [elephant-devel] Removing indexes from existing classes
Message-ID: <24f203480901062223o138701bq69c52e57a25bb22b@mail.gmail.com>

Hi,
Quick question. If I have existing instances of a class with index on
slot X, and I wish to remove that index, is it sufficient simply to
remove the index in the class definition, or do I need to do something
else?
Thanks
Rob.



From eslick at media.mit.edu  Wed Jan  7 08:15:14 2009
From: eslick at media.mit.edu (Ian Eslick)
Date: Wed, 7 Jan 2009 03:15:14 -0500
Subject: [elephant-devel] Removing indexes from existing classes
In-Reply-To: <24f203480901062223o138701bq69c52e57a25bb22b@mail.gmail.com>
References: <24f203480901062223o138701bq69c52e57a25bb22b@mail.gmail.com>
Message-ID: <CC26B1A5-F689-42E6-9B52-B4FC92A5D3C9@media.mit.edu>

Re-evaluating the class definition is sufficient.  The database  
automatically synchronizes to the current class definition.

Beware!

When you remove a slot from the definition, or even rename it, the  
values associated with the slot are dropped from all instances (unless  
you've enabled lazy synchronization).  This is the same behavior as  
normal standard lisp class so it seemed a reasonable starting place.

We should probably make sure there is an optional warning enabled by  
default that warns at either redefinition time or at synchronization  
time and tells the user what data will be lost when synchronization  
occurs.

Ian



On Jan 7, 2009, at 1:23 AM, Robert Synnott wrote:

> Hi,
> Quick question. If I have existing instances of a class with index on
> slot X, and I wish to remove that index, is it sufficient simply to
> remove the index in the class definition, or do I need to do something
> else?
> Thanks
> Rob.
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel




From eslick at media.mit.edu  Wed Jan  7 16:30:42 2009
From: eslick at media.mit.edu (Ian Eslick)
Date: Wed, 7 Jan 2009 11:30:42 -0500
Subject: [elephant-devel] GC
Message-ID: <E2A020E2-E1FF-4EDD-B5FB-99C1C0287EC9@media.mit.edu>

Anyone interested in doing a code review of the GC?  I think i'm  
done.  It really helps to have a second (or third) pair of eyes when  
dealing with concurrency issues.

Ian 



From raison at chatsubo.net  Thu Jan  8 01:09:49 2009
From: raison at chatsubo.net (Kevin Raison)
Date: Wed, 07 Jan 2009 17:09:49 -0800
Subject: [elephant-devel] Strange behavior with indexed-btree
Message-ID: <4965525D.2010106@chatsubo.net>

I am seeing an intermittent error with 1.0 alpha when trying to write to 
  an indexed btree (using BerkeleyDB 4.7 as provided by Ubuntu's package 
repositories):

The slot DB-BDB::INDICES-CACHE is unbound in the object 
#<BDB-INDEXED-BTREE oid:2>

Within the same thread, sometimes this happens and sometimes I am able 
to read and write to the btree.  I am using sbcl 1.0.24 on 32 bit intel 
linux.  Is there something obvious that might help alleviate this or 
should I provide more context?

Thanks!
Kevin



From raison at chatsubo.net  Thu Jan  8 02:10:33 2009
From: raison at chatsubo.net (Kevin Raison)
Date: Wed, 07 Jan 2009 18:10:33 -0800
Subject: [elephant-devel] Strange behavior with indexed-btree
In-Reply-To: <4965525D.2010106@chatsubo.net>
References: <4965525D.2010106@chatsubo.net>
Message-ID: <49656099.6050708@chatsubo.net>

I was finally able to recreate this in the repl (as opposed to seeing it 
in my error logs), so here is a trace:

The slot DB-BDB::INDICES-CACHE is unbound in the object 
#<BDB-INDEXED-BTREE oid:2>.
    [Condition of type UNBOUND-SLOT]

Backtrace:
   0: ((SB-PCL::FAST-METHOD SLOT-UNBOUND (T T T)) #<unavailable 
argument> #<unavailable argument> #<unavailable argument> 
#<BDB-INDEXED-BTREE oid:2> DB-BDB::INDICES-CACHE)
   1: (SB-PCL::SLOT-UNBOUND-INTERNAL #<BDB-INDEXED-BTREE oid:2> 2)
   2: ((SB-PCL::FAST-METHOD (SETF GET-VALUE) (T T 
DB-BDB::BDB-INDEXED-BTREE)) #<unavailable lambda list>)
   3: ((LAMBDA (WORD)) "34766")
   4: (SB-IMPL::MAP1 #<CLOSURE (LAMBDA #) {1003CA4ED9}> (("" "asa" 
"106007" "deny" "inbound" "udp" ...)) :LIST T)
   5: (MAPCAR #<CLOSURE (LAMBDA #) {1003CA4ED9}> ("" "asa" "106007" 
"deny" "inbound" "udp" ...))[:EXTERNAL]
   6: ((SB-PCL::FAST-METHOD INDEX-LOG-ENTRY (LOG-ENTRY)) #<unavailable 
argument> #<unavailable argument> #<LOG-ENTRY oid:6373>)
   7: ((LAMBDA ()))
   8: ((SB-PCL::FAST-METHOD ELEPHANT::EXECUTE-TRANSACTION 
(DB-BDB::BDB-STORE-CONTROLLER T)) #<unavailable argument> #<unavailable 
argument> #<unavailable argument> #<unavailable argument>)[:EXTERNAL]
   9: (ADD-LOG-ENTRY ..)
  10: (SB-INT:SIMPLE-EVAL-IN-LEXENV ..)

And here is the code that sometimes causes the issue:
(defmethod index-log-entry ((log-entry log-entry))
   (let ((*store-controller* *syslog-controller*))
     (let ((btree (get-from-root "log-entry-index" :sc 
*syslog-controller*)))
       (map-words #'(lambda (word)
                      (unless (or (member word *superfluous-words* :test 
#'string-equal)
                                  (< (length word) 2))
                        (unless (existsp word btree)
                          (setf (get-value word btree) (make-pset :sc 
*syslog-controller*)))
                        (insert-item log-entry
                                     (get-value word btree))))
                  (text log-entry)))))


If I drop and recreate the btree, everything runs smoothly (no errors) 
for a few hours and then the above error message starts showing up about 
half to three quarters of the time.

Thanks again,
Kevin


Kevin Raison wrote:
> I am seeing an intermittent error with 1.0 alpha when trying to write to 
>   an indexed btree (using BerkeleyDB 4.7 as provided by Ubuntu's package 
> repositories):
> 
> The slot DB-BDB::INDICES-CACHE is unbound in the object 
> #<BDB-INDEXED-BTREE oid:2>
> 
> Within the same thread, sometimes this happens and sometimes I am able 
> to read and write to the btree.  I am using sbcl 1.0.24 on 32 bit intel 
> linux.  Is there something obvious that might help alleviate this or 
> should I provide more context?
> 
> Thanks!
> Kevin
> 
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel



From raison at chatsubo.net  Thu Jan  8 02:35:00 2009
From: raison at chatsubo.net (Kevin Raison)
Date: Wed, 07 Jan 2009 18:35:00 -0800
Subject: [elephant-devel] Strange behavior with indexed-btree
In-Reply-To: <49656099.6050708@chatsubo.net>
References: <4965525D.2010106@chatsubo.net> <49656099.6050708@chatsubo.net>
Message-ID: <49656654.7050902@chatsubo.net>

I apologize for replying to my own reply, but in the process of 
recreating this from a fresh bdb, i noticed another artifact of the 
error condition.  Before the unbound slot errors begin to appear, I get 
an ELEPHANT:ELEPHANT-TYPE-DESERIALIZATION-ERROR;  just once, and then 
the unbound slot errors start happening.  A trace of the deserialization 
error follows:

Condition ELEPHANT:ELEPHANT-TYPE-DESERIALIZATION-ERROR was signalled.
    [Condition of type ELEPHANT:ELEPHANT-TYPE-DESERIALIZATION-ERROR]

Restarts:
  0: [TERMINATE-THREAD] Terminate this thread (#<THREAD 
"pcall-worker-50" RUNNING {1003C18CE1}>)

Backtrace:
   0: ((LABELS ELEPHANT-SERIALIZER2::%DESERIALIZE) #<unavailable lambda 
list>)
   1: ((LABELS ELEPHANT-SERIALIZER2::%DESERIALIZE) #<unavailable lambda 
list>)
   2: (ELEPHANT-SERIALIZER2::DESERIALIZE #<unavailable lambda list>)
   3: (ELEPHANT::DESERIALIZE ..)
   4: ((SB-PCL::FAST-METHOD ELEPHANT::PERSISTENT-SLOT-READER 
(DB-BDB::BDB-STORE-CONTROLLER T T)) #<unavailable lambda list>)
   5: (SLOT-VALUE #<BDB-INDEXED-BTREE oid:7565> DB-BDB::INDICES)
   6: ((SB-PCL::FAST-METHOD SHARED-INITIALIZE :AFTER 
(DB-BDB::BDB-INDEXED-BTREE T)) #<unavailable lambda list>)[:EXTERNAL]
   7: ((LAMBDA (SB-PCL::.PV. SB-PCL::.NEXT-METHOD-CALL. SB-PCL::.ARG0. 
SB-PCL::.ARG1. SB-INT:&MORE SB-PCL::.DFUN-MORE-CONTEXT. 
SB-PCL::.DFUN-MORE-COUNT.)) ..)
   8: ((SB-PCL::FAST-METHOD SHARED-INITIALIZE :AROUND 
(ELEPHANT:PERSISTENT-OBJECT T)) #<unavailable argument> #<unavailable 
argument> #<unavailable argument> #<unavailable argument>)[:EXTERNAL]
   9: ((SB-PCL::FAST-METHOD ELEPHANT::RECREATE-INSTANCE 
(ELEPHANT:PERSISTENT-COLLECTION)) #<unavailable argument> #<unavailable 
argument> #<unavailable argument>)[:EXTERNAL]
  10: ((SB-PCL::FAST-METHOD ELEPHANT::CONTROLLER-RECREATE-INSTANCE 
(ELEPHANT:STORE-CONTROLLER T)) #<unused argument> #<unused argument> 
#<BDB-STORE-CONTROLLER /srv/console/syslog/> 7565 NIL)
  11: ((LABELS ELEPHANT-SERIALIZER2::%DESERIALIZE) #<unavailable lambda 
list>)
  12: (ELEPHANT-SERIALIZER2::DESERIALIZE #<unavailable lambda list>)
  13: (ELEPHANT::DESERIALIZE ..)
  14: ((SB-PCL::FAST-METHOD ELEPHANT:GET-VALUE (T DB-BDB::BDB-BTREE)) 
#<unavailable lambda list>)
  15: ((SB-PCL::FAST-METHOD CONSOLE::INDEX-LOG-ENTRY 
(CONSOLE::LOG-ENTRY)) #<unavailable argument> #<unavailable argument> 
#<LOG-ENTRY oid:10076>)
  16: ((LAMBDA ()))
  17: ((SB-PCL::FAST-METHOD ELEPHANT::EXECUTE-TRANSACTION 
(DB-BDB::BDB-STORE-CONTROLLER T)) #<unavailable argument> #<unavailable 
argument> #<unavailable argument> #<unavailable argument>)[:EXTERNAL]
  18: (CONSOLE::ADD-LOG-ENTRY ..)


Kevin Raison wrote:
> I was finally able to recreate this in the repl (as opposed to seeing it 
> in my error logs), so here is a trace:
> 
> The slot DB-BDB::INDICES-CACHE is unbound in the object 
> #<BDB-INDEXED-BTREE oid:2>.
>     [Condition of type UNBOUND-SLOT]
> 
> Backtrace:
>    0: ((SB-PCL::FAST-METHOD SLOT-UNBOUND (T T T)) #<unavailable 
> argument> #<unavailable argument> #<unavailable argument> 
> #<BDB-INDEXED-BTREE oid:2> DB-BDB::INDICES-CACHE)
>    1: (SB-PCL::SLOT-UNBOUND-INTERNAL #<BDB-INDEXED-BTREE oid:2> 2)
>    2: ((SB-PCL::FAST-METHOD (SETF GET-VALUE) (T T 
> DB-BDB::BDB-INDEXED-BTREE)) #<unavailable lambda list>)
>    3: ((LAMBDA (WORD)) "34766")
>    4: (SB-IMPL::MAP1 #<CLOSURE (LAMBDA #) {1003CA4ED9}> (("" "asa" 
> "106007" "deny" "inbound" "udp" ...)) :LIST T)
>    5: (MAPCAR #<CLOSURE (LAMBDA #) {1003CA4ED9}> ("" "asa" "106007" 
> "deny" "inbound" "udp" ...))[:EXTERNAL]
>    6: ((SB-PCL::FAST-METHOD INDEX-LOG-ENTRY (LOG-ENTRY)) #<unavailable 
> argument> #<unavailable argument> #<LOG-ENTRY oid:6373>)
>    7: ((LAMBDA ()))
>    8: ((SB-PCL::FAST-METHOD ELEPHANT::EXECUTE-TRANSACTION 
> (DB-BDB::BDB-STORE-CONTROLLER T)) #<unavailable argument> #<unavailable 
> argument> #<unavailable argument> #<unavailable argument>)[:EXTERNAL]
>    9: (ADD-LOG-ENTRY ..)
>   10: (SB-INT:SIMPLE-EVAL-IN-LEXENV ..)
> 
> And here is the code that sometimes causes the issue:
> (defmethod index-log-entry ((log-entry log-entry))
>    (let ((*store-controller* *syslog-controller*))
>      (let ((btree (get-from-root "log-entry-index" :sc 
> *syslog-controller*)))
>        (map-words #'(lambda (word)
>                       (unless (or (member word *superfluous-words* :test 
> #'string-equal)
>                                   (< (length word) 2))
>                         (unless (existsp word btree)
>                           (setf (get-value word btree) (make-pset :sc 
> *syslog-controller*)))
>                         (insert-item log-entry
>                                      (get-value word btree))))
>                   (text log-entry)))))
> 
> 
> If I drop and recreate the btree, everything runs smoothly (no errors) 
> for a few hours and then the above error message starts showing up about 
> half to three quarters of the time.
> 
> Thanks again,
> Kevin
> 
> 
> Kevin Raison wrote:
>> I am seeing an intermittent error with 1.0 alpha when trying to write to 
>>   an indexed btree (using BerkeleyDB 4.7 as provided by Ubuntu's package 
>> repositories):
>>
>> The slot DB-BDB::INDICES-CACHE is unbound in the object 
>> #<BDB-INDEXED-BTREE oid:2>
>>
>> Within the same thread, sometimes this happens and sometimes I am able 
>> to read and write to the btree.  I am using sbcl 1.0.24 on 32 bit intel 
>> linux.  Is there something obvious that might help alleviate this or 
>> should I provide more context?
>>
>> Thanks!
>> Kevin
>>
>> _______________________________________________
>> elephant-devel site list
>> elephant-devel at common-lisp.net
>> http://common-lisp.net/mailman/listinfo/elephant-devel
> 
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel



From eslick at media.mit.edu  Thu Jan  8 03:29:06 2009
From: eslick at media.mit.edu (Ian Eslick)
Date: Wed, 7 Jan 2009 22:29:06 -0500
Subject: [elephant-devel] Rollback in with-transaction
Message-ID: <C07350F2-A697-43A8-B780-48478294D35F@media.mit.edu>

One hole in with-transaction, at least for BDB, has been the fact that  
any non-local exit causes a transaction to fail.  I've now added a  
quick hack that catches all conditions and calls an optional :inhibit- 
rollback-fn to see if the condition is a normal part of success - if  
so, it is re-signaled after the commit.

I don't have a solution to throw, but it's a less common idiom.

Alex, what are you doing in the postmodern API?

Ian 
  



From elliottslaughter at gmail.com  Thu Jan  8 08:22:20 2009
From: elliottslaughter at gmail.com (Elliott Slaughter)
Date: Thu, 8 Jan 2009 00:22:20 -0800
Subject: [elephant-devel] Optimization
In-Reply-To: <79E0CB99-2CA4-4583-A750-AE0CBA4D5C3C@media.mit.edu>
References: <42c0ab790812212135w18d4cdcerb7af5b45a04486f0@mail.gmail.com>
	<20081223092255.GA17361@viridian-project.de>
	<786E3BB7-708F-461E-A6FB-3B392614C4FD@media.mit.edu>
	<42c0ab790812272305k78d8ef1atb4c78df5b9624ee3@mail.gmail.com>
	<CC3600A7-1B01-499B-B45D-D2958995EEDA@media.mit.edu>
	<79E0CB99-2CA4-4583-A750-AE0CBA4D5C3C@media.mit.edu>
Message-ID: <42c0ab790901080022s605eeafh31982acfc5805bd3@mail.gmail.com>

Hi,

On Mon, Dec 29, 2008 at 2:46 PM, Ian Eslick <eslick at media.mit.edu> wrote:

>
> I checked the cache-checkout model to elephant-unstable.
>

Thanks for your work on slot caching. I've been playing around with checkout
caching for the last week or so, and a couple of comments and critiques.

- On checkout, the cached slots of the checked-out object behaves
>
  like a standard built-in object, although with MOP overhead on
>
accesses (see below)
>
- Write operations are also cached
>
- The persistent-checkin operation write the current cached slot state
>
to the DB
>
- You can sync a currently checked out object to make its state
>
persistent
>
- Checkout can be cancelled
>

The checkout model seems just a little awkward to integrate with my code.
Since I am usually working with what is effectively a tree of objects, I
typically want to checkout an entire subtree at any given time. I already
have code which walks over the tree, but since the tree is heterogeneous,
and may not necessarily consist of just cacheable objects, a simple call to
persistent-checkout for each object won't do. The following works, but is a
hack based on the internals of cached-slots.lisp:

(when (eq (ele::%cache-style (class-of object)) :checkout)
  (ele:persistent-checkout object))

Also, there are times when I would like to instantiate and immediately
checkout an object. However, I am experiencing a strange bug where adding an
after method to initialize-instance results in a slot-unbound error later if
make-instance is called from inside with-transaction. Unfortunately, I
haven't been able to reproduce the bug in any self-contained test case (and
I can't say for sure that it isn't my error).

Another issue is that the cache-style declaration in persistent classes
doesn't get inherited by default. This particularly annoying since this
throws an error merely by inheriting cached slots from a superclass.

Finally, I believe (if I understand indexing correctly) that caching write
operations makes it impossible to have indexed cached slots. My most
expensive slot reads (after optimizing all other persistent slots) come from
indexed slots.

That said, I could probably avoid reading indexed slots if I used the query
language discussed on another thread. (Currently I use
get-instances-by-value to cons a list of objects, and then sort by the
indexed slot to effectively iterate over two sorted slots.)

 Standard: 25M/sec
> Cached: 7M/sec
> Persistent (within a transaction): 140k/sec
>
> Cached slot access is now only 4x slower than standard slot access
> Persistent slot access is about 50x slower than cached slot access


I get roughly the same results as you on the micro-benchmarks. When using
checkout caching in the game, I get about a 2x speed increase. It is still
about 3x slower than no database.

Thanks again.

-- 
Elliott Slaughter

"Any road followed precisely to its end leads precisely nowhere." - Frank
Herbert
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20090108/c4cfdc0d/attachment.html>

From elliottslaughter at gmail.com  Thu Jan  8 08:51:29 2009
From: elliottslaughter at gmail.com (Elliott Slaughter)
Date: Thu, 8 Jan 2009 00:51:29 -0800
Subject: [elephant-devel] Windows DLLs for 1.0 Alpha 1
Message-ID: <42c0ab790901080051u7a6c1e3sdcd8fca3903ea1cf@mail.gmail.com>

Hi,
The Windows DLLs linked on the downloads page are incompatible with the BDB
backend in 1.0 Alpha 1. I have compiled binaries for 1.0 Alpha 1, available
at

http://elliottslaughter.net/files/elephant-1-0-a1.zip

Compiled on Windows XP with GCC 3.4.4 and Oracle BDB 4.5.

Enjoy :-)

-- 
Elliott Slaughter

"Any road followed precisely to its end leads precisely nowhere." - Frank
Herbert
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20090108/af6f19e2/attachment.html>

From elliottslaughter at gmail.com  Thu Jan  8 09:14:09 2009
From: elliottslaughter at gmail.com (Elliott Slaughter)
Date: Thu, 8 Jan 2009 01:14:09 -0800
Subject: [elephant-devel] Windows/SBCL bug (AND FIX) when compiling DLLs
	(WAS: partial fix)
Message-ID: <42c0ab790901080114l4821f60fo8051b7ffca6d55b9@mail.gmail.com>

Hi,

On Mon, Dec 15, 2008 at 11:03 PM, Elliott Slaughter <
elliottslaughter at gmail.com> wrote:
>
> I just encountered a peculiarity with the perform method for
> elephant-c-source files, which causes a clean build of elephant-unstable on
> Windows/SBCL to fail.
>

Attached is a patch which fixes this problem in a better way, and safely
deals with different drive letters and paths with spaces.

Enjoy :-)

-- 
Elliott Slaughter

"Any road followed precisely to its end leads precisely nowhere." - Frank
Herbert
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20090108/697da862/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: better-support-for-__prebuilt_libraries-nil_-under-win32_sbcl_.dpatch
Type: application/octet-stream
Size: 22321 bytes
Desc: not available
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20090108/697da862/attachment.obj>

From eslick at media.mit.edu  Thu Jan  8 12:26:44 2009
From: eslick at media.mit.edu (Ian Eslick)
Date: Thu, 8 Jan 2009 07:26:44 -0500
Subject: [elephant-devel] Strange behavior with indexed-btree
In-Reply-To: <49656654.7050902@chatsubo.net>
References: <4965525D.2010106@chatsubo.net> <49656099.6050708@chatsubo.net>
	<49656654.7050902@chatsubo.net>
Message-ID: <F8E97E19-58E9-450C-AC03-03B6C69563A8@media.mit.edu>

Hi Kevin,

Thank you for the report; that's very odd behavior and I haven't seen  
that locally.  I'll look into this tonight.  Are you using a fresh  
repository or an pre-existing one that you upgraded to 1.0?

Ian

On Jan 7, 2009, at 9:35 PM, Kevin Raison wrote:

> I apologize for replying to my own reply, but in the process of
> recreating this from a fresh bdb, i noticed another artifact of the
> error condition.  Before the unbound slot errors begin to appear, I  
> get
> an ELEPHANT:ELEPHANT-TYPE-DESERIALIZATION-ERROR;  just once, and then
> the unbound slot errors start happening.  A trace of the  
> deserialization
> error follows:
>
> Condition ELEPHANT:ELEPHANT-TYPE-DESERIALIZATION-ERROR was signalled.
>    [Condition of type ELEPHANT:ELEPHANT-TYPE-DESERIALIZATION-ERROR]
>
> Restarts:
>  0: [TERMINATE-THREAD] Terminate this thread (#<THREAD
> "pcall-worker-50" RUNNING {1003C18CE1}>)
>
> Backtrace:
>   0: ((LABELS ELEPHANT-SERIALIZER2::%DESERIALIZE) #<unavailable lambda
> list>)
>   1: ((LABELS ELEPHANT-SERIALIZER2::%DESERIALIZE) #<unavailable lambda
> list>)
>   2: (ELEPHANT-SERIALIZER2::DESERIALIZE #<unavailable lambda list>)
>   3: (ELEPHANT::DESERIALIZE ..)
>   4: ((SB-PCL::FAST-METHOD ELEPHANT::PERSISTENT-SLOT-READER
> (DB-BDB::BDB-STORE-CONTROLLER T T)) #<unavailable lambda list>)
>   5: (SLOT-VALUE #<BDB-INDEXED-BTREE oid:7565> DB-BDB::INDICES)
>   6: ((SB-PCL::FAST-METHOD SHARED-INITIALIZE :AFTER
> (DB-BDB::BDB-INDEXED-BTREE T)) #<unavailable lambda list>)[:EXTERNAL]
>   7: ((LAMBDA (SB-PCL::.PV. SB-PCL::.NEXT-METHOD-CALL. SB-PCL::.ARG0.
> SB-PCL::.ARG1. SB-INT:&MORE SB-PCL::.DFUN-MORE-CONTEXT.
> SB-PCL::.DFUN-MORE-COUNT.)) ..)
>   8: ((SB-PCL::FAST-METHOD SHARED-INITIALIZE :AROUND
> (ELEPHANT:PERSISTENT-OBJECT T)) #<unavailable argument> #<unavailable
> argument> #<unavailable argument> #<unavailable argument>)[:EXTERNAL]
>   9: ((SB-PCL::FAST-METHOD ELEPHANT::RECREATE-INSTANCE
> (ELEPHANT:PERSISTENT-COLLECTION)) #<unavailable argument>  
> #<unavailable
> argument> #<unavailable argument>)[:EXTERNAL]
>  10: ((SB-PCL::FAST-METHOD ELEPHANT::CONTROLLER-RECREATE-INSTANCE
> (ELEPHANT:STORE-CONTROLLER T)) #<unused argument> #<unused argument>
> #<BDB-STORE-CONTROLLER /srv/console/syslog/> 7565 NIL)
>  11: ((LABELS ELEPHANT-SERIALIZER2::%DESERIALIZE) #<unavailable lambda
> list>)
>  12: (ELEPHANT-SERIALIZER2::DESERIALIZE #<unavailable lambda list>)
>  13: (ELEPHANT::DESERIALIZE ..)
>  14: ((SB-PCL::FAST-METHOD ELEPHANT:GET-VALUE (T DB-BDB::BDB-BTREE))
> #<unavailable lambda list>)
>  15: ((SB-PCL::FAST-METHOD CONSOLE::INDEX-LOG-ENTRY
> (CONSOLE::LOG-ENTRY)) #<unavailable argument> #<unavailable argument>
> #<LOG-ENTRY oid:10076>)
>  16: ((LAMBDA ()))
>  17: ((SB-PCL::FAST-METHOD ELEPHANT::EXECUTE-TRANSACTION
> (DB-BDB::BDB-STORE-CONTROLLER T)) #<unavailable argument>  
> #<unavailable
> argument> #<unavailable argument> #<unavailable argument>)[:EXTERNAL]
>  18: (CONSOLE::ADD-LOG-ENTRY ..)
>
>
> Kevin Raison wrote:
>> I was finally able to recreate this in the repl (as opposed to  
>> seeing it
>> in my error logs), so here is a trace:
>>
>> The slot DB-BDB::INDICES-CACHE is unbound in the object
>> #<BDB-INDEXED-BTREE oid:2>.
>>    [Condition of type UNBOUND-SLOT]
>>
>> Backtrace:
>>   0: ((SB-PCL::FAST-METHOD SLOT-UNBOUND (T T T)) #<unavailable
>> argument> #<unavailable argument> #<unavailable argument>
>> #<BDB-INDEXED-BTREE oid:2> DB-BDB::INDICES-CACHE)
>>   1: (SB-PCL::SLOT-UNBOUND-INTERNAL #<BDB-INDEXED-BTREE oid:2> 2)
>>   2: ((SB-PCL::FAST-METHOD (SETF GET-VALUE) (T T
>> DB-BDB::BDB-INDEXED-BTREE)) #<unavailable lambda list>)
>>   3: ((LAMBDA (WORD)) "34766")
>>   4: (SB-IMPL::MAP1 #<CLOSURE (LAMBDA #) {1003CA4ED9}> (("" "asa"
>> "106007" "deny" "inbound" "udp" ...)) :LIST T)
>>   5: (MAPCAR #<CLOSURE (LAMBDA #) {1003CA4ED9}> ("" "asa" "106007"
>> "deny" "inbound" "udp" ...))[:EXTERNAL]
>>   6: ((SB-PCL::FAST-METHOD INDEX-LOG-ENTRY (LOG-ENTRY)) #<unavailable
>> argument> #<unavailable argument> #<LOG-ENTRY oid:6373>)
>>   7: ((LAMBDA ()))
>>   8: ((SB-PCL::FAST-METHOD ELEPHANT::EXECUTE-TRANSACTION
>> (DB-BDB::BDB-STORE-CONTROLLER T)) #<unavailable argument>  
>> #<unavailable
>> argument> #<unavailable argument> #<unavailable argument>)[:EXTERNAL]
>>   9: (ADD-LOG-ENTRY ..)
>>  10: (SB-INT:SIMPLE-EVAL-IN-LEXENV ..)
>>
>> And here is the code that sometimes causes the issue:
>> (defmethod index-log-entry ((log-entry log-entry))
>>   (let ((*store-controller* *syslog-controller*))
>>     (let ((btree (get-from-root "log-entry-index" :sc
>> *syslog-controller*)))
>>       (map-words #'(lambda (word)
>>                      (unless (or (member word *superfluous- 
>> words* :test
>> #'string-equal)
>>                                  (< (length word) 2))
>>                        (unless (existsp word btree)
>>                          (setf (get-value word btree) (make-pset :sc
>> *syslog-controller*)))
>>                        (insert-item log-entry
>>                                     (get-value word btree))))
>>                  (text log-entry)))))
>>
>>
>> If I drop and recreate the btree, everything runs smoothly (no  
>> errors)
>> for a few hours and then the above error message starts showing up  
>> about
>> half to three quarters of the time.
>>
>> Thanks again,
>> Kevin
>>
>>
>> Kevin Raison wrote:
>>> I am seeing an intermittent error with 1.0 alpha when trying to  
>>> write to
>>>  an indexed btree (using BerkeleyDB 4.7 as provided by Ubuntu's  
>>> package
>>> repositories):
>>>
>>> The slot DB-BDB::INDICES-CACHE is unbound in the object
>>> #<BDB-INDEXED-BTREE oid:2>
>>>
>>> Within the same thread, sometimes this happens and sometimes I am  
>>> able
>>> to read and write to the btree.  I am using sbcl 1.0.24 on 32 bit  
>>> intel
>>> linux.  Is there something obvious that might help alleviate this or
>>> should I provide more context?
>>>
>>> Thanks!
>>> Kevin
>>>
>>> _______________________________________________
>>> elephant-devel site list
>>> elephant-devel at common-lisp.net
>>> http://common-lisp.net/mailman/listinfo/elephant-devel
>>
>> _______________________________________________
>> elephant-devel site list
>> elephant-devel at common-lisp.net
>> http://common-lisp.net/mailman/listinfo/elephant-devel
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel




From eslick at media.mit.edu  Thu Jan  8 12:26:55 2009
From: eslick at media.mit.edu (Ian Eslick)
Date: Thu, 8 Jan 2009 07:26:55 -0500
Subject: [elephant-devel] Strange behavior with indexed-btree
In-Reply-To: <4965525D.2010106@chatsubo.net>
References: <4965525D.2010106@chatsubo.net>
Message-ID: <695AB8A8-E38A-46E1-AD2F-09C4B237F5BD@media.mit.edu>

PS - Does the test suite pass on your system?

On Jan 7, 2009, at 8:09 PM, Kevin Raison wrote:

> I am seeing an intermittent error with 1.0 alpha when trying to  
> write to
>  an indexed btree (using BerkeleyDB 4.7 as provided by Ubuntu's  
> package
> repositories):
>
> The slot DB-BDB::INDICES-CACHE is unbound in the object
> #<BDB-INDEXED-BTREE oid:2>
>
> Within the same thread, sometimes this happens and sometimes I am able
> to read and write to the btree.  I am using sbcl 1.0.24 on 32 bit  
> intel
> linux.  Is there something obvious that might help alleviate this or
> should I provide more context?
>
> Thanks!
> Kevin
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel




From eslick at media.mit.edu  Thu Jan  8 12:38:13 2009
From: eslick at media.mit.edu (Ian Eslick)
Date: Thu, 8 Jan 2009 07:38:13 -0500
Subject: [elephant-devel] Optimization
In-Reply-To: <42c0ab790901080022s605eeafh31982acfc5805bd3@mail.gmail.com>
References: <42c0ab790812212135w18d4cdcerb7af5b45a04486f0@mail.gmail.com>
	<20081223092255.GA17361@viridian-project.de>
	<786E3BB7-708F-461E-A6FB-3B392614C4FD@media.mit.edu>
	<42c0ab790812272305k78d8ef1atb4c78df5b9624ee3@mail.gmail.com>
	<CC3600A7-1B01-499B-B45D-D2958995EEDA@media.mit.edu>
	<79E0CB99-2CA4-4583-A750-AE0CBA4D5C3C@media.mit.edu>
	<42c0ab790901080022s605eeafh31982acfc5805bd3@mail.gmail.com>
Message-ID: <FC9EF081-C034-4867-B541-7AF88DB2C851@media.mit.edu>

Great feedback!  Thank you.  It's always great to have different use  
cases.

Comments below.

On Jan 8, 2009, at 3:22 AM, Elliott Slaughter wrote:

> Hi,
>
> On Mon, Dec 29, 2008 at 2:46 PM, Ian Eslick <eslick at media.mit.edu>  
> wrote:
>
> I checked the cache-checkout model to elephant-unstable.
>
> Thanks for your work on slot caching. I've been playing around with  
> checkout caching for the last week or so, and a couple of comments  
> and critiques.
>
> - On checkout, the cached slots of the checked-out object behaves
>   like a standard built-in object, although with MOP overhead on
> accesses (see below)
> - Write operations are also cached
> - The persistent-checkin operation write the current cached slot state
> to the DB
> - You can sync a currently checked out object to make its state
> persistent
> - Checkout can be cancelled

> The checkout model seems just a little awkward to integrate with my  
> code. Since I am usually working with what is effectively a tree of  
> objects, I typically want to checkout an entire subtree at any given  
> time. I already have code which walks over the tree, but since the  
> tree is heterogeneous, and may not necessarily consist of just  
> cacheable objects, a simple call to persistent-checkout for each  
> object won't do. The following works, but is a hack based on the  
> internals of cached-slots.lisp:
>
> (when (eq (ele::%cache-style (class-of object)) :checkout)
>   (ele:persistent-checkout object))

We could provide an optional argument to checkout that silently  
succeeds.

> Also, there are times when I would like to instantiate and  
> immediately checkout an object. However, I am experiencing a strange  
> bug where adding an after method to initialize-instance results in a  
> slot-unbound error later if make-instance is called from inside with- 
> transaction. Unfortunately, I haven't been able to reproduce the bug  
> in any self-contained test case (and I can't say for sure that it  
> isn't my error).

I'll do a quick code review tonight and see if something is obvious.

> Another issue is that the cache-style declaration in persistent  
> classes doesn't get inherited by default. This particularly annoying  
> since this throws an error merely by inheriting cached slots from a  
> superclass.

Hmmmm...it should inherit by default unless you are overriding the  
base class slot.  I'll look into this too.

> Finally, I believe (if I understand indexing correctly) that caching  
> write operations makes it impossible to have indexed cached slots.  
> My most expensive slot reads (after optimizing all other persistent  
> slots) come from indexed slots.

That is correct.  We could make a 'write-through' mode, but then we'd  
need 'cached-index' slots as a special case and I'm trying to avoid a  
combinatorial explosion in slot types.

> That said, I could probably avoid reading indexed slots if I used  
> the query language discussed on another thread. (Currently I use get- 
> instances-by-value to cons a list of objects, and then sort by the  
> indexed slot to effectively iterate over two sorted slots.)

The query system would basically do this:

(let ((a (map-inverted-index #'identity 'slot1 :collect t :oids t))
       (b (map-inverted-index #'identity 'slot2 :collect t :oids t)))
   (mapcar #'elephant::controller-recreate-instance
   	  (merge-unique (sort a #'<) (sort b #'<))

I need to export recreate...

You can also do map-inverted-index and accumulate only values matching  
the second value as discussed earlier using an accumulation closure as  
an argument.  For small sets on the first slot, this is probably  
faster.  In general mapping is always more efficient than get- 
instances...


> Standard: 25M/sec
> Cached: 7M/sec
> Persistent (within a transaction): 140k/sec
>
> Cached slot access is now only 4x slower than standard slot access
> Persistent slot access is about 50x slower than cached slot access
>
> I get roughly the same results as you on the micro-benchmarks. When  
> using checkout caching in the game, I get about a 2x speed increase.  
> It is still about 3x slower than no database.

It would be interesting to see some profile data on this to see what,  
if anything, in BDB/Elephant could be improved.

> Thanks again.
>
> -- 
> Elliott Slaughter
>
> "Any road followed precisely to its end leads precisely nowhere." -  
> Frank Herbert
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel




From eslick at media.mit.edu  Thu Jan  8 13:02:20 2009
From: eslick at media.mit.edu (Ian Eslick)
Date: Thu, 8 Jan 2009 08:02:20 -0500
Subject: [elephant-devel] Optimization
In-Reply-To: <FC9EF081-C034-4867-B541-7AF88DB2C851@media.mit.edu>
References: <42c0ab790812212135w18d4cdcerb7af5b45a04486f0@mail.gmail.com>
	<20081223092255.GA17361@viridian-project.de>
	<786E3BB7-708F-461E-A6FB-3B392614C4FD@media.mit.edu>
	<42c0ab790812272305k78d8ef1atb4c78df5b9624ee3@mail.gmail.com>
	<CC3600A7-1B01-499B-B45D-D2958995EEDA@media.mit.edu>
	<79E0CB99-2CA4-4583-A750-AE0CBA4D5C3C@media.mit.edu>
	<42c0ab790901080022s605eeafh31982acfc5805bd3@mail.gmail.com>
	<FC9EF081-C034-4867-B541-7AF88DB2C851@media.mit.edu>
Message-ID: <DC82280E-73A9-4559-959D-6E32E0E4FEF7@media.mit.edu>

By the way, a write through mode is complex because if a transaction  
is rolled back your index could differ from your memory state so you  
would then have to keep track if slot writes to undo them on rollback...

Are you caching blocks of objects for short term operations or long  
term ones?

Sent from my iPhone

On Jan 8, 2009, at 7:38 AM, Ian Eslick <eslick at media.mit.edu> wrote:

> Great feedback!  Thank you.  It's always great to have different use
> cases.
>
> Comments below.
>
> On Jan 8, 2009, at 3:22 AM, Elliott Slaughter wrote:
>
>> Hi,
>>
>> On Mon, Dec 29, 2008 at 2:46 PM, Ian Eslick <eslick at media.mit.edu>
>> wrote:
>>
>> I checked the cache-checkout model to elephant-unstable.
>>
>> Thanks for your work on slot caching. I've been playing around with
>> checkout caching for the last week or so, and a couple of comments
>> and critiques.
>>
>> - On checkout, the cached slots of the checked-out object behaves
>>  like a standard built-in object, although with MOP overhead on
>> accesses (see below)
>> - Write operations are also cached
>> - The persistent-checkin operation write the current cached slot  
>> state
>> to the DB
>> - You can sync a currently checked out object to make its state
>> persistent
>> - Checkout can be cancelled
>
>> The checkout model seems just a little awkward to integrate with my
>> code. Since I am usually working with what is effectively a tree of
>> objects, I typically want to checkout an entire subtree at any given
>> time. I already have code which walks over the tree, but since the
>> tree is heterogeneous, and may not necessarily consist of just
>> cacheable objects, a simple call to persistent-checkout for each
>> object won't do. The following works, but is a hack based on the
>> internals of cached-slots.lisp:
>>
>> (when (eq (ele::%cache-style (class-of object)) :checkout)
>>  (ele:persistent-checkout object))
>
> We could provide an optional argument to checkout that silently
> succeeds.
>
>> Also, there are times when I would like to instantiate and
>> immediately checkout an object. However, I am experiencing a strange
>> bug where adding an after method to initialize-instance results in a
>> slot-unbound error later if make-instance is called from inside with-
>> transaction. Unfortunately, I haven't been able to reproduce the bug
>> in any self-contained test case (and I can't say for sure that it
>> isn't my error).
>
> I'll do a quick code review tonight and see if something is obvious.
>
>> Another issue is that the cache-style declaration in persistent
>> classes doesn't get inherited by default. This particularly annoying
>> since this throws an error merely by inheriting cached slots from a
>> superclass.
>
> Hmmmm...it should inherit by default unless you are overriding the
> base class slot.  I'll look into this too.
>
>> Finally, I believe (if I understand indexing correctly) that caching
>> write operations makes it impossible to have indexed cached slots.
>> My most expensive slot reads (after optimizing all other persistent
>> slots) come from indexed slots.
>
> That is correct.  We could make a 'write-through' mode, but then we'd
> need 'cached-index' slots as a special case and I'm trying to avoid a
> combinatorial explosion in slot types.
>
>> That said, I could probably avoid reading indexed slots if I used
>> the query language discussed on another thread. (Currently I use get-
>> instances-by-value to cons a list of objects, and then sort by the
>> indexed slot to effectively iterate over two sorted slots.)
>
> The query system would basically do this:
>
> (let ((a (map-inverted-index #'identity 'slot1 :collect t :oids t))
>       (b (map-inverted-index #'identity 'slot2 :collect t :oids t)))
>   (mapcar #'elephant::controller-recreate-instance
>         (merge-unique (sort a #'<) (sort b #'<))
>
> I need to export recreate...
>
> You can also do map-inverted-index and accumulate only values matching
> the second value as discussed earlier using an accumulation closure as
> an argument.  For small sets on the first slot, this is probably
> faster.  In general mapping is always more efficient than get-
> instances...
>
>
>> Standard: 25M/sec
>> Cached: 7M/sec
>> Persistent (within a transaction): 140k/sec
>>
>> Cached slot access is now only 4x slower than standard slot access
>> Persistent slot access is about 50x slower than cached slot access
>>
>> I get roughly the same results as you on the micro-benchmarks. When
>> using checkout caching in the game, I get about a 2x speed increase.
>> It is still about 3x slower than no database.
>
> It would be interesting to see some profile data on this to see what,
> if anything, in BDB/Elephant could be improved.
>
>> Thanks again.
>>
>> -- 
>> Elliott Slaughter
>>
>> "Any road followed precisely to its end leads precisely nowhere." -
>> Frank Herbert
>> _______________________________________________
>> elephant-devel site list
>> elephant-devel at common-lisp.net
>> http://common-lisp.net/mailman/listinfo/elephant-devel
>
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel



From eslick at media.mit.edu  Thu Jan  8 13:04:03 2009
From: eslick at media.mit.edu (Ian Eslick)
Date: Thu, 8 Jan 2009 08:04:03 -0500
Subject: [elephant-devel] Strange behavior with indexed-btree
In-Reply-To: <49656654.7050902@chatsubo.net>
References: <4965525D.2010106@chatsubo.net> <49656099.6050708@chatsubo.net>
	<49656654.7050902@chatsubo.net>
Message-ID: <2D292D6D-7019-43D3-906E-1D8380B7A5C0@media.mit.edu>

Sorry, this indicate a fresh DB.  Which version of 1.0 are you using -  
the alpha tag or the latest head?

Sent from my iPhone

On Jan 7, 2009, at 9:35 PM, Kevin Raison <raison at chatsubo.net> wrote:

> I apologize for replying to my own reply, but in the process of
> recreating this from a fresh bdb, i noticed another artifact of the
> error condition.  Before the unbound slot errors begin to appear, I  
> get
> an ELEPHANT:ELEPHANT-TYPE-DESERIALIZATION-ERROR;  just once, and then
> the unbound slot errors start happening.  A trace of the  
> deserialization
> error follows:
>
> Condition ELEPHANT:ELEPHANT-TYPE-DESERIALIZATION-ERROR was signalled.
>    [Condition of type ELEPHANT:ELEPHANT-TYPE-DESERIALIZATION-ERROR]
>
> Restarts:
>  0: [TERMINATE-THREAD] Terminate this thread (#<THREAD
> "pcall-worker-50" RUNNING {1003C18CE1}>)
>
> Backtrace:
>   0: ((LABELS ELEPHANT-SERIALIZER2::%DESERIALIZE) #<unavailable lambda
> list>)
>   1: ((LABELS ELEPHANT-SERIALIZER2::%DESERIALIZE) #<unavailable lambda
> list>)
>   2: (ELEPHANT-SERIALIZER2::DESERIALIZE #<unavailable lambda list>)
>   3: (ELEPHANT::DESERIALIZE ..)
>   4: ((SB-PCL::FAST-METHOD ELEPHANT::PERSISTENT-SLOT-READER
> (DB-BDB::BDB-STORE-CONTROLLER T T)) #<unavailable lambda list>)
>   5: (SLOT-VALUE #<BDB-INDEXED-BTREE oid:7565> DB-BDB::INDICES)
>   6: ((SB-PCL::FAST-METHOD SHARED-INITIALIZE :AFTER
> (DB-BDB::BDB-INDEXED-BTREE T)) #<unavailable lambda list>)[:EXTERNAL]
>   7: ((LAMBDA (SB-PCL::.PV. SB-PCL::.NEXT-METHOD-CALL. SB-PCL::.ARG0.
> SB-PCL::.ARG1. SB-INT:&MORE SB-PCL::.DFUN-MORE-CONTEXT.
> SB-PCL::.DFUN-MORE-COUNT.)) ..)
>   8: ((SB-PCL::FAST-METHOD SHARED-INITIALIZE :AROUND
> (ELEPHANT:PERSISTENT-OBJECT T)) #<unavailable argument> #<unavailable
> argument> #<unavailable argument> #<unavailable argument>)[:EXTERNAL]
>   9: ((SB-PCL::FAST-METHOD ELEPHANT::RECREATE-INSTANCE
> (ELEPHANT:PERSISTENT-COLLECTION)) #<unavailable argument>  
> #<unavailable
> argument> #<unavailable argument>)[:EXTERNAL]
>  10: ((SB-PCL::FAST-METHOD ELEPHANT::CONTROLLER-RECREATE-INSTANCE
> (ELEPHANT:STORE-CONTROLLER T)) #<unused argument> #<unused argument>
> #<BDB-STORE-CONTROLLER /srv/console/syslog/> 7565 NIL)
>  11: ((LABELS ELEPHANT-SERIALIZER2::%DESERIALIZE) #<unavailable lambda
> list>)
>  12: (ELEPHANT-SERIALIZER2::DESERIALIZE #<unavailable lambda list>)
>  13: (ELEPHANT::DESERIALIZE ..)
>  14: ((SB-PCL::FAST-METHOD ELEPHANT:GET-VALUE (T DB-BDB::BDB-BTREE))
> #<unavailable lambda list>)
>  15: ((SB-PCL::FAST-METHOD CONSOLE::INDEX-LOG-ENTRY
> (CONSOLE::LOG-ENTRY)) #<unavailable argument> #<unavailable argument>
> #<LOG-ENTRY oid:10076>)
>  16: ((LAMBDA ()))
>  17: ((SB-PCL::FAST-METHOD ELEPHANT::EXECUTE-TRANSACTION
> (DB-BDB::BDB-STORE-CONTROLLER T)) #<unavailable argument>  
> #<unavailable
> argument> #<unavailable argument> #<unavailable argument>)[:EXTERNAL]
>  18: (CONSOLE::ADD-LOG-ENTRY ..)
>
>
> Kevin Raison wrote:
>> I was finally able to recreate this in the repl (as opposed to  
>> seeing it
>> in my error logs), so here is a trace:
>>
>> The slot DB-BDB::INDICES-CACHE is unbound in the object
>> #<BDB-INDEXED-BTREE oid:2>.
>>    [Condition of type UNBOUND-SLOT]
>>
>> Backtrace:
>>   0: ((SB-PCL::FAST-METHOD SLOT-UNBOUND (T T T)) #<unavailable
>> argument> #<unavailable argument> #<unavailable argument>
>> #<BDB-INDEXED-BTREE oid:2> DB-BDB::INDICES-CACHE)
>>   1: (SB-PCL::SLOT-UNBOUND-INTERNAL #<BDB-INDEXED-BTREE oid:2> 2)
>>   2: ((SB-PCL::FAST-METHOD (SETF GET-VALUE) (T T
>> DB-BDB::BDB-INDEXED-BTREE)) #<unavailable lambda list>)
>>   3: ((LAMBDA (WORD)) "34766")
>>   4: (SB-IMPL::MAP1 #<CLOSURE (LAMBDA #) {1003CA4ED9}> (("" "asa"
>> "106007" "deny" "inbound" "udp" ...)) :LIST T)
>>   5: (MAPCAR #<CLOSURE (LAMBDA #) {1003CA4ED9}> ("" "asa" "106007"
>> "deny" "inbound" "udp" ...))[:EXTERNAL]
>>   6: ((SB-PCL::FAST-METHOD INDEX-LOG-ENTRY (LOG-ENTRY)) #<unavailable
>> argument> #<unavailable argument> #<LOG-ENTRY oid:6373>)
>>   7: ((LAMBDA ()))
>>   8: ((SB-PCL::FAST-METHOD ELEPHANT::EXECUTE-TRANSACTION
>> (DB-BDB::BDB-STORE-CONTROLLER T)) #<unavailable argument>  
>> #<unavailable
>> argument> #<unavailable argument> #<unavailable argument>)[:EXTERNAL]
>>   9: (ADD-LOG-ENTRY ..)
>>  10: (SB-INT:SIMPLE-EVAL-IN-LEXENV ..)
>>
>> And here is the code that sometimes causes the issue:
>> (defmethod index-log-entry ((log-entry log-entry))
>>   (let ((*store-controller* *syslog-controller*))
>>     (let ((btree (get-from-root "log-entry-index" :sc
>> *syslog-controller*)))
>>       (map-words #'(lambda (word)
>>                      (unless (or (member word *superfluous- 
>> words* :test
>> #'string-equal)
>>                                  (< (length word) 2))
>>                        (unless (existsp word btree)
>>                          (setf (get-value word btree) (make-pset :sc
>> *syslog-controller*)))
>>                        (insert-item log-entry
>>                                     (get-value word btree))))
>>                  (text log-entry)))))
>>
>>
>> If I drop and recreate the btree, everything runs smoothly (no  
>> errors)
>> for a few hours and then the above error message starts showing up  
>> about
>> half to three quarters of the time.
>>
>> Thanks again,
>> Kevin
>>
>>
>> Kevin Raison wrote:
>>> I am seeing an intermittent error with 1.0 alpha when trying to  
>>> write to
>>>  an indexed btree (using BerkeleyDB 4.7 as provided by Ubuntu's  
>>> package
>>> repositories):
>>>
>>> The slot DB-BDB::INDICES-CACHE is unbound in the object
>>> #<BDB-INDEXED-BTREE oid:2>
>>>
>>> Within the same thread, sometimes this happens and sometimes I am  
>>> able
>>> to read and write to the btree.  I am using sbcl 1.0.24 on 32 bit  
>>> intel
>>> linux.  Is there something obvious that might help alleviate this or
>>> should I provide more context?
>>>
>>> Thanks!
>>> Kevin
>>>
>>> _______________________________________________
>>> elephant-devel site list
>>> elephant-devel at common-lisp.net
>>> http://common-lisp.net/mailman/listinfo/elephant-devel
>>
>> _______________________________________________
>> elephant-devel site list
>> elephant-devel at common-lisp.net
>> http://common-lisp.net/mailman/listinfo/elephant-devel
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel



From eslick at media.mit.edu  Thu Jan  8 13:52:32 2009
From: eslick at media.mit.edu (Ian Eslick)
Date: Thu, 8 Jan 2009 08:52:32 -0500
Subject: [elephant-devel] Alternative
Message-ID: <9AEB0D39-8F16-44F8-96BC-07B4FF1FA16C@media.mit.edu>

Elliott,

There is a cached-set hack in contrib/eslick that allows you to save a  
tree of standard objects and load them back in so you can do  
completely in-memory ops but easily snapshot them to disk.  You can  
build an in-memory index (e.g. A red-black tree - there is a library  
somewhere with an implementation) to store them in and just pass that  
data structure as the root and it will snapshot all connected elements.

Sent from my iPhone



From raison at chatsubo.net  Thu Jan  8 16:48:48 2009
From: raison at chatsubo.net (Kevin Raison)
Date: Thu, 08 Jan 2009 08:48:48 -0800
Subject: [elephant-devel] Strange behavior with indexed-btree
In-Reply-To: <2D292D6D-7019-43D3-906E-1D8380B7A5C0@media.mit.edu>
References: <4965525D.2010106@chatsubo.net>
	<49656099.6050708@chatsubo.net>	<49656654.7050902@chatsubo.net>
	<2D292D6D-7019-43D3-906E-1D8380B7A5C0@media.mit.edu>
Message-ID: <49662E70.5000601@chatsubo.net>

I am seeing this issue with both the alpha tag and the latest head.

Ian Eslick wrote:
> Sorry, this indicate a fresh DB.  Which version of 1.0 are you using -  
> the alpha tag or the latest head?
> 
> Sent from my iPhone
> 
> On Jan 7, 2009, at 9:35 PM, Kevin Raison <raison at chatsubo.net> wrote:
> 
>> I apologize for replying to my own reply, but in the process of
>> recreating this from a fresh bdb, i noticed another artifact of the
>> error condition.  Before the unbound slot errors begin to appear, I  
>> get
>> an ELEPHANT:ELEPHANT-TYPE-DESERIALIZATION-ERROR;  just once, and then
>> the unbound slot errors start happening.  A trace of the  
>> deserialization
>> error follows:
>>
>> Condition ELEPHANT:ELEPHANT-TYPE-DESERIALIZATION-ERROR was signalled.
>>    [Condition of type ELEPHANT:ELEPHANT-TYPE-DESERIALIZATION-ERROR]
>>
>> Restarts:
>>  0: [TERMINATE-THREAD] Terminate this thread (#<THREAD
>> "pcall-worker-50" RUNNING {1003C18CE1}>)
>>
>> Backtrace:
>>   0: ((LABELS ELEPHANT-SERIALIZER2::%DESERIALIZE) #<unavailable lambda
>> list>)
>>   1: ((LABELS ELEPHANT-SERIALIZER2::%DESERIALIZE) #<unavailable lambda
>> list>)
>>   2: (ELEPHANT-SERIALIZER2::DESERIALIZE #<unavailable lambda list>)
>>   3: (ELEPHANT::DESERIALIZE ..)
>>   4: ((SB-PCL::FAST-METHOD ELEPHANT::PERSISTENT-SLOT-READER
>> (DB-BDB::BDB-STORE-CONTROLLER T T)) #<unavailable lambda list>)
>>   5: (SLOT-VALUE #<BDB-INDEXED-BTREE oid:7565> DB-BDB::INDICES)
>>   6: ((SB-PCL::FAST-METHOD SHARED-INITIALIZE :AFTER
>> (DB-BDB::BDB-INDEXED-BTREE T)) #<unavailable lambda list>)[:EXTERNAL]
>>   7: ((LAMBDA (SB-PCL::.PV. SB-PCL::.NEXT-METHOD-CALL. SB-PCL::.ARG0.
>> SB-PCL::.ARG1. SB-INT:&MORE SB-PCL::.DFUN-MORE-CONTEXT.
>> SB-PCL::.DFUN-MORE-COUNT.)) ..)
>>   8: ((SB-PCL::FAST-METHOD SHARED-INITIALIZE :AROUND
>> (ELEPHANT:PERSISTENT-OBJECT T)) #<unavailable argument> #<unavailable
>> argument> #<unavailable argument> #<unavailable argument>)[:EXTERNAL]
>>   9: ((SB-PCL::FAST-METHOD ELEPHANT::RECREATE-INSTANCE
>> (ELEPHANT:PERSISTENT-COLLECTION)) #<unavailable argument>  
>> #<unavailable
>> argument> #<unavailable argument>)[:EXTERNAL]
>>  10: ((SB-PCL::FAST-METHOD ELEPHANT::CONTROLLER-RECREATE-INSTANCE
>> (ELEPHANT:STORE-CONTROLLER T)) #<unused argument> #<unused argument>
>> #<BDB-STORE-CONTROLLER /srv/console/syslog/> 7565 NIL)
>>  11: ((LABELS ELEPHANT-SERIALIZER2::%DESERIALIZE) #<unavailable lambda
>> list>)
>>  12: (ELEPHANT-SERIALIZER2::DESERIALIZE #<unavailable lambda list>)
>>  13: (ELEPHANT::DESERIALIZE ..)
>>  14: ((SB-PCL::FAST-METHOD ELEPHANT:GET-VALUE (T DB-BDB::BDB-BTREE))
>> #<unavailable lambda list>)
>>  15: ((SB-PCL::FAST-METHOD CONSOLE::INDEX-LOG-ENTRY
>> (CONSOLE::LOG-ENTRY)) #<unavailable argument> #<unavailable argument>
>> #<LOG-ENTRY oid:10076>)
>>  16: ((LAMBDA ()))
>>  17: ((SB-PCL::FAST-METHOD ELEPHANT::EXECUTE-TRANSACTION
>> (DB-BDB::BDB-STORE-CONTROLLER T)) #<unavailable argument>  
>> #<unavailable
>> argument> #<unavailable argument> #<unavailable argument>)[:EXTERNAL]
>>  18: (CONSOLE::ADD-LOG-ENTRY ..)
>>
>>
>> Kevin Raison wrote:
>>> I was finally able to recreate this in the repl (as opposed to  
>>> seeing it
>>> in my error logs), so here is a trace:
>>>
>>> The slot DB-BDB::INDICES-CACHE is unbound in the object
>>> #<BDB-INDEXED-BTREE oid:2>.
>>>    [Condition of type UNBOUND-SLOT]
>>>
>>> Backtrace:
>>>   0: ((SB-PCL::FAST-METHOD SLOT-UNBOUND (T T T)) #<unavailable
>>> argument> #<unavailable argument> #<unavailable argument>
>>> #<BDB-INDEXED-BTREE oid:2> DB-BDB::INDICES-CACHE)
>>>   1: (SB-PCL::SLOT-UNBOUND-INTERNAL #<BDB-INDEXED-BTREE oid:2> 2)
>>>   2: ((SB-PCL::FAST-METHOD (SETF GET-VALUE) (T T
>>> DB-BDB::BDB-INDEXED-BTREE)) #<unavailable lambda list>)
>>>   3: ((LAMBDA (WORD)) "34766")
>>>   4: (SB-IMPL::MAP1 #<CLOSURE (LAMBDA #) {1003CA4ED9}> (("" "asa"
>>> "106007" "deny" "inbound" "udp" ...)) :LIST T)
>>>   5: (MAPCAR #<CLOSURE (LAMBDA #) {1003CA4ED9}> ("" "asa" "106007"
>>> "deny" "inbound" "udp" ...))[:EXTERNAL]
>>>   6: ((SB-PCL::FAST-METHOD INDEX-LOG-ENTRY (LOG-ENTRY)) #<unavailable
>>> argument> #<unavailable argument> #<LOG-ENTRY oid:6373>)
>>>   7: ((LAMBDA ()))
>>>   8: ((SB-PCL::FAST-METHOD ELEPHANT::EXECUTE-TRANSACTION
>>> (DB-BDB::BDB-STORE-CONTROLLER T)) #<unavailable argument>  
>>> #<unavailable
>>> argument> #<unavailable argument> #<unavailable argument>)[:EXTERNAL]
>>>   9: (ADD-LOG-ENTRY ..)
>>>  10: (SB-INT:SIMPLE-EVAL-IN-LEXENV ..)
>>>
>>> And here is the code that sometimes causes the issue:
>>> (defmethod index-log-entry ((log-entry log-entry))
>>>   (let ((*store-controller* *syslog-controller*))
>>>     (let ((btree (get-from-root "log-entry-index" :sc
>>> *syslog-controller*)))
>>>       (map-words #'(lambda (word)
>>>                      (unless (or (member word *superfluous- 
>>> words* :test
>>> #'string-equal)
>>>                                  (< (length word) 2))
>>>                        (unless (existsp word btree)
>>>                          (setf (get-value word btree) (make-pset :sc
>>> *syslog-controller*)))
>>>                        (insert-item log-entry
>>>                                     (get-value word btree))))
>>>                  (text log-entry)))))
>>>
>>>
>>> If I drop and recreate the btree, everything runs smoothly (no  
>>> errors)
>>> for a few hours and then the above error message starts showing up  
>>> about
>>> half to three quarters of the time.
>>>
>>> Thanks again,
>>> Kevin
>>>
>>>
>>> Kevin Raison wrote:
>>>> I am seeing an intermittent error with 1.0 alpha when trying to  
>>>> write to
>>>>  an indexed btree (using BerkeleyDB 4.7 as provided by Ubuntu's  
>>>> package
>>>> repositories):
>>>>
>>>> The slot DB-BDB::INDICES-CACHE is unbound in the object
>>>> #<BDB-INDEXED-BTREE oid:2>
>>>>
>>>> Within the same thread, sometimes this happens and sometimes I am  
>>>> able
>>>> to read and write to the btree.  I am using sbcl 1.0.24 on 32 bit  
>>>> intel
>>>> linux.  Is there something obvious that might help alleviate this or
>>>> should I provide more context?
>>>>
>>>> Thanks!
>>>> Kevin
>>>>
>>>> _______________________________________________
>>>> elephant-devel site list
>>>> elephant-devel at common-lisp.net
>>>> http://common-lisp.net/mailman/listinfo/elephant-devel
>>> _______________________________________________
>>> elephant-devel site list
>>> elephant-devel at common-lisp.net
>>> http://common-lisp.net/mailman/listinfo/elephant-devel
>> _______________________________________________
>> elephant-devel site list
>> elephant-devel at common-lisp.net
>> http://common-lisp.net/mailman/listinfo/elephant-devel
> 
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel



From raison at chatsubo.net  Thu Jan  8 17:14:22 2009
From: raison at chatsubo.net (Kevin Raison)
Date: Thu, 08 Jan 2009 09:14:22 -0800
Subject: [elephant-devel] Strange behavior with indexed-btree
In-Reply-To: <695AB8A8-E38A-46E1-AD2F-09C4B237F5BD@media.mit.edu>
References: <4965525D.2010106@chatsubo.net>
	<695AB8A8-E38A-46E1-AD2F-09C4B237F5BD@media.mit.edu>
Message-ID: <4966346E.6030700@chatsubo.net>

The berkeleydb tests pass 100%.

Ian Eslick wrote:
> PS - Does the test suite pass on your system?
> 
> On Jan 7, 2009, at 8:09 PM, Kevin Raison wrote:
> 
>> I am seeing an intermittent error with 1.0 alpha when trying to  
>> write to
>>  an indexed btree (using BerkeleyDB 4.7 as provided by Ubuntu's  
>> package
>> repositories):
>>
>> The slot DB-BDB::INDICES-CACHE is unbound in the object
>> #<BDB-INDEXED-BTREE oid:2>
>>
>> Within the same thread, sometimes this happens and sometimes I am able
>> to read and write to the btree.  I am using sbcl 1.0.24 on 32 bit  
>> intel
>> linux.  Is there something obvious that might help alleviate this or
>> should I provide more context?
>>
>> Thanks!
>> Kevin
>>
>> _______________________________________________
>> elephant-devel site list
>> elephant-devel at common-lisp.net
>> http://common-lisp.net/mailman/listinfo/elephant-devel
> 
> 
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel



From yarek.kowalik at gmail.com  Fri Jan  9 00:31:39 2009
From: yarek.kowalik at gmail.com (Yarek Kowalik)
Date: Thu, 8 Jan 2009 16:31:39 -0800
Subject: [elephant-devel] Good practices around adding/deleting class slots.
Message-ID: <ecb948b30901081631j7007aebeye174f51f5e925681@mail.gmail.com>

In an evolving site there is a fair chance of adding/changing/deleting slots
from a class.  What are good practices with respect to these actions in
Elephant?  Are there specific steps that need to be taken to migrate data
from an old class definition to a new class definition?  Any gotchas one
needs to be aware of?

BTW, I've read several emails about data migration going by, can someone
please clarify what exactly "data migration" is?  Is it migrating between
stores, or different versions of a class?

Yarek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20090108/d1897f9a/attachment.html>

From yarek.kowalik at gmail.com  Fri Jan  9 01:20:54 2009
From: yarek.kowalik at gmail.com (Yarek Kowalik)
Date: Thu, 8 Jan 2009 17:20:54 -0800
Subject: [elephant-devel] Mirroring classes from 3rd party packages.
Message-ID: <ecb948b30901081720n5786c70fpeb4237d4c43a7f50@mail.gmail.com>

I have a dilemma deciding on the structure on persiting data.  I need your
advice on.

I'm relying on a third party package (Amazon-ECS) that helps me parse Amazon
XML data feed, with the result that the XML is translated into a set of
nested CLOS objects (all classes use metaclass ELEMENT-CLASS).   The XML
documents are represented by a root object, and there subparts of the XML by
the nested objects accessible via root object slots, and so on
recursively.   It's actually a small tree structure of various types of
objecs at each level.   The data within this structure is valuable for
current and future needs for the project and I would like to cache it
instead of fetching from Amazon (there is a limit on how much you can fetch
at one time, plus the volume would make it prohitive).

For the purpose of the application, currently I only need a small subset of
the data.   As a solution I thought that perhaps can I just create a facade
object that will contain the basic data slots I need plus an
OBJECT-REFERENCE field that refers to XMLRoot (and the objects hidden
within)? When this facade object is persisted, will the entire XMLRoot with
its descendends (objects refering to objects either through slots or lists
in slots) be persisted?  That is, when an object is serialized, is the
serialization recursive?

The second option is to build a set of persistend mirror objects for the XML
objects I get and copy the data onto it.  This probably can be automated
somehow.

The third option is to modify the package Amazon ECS parsing package to use
persistent objects by default.  It's problematic, because now I will
effectively create a branch of the package and maintain it separately from
the core.  I would not want to persist all of the classes, so I would have
to be selective - it isn't simply making Amazon-ECS' ELEMENT-CLASS an
Elephant's PERSISTENT-CLASS.

Any clarifications and advice would be much appreciated.

Thanks,

Yarek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20090108/029a0af0/attachment.html>

From eslick at media.mit.edu  Fri Jan  9 03:16:07 2009
From: eslick at media.mit.edu (Ian Eslick)
Date: Thu, 8 Jan 2009 22:16:07 -0500
Subject: [elephant-devel] Mirroring classes from 3rd party packages.
In-Reply-To: <ecb948b30901081720n5786c70fpeb4237d4c43a7f50@mail.gmail.com>
References: <ecb948b30901081720n5786c70fpeb4237d4c43a7f50@mail.gmail.com>
Message-ID: <84B4D57D-42CE-4407-9BAB-EA0AB103BC45@media.mit.edu>

Yes, serialization is recursive and keeps track of references so it  
deals with circular and lattice-like references.  This is all  
serialized into a single buffer, but for moderately sized, self- 
contained trees you're probably OK.  Remember that when you load that  
value back in it serializes fresh standard objects for each node in  
your tree.  Also, if your tree is too big you can blow out the  
serializer buffer space (but that's many kilobytes of data).

While it might be a little heavy weight for you, a good way to save a  
complex data structure that you want to load/store to elephant is to  
use my snapshot-set hack in src/contrib/eslick/snapshot-set.lisp  
(which I mistakenly called cached-set in an email in the last day or  
so).  I use it as an example in the manual.  For example, you could  
subclass it for your metadata object and store your tree to its root...

An intro e-mail has some more details:
http://www.mail-archive.com/elephant-devel at common-lisp.net/msg00537.html

Ian


On Jan 8, 2009, at 8:20 PM, Yarek Kowalik wrote:

> I have a dilemma deciding on the structure on persiting data.  I  
> need your advice on.
>
> I'm relying on a third party package (Amazon-ECS) that helps me  
> parse Amazon XML data feed, with the result that the XML is  
> translated into a set of nested CLOS objects (all classes use  
> metaclass ELEMENT-CLASS).   The XML documents are represented by a  
> root object, and there subparts of the XML by the nested objects  
> accessible via root object slots, and so on recursively.   It's  
> actually a small tree structure of various types of objecs at each  
> level.   The data within this structure is valuable for current and  
> future needs for the project and I would like to cache it instead of  
> fetching from Amazon (there is a limit on how much you can fetch at  
> one time, plus the volume would make it prohitive).
>
> For the purpose of the application, currently I only need a small  
> subset of the data.   As a solution I thought that perhaps can I  
> just create a facade object that will contain the basic data slots I  
> need plus an OBJECT-REFERENCE field that refers to XMLRoot (and the  
> objects hidden within)? When this facade object is persisted, will  
> the entire XMLRoot with its descendends (objects refering to objects  
> either through slots or lists in slots) be persisted?  That is, when  
> an object is serialized, is the serialization recursive?
>
> The second option is to build a set of persistend mirror objects for  
> the XML objects I get and copy the data onto it.  This probably can  
> be automated somehow.
>
> The third option is to modify the package Amazon ECS parsing package  
> to use persistent objects by default.  It's problematic, because now  
> I will effectively create a branch of the package and maintain it  
> separately from the core.  I would not want to persist all of the  
> classes, so I would have to be selective - it isn't simply making  
> Amazon-ECS' ELEMENT-CLASS an Elephant's PERSISTENT-CLASS.
>
> Any clarifications and advice would be much appreciated.
>
> Thanks,
>
> Yarek
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel




From eslick at media.mit.edu  Fri Jan  9 03:21:59 2009
From: eslick at media.mit.edu (Ian Eslick)
Date: Thu, 8 Jan 2009 22:21:59 -0500
Subject: [elephant-devel] Good practices around adding/deleting class
	slots.
In-Reply-To: <ecb948b30901081631j7007aebeye174f51f5e925681@mail.gmail.com>
References: <ecb948b30901081631j7007aebeye174f51f5e925681@mail.gmail.com>
Message-ID: <E2C68711-DB18-4F62-AF96-B411F04D4566@media.mit.edu>

Hi Yarek,

There are no problems adding or deleting slots.  The semantics are  
basically the same as redefining slots for standard classes.  You can  
change the type of a slot (e.g. cached, index, etc) without any data  
loss.  If you change the name or drop a slot the data associated with  
that is lost.

Currently I don't have a mechanism to handle the general case along  
the lines of  update-instance-for-redefined-class that will work with  
objects on the persistent heap (not that there isn't a way to do this,  
I just haven't done it yet).

For now, the fool-proof method is to only add slots, copy or modify  
any data, then delete them.  This is probably a good discipline anyway  
if you're making schema changes against live databases.

I'll also be adding some default "are you sure you want to throw away  
all your data?" messages into Elephant at appropriate places for 1.0.   
This will help you be conscious about when you are likely to be  
throwing data away.

Ian

On Jan 8, 2009, at 7:31 PM, Yarek Kowalik wrote:

> In an evolving site there is a fair chance of adding/changing/ 
> deleting slots from a class.  What are good practices with respect  
> to these actions in Elephant?  Are there specific steps that need to  
> be taken to migrate data from an old class definition to a new class  
> definition?  Any gotchas one needs to be aware of?
>
> BTW, I've read several emails about data migration going by, can  
> someone please clarify what exactly "data migration" is?  Is it  
> migrating between stores, or different versions of a class?
>
> Yarek
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel




From eslick at media.mit.edu  Fri Jan  9 03:59:56 2009
From: eslick at media.mit.edu (Ian Eslick)
Date: Thu, 8 Jan 2009 22:59:56 -0500
Subject: [elephant-devel] Strange behavior with indexed-btree
In-Reply-To: <49656099.6050708@chatsubo.net>
References: <4965525D.2010106@chatsubo.net> <49656099.6050708@chatsubo.net>
Message-ID: <C526FC8C-952F-48F8-BE57-1606443BBD01@media.mit.edu>

Hi Kevin,

I have some theories, but no conclusion yet.  I did revert two very  
small changes which could have been responsible.  Please rebuild and  
see if you can reproduce.

Thank you,
Ian

On Jan 7, 2009, at 9:10 PM, Kevin Raison wrote:

> I was finally able to recreate this in the repl (as opposed to  
> seeing it
> in my error logs), so here is a trace:
>
> The slot DB-BDB::INDICES-CACHE is unbound in the object
> #<BDB-INDEXED-BTREE oid:2>.
>    [Condition of type UNBOUND-SLOT]
>
> Backtrace:
>   0: ((SB-PCL::FAST-METHOD SLOT-UNBOUND (T T T)) #<unavailable
> argument> #<unavailable argument> #<unavailable argument>
> #<BDB-INDEXED-BTREE oid:2> DB-BDB::INDICES-CACHE)
>   1: (SB-PCL::SLOT-UNBOUND-INTERNAL #<BDB-INDEXED-BTREE oid:2> 2)
>   2: ((SB-PCL::FAST-METHOD (SETF GET-VALUE) (T T
> DB-BDB::BDB-INDEXED-BTREE)) #<unavailable lambda list>)
>   3: ((LAMBDA (WORD)) "34766")
>   4: (SB-IMPL::MAP1 #<CLOSURE (LAMBDA #) {1003CA4ED9}> (("" "asa"
> "106007" "deny" "inbound" "udp" ...)) :LIST T)
>   5: (MAPCAR #<CLOSURE (LAMBDA #) {1003CA4ED9}> ("" "asa" "106007"
> "deny" "inbound" "udp" ...))[:EXTERNAL]
>   6: ((SB-PCL::FAST-METHOD INDEX-LOG-ENTRY (LOG-ENTRY)) #<unavailable
> argument> #<unavailable argument> #<LOG-ENTRY oid:6373>)
>   7: ((LAMBDA ()))
>   8: ((SB-PCL::FAST-METHOD ELEPHANT::EXECUTE-TRANSACTION
> (DB-BDB::BDB-STORE-CONTROLLER T)) #<unavailable argument>  
> #<unavailable
> argument> #<unavailable argument> #<unavailable argument>)[:EXTERNAL]
>   9: (ADD-LOG-ENTRY ..)
>  10: (SB-INT:SIMPLE-EVAL-IN-LEXENV ..)
>
> And here is the code that sometimes causes the issue:
> (defmethod index-log-entry ((log-entry log-entry))
>   (let ((*store-controller* *syslog-controller*))
>     (let ((btree (get-from-root "log-entry-index" :sc
> *syslog-controller*)))
>       (map-words #'(lambda (word)
>                      (unless (or (member word *superfluous- 
> words* :test
> #'string-equal)
>                                  (< (length word) 2))
>                        (unless (existsp word btree)
>                          (setf (get-value word btree) (make-pset :sc
> *syslog-controller*)))
>                        (insert-item log-entry
>                                     (get-value word btree))))
>                  (text log-entry)))))
>
>
> If I drop and recreate the btree, everything runs smoothly (no errors)
> for a few hours and then the above error message starts showing up  
> about
> half to three quarters of the time.
>
> Thanks again,
> Kevin
>
>
> Kevin Raison wrote:
>> I am seeing an intermittent error with 1.0 alpha when trying to  
>> write to
>>  an indexed btree (using BerkeleyDB 4.7 as provided by Ubuntu's  
>> package
>> repositories):
>>
>> The slot DB-BDB::INDICES-CACHE is unbound in the object
>> #<BDB-INDEXED-BTREE oid:2>
>>
>> Within the same thread, sometimes this happens and sometimes I am  
>> able
>> to read and write to the btree.  I am using sbcl 1.0.24 on 32 bit  
>> intel
>> linux.  Is there something obvious that might help alleviate this or
>> should I provide more context?
>>
>> Thanks!
>> Kevin
>>
>> _______________________________________________
>> elephant-devel site list
>> elephant-devel at common-lisp.net
>> http://common-lisp.net/mailman/listinfo/elephant-devel
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel




From eslick at media.mit.edu  Fri Jan  9 04:01:50 2009
From: eslick at media.mit.edu (Ian Eslick)
Date: Thu, 8 Jan 2009 23:01:50 -0500
Subject: [elephant-devel] Strange behavior with indexed-btree
In-Reply-To: <49656099.6050708@chatsubo.net>
References: <4965525D.2010106@chatsubo.net> <49656099.6050708@chatsubo.net>
Message-ID: <AF76A669-036C-4597-9FB1-3DEA1C39DCB2@media.mit.edu>

Also try pushing :elephant-without-optimize onto *features* prior to  
doing a force rebuild of elephant.  This should give you some more  
information and rule out any optimization/declaration related bugs.

Thank you,
Ian


On Jan 7, 2009, at 9:10 PM, Kevin Raison wrote:

> I was finally able to recreate this in the repl (as opposed to  
> seeing it
> in my error logs), so here is a trace:
>
> The slot DB-BDB::INDICES-CACHE is unbound in the object
> #<BDB-INDEXED-BTREE oid:2>.
>    [Condition of type UNBOUND-SLOT]
>
> Backtrace:
>   0: ((SB-PCL::FAST-METHOD SLOT-UNBOUND (T T T)) #<unavailable
> argument> #<unavailable argument> #<unavailable argument>
> #<BDB-INDEXED-BTREE oid:2> DB-BDB::INDICES-CACHE)
>   1: (SB-PCL::SLOT-UNBOUND-INTERNAL #<BDB-INDEXED-BTREE oid:2> 2)
>   2: ((SB-PCL::FAST-METHOD (SETF GET-VALUE) (T T
> DB-BDB::BDB-INDEXED-BTREE)) #<unavailable lambda list>)
>   3: ((LAMBDA (WORD)) "34766")
>   4: (SB-IMPL::MAP1 #<CLOSURE (LAMBDA #) {1003CA4ED9}> (("" "asa"
> "106007" "deny" "inbound" "udp" ...)) :LIST T)
>   5: (MAPCAR #<CLOSURE (LAMBDA #) {1003CA4ED9}> ("" "asa" "106007"
> "deny" "inbound" "udp" ...))[:EXTERNAL]
>   6: ((SB-PCL::FAST-METHOD INDEX-LOG-ENTRY (LOG-ENTRY)) #<unavailable
> argument> #<unavailable argument> #<LOG-ENTRY oid:6373>)
>   7: ((LAMBDA ()))
>   8: ((SB-PCL::FAST-METHOD ELEPHANT::EXECUTE-TRANSACTION
> (DB-BDB::BDB-STORE-CONTROLLER T)) #<unavailable argument>  
> #<unavailable
> argument> #<unavailable argument> #<unavailable argument>)[:EXTERNAL]
>   9: (ADD-LOG-ENTRY ..)
>  10: (SB-INT:SIMPLE-EVAL-IN-LEXENV ..)
>
> And here is the code that sometimes causes the issue:
> (defmethod index-log-entry ((log-entry log-entry))
>   (let ((*store-controller* *syslog-controller*))
>     (let ((btree (get-from-root "log-entry-index" :sc
> *syslog-controller*)))
>       (map-words #'(lambda (word)
>                      (unless (or (member word *superfluous- 
> words* :test
> #'string-equal)
>                                  (< (length word) 2))
>                        (unless (existsp word btree)
>                          (setf (get-value word btree) (make-pset :sc
> *syslog-controller*)))
>                        (insert-item log-entry
>                                     (get-value word btree))))
>                  (text log-entry)))))
>
>
> If I drop and recreate the btree, everything runs smoothly (no errors)
> for a few hours and then the above error message starts showing up  
> about
> half to three quarters of the time.
>
> Thanks again,
> Kevin
>
>
> Kevin Raison wrote:
>> I am seeing an intermittent error with 1.0 alpha when trying to  
>> write to
>>  an indexed btree (using BerkeleyDB 4.7 as provided by Ubuntu's  
>> package
>> repositories):
>>
>> The slot DB-BDB::INDICES-CACHE is unbound in the object
>> #<BDB-INDEXED-BTREE oid:2>
>>
>> Within the same thread, sometimes this happens and sometimes I am  
>> able
>> to read and write to the btree.  I am using sbcl 1.0.24 on 32 bit  
>> intel
>> linux.  Is there something obvious that might help alleviate this or
>> should I provide more context?
>>
>> Thanks!
>> Kevin
>>
>> _______________________________________________
>> elephant-devel site list
>> elephant-devel at common-lisp.net
>> http://common-lisp.net/mailman/listinfo/elephant-devel
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel




From raison at chatsubo.net  Fri Jan  9 17:45:31 2009
From: raison at chatsubo.net (Kevin Raison)
Date: Fri, 09 Jan 2009 09:45:31 -0800
Subject: [elephant-devel] Strange behavior with indexed-btree
In-Reply-To: <AF76A669-036C-4597-9FB1-3DEA1C39DCB2@media.mit.edu>
References: <4965525D.2010106@chatsubo.net> <49656099.6050708@chatsubo.net>
	<AF76A669-036C-4597-9FB1-3DEA1C39DCB2@media.mit.edu>
Message-ID: <49678D3B.4050106@chatsubo.net>

Ian, I pulled all new patches from darcs and did a rebuild as you 
suggested.  BDB test are now failing:


  Did 491 checks.
     Pass: 483 (98%)
     Skip: 0 ( 0%)
     Fail: 8 ( 1%)

  Failure Details:
  --------------------------------
  INDEXING-CHANGE-CLASS []:
       1 evaluated to 1, which is not = to 2..
  --------------------------------
  --------------------------------
  INDEXING-CHANGE-CLASS []:
       1 evaluated to 1, which is not = to 2..
  --------------------------------
  --------------------------------
  INDEXING-HIERARCHY []:
       (POP #:RESULT147) evaluated to (1 1 2 2 3 3), which is not EQUALP 
to (1 2 3)..
  --------------------------------
  --------------------------------
  INDEXING-HIERARCHY []:
       (POP #:RESULT147) evaluated to (1 1 2 2 3 3), which is not EQUALP 
to (1 2 3)..
  --------------------------------
  --------------------------------
  INDEXING-HIERARCHY []:
       (POP #:RESULT147) evaluated to (1 2 4 1 2 4), which is not EQUALP 
to (1 2 4)..
  --------------------------------
  --------------------------------
  INDEXING-HIERARCHY []:
       (POP #:RESULT147) evaluated to (2 3 2 3), which is not EQUALP to 
(2 3)..
  --------------------------------
  --------------------------------
  INDEXING-HIERARCHY []:
       (POP #:RESULT147) evaluated to (4 4), which is not EQUALP to (4)..
  --------------------------------
  --------------------------------
  INHIBIT-ROLLBACK []:
       Unexpected Error: #<IT.BESE.FIVEAM::TEST-FAILURE {1004A633D1}>
#<TEST-FAILURE {1004A633D1}>..
  --------------------------------

I will let you know later today if I can reproduce my deserialization 
errors.

-Kevin


Ian Eslick wrote:
> Also try pushing :elephant-without-optimize onto *features* prior to  
> doing a force rebuild of elephant.  This should give you some more  
> information and rule out any optimization/declaration related bugs.
> 
> Thank you,
> Ian
> 
> 
> On Jan 7, 2009, at 9:10 PM, Kevin Raison wrote:
> 
>> I was finally able to recreate this in the repl (as opposed to  
>> seeing it
>> in my error logs), so here is a trace:
>>
>> The slot DB-BDB::INDICES-CACHE is unbound in the object
>> #<BDB-INDEXED-BTREE oid:2>.
>>    [Condition of type UNBOUND-SLOT]
>>
>> Backtrace:
>>   0: ((SB-PCL::FAST-METHOD SLOT-UNBOUND (T T T)) #<unavailable
>> argument> #<unavailable argument> #<unavailable argument>
>> #<BDB-INDEXED-BTREE oid:2> DB-BDB::INDICES-CACHE)
>>   1: (SB-PCL::SLOT-UNBOUND-INTERNAL #<BDB-INDEXED-BTREE oid:2> 2)
>>   2: ((SB-PCL::FAST-METHOD (SETF GET-VALUE) (T T
>> DB-BDB::BDB-INDEXED-BTREE)) #<unavailable lambda list>)
>>   3: ((LAMBDA (WORD)) "34766")
>>   4: (SB-IMPL::MAP1 #<CLOSURE (LAMBDA #) {1003CA4ED9}> (("" "asa"
>> "106007" "deny" "inbound" "udp" ...)) :LIST T)
>>   5: (MAPCAR #<CLOSURE (LAMBDA #) {1003CA4ED9}> ("" "asa" "106007"
>> "deny" "inbound" "udp" ...))[:EXTERNAL]
>>   6: ((SB-PCL::FAST-METHOD INDEX-LOG-ENTRY (LOG-ENTRY)) #<unavailable
>> argument> #<unavailable argument> #<LOG-ENTRY oid:6373>)
>>   7: ((LAMBDA ()))
>>   8: ((SB-PCL::FAST-METHOD ELEPHANT::EXECUTE-TRANSACTION
>> (DB-BDB::BDB-STORE-CONTROLLER T)) #<unavailable argument>  
>> #<unavailable
>> argument> #<unavailable argument> #<unavailable argument>)[:EXTERNAL]
>>   9: (ADD-LOG-ENTRY ..)
>>  10: (SB-INT:SIMPLE-EVAL-IN-LEXENV ..)
>>
>> And here is the code that sometimes causes the issue:
>> (defmethod index-log-entry ((log-entry log-entry))
>>   (let ((*store-controller* *syslog-controller*))
>>     (let ((btree (get-from-root "log-entry-index" :sc
>> *syslog-controller*)))
>>       (map-words #'(lambda (word)
>>                      (unless (or (member word *superfluous- 
>> words* :test
>> #'string-equal)
>>                                  (< (length word) 2))
>>                        (unless (existsp word btree)
>>                          (setf (get-value word btree) (make-pset :sc
>> *syslog-controller*)))
>>                        (insert-item log-entry
>>                                     (get-value word btree))))
>>                  (text log-entry)))))
>>
>>
>> If I drop and recreate the btree, everything runs smoothly (no errors)
>> for a few hours and then the above error message starts showing up  
>> about
>> half to three quarters of the time.
>>
>> Thanks again,
>> Kevin
>>
>>
>> Kevin Raison wrote:
>>> I am seeing an intermittent error with 1.0 alpha when trying to  
>>> write to
>>>  an indexed btree (using BerkeleyDB 4.7 as provided by Ubuntu's  
>>> package
>>> repositories):
>>>
>>> The slot DB-BDB::INDICES-CACHE is unbound in the object
>>> #<BDB-INDEXED-BTREE oid:2>
>>>
>>> Within the same thread, sometimes this happens and sometimes I am  
>>> able
>>> to read and write to the btree.  I am using sbcl 1.0.24 on 32 bit  
>>> intel
>>> linux.  Is there something obvious that might help alleviate this or
>>> should I provide more context?
>>>
>>> Thanks!
>>> Kevin
>>>
>>> _______________________________________________
>>> elephant-devel site list
>>> elephant-devel at common-lisp.net
>>> http://common-lisp.net/mailman/listinfo/elephant-devel
>> _______________________________________________
>> elephant-devel site list
>> elephant-devel at common-lisp.net
>> http://common-lisp.net/mailman/listinfo/elephant-devel
> 
> 
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel



From eslick at media.mit.edu  Sat Jan 10 00:54:22 2009
From: eslick at media.mit.edu (Ian Eslick)
Date: Fri, 9 Jan 2009 19:54:22 -0500
Subject: [elephant-devel] Strange behavior with indexed-btree
In-Reply-To: <49678D3B.4050106@chatsubo.net>
References: <4965525D.2010106@chatsubo.net> <49656099.6050708@chatsubo.net>
	<AF76A669-036C-4597-9FB1-3DEA1C39DCB2@media.mit.edu>
	<49678D3B.4050106@chatsubo.net>
Message-ID: <B6EC4042-9832-4A3F-8DA2-43097C3E852B@media.mit.edu>

Sounds like there is prior state in the db.  Try running delscript.sh  
in the tests directory to clear out all the prior state.

I'll check in some further fixed tomorrow that may address the prior  
problem you had.

Sent from my iPhone

On Jan 9, 2009, at 12:45 PM, Kevin Raison <raison at chatsubo.net> wrote:

> Ian, I pulled all new patches from darcs and did a rebuild as you
> suggested.  BDB test are now failing:
>
>
>  Did 491 checks.
>     Pass: 483 (98%)
>     Skip: 0 ( 0%)
>     Fail: 8 ( 1%)
>
>  Failure Details:
>  --------------------------------
>  INDEXING-CHANGE-CLASS []:
>       1 evaluated to 1, which is not = to 2..
>  --------------------------------
>  --------------------------------
>  INDEXING-CHANGE-CLASS []:
>       1 evaluated to 1, which is not = to 2..
>  --------------------------------
>  --------------------------------
>  INDEXING-HIERARCHY []:
>       (POP #:RESULT147) evaluated to (1 1 2 2 3 3), which is not  
> EQUALP
> to (1 2 3)..
>  --------------------------------
>  --------------------------------
>  INDEXING-HIERARCHY []:
>       (POP #:RESULT147) evaluated to (1 1 2 2 3 3), which is not  
> EQUALP
> to (1 2 3)..
>  --------------------------------
>  --------------------------------
>  INDEXING-HIERARCHY []:
>       (POP #:RESULT147) evaluated to (1 2 4 1 2 4), which is not  
> EQUALP
> to (1 2 4)..
>  --------------------------------
>  --------------------------------
>  INDEXING-HIERARCHY []:
>       (POP #:RESULT147) evaluated to (2 3 2 3), which is not EQUALP to
> (2 3)..
>  --------------------------------
>  --------------------------------
>  INDEXING-HIERARCHY []:
>       (POP #:RESULT147) evaluated to (4 4), which is not EQUALP to  
> (4)..
>  --------------------------------
>  --------------------------------
>  INHIBIT-ROLLBACK []:
>       Unexpected Error: #<IT.BESE.FIVEAM::TEST-FAILURE {1004A633D1}>
> #<TEST-FAILURE {1004A633D1}>..
>  --------------------------------
>
> I will let you know later today if I can reproduce my deserialization
> errors.
>
> -Kevin
>
>
> Ian Eslick wrote:
>> Also try pushing :elephant-without-optimize onto *features* prior to
>> doing a force rebuild of elephant.  This should give you some more
>> information and rule out any optimization/declaration related bugs.
>>
>> Thank you,
>> Ian
>>
>>
>> On Jan 7, 2009, at 9:10 PM, Kevin Raison wrote:
>>
>>> I was finally able to recreate this in the repl (as opposed to
>>> seeing it
>>> in my error logs), so here is a trace:
>>>
>>> The slot DB-BDB::INDICES-CACHE is unbound in the object
>>> #<BDB-INDEXED-BTREE oid:2>.
>>>   [Condition of type UNBOUND-SLOT]
>>>
>>> Backtrace:
>>>  0: ((SB-PCL::FAST-METHOD SLOT-UNBOUND (T T T)) #<unavailable
>>> argument> #<unavailable argument> #<unavailable argument>
>>> #<BDB-INDEXED-BTREE oid:2> DB-BDB::INDICES-CACHE)
>>>  1: (SB-PCL::SLOT-UNBOUND-INTERNAL #<BDB-INDEXED-BTREE oid:2> 2)
>>>  2: ((SB-PCL::FAST-METHOD (SETF GET-VALUE) (T T
>>> DB-BDB::BDB-INDEXED-BTREE)) #<unavailable lambda list>)
>>>  3: ((LAMBDA (WORD)) "34766")
>>>  4: (SB-IMPL::MAP1 #<CLOSURE (LAMBDA #) {1003CA4ED9}> (("" "asa"
>>> "106007" "deny" "inbound" "udp" ...)) :LIST T)
>>>  5: (MAPCAR #<CLOSURE (LAMBDA #) {1003CA4ED9}> ("" "asa" "106007"
>>> "deny" "inbound" "udp" ...))[:EXTERNAL]
>>>  6: ((SB-PCL::FAST-METHOD INDEX-LOG-ENTRY (LOG-ENTRY)) #<unavailable
>>> argument> #<unavailable argument> #<LOG-ENTRY oid:6373>)
>>>  7: ((LAMBDA ()))
>>>  8: ((SB-PCL::FAST-METHOD ELEPHANT::EXECUTE-TRANSACTION
>>> (DB-BDB::BDB-STORE-CONTROLLER T)) #<unavailable argument>
>>> #<unavailable
>>> argument> #<unavailable argument> #<unavailable argument>) 
>>> [:EXTERNAL]
>>>  9: (ADD-LOG-ENTRY ..)
>>> 10: (SB-INT:SIMPLE-EVAL-IN-LEXENV ..)
>>>
>>> And here is the code that sometimes causes the issue:
>>> (defmethod index-log-entry ((log-entry log-entry))
>>>  (let ((*store-controller* *syslog-controller*))
>>>    (let ((btree (get-from-root "log-entry-index" :sc
>>> *syslog-controller*)))
>>>      (map-words #'(lambda (word)
>>>                     (unless (or (member word *superfluous-
>>> words* :test
>>> #'string-equal)
>>>                                 (< (length word) 2))
>>>                       (unless (existsp word btree)
>>>                         (setf (get-value word btree) (make-pset :sc
>>> *syslog-controller*)))
>>>                       (insert-item log-entry
>>>                                    (get-value word btree))))
>>>                 (text log-entry)))))
>>>
>>>
>>> If I drop and recreate the btree, everything runs smoothly (no  
>>> errors)
>>> for a few hours and then the above error message starts showing up
>>> about
>>> half to three quarters of the time.
>>>
>>> Thanks again,
>>> Kevin
>>>
>>>
>>> Kevin Raison wrote:
>>>> I am seeing an intermittent error with 1.0 alpha when trying to
>>>> write to
>>>> an indexed btree (using BerkeleyDB 4.7 as provided by Ubuntu's
>>>> package
>>>> repositories):
>>>>
>>>> The slot DB-BDB::INDICES-CACHE is unbound in the object
>>>> #<BDB-INDEXED-BTREE oid:2>
>>>>
>>>> Within the same thread, sometimes this happens and sometimes I am
>>>> able
>>>> to read and write to the btree.  I am using sbcl 1.0.24 on 32 bit
>>>> intel
>>>> linux.  Is there something obvious that might help alleviate this  
>>>> or
>>>> should I provide more context?
>>>>
>>>> Thanks!
>>>> Kevin
>>>>
>>>> _______________________________________________
>>>> elephant-devel site list
>>>> elephant-devel at common-lisp.net
>>>> http://common-lisp.net/mailman/listinfo/elephant-devel
>>> _______________________________________________
>>> elephant-devel site list
>>> elephant-devel at common-lisp.net
>>> http://common-lisp.net/mailman/listinfo/elephant-devel
>>
>>
>> _______________________________________________
>> elephant-devel site list
>> elephant-devel at common-lisp.net
>> http://common-lisp.net/mailman/listinfo/elephant-devel
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel



From raison at chatsubo.net  Sat Jan 10 19:04:03 2009
From: raison at chatsubo.net (Kevin Raison)
Date: Sat, 10 Jan 2009 11:04:03 -0800
Subject: [elephant-devel] Strange behavior with indexed-btree
In-Reply-To: <AF76A669-036C-4597-9FB1-3DEA1C39DCB2@media.mit.edu>
References: <4965525D.2010106@chatsubo.net> <49656099.6050708@chatsubo.net>
	<AF76A669-036C-4597-9FB1-3DEA1C39DCB2@media.mit.edu>
Message-ID: <4968F123.5090802@chatsubo.net>

After pulling the latest patches and rebuilding without optimizations as 
you suggest, I get the following errors, the second of which is slightly 
different than what I was receiving previously.

First, a deserialization error:
Condition ELEPHANT-TYPE-DESERIALIZATION-ERROR was signalled.
    [Condition of type ELEPHANT-TYPE-DESERIALIZATION-ERROR]

Restarts:
  0: [RETRY] Retry SLIME REPL evaluation request.
  1: [ABORT] Return to SLIME's top level.
  2: [TERMINATE-THREAD] Terminate this thread (#<THREAD "repl-thread" 
RUNNING {10034F31F1}>)

Backtrace:
   0: ((LAMBDA (SWANK-BACKEND::DEBUGGER-LOOP-FN)) #<FUNCTION (LAMBDA #) 
{10040B2389}>)
   1: (SWANK::DEBUG-IN-EMACS #<ELEPHANT-TYPE-DESERIALIZATION-ERROR 
{1002CB3531}>)
   2: (SWANK:INVOKE-SLIME-DEBUGGER #<ELEPHANT-TYPE-DESERIALIZATION-ERROR 
{1002CB3531}>)
   3: ((LAMBDA (SWANK-BACKEND::HOOK SWANK-BACKEND::FUN)) #<FUNCTION 
SWANK:SWANK-DEBUGGER-HOOK> #<CLOSURE (LAMBDA #) {1002CB3AC9}>)
   4: (INVOKE-DEBUGGER #<ELEPHANT-TYPE-DESERIALIZATION-ERROR {1002CB3531}>)
   5: ((FLET #:FUN23) #<ELEPHANT-TYPE-DESERIALIZATION-ERROR {1002CB3531}>)
   6: ((SB-PCL::FAST-METHOD ELEPHANT::EXECUTE-TRANSACTION 
(DB-BDB::BDB-STORE-CONTROLLER T)) ..)[:EXTERNAL]
   7: (SB-INT:SIMPLE-EVAL-IN-LEXENV (REINDEX-LOG-ENTRIES) #<NULL-LEXENV>)
   8: (SWANK::EVAL-REGION "(reindex-log-entries)\n")
   9: ((LAMBDA ()))
  10: (SWANK::TRACK-PACKAGE #<CLOSURE (LAMBDA #) {1002C596F9}>)
  11: (SWANK::CALL-WITH-RETRY-RESTART "Retry SLIME REPL evaluation 
request." #<CLOSURE (LAMBDA #) {1002C59619}>)
  12: (SWANK::CALL-WITH-BUFFER-SYNTAX NIL #<CLOSURE (LAMBDA #) 
{1002C595E9}>)
  13: (SWANK::REPL-EVAL "(reindex-log-entries)\n")

And then this one with any subsequent access to the bdb:

Bad type argument: 
 

   BDB-DB-ERROR
    [Condition of type SIMPLE-TYPE-ERROR]

Restarts:
  0: [RETRY] Retry SLIME REPL evaluation request.
  1: [ABORT] Return to SLIME's top level.
  2: [TERMINATE-THREAD] Terminate this thread (#<THREAD "repl-thread" 
RUNNING {10034F31F1}>)

Backtrace:
   0: (MAKE-CONDITION BDB-DB-ERROR)[:EXTERNAL]
   1: (ERROR BDB-DB-ERROR)[:EXTERNAL]
   2: ((SB-PCL::FAST-METHOD ELEPHANT::EXECUTE-TRANSACTION 
(DB-BDB::BDB-STORE-CONTROLLER T)) #<unused argument> #<unused argument> 
#<BDB-STORE-CONTROLLER /srv/console/db/> #<CLOSURE (LAMBDA #) {10037C53D9$
   3: (ADD-LOG-ENTRY)[:EXTERNAL]
   4: (SB-INT:SIMPLE-EVAL-IN-LEXENV (ADD-LOG-ENTRY :SEVERITY 2 
:HOST-UUID "123" :TIMESTAMP ...) #<NULL-LEXENV>)
   5: (SWANK::EVAL-REGION "(add-log-entry :severity 2 :host-uuid \"123\" 
:timestamp (get-universal-time)\n                       :program 
\"blah\" :text \"test entry\")\n")
   6: ((LAMBDA ()))
   7: (SWANK::TRACK-PACKAGE #<CLOSURE (LAMBDA #) {10037C1BF9}>)
   8: (SWANK::CALL-WITH-RETRY-RESTART "Retry SLIME REPL evaluation 
request." #<CLOSURE (LAMBDA #) {1003636499}>)
   9: (SWANK::CALL-WITH-BUFFER-SYNTAX NIL #<CLOSURE (LAMBDA #) 
{1003636469}>)
  10: (SWANK::REPL-EVAL "(add-log-entry :severity 2 :host-uuid \"123\" 
:timestamp (get-universal-time)\n                 :program \"blah\" 
:text \"test entry\")\n")
  11: (SB-INT:SIMPLE-EVAL-IN-LEXENV (SWANK:LISTENER-EVAL "(add-log-entry 
:severity 2 :host-uuid \"123\" :timestamp (get-universal-time)\n 
                 :program \"blah\" :text \"test entry\")\n"$
  12: (SWANK::EVAL-FOR-EMACS (SWANK:LISTENER-EVAL "(add-log-entry 
:severity 2 :host-uuid \"123\" :timestamp (get-universal-time)\n 
                :program \"blah\" :text \"test entry\")\n") "CONSO$
  13: (SWANK::PROCESS-REQUESTS NIL)
  14: ((LAMBDA ()))
  15: ((LAMBDA (SWANK-BACKEND::HOOK SWANK-BACKEND::FUN)) #<FUNCTION 
SWANK:SWANK-DEBUGGER-HOOK> #<CLOSURE (LAMBDA #) {10038AD269}>)
  16: (SWANK::CALL-WITH-REDIRECTED-IO #<SWANK::CONNECTION {10032FB281}> 
#<CLOSURE (LAMBDA #) {10038AD289}>)
  17: (SWANK::CALL-WITH-CONNECTION #<SWANK::CONNECTION {10032FB281}> 
#<CLOSURE (LAMBDA #) {10038AD269}>)
  18: (SWANK::HANDLE-REQUESTS #<SWANK::CONNECTION {10032FB281}> NIL)
  19: (SWANK::CALL-WITH-BINDINGS NIL #<CLOSURE (LAMBDA #) {10034F5189}>)
  20: ((FLET SB-THREAD::WITH-MUTEX-THUNK))
  21: ((FLET #:WITHOUT-INTERRUPTS-BODY-[CALL-WITH-MUTEX]477))
  22: (SB-THREAD::CALL-WITH-MUTEX ..)
  23: ((LAMBDA ()))
  24: ("foreign function: call_into_lisp")
  25: ("foreign function: new_thread_trampoline")


Ian Eslick wrote:
> Also try pushing :elephant-without-optimize onto *features* prior to  
> doing a force rebuild of elephant.  This should give you some more  
> information and rule out any optimization/declaration related bugs.
> 
> Thank you,
> Ian
> 
> 
> On Jan 7, 2009, at 9:10 PM, Kevin Raison wrote:
> 
>> I was finally able to recreate this in the repl (as opposed to  
>> seeing it
>> in my error logs), so here is a trace:
>>
>> The slot DB-BDB::INDICES-CACHE is unbound in the object
>> #<BDB-INDEXED-BTREE oid:2>.
>>    [Condition of type UNBOUND-SLOT]
>>
>> Backtrace:
>>   0: ((SB-PCL::FAST-METHOD SLOT-UNBOUND (T T T)) #<unavailable
>> argument> #<unavailable argument> #<unavailable argument>
>> #<BDB-INDEXED-BTREE oid:2> DB-BDB::INDICES-CACHE)
>>   1: (SB-PCL::SLOT-UNBOUND-INTERNAL #<BDB-INDEXED-BTREE oid:2> 2)
>>   2: ((SB-PCL::FAST-METHOD (SETF GET-VALUE) (T T
>> DB-BDB::BDB-INDEXED-BTREE)) #<unavailable lambda list>)
>>   3: ((LAMBDA (WORD)) "34766")
>>   4: (SB-IMPL::MAP1 #<CLOSURE (LAMBDA #) {1003CA4ED9}> (("" "asa"
>> "106007" "deny" "inbound" "udp" ...)) :LIST T)
>>   5: (MAPCAR #<CLOSURE (LAMBDA #) {1003CA4ED9}> ("" "asa" "106007"
>> "deny" "inbound" "udp" ...))[:EXTERNAL]
>>   6: ((SB-PCL::FAST-METHOD INDEX-LOG-ENTRY (LOG-ENTRY)) #<unavailable
>> argument> #<unavailable argument> #<LOG-ENTRY oid:6373>)
>>   7: ((LAMBDA ()))
>>   8: ((SB-PCL::FAST-METHOD ELEPHANT::EXECUTE-TRANSACTION
>> (DB-BDB::BDB-STORE-CONTROLLER T)) #<unavailable argument>  
>> #<unavailable
>> argument> #<unavailable argument> #<unavailable argument>)[:EXTERNAL]
>>   9: (ADD-LOG-ENTRY ..)
>>  10: (SB-INT:SIMPLE-EVAL-IN-LEXENV ..)
>>
>> And here is the code that sometimes causes the issue:
>> (defmethod index-log-entry ((log-entry log-entry))
>>   (let ((*store-controller* *syslog-controller*))
>>     (let ((btree (get-from-root "log-entry-index" :sc
>> *syslog-controller*)))
>>       (map-words #'(lambda (word)
>>                      (unless (or (member word *superfluous- 
>> words* :test
>> #'string-equal)
>>                                  (< (length word) 2))
>>                        (unless (existsp word btree)
>>                          (setf (get-value word btree) (make-pset :sc
>> *syslog-controller*)))
>>                        (insert-item log-entry
>>                                     (get-value word btree))))
>>                  (text log-entry)))))
>>
>>
>> If I drop and recreate the btree, everything runs smoothly (no errors)
>> for a few hours and then the above error message starts showing up  
>> about
>> half to three quarters of the time.
>>
>> Thanks again,
>> Kevin
>>
>>
>> Kevin Raison wrote:
>>> I am seeing an intermittent error with 1.0 alpha when trying to  
>>> write to
>>>  an indexed btree (using BerkeleyDB 4.7 as provided by Ubuntu's  
>>> package
>>> repositories):
>>>
>>> The slot DB-BDB::INDICES-CACHE is unbound in the object
>>> #<BDB-INDEXED-BTREE oid:2>
>>>
>>> Within the same thread, sometimes this happens and sometimes I am  
>>> able
>>> to read and write to the btree.  I am using sbcl 1.0.24 on 32 bit  
>>> intel
>>> linux.  Is there something obvious that might help alleviate this or
>>> should I provide more context?
>>>
>>> Thanks!
>>> Kevin
>>>
>>> _______________________________________________
>>> elephant-devel site list
>>> elephant-devel at common-lisp.net
>>> http://common-lisp.net/mailman/listinfo/elephant-devel
>> _______________________________________________
>> elephant-devel site list
>> elephant-devel at common-lisp.net
>> http://common-lisp.net/mailman/listinfo/elephant-devel
> 
> 
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel



From eslick at media.mit.edu  Sat Jan 10 20:13:17 2009
From: eslick at media.mit.edu (Ian Eslick)
Date: Sat, 10 Jan 2009 15:13:17 -0500
Subject: [elephant-devel] Strange behavior with indexed-btree
In-Reply-To: <4968F123.5090802@chatsubo.net>
References: <4965525D.2010106@chatsubo.net> <49656099.6050708@chatsubo.net>
	<AF76A669-036C-4597-9FB1-3DEA1C39DCB2@media.mit.edu>
	<4968F123.5090802@chatsubo.net>
Message-ID: <841C8B93-7A80-4934-9ACE-31262416C283@media.mit.edu>

I just checked in a few more fixes a few minutes ago.  I finally was  
able to reproduce some of this locally.  Try cleaning out the test  
database and re-running the tests.  Everything passes for me on a  
fresh DB and my own application is running fine too.

Ian

On Jan 10, 2009, at 2:04 PM, Kevin Raison wrote:

> After pulling the latest patches and rebuilding without  
> optimizations as
> you suggest, I get the following errors, the second of which is  
> slightly
> different than what I was receiving previously.
>
> First, a deserialization error:
> Condition ELEPHANT-TYPE-DESERIALIZATION-ERROR was signalled.
>    [Condition of type ELEPHANT-TYPE-DESERIALIZATION-ERROR]
>
> Restarts:
>  0: [RETRY] Retry SLIME REPL evaluation request.
>  1: [ABORT] Return to SLIME's top level.
>  2: [TERMINATE-THREAD] Terminate this thread (#<THREAD "repl-thread"
> RUNNING {10034F31F1}>)
>
> Backtrace:
>   0: ((LAMBDA (SWANK-BACKEND::DEBUGGER-LOOP-FN)) #<FUNCTION (LAMBDA #)
> {10040B2389}>)
>   1: (SWANK::DEBUG-IN-EMACS #<ELEPHANT-TYPE-DESERIALIZATION-ERROR
> {1002CB3531}>)
>   2: (SWANK:INVOKE-SLIME-DEBUGGER #<ELEPHANT-TYPE-DESERIALIZATION- 
> ERROR
> {1002CB3531}>)
>   3: ((LAMBDA (SWANK-BACKEND::HOOK SWANK-BACKEND::FUN)) #<FUNCTION
> SWANK:SWANK-DEBUGGER-HOOK> #<CLOSURE (LAMBDA #) {1002CB3AC9}>)
>   4: (INVOKE-DEBUGGER #<ELEPHANT-TYPE-DESERIALIZATION-ERROR  
> {1002CB3531}>)
>   5: ((FLET #:FUN23) #<ELEPHANT-TYPE-DESERIALIZATION-ERROR  
> {1002CB3531}>)
>   6: ((SB-PCL::FAST-METHOD ELEPHANT::EXECUTE-TRANSACTION
> (DB-BDB::BDB-STORE-CONTROLLER T)) ..)[:EXTERNAL]
>   7: (SB-INT:SIMPLE-EVAL-IN-LEXENV (REINDEX-LOG-ENTRIES) #<NULL- 
> LEXENV>)
>   8: (SWANK::EVAL-REGION "(reindex-log-entries)\n")
>   9: ((LAMBDA ()))
>  10: (SWANK::TRACK-PACKAGE #<CLOSURE (LAMBDA #) {1002C596F9}>)
>  11: (SWANK::CALL-WITH-RETRY-RESTART "Retry SLIME REPL evaluation
> request." #<CLOSURE (LAMBDA #) {1002C59619}>)
>  12: (SWANK::CALL-WITH-BUFFER-SYNTAX NIL #<CLOSURE (LAMBDA #)
> {1002C595E9}>)
>  13: (SWANK::REPL-EVAL "(reindex-log-entries)\n")
>
> And then this one with any subsequent access to the bdb:
>
> Bad type argument:
>
>
>   BDB-DB-ERROR
>    [Condition of type SIMPLE-TYPE-ERROR]
>
> Restarts:
>  0: [RETRY] Retry SLIME REPL evaluation request.
>  1: [ABORT] Return to SLIME's top level.
>  2: [TERMINATE-THREAD] Terminate this thread (#<THREAD "repl-thread"
> RUNNING {10034F31F1}>)
>
> Backtrace:
>   0: (MAKE-CONDITION BDB-DB-ERROR)[:EXTERNAL]
>   1: (ERROR BDB-DB-ERROR)[:EXTERNAL]
>   2: ((SB-PCL::FAST-METHOD ELEPHANT::EXECUTE-TRANSACTION
> (DB-BDB::BDB-STORE-CONTROLLER T)) #<unused argument> #<unused  
> argument>
> #<BDB-STORE-CONTROLLER /srv/console/db/> #<CLOSURE (LAMBDA #)  
> {10037C53D9$
>   3: (ADD-LOG-ENTRY)[:EXTERNAL]
>   4: (SB-INT:SIMPLE-EVAL-IN-LEXENV (ADD-LOG-ENTRY :SEVERITY 2
> :HOST-UUID "123" :TIMESTAMP ...) #<NULL-LEXENV>)
>   5: (SWANK::EVAL-REGION "(add-log-entry :severity 2 :host-uuid  
> \"123\"
> :timestamp (get-universal-time)\n                       :program
> \"blah\" :text \"test entry\")\n")
>   6: ((LAMBDA ()))
>   7: (SWANK::TRACK-PACKAGE #<CLOSURE (LAMBDA #) {10037C1BF9}>)
>   8: (SWANK::CALL-WITH-RETRY-RESTART "Retry SLIME REPL evaluation
> request." #<CLOSURE (LAMBDA #) {1003636499}>)
>   9: (SWANK::CALL-WITH-BUFFER-SYNTAX NIL #<CLOSURE (LAMBDA #)
> {1003636469}>)
>  10: (SWANK::REPL-EVAL "(add-log-entry :severity 2 :host-uuid \"123\"
> :timestamp (get-universal-time)\n                 :program \"blah\"
> :text \"test entry\")\n")
>  11: (SB-INT:SIMPLE-EVAL-IN-LEXENV (SWANK:LISTENER-EVAL "(add-log- 
> entry
> :severity 2 :host-uuid \"123\" :timestamp (get-universal-time)\n
>                 :program \"blah\" :text \"test entry\")\n"$
>  12: (SWANK::EVAL-FOR-EMACS (SWANK:LISTENER-EVAL "(add-log-entry
> :severity 2 :host-uuid \"123\" :timestamp (get-universal-time)\n
>                :program \"blah\" :text \"test entry\")\n") "CONSO$
>  13: (SWANK::PROCESS-REQUESTS NIL)
>  14: ((LAMBDA ()))
>  15: ((LAMBDA (SWANK-BACKEND::HOOK SWANK-BACKEND::FUN)) #<FUNCTION
> SWANK:SWANK-DEBUGGER-HOOK> #<CLOSURE (LAMBDA #) {10038AD269}>)
>  16: (SWANK::CALL-WITH-REDIRECTED-IO #<SWANK::CONNECTION {10032FB281}>
> #<CLOSURE (LAMBDA #) {10038AD289}>)
>  17: (SWANK::CALL-WITH-CONNECTION #<SWANK::CONNECTION {10032FB281}>
> #<CLOSURE (LAMBDA #) {10038AD269}>)
>  18: (SWANK::HANDLE-REQUESTS #<SWANK::CONNECTION {10032FB281}> NIL)
>  19: (SWANK::CALL-WITH-BINDINGS NIL #<CLOSURE (LAMBDA #)  
> {10034F5189}>)
>  20: ((FLET SB-THREAD::WITH-MUTEX-THUNK))
>  21: ((FLET #:WITHOUT-INTERRUPTS-BODY-[CALL-WITH-MUTEX]477))
>  22: (SB-THREAD::CALL-WITH-MUTEX ..)
>  23: ((LAMBDA ()))
>  24: ("foreign function: call_into_lisp")
>  25: ("foreign function: new_thread_trampoline")
>
>
> Ian Eslick wrote:
>> Also try pushing :elephant-without-optimize onto *features* prior to
>> doing a force rebuild of elephant.  This should give you some more
>> information and rule out any optimization/declaration related bugs.
>>
>> Thank you,
>> Ian
>>
>>
>> On Jan 7, 2009, at 9:10 PM, Kevin Raison wrote:
>>
>>> I was finally able to recreate this in the repl (as opposed to
>>> seeing it
>>> in my error logs), so here is a trace:
>>>
>>> The slot DB-BDB::INDICES-CACHE is unbound in the object
>>> #<BDB-INDEXED-BTREE oid:2>.
>>>   [Condition of type UNBOUND-SLOT]
>>>
>>> Backtrace:
>>>  0: ((SB-PCL::FAST-METHOD SLOT-UNBOUND (T T T)) #<unavailable
>>> argument> #<unavailable argument> #<unavailable argument>
>>> #<BDB-INDEXED-BTREE oid:2> DB-BDB::INDICES-CACHE)
>>>  1: (SB-PCL::SLOT-UNBOUND-INTERNAL #<BDB-INDEXED-BTREE oid:2> 2)
>>>  2: ((SB-PCL::FAST-METHOD (SETF GET-VALUE) (T T
>>> DB-BDB::BDB-INDEXED-BTREE)) #<unavailable lambda list>)
>>>  3: ((LAMBDA (WORD)) "34766")
>>>  4: (SB-IMPL::MAP1 #<CLOSURE (LAMBDA #) {1003CA4ED9}> (("" "asa"
>>> "106007" "deny" "inbound" "udp" ...)) :LIST T)
>>>  5: (MAPCAR #<CLOSURE (LAMBDA #) {1003CA4ED9}> ("" "asa" "106007"
>>> "deny" "inbound" "udp" ...))[:EXTERNAL]
>>>  6: ((SB-PCL::FAST-METHOD INDEX-LOG-ENTRY (LOG-ENTRY)) #<unavailable
>>> argument> #<unavailable argument> #<LOG-ENTRY oid:6373>)
>>>  7: ((LAMBDA ()))
>>>  8: ((SB-PCL::FAST-METHOD ELEPHANT::EXECUTE-TRANSACTION
>>> (DB-BDB::BDB-STORE-CONTROLLER T)) #<unavailable argument>
>>> #<unavailable
>>> argument> #<unavailable argument> #<unavailable argument>) 
>>> [:EXTERNAL]
>>>  9: (ADD-LOG-ENTRY ..)
>>> 10: (SB-INT:SIMPLE-EVAL-IN-LEXENV ..)
>>>
>>> And here is the code that sometimes causes the issue:
>>> (defmethod index-log-entry ((log-entry log-entry))
>>>  (let ((*store-controller* *syslog-controller*))
>>>    (let ((btree (get-from-root "log-entry-index" :sc
>>> *syslog-controller*)))
>>>      (map-words #'(lambda (word)
>>>                     (unless (or (member word *superfluous-
>>> words* :test
>>> #'string-equal)
>>>                                 (< (length word) 2))
>>>                       (unless (existsp word btree)
>>>                         (setf (get-value word btree) (make-pset :sc
>>> *syslog-controller*)))
>>>                       (insert-item log-entry
>>>                                    (get-value word btree))))
>>>                 (text log-entry)))))
>>>
>>>
>>> If I drop and recreate the btree, everything runs smoothly (no  
>>> errors)
>>> for a few hours and then the above error message starts showing up
>>> about
>>> half to three quarters of the time.
>>>
>>> Thanks again,
>>> Kevin
>>>
>>>
>>> Kevin Raison wrote:
>>>> I am seeing an intermittent error with 1.0 alpha when trying to
>>>> write to
>>>> an indexed btree (using BerkeleyDB 4.7 as provided by Ubuntu's
>>>> package
>>>> repositories):
>>>>
>>>> The slot DB-BDB::INDICES-CACHE is unbound in the object
>>>> #<BDB-INDEXED-BTREE oid:2>
>>>>
>>>> Within the same thread, sometimes this happens and sometimes I am
>>>> able
>>>> to read and write to the btree.  I am using sbcl 1.0.24 on 32 bit
>>>> intel
>>>> linux.  Is there something obvious that might help alleviate this  
>>>> or
>>>> should I provide more context?
>>>>
>>>> Thanks!
>>>> Kevin
>>>>
>>>> _______________________________________________
>>>> elephant-devel site list
>>>> elephant-devel at common-lisp.net
>>>> http://common-lisp.net/mailman/listinfo/elephant-devel
>>> _______________________________________________
>>> elephant-devel site list
>>> elephant-devel at common-lisp.net
>>> http://common-lisp.net/mailman/listinfo/elephant-devel
>>
>>
>> _______________________________________________
>> elephant-devel site list
>> elephant-devel at common-lisp.net
>> http://common-lisp.net/mailman/listinfo/elephant-devel
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel




From raison at chatsubo.net  Sat Jan 10 21:05:50 2009
From: raison at chatsubo.net (Kevin Raison)
Date: Sat, 10 Jan 2009 13:05:50 -0800
Subject: [elephant-devel] Strange behavior with indexed-btree
In-Reply-To: <841C8B93-7A80-4934-9ACE-31262416C283@media.mit.edu>
References: <4965525D.2010106@chatsubo.net>
	<49656099.6050708@chatsubo.net>	<AF76A669-036C-4597-9FB1-3DEA1C39DCB2@media.mit.edu>	<4968F123.5090802@chatsubo.net>
	<841C8B93-7A80-4934-9ACE-31262416C283@media.mit.edu>
Message-ID: <49690DAE.7030004@chatsubo.net>

Thanks, Ian.  All tests pass for me as well.  I am running my 
application with the new code and will let you know how it goes.

Thanks again!
Kevin

Ian Eslick wrote:
> I just checked in a few more fixes a few minutes ago.  I finally was  
> able to reproduce some of this locally.  Try cleaning out the test  
> database and re-running the tests.  Everything passes for me on a  
> fresh DB and my own application is running fine too.
> 
> Ian
> 
> On Jan 10, 2009, at 2:04 PM, Kevin Raison wrote:
> 
>> After pulling the latest patches and rebuilding without  
>> optimizations as
>> you suggest, I get the following errors, the second of which is  
>> slightly
>> different than what I was receiving previously.
>>
>> First, a deserialization error:
>> Condition ELEPHANT-TYPE-DESERIALIZATION-ERROR was signalled.
>>    [Condition of type ELEPHANT-TYPE-DESERIALIZATION-ERROR]
>>
>> Restarts:
>>  0: [RETRY] Retry SLIME REPL evaluation request.
>>  1: [ABORT] Return to SLIME's top level.
>>  2: [TERMINATE-THREAD] Terminate this thread (#<THREAD "repl-thread"
>> RUNNING {10034F31F1}>)
>>
>> Backtrace:
>>   0: ((LAMBDA (SWANK-BACKEND::DEBUGGER-LOOP-FN)) #<FUNCTION (LAMBDA #)
>> {10040B2389}>)
>>   1: (SWANK::DEBUG-IN-EMACS #<ELEPHANT-TYPE-DESERIALIZATION-ERROR
>> {1002CB3531}>)
>>   2: (SWANK:INVOKE-SLIME-DEBUGGER #<ELEPHANT-TYPE-DESERIALIZATION- 
>> ERROR
>> {1002CB3531}>)
>>   3: ((LAMBDA (SWANK-BACKEND::HOOK SWANK-BACKEND::FUN)) #<FUNCTION
>> SWANK:SWANK-DEBUGGER-HOOK> #<CLOSURE (LAMBDA #) {1002CB3AC9}>)
>>   4: (INVOKE-DEBUGGER #<ELEPHANT-TYPE-DESERIALIZATION-ERROR  
>> {1002CB3531}>)
>>   5: ((FLET #:FUN23) #<ELEPHANT-TYPE-DESERIALIZATION-ERROR  
>> {1002CB3531}>)
>>   6: ((SB-PCL::FAST-METHOD ELEPHANT::EXECUTE-TRANSACTION
>> (DB-BDB::BDB-STORE-CONTROLLER T)) ..)[:EXTERNAL]
>>   7: (SB-INT:SIMPLE-EVAL-IN-LEXENV (REINDEX-LOG-ENTRIES) #<NULL- 
>> LEXENV>)
>>   8: (SWANK::EVAL-REGION "(reindex-log-entries)\n")
>>   9: ((LAMBDA ()))
>>  10: (SWANK::TRACK-PACKAGE #<CLOSURE (LAMBDA #) {1002C596F9}>)
>>  11: (SWANK::CALL-WITH-RETRY-RESTART "Retry SLIME REPL evaluation
>> request." #<CLOSURE (LAMBDA #) {1002C59619}>)
>>  12: (SWANK::CALL-WITH-BUFFER-SYNTAX NIL #<CLOSURE (LAMBDA #)
>> {1002C595E9}>)
>>  13: (SWANK::REPL-EVAL "(reindex-log-entries)\n")
>>
>> And then this one with any subsequent access to the bdb:
>>
>> Bad type argument:
>>
>>
>>   BDB-DB-ERROR
>>    [Condition of type SIMPLE-TYPE-ERROR]
>>
>> Restarts:
>>  0: [RETRY] Retry SLIME REPL evaluation request.
>>  1: [ABORT] Return to SLIME's top level.
>>  2: [TERMINATE-THREAD] Terminate this thread (#<THREAD "repl-thread"
>> RUNNING {10034F31F1}>)
>>
>> Backtrace:
>>   0: (MAKE-CONDITION BDB-DB-ERROR)[:EXTERNAL]
>>   1: (ERROR BDB-DB-ERROR)[:EXTERNAL]
>>   2: ((SB-PCL::FAST-METHOD ELEPHANT::EXECUTE-TRANSACTION
>> (DB-BDB::BDB-STORE-CONTROLLER T)) #<unused argument> #<unused  
>> argument>
>> #<BDB-STORE-CONTROLLER /srv/console/db/> #<CLOSURE (LAMBDA #)  
>> {10037C53D9$
>>   3: (ADD-LOG-ENTRY)[:EXTERNAL]
>>   4: (SB-INT:SIMPLE-EVAL-IN-LEXENV (ADD-LOG-ENTRY :SEVERITY 2
>> :HOST-UUID "123" :TIMESTAMP ...) #<NULL-LEXENV>)
>>   5: (SWANK::EVAL-REGION "(add-log-entry :severity 2 :host-uuid  
>> \"123\"
>> :timestamp (get-universal-time)\n                       :program
>> \"blah\" :text \"test entry\")\n")
>>   6: ((LAMBDA ()))
>>   7: (SWANK::TRACK-PACKAGE #<CLOSURE (LAMBDA #) {10037C1BF9}>)
>>   8: (SWANK::CALL-WITH-RETRY-RESTART "Retry SLIME REPL evaluation
>> request." #<CLOSURE (LAMBDA #) {1003636499}>)
>>   9: (SWANK::CALL-WITH-BUFFER-SYNTAX NIL #<CLOSURE (LAMBDA #)
>> {1003636469}>)
>>  10: (SWANK::REPL-EVAL "(add-log-entry :severity 2 :host-uuid \"123\"
>> :timestamp (get-universal-time)\n                 :program \"blah\"
>> :text \"test entry\")\n")
>>  11: (SB-INT:SIMPLE-EVAL-IN-LEXENV (SWANK:LISTENER-EVAL "(add-log- 
>> entry
>> :severity 2 :host-uuid \"123\" :timestamp (get-universal-time)\n
>>                 :program \"blah\" :text \"test entry\")\n"$
>>  12: (SWANK::EVAL-FOR-EMACS (SWANK:LISTENER-EVAL "(add-log-entry
>> :severity 2 :host-uuid \"123\" :timestamp (get-universal-time)\n
>>                :program \"blah\" :text \"test entry\")\n") "CONSO$
>>  13: (SWANK::PROCESS-REQUESTS NIL)
>>  14: ((LAMBDA ()))
>>  15: ((LAMBDA (SWANK-BACKEND::HOOK SWANK-BACKEND::FUN)) #<FUNCTION
>> SWANK:SWANK-DEBUGGER-HOOK> #<CLOSURE (LAMBDA #) {10038AD269}>)
>>  16: (SWANK::CALL-WITH-REDIRECTED-IO #<SWANK::CONNECTION {10032FB281}>
>> #<CLOSURE (LAMBDA #) {10038AD289}>)
>>  17: (SWANK::CALL-WITH-CONNECTION #<SWANK::CONNECTION {10032FB281}>
>> #<CLOSURE (LAMBDA #) {10038AD269}>)
>>  18: (SWANK::HANDLE-REQUESTS #<SWANK::CONNECTION {10032FB281}> NIL)
>>  19: (SWANK::CALL-WITH-BINDINGS NIL #<CLOSURE (LAMBDA #)  
>> {10034F5189}>)
>>  20: ((FLET SB-THREAD::WITH-MUTEX-THUNK))
>>  21: ((FLET #:WITHOUT-INTERRUPTS-BODY-[CALL-WITH-MUTEX]477))
>>  22: (SB-THREAD::CALL-WITH-MUTEX ..)
>>  23: ((LAMBDA ()))
>>  24: ("foreign function: call_into_lisp")
>>  25: ("foreign function: new_thread_trampoline")
>>
>>
>> Ian Eslick wrote:
>>> Also try pushing :elephant-without-optimize onto *features* prior to
>>> doing a force rebuild of elephant.  This should give you some more
>>> information and rule out any optimization/declaration related bugs.
>>>
>>> Thank you,
>>> Ian
>>>
>>>
>>> On Jan 7, 2009, at 9:10 PM, Kevin Raison wrote:
>>>
>>>> I was finally able to recreate this in the repl (as opposed to
>>>> seeing it
>>>> in my error logs), so here is a trace:
>>>>
>>>> The slot DB-BDB::INDICES-CACHE is unbound in the object
>>>> #<BDB-INDEXED-BTREE oid:2>.
>>>>   [Condition of type UNBOUND-SLOT]
>>>>
>>>> Backtrace:
>>>>  0: ((SB-PCL::FAST-METHOD SLOT-UNBOUND (T T T)) #<unavailable
>>>> argument> #<unavailable argument> #<unavailable argument>
>>>> #<BDB-INDEXED-BTREE oid:2> DB-BDB::INDICES-CACHE)
>>>>  1: (SB-PCL::SLOT-UNBOUND-INTERNAL #<BDB-INDEXED-BTREE oid:2> 2)
>>>>  2: ((SB-PCL::FAST-METHOD (SETF GET-VALUE) (T T
>>>> DB-BDB::BDB-INDEXED-BTREE)) #<unavailable lambda list>)
>>>>  3: ((LAMBDA (WORD)) "34766")
>>>>  4: (SB-IMPL::MAP1 #<CLOSURE (LAMBDA #) {1003CA4ED9}> (("" "asa"
>>>> "106007" "deny" "inbound" "udp" ...)) :LIST T)
>>>>  5: (MAPCAR #<CLOSURE (LAMBDA #) {1003CA4ED9}> ("" "asa" "106007"
>>>> "deny" "inbound" "udp" ...))[:EXTERNAL]
>>>>  6: ((SB-PCL::FAST-METHOD INDEX-LOG-ENTRY (LOG-ENTRY)) #<unavailable
>>>> argument> #<unavailable argument> #<LOG-ENTRY oid:6373>)
>>>>  7: ((LAMBDA ()))
>>>>  8: ((SB-PCL::FAST-METHOD ELEPHANT::EXECUTE-TRANSACTION
>>>> (DB-BDB::BDB-STORE-CONTROLLER T)) #<unavailable argument>
>>>> #<unavailable
>>>> argument> #<unavailable argument> #<unavailable argument>) 
>>>> [:EXTERNAL]
>>>>  9: (ADD-LOG-ENTRY ..)
>>>> 10: (SB-INT:SIMPLE-EVAL-IN-LEXENV ..)
>>>>
>>>> And here is the code that sometimes causes the issue:
>>>> (defmethod index-log-entry ((log-entry log-entry))
>>>>  (let ((*store-controller* *syslog-controller*))
>>>>    (let ((btree (get-from-root "log-entry-index" :sc
>>>> *syslog-controller*)))
>>>>      (map-words #'(lambda (word)
>>>>                     (unless (or (member word *superfluous-
>>>> words* :test
>>>> #'string-equal)
>>>>                                 (< (length word) 2))
>>>>                       (unless (existsp word btree)
>>>>                         (setf (get-value word btree) (make-pset :sc
>>>> *syslog-controller*)))
>>>>                       (insert-item log-entry
>>>>                                    (get-value word btree))))
>>>>                 (text log-entry)))))
>>>>
>>>>
>>>> If I drop and recreate the btree, everything runs smoothly (no  
>>>> errors)
>>>> for a few hours and then the above error message starts showing up
>>>> about
>>>> half to three quarters of the time.
>>>>
>>>> Thanks again,
>>>> Kevin
>>>>
>>>>
>>>> Kevin Raison wrote:
>>>>> I am seeing an intermittent error with 1.0 alpha when trying to
>>>>> write to
>>>>> an indexed btree (using BerkeleyDB 4.7 as provided by Ubuntu's
>>>>> package
>>>>> repositories):
>>>>>
>>>>> The slot DB-BDB::INDICES-CACHE is unbound in the object
>>>>> #<BDB-INDEXED-BTREE oid:2>
>>>>>
>>>>> Within the same thread, sometimes this happens and sometimes I am
>>>>> able
>>>>> to read and write to the btree.  I am using sbcl 1.0.24 on 32 bit
>>>>> intel
>>>>> linux.  Is there something obvious that might help alleviate this  
>>>>> or
>>>>> should I provide more context?
>>>>>
>>>>> Thanks!
>>>>> Kevin
>>>>>
>>>>> _______________________________________________
>>>>> elephant-devel site list
>>>>> elephant-devel at common-lisp.net
>>>>> http://common-lisp.net/mailman/listinfo/elephant-devel
>>>> _______________________________________________
>>>> elephant-devel site list
>>>> elephant-devel at common-lisp.net
>>>> http://common-lisp.net/mailman/listinfo/elephant-devel
>>>
>>> _______________________________________________
>>> elephant-devel site list
>>> elephant-devel at common-lisp.net
>>> http://common-lisp.net/mailman/listinfo/elephant-devel
>> _______________________________________________
>> elephant-devel site list
>> elephant-devel at common-lisp.net
>> http://common-lisp.net/mailman/listinfo/elephant-devel
> 
> 
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel



From raison at chatsubo.net  Sat Jan 10 23:27:43 2009
From: raison at chatsubo.net (Kevin Raison)
Date: Sat, 10 Jan 2009 15:27:43 -0800
Subject: [elephant-devel] Strange behavior with indexed-btree
In-Reply-To: <49690DAE.7030004@chatsubo.net>
References: <4965525D.2010106@chatsubo.net>	<49656099.6050708@chatsubo.net>	<AF76A669-036C-4597-9FB1-3DEA1C39DCB2@media.mit.edu>	<4968F123.5090802@chatsubo.net>	<841C8B93-7A80-4934-9ACE-31262416C283@media.mit.edu>
	<49690DAE.7030004@chatsubo.net>
Message-ID: <49692EEF.7070208@chatsubo.net>

Unfortunately, things remain as before.  Here is some additional debug 
output that I found in the *inferior-lisp* buffer of my emacs session:

Deserialization error in map: returning nil for element
transaction has active cursors
PANIC: Invalid argument
PANIC: DB_RUNRECOVERY: Fatal error, run database recovery
PANIC: fatal region error detected; run recovery 
 


The deserialization error happens in one of two spots:
- when trying to get-from-root a btree or indexed btree
- when trying to get a pset via a call to get-value on the btree

Running recovery does not help;  the errors start as soon as I load and 
execute my code post-recovery.

Ian, can you enlighten me as to where in the code you believe these 
errors to originate?  Also, can you think of anything that I might be 
doing incorrectly in my code that would cause this sort of behavior? 
Here is a brief outline of what I am doing:

There is
  - a waitqueue (via sbcl's sb-thread:make-waitqueue)
  - a tcp listener thread that writes its input to an incoming work queue
  - 3 reader threads who use the waitqueue to get work items from the 
work queue
The readers parse the incoming data and create a persistent object from 
it.  They also index the individual words of one of the persistent 
object's slots using a btree.  These actions are contained in a 
with-transaction block.
There is also a hunchentoot web server thread running.  Users can search 
the indexed text via this web interface.

It is really rather simple.  The errors crop up randomly, sometimes when 
I am running searches via the web interface while the listener and 
reader threads are active, and sometimes while the listener and reader 
threads are doing their thing without any web-based reads happening.

Cheers,
Kevin


Kevin Raison wrote:
> Thanks, Ian.  All tests pass for me as well.  I am running my 
> application with the new code and will let you know how it goes.
> 
> Thanks again!
> Kevin
> 
> Ian Eslick wrote:
>> I just checked in a few more fixes a few minutes ago.  I finally was  
>> able to reproduce some of this locally.  Try cleaning out the test  
>> database and re-running the tests.  Everything passes for me on a  
>> fresh DB and my own application is running fine too.
>>
>> Ian
>>
>> On Jan 10, 2009, at 2:04 PM, Kevin Raison wrote:
>>
>>> After pulling the latest patches and rebuilding without  
>>> optimizations as
>>> you suggest, I get the following errors, the second of which is  
>>> slightly
>>> different than what I was receiving previously.
>>>
>>> First, a deserialization error:
>>> Condition ELEPHANT-TYPE-DESERIALIZATION-ERROR was signalled.
>>>    [Condition of type ELEPHANT-TYPE-DESERIALIZATION-ERROR]
>>>
>>> Restarts:
>>>  0: [RETRY] Retry SLIME REPL evaluation request.
>>>  1: [ABORT] Return to SLIME's top level.
>>>  2: [TERMINATE-THREAD] Terminate this thread (#<THREAD "repl-thread"
>>> RUNNING {10034F31F1}>)
>>>
>>> Backtrace:
>>>   0: ((LAMBDA (SWANK-BACKEND::DEBUGGER-LOOP-FN)) #<FUNCTION (LAMBDA #)
>>> {10040B2389}>)
>>>   1: (SWANK::DEBUG-IN-EMACS #<ELEPHANT-TYPE-DESERIALIZATION-ERROR
>>> {1002CB3531}>)
>>>   2: (SWANK:INVOKE-SLIME-DEBUGGER #<ELEPHANT-TYPE-DESERIALIZATION- 
>>> ERROR
>>> {1002CB3531}>)
>>>   3: ((LAMBDA (SWANK-BACKEND::HOOK SWANK-BACKEND::FUN)) #<FUNCTION
>>> SWANK:SWANK-DEBUGGER-HOOK> #<CLOSURE (LAMBDA #) {1002CB3AC9}>)
>>>   4: (INVOKE-DEBUGGER #<ELEPHANT-TYPE-DESERIALIZATION-ERROR  
>>> {1002CB3531}>)
>>>   5: ((FLET #:FUN23) #<ELEPHANT-TYPE-DESERIALIZATION-ERROR  
>>> {1002CB3531}>)
>>>   6: ((SB-PCL::FAST-METHOD ELEPHANT::EXECUTE-TRANSACTION
>>> (DB-BDB::BDB-STORE-CONTROLLER T)) ..)[:EXTERNAL]
>>>   7: (SB-INT:SIMPLE-EVAL-IN-LEXENV (REINDEX-LOG-ENTRIES) #<NULL- 
>>> LEXENV>)
>>>   8: (SWANK::EVAL-REGION "(reindex-log-entries)\n")
>>>   9: ((LAMBDA ()))
>>>  10: (SWANK::TRACK-PACKAGE #<CLOSURE (LAMBDA #) {1002C596F9}>)
>>>  11: (SWANK::CALL-WITH-RETRY-RESTART "Retry SLIME REPL evaluation
>>> request." #<CLOSURE (LAMBDA #) {1002C59619}>)
>>>  12: (SWANK::CALL-WITH-BUFFER-SYNTAX NIL #<CLOSURE (LAMBDA #)
>>> {1002C595E9}>)
>>>  13: (SWANK::REPL-EVAL "(reindex-log-entries)\n")
>>>
>>> And then this one with any subsequent access to the bdb:
>>>
>>> Bad type argument:
>>>
>>>
>>>   BDB-DB-ERROR
>>>    [Condition of type SIMPLE-TYPE-ERROR]
>>>
>>> Restarts:
>>>  0: [RETRY] Retry SLIME REPL evaluation request.
>>>  1: [ABORT] Return to SLIME's top level.
>>>  2: [TERMINATE-THREAD] Terminate this thread (#<THREAD "repl-thread"
>>> RUNNING {10034F31F1}>)
>>>
>>> Backtrace:
>>>   0: (MAKE-CONDITION BDB-DB-ERROR)[:EXTERNAL]
>>>   1: (ERROR BDB-DB-ERROR)[:EXTERNAL]
>>>   2: ((SB-PCL::FAST-METHOD ELEPHANT::EXECUTE-TRANSACTION
>>> (DB-BDB::BDB-STORE-CONTROLLER T)) #<unused argument> #<unused  
>>> argument>
>>> #<BDB-STORE-CONTROLLER /srv/console/db/> #<CLOSURE (LAMBDA #)  
>>> {10037C53D9$
>>>   3: (ADD-LOG-ENTRY)[:EXTERNAL]
>>>   4: (SB-INT:SIMPLE-EVAL-IN-LEXENV (ADD-LOG-ENTRY :SEVERITY 2
>>> :HOST-UUID "123" :TIMESTAMP ...) #<NULL-LEXENV>)
>>>   5: (SWANK::EVAL-REGION "(add-log-entry :severity 2 :host-uuid  
>>> \"123\"
>>> :timestamp (get-universal-time)\n                       :program
>>> \"blah\" :text \"test entry\")\n")
>>>   6: ((LAMBDA ()))
>>>   7: (SWANK::TRACK-PACKAGE #<CLOSURE (LAMBDA #) {10037C1BF9}>)
>>>   8: (SWANK::CALL-WITH-RETRY-RESTART "Retry SLIME REPL evaluation
>>> request." #<CLOSURE (LAMBDA #) {1003636499}>)
>>>   9: (SWANK::CALL-WITH-BUFFER-SYNTAX NIL #<CLOSURE (LAMBDA #)
>>> {1003636469}>)
>>>  10: (SWANK::REPL-EVAL "(add-log-entry :severity 2 :host-uuid \"123\"
>>> :timestamp (get-universal-time)\n                 :program \"blah\"
>>> :text \"test entry\")\n")
>>>  11: (SB-INT:SIMPLE-EVAL-IN-LEXENV (SWANK:LISTENER-EVAL "(add-log- 
>>> entry
>>> :severity 2 :host-uuid \"123\" :timestamp (get-universal-time)\n
>>>                 :program \"blah\" :text \"test entry\")\n"$
>>>  12: (SWANK::EVAL-FOR-EMACS (SWANK:LISTENER-EVAL "(add-log-entry
>>> :severity 2 :host-uuid \"123\" :timestamp (get-universal-time)\n
>>>                :program \"blah\" :text \"test entry\")\n") "CONSO$
>>>  13: (SWANK::PROCESS-REQUESTS NIL)
>>>  14: ((LAMBDA ()))
>>>  15: ((LAMBDA (SWANK-BACKEND::HOOK SWANK-BACKEND::FUN)) #<FUNCTION
>>> SWANK:SWANK-DEBUGGER-HOOK> #<CLOSURE (LAMBDA #) {10038AD269}>)
>>>  16: (SWANK::CALL-WITH-REDIRECTED-IO #<SWANK::CONNECTION {10032FB281}>
>>> #<CLOSURE (LAMBDA #) {10038AD289}>)
>>>  17: (SWANK::CALL-WITH-CONNECTION #<SWANK::CONNECTION {10032FB281}>
>>> #<CLOSURE (LAMBDA #) {10038AD269}>)
>>>  18: (SWANK::HANDLE-REQUESTS #<SWANK::CONNECTION {10032FB281}> NIL)
>>>  19: (SWANK::CALL-WITH-BINDINGS NIL #<CLOSURE (LAMBDA #)  
>>> {10034F5189}>)
>>>  20: ((FLET SB-THREAD::WITH-MUTEX-THUNK))
>>>  21: ((FLET #:WITHOUT-INTERRUPTS-BODY-[CALL-WITH-MUTEX]477))
>>>  22: (SB-THREAD::CALL-WITH-MUTEX ..)
>>>  23: ((LAMBDA ()))
>>>  24: ("foreign function: call_into_lisp")
>>>  25: ("foreign function: new_thread_trampoline")
>>>
>>>
>>> Ian Eslick wrote:
>>>> Also try pushing :elephant-without-optimize onto *features* prior to
>>>> doing a force rebuild of elephant.  This should give you some more
>>>> information and rule out any optimization/declaration related bugs.
>>>>
>>>> Thank you,
>>>> Ian
>>>>
>>>>
>>>> On Jan 7, 2009, at 9:10 PM, Kevin Raison wrote:
>>>>
>>>>> I was finally able to recreate this in the repl (as opposed to
>>>>> seeing it
>>>>> in my error logs), so here is a trace:
>>>>>
>>>>> The slot DB-BDB::INDICES-CACHE is unbound in the object
>>>>> #<BDB-INDEXED-BTREE oid:2>.
>>>>>   [Condition of type UNBOUND-SLOT]
>>>>>
>>>>> Backtrace:
>>>>>  0: ((SB-PCL::FAST-METHOD SLOT-UNBOUND (T T T)) #<unavailable
>>>>> argument> #<unavailable argument> #<unavailable argument>
>>>>> #<BDB-INDEXED-BTREE oid:2> DB-BDB::INDICES-CACHE)
>>>>>  1: (SB-PCL::SLOT-UNBOUND-INTERNAL #<BDB-INDEXED-BTREE oid:2> 2)
>>>>>  2: ((SB-PCL::FAST-METHOD (SETF GET-VALUE) (T T
>>>>> DB-BDB::BDB-INDEXED-BTREE)) #<unavailable lambda list>)
>>>>>  3: ((LAMBDA (WORD)) "34766")
>>>>>  4: (SB-IMPL::MAP1 #<CLOSURE (LAMBDA #) {1003CA4ED9}> (("" "asa"
>>>>> "106007" "deny" "inbound" "udp" ...)) :LIST T)
>>>>>  5: (MAPCAR #<CLOSURE (LAMBDA #) {1003CA4ED9}> ("" "asa" "106007"
>>>>> "deny" "inbound" "udp" ...))[:EXTERNAL]
>>>>>  6: ((SB-PCL::FAST-METHOD INDEX-LOG-ENTRY (LOG-ENTRY)) #<unavailable
>>>>> argument> #<unavailable argument> #<LOG-ENTRY oid:6373>)
>>>>>  7: ((LAMBDA ()))
>>>>>  8: ((SB-PCL::FAST-METHOD ELEPHANT::EXECUTE-TRANSACTION
>>>>> (DB-BDB::BDB-STORE-CONTROLLER T)) #<unavailable argument>
>>>>> #<unavailable
>>>>> argument> #<unavailable argument> #<unavailable argument>) 
>>>>> [:EXTERNAL]
>>>>>  9: (ADD-LOG-ENTRY ..)
>>>>> 10: (SB-INT:SIMPLE-EVAL-IN-LEXENV ..)
>>>>>
>>>>> And here is the code that sometimes causes the issue:
>>>>> (defmethod index-log-entry ((log-entry log-entry))
>>>>>  (let ((*store-controller* *syslog-controller*))
>>>>>    (let ((btree (get-from-root "log-entry-index" :sc
>>>>> *syslog-controller*)))
>>>>>      (map-words #'(lambda (word)
>>>>>                     (unless (or (member word *superfluous-
>>>>> words* :test
>>>>> #'string-equal)
>>>>>                                 (< (length word) 2))
>>>>>                       (unless (existsp word btree)
>>>>>                         (setf (get-value word btree) (make-pset :sc
>>>>> *syslog-controller*)))
>>>>>                       (insert-item log-entry
>>>>>                                    (get-value word btree))))
>>>>>                 (text log-entry)))))
>>>>>
>>>>>
>>>>> If I drop and recreate the btree, everything runs smoothly (no  
>>>>> errors)
>>>>> for a few hours and then the above error message starts showing up
>>>>> about
>>>>> half to three quarters of the time.
>>>>>
>>>>> Thanks again,
>>>>> Kevin
>>>>>
>>>>>
>>>>> Kevin Raison wrote:
>>>>>> I am seeing an intermittent error with 1.0 alpha when trying to
>>>>>> write to
>>>>>> an indexed btree (using BerkeleyDB 4.7 as provided by Ubuntu's
>>>>>> package
>>>>>> repositories):
>>>>>>
>>>>>> The slot DB-BDB::INDICES-CACHE is unbound in the object
>>>>>> #<BDB-INDEXED-BTREE oid:2>
>>>>>>
>>>>>> Within the same thread, sometimes this happens and sometimes I am
>>>>>> able
>>>>>> to read and write to the btree.  I am using sbcl 1.0.24 on 32 bit
>>>>>> intel
>>>>>> linux.  Is there something obvious that might help alleviate this  
>>>>>> or
>>>>>> should I provide more context?
>>>>>>
>>>>>> Thanks!
>>>>>> Kevin
>>>>>>
>>>>>> _______________________________________________
>>>>>> elephant-devel site list
>>>>>> elephant-devel at common-lisp.net
>>>>>> http://common-lisp.net/mailman/listinfo/elephant-devel
>>>>> _______________________________________________
>>>>> elephant-devel site list
>>>>> elephant-devel at common-lisp.net
>>>>> http://common-lisp.net/mailman/listinfo/elephant-devel
>>>> _______________________________________________
>>>> elephant-devel site list
>>>> elephant-devel at common-lisp.net
>>>> http://common-lisp.net/mailman/listinfo/elephant-devel
>>> _______________________________________________
>>> elephant-devel site list
>>> elephant-devel at common-lisp.net
>>> http://common-lisp.net/mailman/listinfo/elephant-devel
>>
>> _______________________________________________
>> elephant-devel site list
>> elephant-devel at common-lisp.net
>> http://common-lisp.net/mailman/listinfo/elephant-devel
> 
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel



From eslick at media.mit.edu  Sun Jan 11 03:51:31 2009
From: eslick at media.mit.edu (Ian Eslick)
Date: Sat, 10 Jan 2009 22:51:31 -0500
Subject: [elephant-devel] Strange behavior with indexed-btree
In-Reply-To: <49692EEF.7070208@chatsubo.net>
References: <4965525D.2010106@chatsubo.net>	<49656099.6050708@chatsubo.net>	<AF76A669-036C-4597-9FB1-3DEA1C39DCB2@media.mit.edu>	<4968F123.5090802@chatsubo.net>	<841C8B93-7A80-4934-9ACE-31262416C283@media.mit.edu>
	<49690DAE.7030004@chatsubo.net> <49692EEF.7070208@chatsubo.net>
Message-ID: <CD09C818-5174-4FA5-B192-0F67C3514077@media.mit.edu>

This is very odd behavior.  I have a live webapp that uses all this  
stuff pretty heavily.

I did see this active cursor message earlier, but it was due to some  
debugging churn in the BDB transaction handler a day or two ago.   
Those issues should all be resolved now, but it's possible something  
isn't working right.  Since I did make some changes to the main  
transaction macros and that means that everything depending on those  
macros needs to be compiled - asdf should do that automatically but  
you may want to force a rebuild on the new elephant tree and your app.

Sometimes I find it useful to manually run a full, recover-fatal pass  
manually just to be sure BDB itself is happy.  (db_recover -f)

Deserialization errors have become pretty rare.  I suppose it is  
possible that you have a database that has a value that somehow got  
corrupted.  Have you tried your app on a fresh database?  It might be  
worth doing that to see if you can reproduce the problem.  (How  
critical is the data you have in there now?)

By the way:
- Do you have a 32 or 64 bit BDB/SBCL?
- Have you double checked your my-config.sexp to make sure everything  
is set right?

Can you produce a stand-alone test case I can try locally?  If there  
aren't too many dependencies, I'd be happy to try reproduce this  
locally or login to a machine and look at it remotely.

By the way, stack traces should be a bit more meaningful now if you  
want to send me the latest full stack trace.

Another thing you can try is tracing 'get-value' and (setf get-value)  
to see all the data you are reading/writing; perhaps it's one  
particular value that is causing the serializer to choke?

Ian

On Jan 10, 2009, at 6:27 PM, Kevin Raison wrote:

> Unfortunately, things remain as before.  Here is some additional debug
> output that I found in the *inferior-lisp* buffer of my emacs session:
>
> Deserialization error in map: returning nil for element
> transaction has active cursors
> PANIC: Invalid argument
> PANIC: DB_RUNRECOVERY: Fatal error, run database recovery
> PANIC: fatal region error detected; run recovery
>
>
>
> The deserialization error happens in one of two spots:
> - when trying to get-from-root a btree or indexed btree
> - when trying to get a pset via a call to get-value on the btree
>
> Running recovery does not help;  the errors start as soon as I load  
> and
> execute my code post-recovery.
>
> Ian, can you enlighten me as to where in the code you believe these
> errors to originate?  Also, can you think of anything that I might be
> doing incorrectly in my code that would cause this sort of behavior?
> Here is a brief outline of what I am doing:
>
> There is
>  - a waitqueue (via sbcl's sb-thread:make-waitqueue)
>  - a tcp listener thread that writes its input to an incoming work  
> queue
>  - 3 reader threads who use the waitqueue to get work items from the
> work queue
> The readers parse the incoming data and create a persistent object  
> from
> it.  They also index the individual words of one of the persistent
> object's slots using a btree.  These actions are contained in a
> with-transaction block.
> There is also a hunchentoot web server thread running.  Users can  
> search
> the indexed text via this web interface.
>
> It is really rather simple.  The errors crop up randomly, sometimes  
> when
> I am running searches via the web interface while the listener and
> reader threads are active, and sometimes while the listener and reader
> threads are doing their thing without any web-based reads happening.
>
> Cheers,
> Kevin
>
>
> Kevin Raison wrote:
>> Thanks, Ian.  All tests pass for me as well.  I am running my
>> application with the new code and will let you know how it goes.
>>
>> Thanks again!
>> Kevin
>>
>> Ian Eslick wrote:
>>> I just checked in a few more fixes a few minutes ago.  I finally was
>>> able to reproduce some of this locally.  Try cleaning out the test
>>> database and re-running the tests.  Everything passes for me on a
>>> fresh DB and my own application is running fine too.
>>>
>>> Ian
>>>
>>> On Jan 10, 2009, at 2:04 PM, Kevin Raison wrote:
>>>
>>>> After pulling the latest patches and rebuilding without
>>>> optimizations as
>>>> you suggest, I get the following errors, the second of which is
>>>> slightly
>>>> different than what I was receiving previously.
>>>>
>>>> First, a deserialization error:
>>>> Condition ELEPHANT-TYPE-DESERIALIZATION-ERROR was signalled.
>>>>   [Condition of type ELEPHANT-TYPE-DESERIALIZATION-ERROR]
>>>>
>>>> Restarts:
>>>> 0: [RETRY] Retry SLIME REPL evaluation request.
>>>> 1: [ABORT] Return to SLIME's top level.
>>>> 2: [TERMINATE-THREAD] Terminate this thread (#<THREAD "repl-thread"
>>>> RUNNING {10034F31F1}>)
>>>>
>>>> Backtrace:
>>>>  0: ((LAMBDA (SWANK-BACKEND::DEBUGGER-LOOP-FN)) #<FUNCTION  
>>>> (LAMBDA #)
>>>> {10040B2389}>)
>>>>  1: (SWANK::DEBUG-IN-EMACS #<ELEPHANT-TYPE-DESERIALIZATION-ERROR
>>>> {1002CB3531}>)
>>>>  2: (SWANK:INVOKE-SLIME-DEBUGGER #<ELEPHANT-TYPE-DESERIALIZATION-
>>>> ERROR
>>>> {1002CB3531}>)
>>>>  3: ((LAMBDA (SWANK-BACKEND::HOOK SWANK-BACKEND::FUN)) #<FUNCTION
>>>> SWANK:SWANK-DEBUGGER-HOOK> #<CLOSURE (LAMBDA #) {1002CB3AC9}>)
>>>>  4: (INVOKE-DEBUGGER #<ELEPHANT-TYPE-DESERIALIZATION-ERROR
>>>> {1002CB3531}>)
>>>>  5: ((FLET #:FUN23) #<ELEPHANT-TYPE-DESERIALIZATION-ERROR
>>>> {1002CB3531}>)
>>>>  6: ((SB-PCL::FAST-METHOD ELEPHANT::EXECUTE-TRANSACTION
>>>> (DB-BDB::BDB-STORE-CONTROLLER T)) ..)[:EXTERNAL]
>>>>  7: (SB-INT:SIMPLE-EVAL-IN-LEXENV (REINDEX-LOG-ENTRIES) #<NULL-
>>>> LEXENV>)
>>>>  8: (SWANK::EVAL-REGION "(reindex-log-entries)\n")
>>>>  9: ((LAMBDA ()))
>>>> 10: (SWANK::TRACK-PACKAGE #<CLOSURE (LAMBDA #) {1002C596F9}>)
>>>> 11: (SWANK::CALL-WITH-RETRY-RESTART "Retry SLIME REPL evaluation
>>>> request." #<CLOSURE (LAMBDA #) {1002C59619}>)
>>>> 12: (SWANK::CALL-WITH-BUFFER-SYNTAX NIL #<CLOSURE (LAMBDA #)
>>>> {1002C595E9}>)
>>>> 13: (SWANK::REPL-EVAL "(reindex-log-entries)\n")
>>>>
>>>> And then this one with any subsequent access to the bdb:
>>>>
>>>> Bad type argument:
>>>>
>>>>
>>>>  BDB-DB-ERROR
>>>>   [Condition of type SIMPLE-TYPE-ERROR]
>>>>
>>>> Restarts:
>>>> 0: [RETRY] Retry SLIME REPL evaluation request.
>>>> 1: [ABORT] Return to SLIME's top level.
>>>> 2: [TERMINATE-THREAD] Terminate this thread (#<THREAD "repl-thread"
>>>> RUNNING {10034F31F1}>)
>>>>
>>>> Backtrace:
>>>>  0: (MAKE-CONDITION BDB-DB-ERROR)[:EXTERNAL]
>>>>  1: (ERROR BDB-DB-ERROR)[:EXTERNAL]
>>>>  2: ((SB-PCL::FAST-METHOD ELEPHANT::EXECUTE-TRANSACTION
>>>> (DB-BDB::BDB-STORE-CONTROLLER T)) #<unused argument> #<unused
>>>> argument>
>>>> #<BDB-STORE-CONTROLLER /srv/console/db/> #<CLOSURE (LAMBDA #)
>>>> {10037C53D9$
>>>>  3: (ADD-LOG-ENTRY)[:EXTERNAL]
>>>>  4: (SB-INT:SIMPLE-EVAL-IN-LEXENV (ADD-LOG-ENTRY :SEVERITY 2
>>>> :HOST-UUID "123" :TIMESTAMP ...) #<NULL-LEXENV>)
>>>>  5: (SWANK::EVAL-REGION "(add-log-entry :severity 2 :host-uuid
>>>> \"123\"
>>>> :timestamp (get-universal-time)\n                       :program
>>>> \"blah\" :text \"test entry\")\n")
>>>>  6: ((LAMBDA ()))
>>>>  7: (SWANK::TRACK-PACKAGE #<CLOSURE (LAMBDA #) {10037C1BF9}>)
>>>>  8: (SWANK::CALL-WITH-RETRY-RESTART "Retry SLIME REPL evaluation
>>>> request." #<CLOSURE (LAMBDA #) {1003636499}>)
>>>>  9: (SWANK::CALL-WITH-BUFFER-SYNTAX NIL #<CLOSURE (LAMBDA #)
>>>> {1003636469}>)
>>>> 10: (SWANK::REPL-EVAL "(add-log-entry :severity 2 :host-uuid  
>>>> \"123\"
>>>> :timestamp (get-universal-time)\n                 :program \"blah\"
>>>> :text \"test entry\")\n")
>>>> 11: (SB-INT:SIMPLE-EVAL-IN-LEXENV (SWANK:LISTENER-EVAL "(add-log-
>>>> entry
>>>> :severity 2 :host-uuid \"123\" :timestamp (get-universal-time)\n
>>>>                :program \"blah\" :text \"test entry\")\n"$
>>>> 12: (SWANK::EVAL-FOR-EMACS (SWANK:LISTENER-EVAL "(add-log-entry
>>>> :severity 2 :host-uuid \"123\" :timestamp (get-universal-time)\n
>>>>               :program \"blah\" :text \"test entry\")\n") "CONSO$
>>>> 13: (SWANK::PROCESS-REQUESTS NIL)
>>>> 14: ((LAMBDA ()))
>>>> 15: ((LAMBDA (SWANK-BACKEND::HOOK SWANK-BACKEND::FUN)) #<FUNCTION
>>>> SWANK:SWANK-DEBUGGER-HOOK> #<CLOSURE (LAMBDA #) {10038AD269}>)
>>>> 16: (SWANK::CALL-WITH-REDIRECTED-IO #<SWANK::CONNECTION  
>>>> {10032FB281}>
>>>> #<CLOSURE (LAMBDA #) {10038AD289}>)
>>>> 17: (SWANK::CALL-WITH-CONNECTION #<SWANK::CONNECTION {10032FB281}>
>>>> #<CLOSURE (LAMBDA #) {10038AD269}>)
>>>> 18: (SWANK::HANDLE-REQUESTS #<SWANK::CONNECTION {10032FB281}> NIL)
>>>> 19: (SWANK::CALL-WITH-BINDINGS NIL #<CLOSURE (LAMBDA #)
>>>> {10034F5189}>)
>>>> 20: ((FLET SB-THREAD::WITH-MUTEX-THUNK))
>>>> 21: ((FLET #:WITHOUT-INTERRUPTS-BODY-[CALL-WITH-MUTEX]477))
>>>> 22: (SB-THREAD::CALL-WITH-MUTEX ..)
>>>> 23: ((LAMBDA ()))
>>>> 24: ("foreign function: call_into_lisp")
>>>> 25: ("foreign function: new_thread_trampoline")
>>>>
>>>>
>>>> Ian Eslick wrote:
>>>>> Also try pushing :elephant-without-optimize onto *features*  
>>>>> prior to
>>>>> doing a force rebuild of elephant.  This should give you some more
>>>>> information and rule out any optimization/declaration related  
>>>>> bugs.
>>>>>
>>>>> Thank you,
>>>>> Ian
>>>>>
>>>>>
>>>>> On Jan 7, 2009, at 9:10 PM, Kevin Raison wrote:
>>>>>
>>>>>> I was finally able to recreate this in the repl (as opposed to
>>>>>> seeing it
>>>>>> in my error logs), so here is a trace:
>>>>>>
>>>>>> The slot DB-BDB::INDICES-CACHE is unbound in the object
>>>>>> #<BDB-INDEXED-BTREE oid:2>.
>>>>>>  [Condition of type UNBOUND-SLOT]
>>>>>>
>>>>>> Backtrace:
>>>>>> 0: ((SB-PCL::FAST-METHOD SLOT-UNBOUND (T T T)) #<unavailable
>>>>>> argument> #<unavailable argument> #<unavailable argument>
>>>>>> #<BDB-INDEXED-BTREE oid:2> DB-BDB::INDICES-CACHE)
>>>>>> 1: (SB-PCL::SLOT-UNBOUND-INTERNAL #<BDB-INDEXED-BTREE oid:2> 2)
>>>>>> 2: ((SB-PCL::FAST-METHOD (SETF GET-VALUE) (T T
>>>>>> DB-BDB::BDB-INDEXED-BTREE)) #<unavailable lambda list>)
>>>>>> 3: ((LAMBDA (WORD)) "34766")
>>>>>> 4: (SB-IMPL::MAP1 #<CLOSURE (LAMBDA #) {1003CA4ED9}> (("" "asa"
>>>>>> "106007" "deny" "inbound" "udp" ...)) :LIST T)
>>>>>> 5: (MAPCAR #<CLOSURE (LAMBDA #) {1003CA4ED9}> ("" "asa" "106007"
>>>>>> "deny" "inbound" "udp" ...))[:EXTERNAL]
>>>>>> 6: ((SB-PCL::FAST-METHOD INDEX-LOG-ENTRY (LOG-ENTRY))  
>>>>>> #<unavailable
>>>>>> argument> #<unavailable argument> #<LOG-ENTRY oid:6373>)
>>>>>> 7: ((LAMBDA ()))
>>>>>> 8: ((SB-PCL::FAST-METHOD ELEPHANT::EXECUTE-TRANSACTION
>>>>>> (DB-BDB::BDB-STORE-CONTROLLER T)) #<unavailable argument>
>>>>>> #<unavailable
>>>>>> argument> #<unavailable argument> #<unavailable argument>)
>>>>>> [:EXTERNAL]
>>>>>> 9: (ADD-LOG-ENTRY ..)
>>>>>> 10: (SB-INT:SIMPLE-EVAL-IN-LEXENV ..)
>>>>>>
>>>>>> And here is the code that sometimes causes the issue:
>>>>>> (defmethod index-log-entry ((log-entry log-entry))
>>>>>> (let ((*store-controller* *syslog-controller*))
>>>>>>   (let ((btree (get-from-root "log-entry-index" :sc
>>>>>> *syslog-controller*)))
>>>>>>     (map-words #'(lambda (word)
>>>>>>                    (unless (or (member word *superfluous-
>>>>>> words* :test
>>>>>> #'string-equal)
>>>>>>                                (< (length word) 2))
>>>>>>                      (unless (existsp word btree)
>>>>>>                        (setf (get-value word btree) (make- 
>>>>>> pset :sc
>>>>>> *syslog-controller*)))
>>>>>>                      (insert-item log-entry
>>>>>>                                   (get-value word btree))))
>>>>>>                (text log-entry)))))
>>>>>>
>>>>>>
>>>>>> If I drop and recreate the btree, everything runs smoothly (no
>>>>>> errors)
>>>>>> for a few hours and then the above error message starts showing  
>>>>>> up
>>>>>> about
>>>>>> half to three quarters of the time.
>>>>>>
>>>>>> Thanks again,
>>>>>> Kevin
>>>>>>
>>>>>>
>>>>>> Kevin Raison wrote:
>>>>>>> I am seeing an intermittent error with 1.0 alpha when trying to
>>>>>>> write to
>>>>>>> an indexed btree (using BerkeleyDB 4.7 as provided by Ubuntu's
>>>>>>> package
>>>>>>> repositories):
>>>>>>>
>>>>>>> The slot DB-BDB::INDICES-CACHE is unbound in the object
>>>>>>> #<BDB-INDEXED-BTREE oid:2>
>>>>>>>
>>>>>>> Within the same thread, sometimes this happens and sometimes I  
>>>>>>> am
>>>>>>> able
>>>>>>> to read and write to the btree.  I am using sbcl 1.0.24 on 32  
>>>>>>> bit
>>>>>>> intel
>>>>>>> linux.  Is there something obvious that might help alleviate  
>>>>>>> this
>>>>>>> or
>>>>>>> should I provide more context?
>>>>>>>
>>>>>>> Thanks!
>>>>>>> Kevin
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> elephant-devel site list
>>>>>>> elephant-devel at common-lisp.net
>>>>>>> http://common-lisp.net/mailman/listinfo/elephant-devel
>>>>>> _______________________________________________
>>>>>> elephant-devel site list
>>>>>> elephant-devel at common-lisp.net
>>>>>> http://common-lisp.net/mailman/listinfo/elephant-devel
>>>>> _______________________________________________
>>>>> elephant-devel site list
>>>>> elephant-devel at common-lisp.net
>>>>> http://common-lisp.net/mailman/listinfo/elephant-devel
>>>> _______________________________________________
>>>> elephant-devel site list
>>>> elephant-devel at common-lisp.net
>>>> http://common-lisp.net/mailman/listinfo/elephant-devel
>>>
>>> _______________________________________________
>>> elephant-devel site list
>>> elephant-devel at common-lisp.net
>>> http://common-lisp.net/mailman/listinfo/elephant-devel
>>
>> _______________________________________________
>> elephant-devel site list
>> elephant-devel at common-lisp.net
>> http://common-lisp.net/mailman/listinfo/elephant-devel
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel




From raison at chatsubo.net  Sun Jan 11 04:46:01 2009
From: raison at chatsubo.net (Kevin Raison)
Date: Sat, 10 Jan 2009 20:46:01 -0800
Subject: [elephant-devel] Strange behavior with indexed-btree
In-Reply-To: <CD09C818-5174-4FA5-B192-0F67C3514077@media.mit.edu>
References: <4965525D.2010106@chatsubo.net>	<49656099.6050708@chatsubo.net>	<AF76A669-036C-4597-9FB1-3DEA1C39DCB2@media.mit.edu>	<4968F123.5090802@chatsubo.net>	<841C8B93-7A80-4934-9ACE-31262416C283@media.mit.edu>	<49690DAE.7030004@chatsubo.net>
	<49692EEF.7070208@chatsubo.net>
	<CD09C818-5174-4FA5-B192-0F67C3514077@media.mit.edu>
Message-ID: <49697989.2030605@chatsubo.net>

Ian, luckily I am in the development stages of my app, so I can freely 
throw away my data whenever I like (it is just syslog data from some 
very chatty networking  hardware, so there is always more to be had).  I 
have experienced these issues on 32 and 64 bit systems, though my main 
system is 64 bit.  I just realized that while I have rebuilt elephant 
several times, I have not recompiled my own code in a while.  I will try 
that, and if it fails, I'll figure out some way to to get you access to 
a test case.  Thanks again for all of your help with this;  I hope I am 
contributing to the stabilization of 1.0, and not just chasing my own bugs!

-Kevin

Ian Eslick wrote:
> This is very odd behavior.  I have a live webapp that uses all this  
> stuff pretty heavily.
> 
> I did see this active cursor message earlier, but it was due to some  
> debugging churn in the BDB transaction handler a day or two ago.   
> Those issues should all be resolved now, but it's possible something  
> isn't working right.  Since I did make some changes to the main  
> transaction macros and that means that everything depending on those  
> macros needs to be compiled - asdf should do that automatically but  
> you may want to force a rebuild on the new elephant tree and your app.
> 
> Sometimes I find it useful to manually run a full, recover-fatal pass  
> manually just to be sure BDB itself is happy.  (db_recover -f)
> 
> Deserialization errors have become pretty rare.  I suppose it is  
> possible that you have a database that has a value that somehow got  
> corrupted.  Have you tried your app on a fresh database?  It might be  
> worth doing that to see if you can reproduce the problem.  (How  
> critical is the data you have in there now?)
> 
> By the way:
> - Do you have a 32 or 64 bit BDB/SBCL?
> - Have you double checked your my-config.sexp to make sure everything  
> is set right?
> 
> Can you produce a stand-alone test case I can try locally?  If there  
> aren't too many dependencies, I'd be happy to try reproduce this  
> locally or login to a machine and look at it remotely.
> 
> By the way, stack traces should be a bit more meaningful now if you  
> want to send me the latest full stack trace.
> 
> Another thing you can try is tracing 'get-value' and (setf get-value)  
> to see all the data you are reading/writing; perhaps it's one  
> particular value that is causing the serializer to choke?
> 
> Ian
> 
> On Jan 10, 2009, at 6:27 PM, Kevin Raison wrote:
> 
>> Unfortunately, things remain as before.  Here is some additional debug
>> output that I found in the *inferior-lisp* buffer of my emacs session:
>>
>> Deserialization error in map: returning nil for element
>> transaction has active cursors
>> PANIC: Invalid argument
>> PANIC: DB_RUNRECOVERY: Fatal error, run database recovery
>> PANIC: fatal region error detected; run recovery
>>
>>
>>
>> The deserialization error happens in one of two spots:
>> - when trying to get-from-root a btree or indexed btree
>> - when trying to get a pset via a call to get-value on the btree
>>
>> Running recovery does not help;  the errors start as soon as I load  
>> and
>> execute my code post-recovery.
>>
>> Ian, can you enlighten me as to where in the code you believe these
>> errors to originate?  Also, can you think of anything that I might be
>> doing incorrectly in my code that would cause this sort of behavior?
>> Here is a brief outline of what I am doing:
>>
>> There is
>>  - a waitqueue (via sbcl's sb-thread:make-waitqueue)
>>  - a tcp listener thread that writes its input to an incoming work  
>> queue
>>  - 3 reader threads who use the waitqueue to get work items from the
>> work queue
>> The readers parse the incoming data and create a persistent object  
>> from
>> it.  They also index the individual words of one of the persistent
>> object's slots using a btree.  These actions are contained in a
>> with-transaction block.
>> There is also a hunchentoot web server thread running.  Users can  
>> search
>> the indexed text via this web interface.
>>
>> It is really rather simple.  The errors crop up randomly, sometimes  
>> when
>> I am running searches via the web interface while the listener and
>> reader threads are active, and sometimes while the listener and reader
>> threads are doing their thing without any web-based reads happening.
>>
>> Cheers,
>> Kevin
>>
>>
>> Kevin Raison wrote:
>>> Thanks, Ian.  All tests pass for me as well.  I am running my
>>> application with the new code and will let you know how it goes.
>>>
>>> Thanks again!
>>> Kevin
>>>
>>> Ian Eslick wrote:
>>>> I just checked in a few more fixes a few minutes ago.  I finally was
>>>> able to reproduce some of this locally.  Try cleaning out the test
>>>> database and re-running the tests.  Everything passes for me on a
>>>> fresh DB and my own application is running fine too.
>>>>
>>>> Ian
>>>>
>>>> On Jan 10, 2009, at 2:04 PM, Kevin Raison wrote:
>>>>
>>>>> After pulling the latest patches and rebuilding without
>>>>> optimizations as
>>>>> you suggest, I get the following errors, the second of which is
>>>>> slightly
>>>>> different than what I was receiving previously.
>>>>>
>>>>> First, a deserialization error:
>>>>> Condition ELEPHANT-TYPE-DESERIALIZATION-ERROR was signalled.
>>>>>   [Condition of type ELEPHANT-TYPE-DESERIALIZATION-ERROR]
>>>>>
>>>>> Restarts:
>>>>> 0: [RETRY] Retry SLIME REPL evaluation request.
>>>>> 1: [ABORT] Return to SLIME's top level.
>>>>> 2: [TERMINATE-THREAD] Terminate this thread (#<THREAD "repl-thread"
>>>>> RUNNING {10034F31F1}>)
>>>>>
>>>>> Backtrace:
>>>>>  0: ((LAMBDA (SWANK-BACKEND::DEBUGGER-LOOP-FN)) #<FUNCTION  
>>>>> (LAMBDA #)
>>>>> {10040B2389}>)
>>>>>  1: (SWANK::DEBUG-IN-EMACS #<ELEPHANT-TYPE-DESERIALIZATION-ERROR
>>>>> {1002CB3531}>)
>>>>>  2: (SWANK:INVOKE-SLIME-DEBUGGER #<ELEPHANT-TYPE-DESERIALIZATION-
>>>>> ERROR
>>>>> {1002CB3531}>)
>>>>>  3: ((LAMBDA (SWANK-BACKEND::HOOK SWANK-BACKEND::FUN)) #<FUNCTION
>>>>> SWANK:SWANK-DEBUGGER-HOOK> #<CLOSURE (LAMBDA #) {1002CB3AC9}>)
>>>>>  4: (INVOKE-DEBUGGER #<ELEPHANT-TYPE-DESERIALIZATION-ERROR
>>>>> {1002CB3531}>)
>>>>>  5: ((FLET #:FUN23) #<ELEPHANT-TYPE-DESERIALIZATION-ERROR
>>>>> {1002CB3531}>)
>>>>>  6: ((SB-PCL::FAST-METHOD ELEPHANT::EXECUTE-TRANSACTION
>>>>> (DB-BDB::BDB-STORE-CONTROLLER T)) ..)[:EXTERNAL]
>>>>>  7: (SB-INT:SIMPLE-EVAL-IN-LEXENV (REINDEX-LOG-ENTRIES) #<NULL-
>>>>> LEXENV>)
>>>>>  8: (SWANK::EVAL-REGION "(reindex-log-entries)\n")
>>>>>  9: ((LAMBDA ()))
>>>>> 10: (SWANK::TRACK-PACKAGE #<CLOSURE (LAMBDA #) {1002C596F9}>)
>>>>> 11: (SWANK::CALL-WITH-RETRY-RESTART "Retry SLIME REPL evaluation
>>>>> request." #<CLOSURE (LAMBDA #) {1002C59619}>)
>>>>> 12: (SWANK::CALL-WITH-BUFFER-SYNTAX NIL #<CLOSURE (LAMBDA #)
>>>>> {1002C595E9}>)
>>>>> 13: (SWANK::REPL-EVAL "(reindex-log-entries)\n")
>>>>>
>>>>> And then this one with any subsequent access to the bdb:
>>>>>
>>>>> Bad type argument:
>>>>>
>>>>>
>>>>>  BDB-DB-ERROR
>>>>>   [Condition of type SIMPLE-TYPE-ERROR]
>>>>>
>>>>> Restarts:
>>>>> 0: [RETRY] Retry SLIME REPL evaluation request.
>>>>> 1: [ABORT] Return to SLIME's top level.
>>>>> 2: [TERMINATE-THREAD] Terminate this thread (#<THREAD "repl-thread"
>>>>> RUNNING {10034F31F1}>)
>>>>>
>>>>> Backtrace:
>>>>>  0: (MAKE-CONDITION BDB-DB-ERROR)[:EXTERNAL]
>>>>>  1: (ERROR BDB-DB-ERROR)[:EXTERNAL]
>>>>>  2: ((SB-PCL::FAST-METHOD ELEPHANT::EXECUTE-TRANSACTION
>>>>> (DB-BDB::BDB-STORE-CONTROLLER T)) #<unused argument> #<unused
>>>>> argument>
>>>>> #<BDB-STORE-CONTROLLER /srv/console/db/> #<CLOSURE (LAMBDA #)
>>>>> {10037C53D9$
>>>>>  3: (ADD-LOG-ENTRY)[:EXTERNAL]
>>>>>  4: (SB-INT:SIMPLE-EVAL-IN-LEXENV (ADD-LOG-ENTRY :SEVERITY 2
>>>>> :HOST-UUID "123" :TIMESTAMP ...) #<NULL-LEXENV>)
>>>>>  5: (SWANK::EVAL-REGION "(add-log-entry :severity 2 :host-uuid
>>>>> \"123\"
>>>>> :timestamp (get-universal-time)\n                       :program
>>>>> \"blah\" :text \"test entry\")\n")
>>>>>  6: ((LAMBDA ()))
>>>>>  7: (SWANK::TRACK-PACKAGE #<CLOSURE (LAMBDA #) {10037C1BF9}>)
>>>>>  8: (SWANK::CALL-WITH-RETRY-RESTART "Retry SLIME REPL evaluation
>>>>> request." #<CLOSURE (LAMBDA #) {1003636499}>)
>>>>>  9: (SWANK::CALL-WITH-BUFFER-SYNTAX NIL #<CLOSURE (LAMBDA #)
>>>>> {1003636469}>)
>>>>> 10: (SWANK::REPL-EVAL "(add-log-entry :severity 2 :host-uuid  
>>>>> \"123\"
>>>>> :timestamp (get-universal-time)\n                 :program \"blah\"
>>>>> :text \"test entry\")\n")
>>>>> 11: (SB-INT:SIMPLE-EVAL-IN-LEXENV (SWANK:LISTENER-EVAL "(add-log-
>>>>> entry
>>>>> :severity 2 :host-uuid \"123\" :timestamp (get-universal-time)\n
>>>>>                :program \"blah\" :text \"test entry\")\n"$
>>>>> 12: (SWANK::EVAL-FOR-EMACS (SWANK:LISTENER-EVAL "(add-log-entry
>>>>> :severity 2 :host-uuid \"123\" :timestamp (get-universal-time)\n
>>>>>               :program \"blah\" :text \"test entry\")\n") "CONSO$
>>>>> 13: (SWANK::PROCESS-REQUESTS NIL)
>>>>> 14: ((LAMBDA ()))
>>>>> 15: ((LAMBDA (SWANK-BACKEND::HOOK SWANK-BACKEND::FUN)) #<FUNCTION
>>>>> SWANK:SWANK-DEBUGGER-HOOK> #<CLOSURE (LAMBDA #) {10038AD269}>)
>>>>> 16: (SWANK::CALL-WITH-REDIRECTED-IO #<SWANK::CONNECTION  
>>>>> {10032FB281}>
>>>>> #<CLOSURE (LAMBDA #) {10038AD289}>)
>>>>> 17: (SWANK::CALL-WITH-CONNECTION #<SWANK::CONNECTION {10032FB281}>
>>>>> #<CLOSURE (LAMBDA #) {10038AD269}>)
>>>>> 18: (SWANK::HANDLE-REQUESTS #<SWANK::CONNECTION {10032FB281}> NIL)
>>>>> 19: (SWANK::CALL-WITH-BINDINGS NIL #<CLOSURE (LAMBDA #)
>>>>> {10034F5189}>)
>>>>> 20: ((FLET SB-THREAD::WITH-MUTEX-THUNK))
>>>>> 21: ((FLET #:WITHOUT-INTERRUPTS-BODY-[CALL-WITH-MUTEX]477))
>>>>> 22: (SB-THREAD::CALL-WITH-MUTEX ..)
>>>>> 23: ((LAMBDA ()))
>>>>> 24: ("foreign function: call_into_lisp")
>>>>> 25: ("foreign function: new_thread_trampoline")
>>>>>
>>>>>
>>>>> Ian Eslick wrote:
>>>>>> Also try pushing :elephant-without-optimize onto *features*  
>>>>>> prior to
>>>>>> doing a force rebuild of elephant.  This should give you some more
>>>>>> information and rule out any optimization/declaration related  
>>>>>> bugs.
>>>>>>
>>>>>> Thank you,
>>>>>> Ian
>>>>>>
>>>>>>
>>>>>> On Jan 7, 2009, at 9:10 PM, Kevin Raison wrote:
>>>>>>
>>>>>>> I was finally able to recreate this in the repl (as opposed to
>>>>>>> seeing it
>>>>>>> in my error logs), so here is a trace:
>>>>>>>
>>>>>>> The slot DB-BDB::INDICES-CACHE is unbound in the object
>>>>>>> #<BDB-INDEXED-BTREE oid:2>.
>>>>>>>  [Condition of type UNBOUND-SLOT]
>>>>>>>
>>>>>>> Backtrace:
>>>>>>> 0: ((SB-PCL::FAST-METHOD SLOT-UNBOUND (T T T)) #<unavailable
>>>>>>> argument> #<unavailable argument> #<unavailable argument>
>>>>>>> #<BDB-INDEXED-BTREE oid:2> DB-BDB::INDICES-CACHE)
>>>>>>> 1: (SB-PCL::SLOT-UNBOUND-INTERNAL #<BDB-INDEXED-BTREE oid:2> 2)
>>>>>>> 2: ((SB-PCL::FAST-METHOD (SETF GET-VALUE) (T T
>>>>>>> DB-BDB::BDB-INDEXED-BTREE)) #<unavailable lambda list>)
>>>>>>> 3: ((LAMBDA (WORD)) "34766")
>>>>>>> 4: (SB-IMPL::MAP1 #<CLOSURE (LAMBDA #) {1003CA4ED9}> (("" "asa"
>>>>>>> "106007" "deny" "inbound" "udp" ...)) :LIST T)
>>>>>>> 5: (MAPCAR #<CLOSURE (LAMBDA #) {1003CA4ED9}> ("" "asa" "106007"
>>>>>>> "deny" "inbound" "udp" ...))[:EXTERNAL]
>>>>>>> 6: ((SB-PCL::FAST-METHOD INDEX-LOG-ENTRY (LOG-ENTRY))  
>>>>>>> #<unavailable
>>>>>>> argument> #<unavailable argument> #<LOG-ENTRY oid:6373>)
>>>>>>> 7: ((LAMBDA ()))
>>>>>>> 8: ((SB-PCL::FAST-METHOD ELEPHANT::EXECUTE-TRANSACTION
>>>>>>> (DB-BDB::BDB-STORE-CONTROLLER T)) #<unavailable argument>
>>>>>>> #<unavailable
>>>>>>> argument> #<unavailable argument> #<unavailable argument>)
>>>>>>> [:EXTERNAL]
>>>>>>> 9: (ADD-LOG-ENTRY ..)
>>>>>>> 10: (SB-INT:SIMPLE-EVAL-IN-LEXENV ..)
>>>>>>>
>>>>>>> And here is the code that sometimes causes the issue:
>>>>>>> (defmethod index-log-entry ((log-entry log-entry))
>>>>>>> (let ((*store-controller* *syslog-controller*))
>>>>>>>   (let ((btree (get-from-root "log-entry-index" :sc
>>>>>>> *syslog-controller*)))
>>>>>>>     (map-words #'(lambda (word)
>>>>>>>                    (unless (or (member word *superfluous-
>>>>>>> words* :test
>>>>>>> #'string-equal)
>>>>>>>                                (< (length word) 2))
>>>>>>>                      (unless (existsp word btree)
>>>>>>>                        (setf (get-value word btree) (make- 
>>>>>>> pset :sc
>>>>>>> *syslog-controller*)))
>>>>>>>                      (insert-item log-entry
>>>>>>>                                   (get-value word btree))))
>>>>>>>                (text log-entry)))))
>>>>>>>
>>>>>>>
>>>>>>> If I drop and recreate the btree, everything runs smoothly (no
>>>>>>> errors)
>>>>>>> for a few hours and then the above error message starts showing  
>>>>>>> up
>>>>>>> about
>>>>>>> half to three quarters of the time.
>>>>>>>
>>>>>>> Thanks again,
>>>>>>> Kevin
>>>>>>>
>>>>>>>
>>>>>>> Kevin Raison wrote:
>>>>>>>> I am seeing an intermittent error with 1.0 alpha when trying to
>>>>>>>> write to
>>>>>>>> an indexed btree (using BerkeleyDB 4.7 as provided by Ubuntu's
>>>>>>>> package
>>>>>>>> repositories):
>>>>>>>>
>>>>>>>> The slot DB-BDB::INDICES-CACHE is unbound in the object
>>>>>>>> #<BDB-INDEXED-BTREE oid:2>
>>>>>>>>
>>>>>>>> Within the same thread, sometimes this happens and sometimes I  
>>>>>>>> am
>>>>>>>> able
>>>>>>>> to read and write to the btree.  I am using sbcl 1.0.24 on 32  
>>>>>>>> bit
>>>>>>>> intel
>>>>>>>> linux.  Is there something obvious that might help alleviate  
>>>>>>>> this
>>>>>>>> or
>>>>>>>> should I provide more context?
>>>>>>>>
>>>>>>>> Thanks!
>>>>>>>> Kevin
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> elephant-devel site list
>>>>>>>> elephant-devel at common-lisp.net
>>>>>>>> http://common-lisp.net/mailman/listinfo/elephant-devel
>>>>>>> _______________________________________________
>>>>>>> elephant-devel site list
>>>>>>> elephant-devel at common-lisp.net
>>>>>>> http://common-lisp.net/mailman/listinfo/elephant-devel
>>>>>> _______________________________________________
>>>>>> elephant-devel site list
>>>>>> elephant-devel at common-lisp.net
>>>>>> http://common-lisp.net/mailman/listinfo/elephant-devel
>>>>> _______________________________________________
>>>>> elephant-devel site list
>>>>> elephant-devel at common-lisp.net
>>>>> http://common-lisp.net/mailman/listinfo/elephant-devel
>>>> _______________________________________________
>>>> elephant-devel site list
>>>> elephant-devel at common-lisp.net
>>>> http://common-lisp.net/mailman/listinfo/elephant-devel
>>> _______________________________________________
>>> elephant-devel site list
>>> elephant-devel at common-lisp.net
>>> http://common-lisp.net/mailman/listinfo/elephant-devel
>> _______________________________________________
>> elephant-devel site list
>> elephant-devel at common-lisp.net
>> http://common-lisp.net/mailman/listinfo/elephant-devel
> 
> 
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel



From ihuquan at gmail.com  Sun Jan 11 16:03:31 2009
From: ihuquan at gmail.com (quan hu)
Date: Mon, 12 Jan 2009 00:03:31 +0800
Subject: [elephant-devel] Good practices around adding/deleting class
	slots.
In-Reply-To: <ecb948b30901081631j7007aebeye174f51f5e925681@mail.gmail.com>
References: <ecb948b30901081631j7007aebeye174f51f5e925681@mail.gmail.com>
Message-ID: <1cc8deb80901110803y31e71040w8c05b9dcf19eb962@mail.gmail.com>

Hi Yarek,
    Regarding to data migration, it is migration between different stores.

    My understanding of the typical cases are:

     1.  Change the backend. e.g. change BDB to Postmodern .
     2.  Upgrade. e.g. your application uses elephant 0.9.1 and BDB 4.5; To
upgrade elephant from 0.9.1 to 1.0, you need to install BDB 4.7 and run the
migration process.
     3.  Garbage collection. Prior to 1.0, doing a DB migration is the only
way for it. From a recent email, Ian released the online garbage collection
feature in 1.0. It is in experiment status.

Regards
Quan



2009/1/9 Yarek Kowalik <yarek.kowalik at gmail.com>

> In an evolving site there is a fair chance of adding/changing/deleting
> slots from a class.  What are good practices with respect to these actions
> in Elephant?  Are there specific steps that need to be taken to migrate data
> from an old class definition to a new class definition?  Any gotchas one
> needs to be aware of?
>
> BTW, I've read several emails about data migration going by, can someone
> please clarify what exactly "data migration" is?  Is it migrating between
> stores, or different versions of a class?
>
> Yarek
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20090112/90b205eb/attachment.html>

From sky at viridian-project.de  Tue Jan 13 09:42:30 2009
From: sky at viridian-project.de (Leslie P. Polzer)
Date: Tue, 13 Jan 2009 10:42:30 +0100 (CET)
Subject: [elephant-devel] CLSQL Store
In-Reply-To: <1F1F136C-D922-46F8-BAC3-B252D34BF8BC@media.mit.edu>
References: <ecb948b30901021117t4733cbf0m83306c66a863c36d@mail.gmail.com>
	<gjmef5$l64$1@ger.gmane.org>
	<1F1F136C-D922-46F8-BAC3-B252D34BF8BC@media.mit.edu>
Message-ID: <63936.88.73.192.17.1231839750.squirrel@mail.stardawn.org>


> Is anyone still using the CL-SQL store?  Robert and I have discussed
> deprecating it in this or a future release, although SQLite support is
> the only way to use Elephant with a cheap, free, easy to install/
> distribute backend.

I would prefer to ditch CLSQL and SQLite altogether and put
that energy into making BDB installation easier, as proposed
in

  http://www.mail-archive.com/elephant-devel at common-lisp.net/msg01761.html

 Leslie


-- 
LinkedIn Profile: http://www.linkedin.com/in/polzer
Xing Profile: https://www.xing.com/profile/LeslieP_Polzer
Blog: http://blog.viridian-project.de/




From elliottslaughter at gmail.com  Tue Jan 13 09:15:59 2009
From: elliottslaughter at gmail.com (Elliott Slaughter)
Date: Tue, 13 Jan 2009 01:15:59 -0800
Subject: [elephant-devel] Optimization
In-Reply-To: <DC82280E-73A9-4559-959D-6E32E0E4FEF7@media.mit.edu>
References: <42c0ab790812212135w18d4cdcerb7af5b45a04486f0@mail.gmail.com>
	<20081223092255.GA17361@viridian-project.de>
	<786E3BB7-708F-461E-A6FB-3B392614C4FD@media.mit.edu>
	<42c0ab790812272305k78d8ef1atb4c78df5b9624ee3@mail.gmail.com>
	<CC3600A7-1B01-499B-B45D-D2958995EEDA@media.mit.edu>
	<79E0CB99-2CA4-4583-A750-AE0CBA4D5C3C@media.mit.edu>
	<42c0ab790901080022s605eeafh31982acfc5805bd3@mail.gmail.com>
	<FC9EF081-C034-4867-B541-7AF88DB2C851@media.mit.edu>
	<DC82280E-73A9-4559-959D-6E32E0E4FEF7@media.mit.edu>
Message-ID: <42c0ab790901130115l62797a3dn8c1412e5a6a10d37@mail.gmail.com>

Thanks for your comments. Let me try to provide you with a little more
information.

On Thu, Jan 8, 2009 at 4:38 AM, Ian Eslick <eslick at media.mit.edu> wrote:
>
> On Jan 8, 2009, at 3:22 AM, Elliott Slaughter wrote:
>
> > Another issue is that the cache-style declaration in persistent
> > classes doesn't get inherited by default. This particularly annoying
> > since this throws an error merely by inheriting cached slots from a
> > superclass.
>
> Hmmmm...it should inherit by default unless you are overriding the
> base class slot.  I'll look into this too.


It looks like cache-style is inherited when no new cached slots are
declared. But when new cached slots are declared, it produces a rather
strange error. I have attached a test case for you to examine.

> That said, I could probably avoid reading indexed slots if I used
> > the query language discussed on another thread. (Currently I use get-
> > instances-by-value to cons a list of objects, and then sort by the
> > indexed slot to effectively iterate over two sorted slots.)
>
> The query system would basically do this:
>
> (let ((a (map-inverted-index #'identity 'slot1 :collect t :oids t))
>       (b (map-inverted-index #'identity 'slot2 :collect t :oids t)))
>   (mapcar #'elephant::controller-recreate-instance
>          (merge-unique (sort a #'<) (sort b #'<))


What I need to do is more like

(let ((list (map-inverted-index #'identity 'slot1 :value foo :collect t)))
  (sort list #'< :key #'slot2))

which is effectively what I am currently doing with get-instances-by-value.
Is there a better way to efficiently iterate over one value in the first
slot, with the ordering of the second slot? (Should I look into creating
another level of btree for holding each subset of object in order?)

You can also do map-inverted-index and accumulate only values matching
> the second value as discussed earlier using an accumulation closure as
> an argument.  For small sets on the first slot, this is probably
> faster.  In general mapping is always more efficient than get-
> instances...


Most of the time, the sets will be close in size. Less frequently, the
second set will be much larger.

> I get roughly the same results as you on the micro-benchmarks. When
> > using checkout caching in the game, I get about a 2x speed increase.
> > It is still about 3x slower than no database.
>
> It would be interesting to see some profile data on this to see what,
> if anything, in BDB/Elephant could be improved.


Do you want profile results from my package, or from Elephant internals?
Either way, I have attached to profile reports. I hope you find them
helpful.

On Thu, Jan 8, 2009 at 5:02 AM, Ian Eslick <eslick at media.mit.edu> wrote:

> By the way, a write through mode is complex because if a transaction
> is rolled back your index could differ from your memory state so you
> would then have to keep track if slot writes to undo them on rollback...


I don't currently (or expect to) rollback changes, so this isn't an issue
with my particular application.

Are you caching blocks of objects for short term operations or long
> term ones?
>

I would like to cache all objects in the current room, so the only major
context switches should be when the player moves between rooms. So long
term.

-- 
Elliott Slaughter

"Any road followed precisely to its end leads precisely nowhere." - Frank
Herbert
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20090113/b37e0438/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: class_cache-style_inheritence_bug.lisp
Type: text/x-lisp-source
Size: 465 bytes
Desc: not available
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20090113/b37e0438/attachment.bin>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: profile_ele.txt
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20090113/b37e0438/attachment.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: profile_bt-ele.txt
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20090113/b37e0438/attachment-0001.txt>

From yarek.kowalik at gmail.com  Wed Jan 14 01:25:55 2009
From: yarek.kowalik at gmail.com (Yarek Kowalik)
Date: Tue, 13 Jan 2009 17:25:55 -0800
Subject: [elephant-devel] Querying for objects on two slots
In-Reply-To: <gjvlkt$8hh$1@ger.gmane.org>
References: <ecb948b30901052236t4762d164p492ffac56e040152@mail.gmail.com>
	<gjvlkt$8hh$1@ger.gmane.org>
Message-ID: <ecb948b30901131725t308514cj7540f75e5314aae2@mail.gmail.com>

When serializing tuples, is the string representation best:  You suggest
using (format t "~A ~A" a b) - is that efficient enough?  what about doing
(cons a b) = is there a way to index and search for conses? Any other ideas?

Yarek

On Tue, Jan 6, 2009 at 5:17 AM, Alex Mizrahi <killerstorm at newmail.ru> wrote:

> YK> Is this a reasonable way of finding an object of type
> YK> 'my-class that matches on values val-a and val-b for slots a and b?
>
> yep, it is reasonable if you have relatively low number of objects
> in returned by (get-instances-by-value 'my-class 'slot-b val-b) query.
> if number of objects is significant and you get a slowdown because
> of this, you might want to optimize this. a trivial thing is to try it
> symmertrically with slot-a -- whatever returns less objects is better.
> less trivial optimizations would be to work on lower-level -- via
> map-inverted-index (to avoid allocating whole list but instead test objects
> one by one) or even cursor API (this way you can retrieve oids rather
> than objects, which should be faster, and also iterating both indices at
> once might be a significant benefit if values are not uniformly
> distributed).
>
> but the most optimal way doing this in case of high number of objects
> in both slot-a and slot-b queries would be building and using multi-column
> index. unfortunately, Elephant does not help you with it -- either you'll
> have
> to serialize slot tuples into a string (e.g (format nil "~a_~a" slot-a
> slot-b)),
> or reorganize your data to use a custom index structure (like btree of
> btrees).
>
> there are also backend-specific considerations. for postmodern
>
>  (intersection (get-instances-by-value 'my-class 'slot-b val-b)
>                    (get-instances-by-value 'my-class 'slot-a val-a))
>
> would be much faster then testing objects one by one, for BDB -- i doubt
> so.
>
> LP> Use MAP-CLASS, this will considerably speed up the query.
>
> how is that? using at least one index is much better than using no index at
> all.
>
>
>
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20090113/28945e99/attachment.html>

From rsynnott at gmail.com  Wed Jan 14 01:51:48 2009
From: rsynnott at gmail.com (Robert Synnott)
Date: Wed, 14 Jan 2009 01:51:48 +0000
Subject: [elephant-devel] Querying for objects on two slots
In-Reply-To: <ecb948b30901131725t308514cj7540f75e5314aae2@mail.gmail.com>
References: <ecb948b30901052236t4762d164p492ffac56e040152@mail.gmail.com>
	<gjvlkt$8hh$1@ger.gmane.org>
	<ecb948b30901131725t308514cj7540f75e5314aae2@mail.gmail.com>
Message-ID: <24f203480901131751o78df72e8x18013802642ba7e2@mail.gmail.com>

I don't believe the Postmodern backend, at least, allows search by conses.

If you want sensible (ordered) results with a and b if a and/or b are
integers, by the way, you should zero-pad them; otherwise, say, '1 2'
comes after '1 12', which is probably not what you want
Rob

2009/1/14 Yarek Kowalik <yarek.kowalik at gmail.com>:
> When serializing tuples, is the string representation best:  You suggest
> using (format t "~A ~A" a b) - is that efficient enough?  what about doing
> (cons a b) = is there a way to index and search for conses? Any other ideas?
>
> Yarek
>
> On Tue, Jan 6, 2009 at 5:17 AM, Alex Mizrahi <killerstorm at newmail.ru> wrote:
>>
>> YK> Is this a reasonable way of finding an object of type
>> YK> 'my-class that matches on values val-a and val-b for slots a and b?
>>
>> yep, it is reasonable if you have relatively low number of objects
>> in returned by (get-instances-by-value 'my-class 'slot-b val-b) query.
>> if number of objects is significant and you get a slowdown because
>> of this, you might want to optimize this. a trivial thing is to try it
>> symmertrically with slot-a -- whatever returns less objects is better.
>> less trivial optimizations would be to work on lower-level -- via
>> map-inverted-index (to avoid allocating whole list but instead test
>> objects
>> one by one) or even cursor API (this way you can retrieve oids rather
>> than objects, which should be faster, and also iterating both indices at
>> once might be a significant benefit if values are not uniformly
>> distributed).
>>
>> but the most optimal way doing this in case of high number of objects
>> in both slot-a and slot-b queries would be building and using multi-column
>> index. unfortunately, Elephant does not help you with it -- either you'll
>> have
>> to serialize slot tuples into a string (e.g (format nil "~a_~a" slot-a
>> slot-b)),
>> or reorganize your data to use a custom index structure (like btree of
>> btrees).
>>
>> there are also backend-specific considerations. for postmodern
>>
>>  (intersection (get-instances-by-value 'my-class 'slot-b val-b)
>>                    (get-instances-by-value 'my-class 'slot-a val-a))
>>
>> would be much faster then testing objects one by one, for BDB -- i doubt
>> so.
>>
>> LP> Use MAP-CLASS, this will considerably speed up the query.
>>
>> how is that? using at least one index is much better than using no index
>> at
>> all.
>>
>>
>>
>>
>> _______________________________________________
>> elephant-devel site list
>> elephant-devel at common-lisp.net
>> http://common-lisp.net/mailman/listinfo/elephant-devel
>
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel
>



-- 
Robert Synnott
http://myblog.rsynnott.com
MSN: rsynnott at gmail.com
Jabber: rsynnott at gmail.com



From eslick at media.mit.edu  Wed Jan 14 01:57:00 2009
From: eslick at media.mit.edu (Ian Eslick)
Date: Tue, 13 Jan 2009 20:57:00 -0500
Subject: [elephant-devel] Querying for objects on two slots
In-Reply-To: <ecb948b30901131725t308514cj7540f75e5314aae2@mail.gmail.com>
References: <ecb948b30901052236t4762d164p492ffac56e040152@mail.gmail.com>
	<gjvlkt$8hh$1@ger.gmane.org>
	<ecb948b30901131725t308514cj7540f75e5314aae2@mail.gmail.com>
Message-ID: <FA60594D-CCE0-45C4-972F-041C715FED35@media.mit.edu>

I'd like to add a mechanism for sorting on small tuples, but it would  
be different for each backend.  I think I know how to do it for BDB,  
but it will take some work. CLSQL could do it pretty easily (however  
see my response to Leslie's recent mail).

Alex, what would the issues be for the postmodern backend?  Could we  
use a behind the scenes convert-to-string strategy so we had a common  
API for indexing on tuples?

Unless we have a relatively straightforward solution for all back- 
ends, this will have to be a post-1.0 feature.  I'll have to look into  
the idea I had for BDB again before committing either way.

Ian

PS - I just tagged ELEPHANT-0-1-A2 after fixing a few bugs and adding  
some additional small features.  Thanks to Kevin Raison for helping  
track down a bug I introduced...

On Jan 13, 2009, at 8:25 PM, Yarek Kowalik wrote:

> When serializing tuples, is the string representation best:  You  
> suggest using (format t "~A ~A" a b) - is that efficient enough?   
> what about doing (cons a b) = is there a way to index and search for  
> conses? Any other ideas?
>
> Yarek
>
> On Tue, Jan 6, 2009 at 5:17 AM, Alex Mizrahi  
> <killerstorm at newmail.ru> wrote:
> YK> Is this a reasonable way of finding an object of type
> YK> 'my-class that matches on values val-a and val-b for slots a and  
> b?
>
> yep, it is reasonable if you have relatively low number of objects
> in returned by (get-instances-by-value 'my-class 'slot-b val-b) query.
> if number of objects is significant and you get a slowdown because
> of this, you might want to optimize this. a trivial thing is to try it
> symmertrically with slot-a -- whatever returns less objects is better.
> less trivial optimizations would be to work on lower-level -- via
> map-inverted-index (to avoid allocating whole list but instead test  
> objects
> one by one) or even cursor API (this way you can retrieve oids rather
> than objects, which should be faster, and also iterating both  
> indices at
> once might be a significant benefit if values are not uniformly
> distributed).
>
> but the most optimal way doing this in case of high number of objects
> in both slot-a and slot-b queries would be building and using multi- 
> column
> index. unfortunately, Elephant does not help you with it -- either  
> you'll
> have
> to serialize slot tuples into a string (e.g (format nil "~a_~a" slot-a
> slot-b)),
> or reorganize your data to use a custom index structure (like btree of
> btrees).
>
> there are also backend-specific considerations. for postmodern
>
>  (intersection (get-instances-by-value 'my-class 'slot-b val-b)
>                    (get-instances-by-value 'my-class 'slot-a val-a))
>
> would be much faster then testing objects one by one, for BDB -- i  
> doubt so.
>
> LP> Use MAP-CLASS, this will considerably speed up the query.
>
> how is that? using at least one index is much better than using no  
> index at
> all.
>
>
>
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel




From eslick at media.mit.edu  Wed Jan 14 03:16:15 2009
From: eslick at media.mit.edu (Ian Eslick)
Date: Tue, 13 Jan 2009 22:16:15 -0500
Subject: [elephant-devel] Querying for objects on two slots
In-Reply-To: <24f203480901131751o78df72e8x18013802642ba7e2@mail.gmail.com>
References: <ecb948b30901052236t4762d164p492ffac56e040152@mail.gmail.com>
	<gjvlkt$8hh$1@ger.gmane.org>
	<ecb948b30901131725t308514cj7540f75e5314aae2@mail.gmail.com>
	<24f203480901131751o78df72e8x18013802642ba7e2@mail.gmail.com>
Message-ID: <824FD3F0-9648-4FE0-9F7E-58AC238EE46A@media.mit.edu>

There is no good general solution to this is you are going to change  
encodings from something open-ended to something of fixed radix (e.g.  
a pair of integers => fixed length string field).

We might be able to provide some support for specific kinds of items  
defined by a simple specification language.

e.g. '(tuple (integer 16) (integer 16))          ; (to-user-id, from- 
user-id)
      '(tuple (string 40) boolean (integer 8))    ; (last-name, sex,  
age)

For example we could export a multi-item index with a new API and let  
the backend decide how to do it or tell you it can't.  For classes  
this could be an extension so the derived index model.  A specific  
data store could check for violations when you insert and you can do a  
bulk upgrade to rebuild the index if you need to.

Ian

On Jan 13, 2009, at 8:51 PM, Robert Synnott wrote:

> I don't believe the Postmodern backend, at least, allows search by  
> conses.
>
> If you want sensible (ordered) results with a and b if a and/or b are
> integers, by the way, you should zero-pad them; otherwise, say, '1 2'
> comes after '1 12', which is probably not what you want
> Rob
>
> 2009/1/14 Yarek Kowalik <yarek.kowalik at gmail.com>:
>> When serializing tuples, is the string representation best:  You  
>> suggest
>> using (format t "~A ~A" a b) - is that efficient enough?  what  
>> about doing
>> (cons a b) = is there a way to index and search for conses? Any  
>> other ideas?
>>
>> Yarek
>>
>> On Tue, Jan 6, 2009 at 5:17 AM, Alex Mizrahi  
>> <killerstorm at newmail.ru> wrote:
>>>
>>> YK> Is this a reasonable way of finding an object of type
>>> YK> 'my-class that matches on values val-a and val-b for slots a  
>>> and b?
>>>
>>> yep, it is reasonable if you have relatively low number of objects
>>> in returned by (get-instances-by-value 'my-class 'slot-b val-b)  
>>> query.
>>> if number of objects is significant and you get a slowdown because
>>> of this, you might want to optimize this. a trivial thing is to  
>>> try it
>>> symmertrically with slot-a -- whatever returns less objects is  
>>> better.
>>> less trivial optimizations would be to work on lower-level -- via
>>> map-inverted-index (to avoid allocating whole list but instead test
>>> objects
>>> one by one) or even cursor API (this way you can retrieve oids  
>>> rather
>>> than objects, which should be faster, and also iterating both  
>>> indices at
>>> once might be a significant benefit if values are not uniformly
>>> distributed).
>>>
>>> but the most optimal way doing this in case of high number of  
>>> objects
>>> in both slot-a and slot-b queries would be building and using  
>>> multi-column
>>> index. unfortunately, Elephant does not help you with it -- either  
>>> you'll
>>> have
>>> to serialize slot tuples into a string (e.g (format nil "~a_~a"  
>>> slot-a
>>> slot-b)),
>>> or reorganize your data to use a custom index structure (like  
>>> btree of
>>> btrees).
>>>
>>> there are also backend-specific considerations. for postmodern
>>>
>>> (intersection (get-instances-by-value 'my-class 'slot-b val-b)
>>>                   (get-instances-by-value 'my-class 'slot-a val-a))
>>>
>>> would be much faster then testing objects one by one, for BDB -- i  
>>> doubt
>>> so.
>>>
>>> LP> Use MAP-CLASS, this will considerably speed up the query.
>>>
>>> how is that? using at least one index is much better than using no  
>>> index
>>> at
>>> all.
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> elephant-devel site list
>>> elephant-devel at common-lisp.net
>>> http://common-lisp.net/mailman/listinfo/elephant-devel
>>
>>
>> _______________________________________________
>> elephant-devel site list
>> elephant-devel at common-lisp.net
>> http://common-lisp.net/mailman/listinfo/elephant-devel
>>
>
>
>
> -- 
> Robert Synnott
> http://myblog.rsynnott.com
> MSN: rsynnott at gmail.com
> Jabber: rsynnott at gmail.com
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel




From killerstorm at newmail.ru  Wed Jan 14 19:03:28 2009
From: killerstorm at newmail.ru (Alex Mizrahi)
Date: Wed, 14 Jan 2009 21:03:28 +0200
Subject: [elephant-devel] Querying for objects on two slots
References: <ecb948b30901052236t4762d164p492ffac56e040152@mail.gmail.com><gjvlkt$8hh$1@ger.gmane.org><ecb948b30901131725t308514cj7540f75e5314aae2@mail.gmail.com>
	<FA60594D-CCE0-45C4-972F-041C715FED35@media.mit.edu>
Message-ID: <gklcu6$tp2$1@ger.gmane.org>

 IE> Alex, what would the issues be for the postmodern backend?
 IE>   Could we use a behind the scenes convert-to-string strategy so we had
 IE> a common API for indexing on tuples?

doing it in "proper way" (sorting on multiple columns) will be quite hard
in postmodern, as it means complicating cursor stuff even further.
as least i don't know how to do this anyhow easy.

so if you know how to do this converting everything to strings and sorting 
them (if i understand you correctly),
it would be much better. does it need any support from backend or it should
work totally transparently?

p.s.  i've fixed the nasty upgrade-btree issue for postmodern, now i'm 
reviewing stuff, testing
and fixing leftover small issues, so i hope soon postmodern would be 
"ready".

by the way, my email killerstorm at newmail.ru went down few days ago, so if
anybody wants to communicate with me it would be better to use 
alex.mizrahi at gmail.com
address instead. 






From eslick at media.mit.edu  Wed Jan 14 19:30:23 2009
From: eslick at media.mit.edu (Ian Eslick)
Date: Wed, 14 Jan 2009 14:30:23 -0500
Subject: [elephant-devel] Querying for objects on two slots
In-Reply-To: <gklcu6$tp2$1@ger.gmane.org>
References: <ecb948b30901052236t4762d164p492ffac56e040152@mail.gmail.com><gjvlkt$8hh$1@ger.gmane.org><ecb948b30901131725t308514cj7540f75e5314aae2@mail.gmail.com>
	<FA60594D-CCE0-45C4-972F-041C715FED35@media.mit.edu>
	<gklcu6$tp2$1@ger.gmane.org>
Message-ID: <9B5F9785-AD97-4EC1-B8A9-FA93B774EFD5@media.mit.edu>

Thanks Alex,

I think a toolkit/wrapper around the current model of derived indices  
where you can define a derived index with a function where that  
function is parameterized by the slots it indexes and a specification  
of the maximum size of that slot, etc would be easy to include as an  
aid.

The interface could perhaps be executed by methods that are defined  
either generically (via strings or bignums, for example) or on a per- 
store basis that overrides the default representation for efficiency.

If I can see a quick way to do this I'll add it as a small optional  
toolkit for 1.0.

Ian

On Jan 14, 2009, at 2:03 PM, Alex Mizrahi wrote:

> IE> Alex, what would the issues be for the postmodern backend?
> IE>   Could we use a behind the scenes convert-to-string strategy so  
> we had
> IE> a common API for indexing on tuples?
>
> doing it in "proper way" (sorting on multiple columns) will be quite  
> hard
> in postmodern, as it means complicating cursor stuff even further.
> as least i don't know how to do this anyhow easy.
>
> so if you know how to do this converting everything to strings and  
> sorting
> them (if i understand you correctly),
> it would be much better. does it need any support from backend or it  
> should
> work totally transparently?
>
> p.s.  i've fixed the nasty upgrade-btree issue for postmodern, now i'm
> reviewing stuff, testing
> and fixing leftover small issues, so i hope soon postmodern would be
> "ready".
>
> by the way, my email killerstorm at newmail.ru went down few days ago,  
> so if
> anybody wants to communicate with me it would be better to use
> alex.mizrahi at gmail.com
> address instead.
>
>
>
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel




From yarek.kowalik at gmail.com  Wed Jan 14 20:07:07 2009
From: yarek.kowalik at gmail.com (Yarek Kowalik)
Date: Wed, 14 Jan 2009 12:07:07 -0800
Subject: [elephant-devel] Querying for objects on two slots
In-Reply-To: <24f203480901131751o78df72e8x18013802642ba7e2@mail.gmail.com>
References: <ecb948b30901052236t4762d164p492ffac56e040152@mail.gmail.com>
	<gjvlkt$8hh$1@ger.gmane.org>
	<ecb948b30901131725t308514cj7540f75e5314aae2@mail.gmail.com>
	<24f203480901131751o78df72e8x18013802642ba7e2@mail.gmail.com>
Message-ID: <ecb948b30901141207r157d77eau2ded79b7f2ab41ad@mail.gmail.com>

Out of curiosity, the OIDs in elephant are they 32 bit integers or 64?  I'm
running on Ubuntu-64/SBCL-64.

I'm thinking that my 'query' slot vale would be generated something like
this (for 32 bit values):

(format t "~8,0X ~8,0X" slot-a-val slot-b-val)

Yarek

On Tue, Jan 13, 2009 at 5:51 PM, Robert Synnott <rsynnott at gmail.com> wrote:

> I don't believe the Postmodern backend, at least, allows search by conses.
>
> If you want sensible (ordered) results with a and b if a and/or b are
> integers, by the way, you should zero-pad them; otherwise, say, '1 2'
> comes after '1 12', which is probably not what you want
> Rob
>
> 2009/1/14 Yarek Kowalik <yarek.kowalik at gmail.com>:
> > When serializing tuples, is the string representation best:  You suggest
> > using (format t "~A ~A" a b) - is that efficient enough?  what about
> doing
> > (cons a b) = is there a way to index and search for conses? Any other
> ideas?
> >
> > Yarek
> >
> > On Tue, Jan 6, 2009 at 5:17 AM, Alex Mizrahi <killerstorm at newmail.ru>
> wrote:
> >>
> >> YK> Is this a reasonable way of finding an object of type
> >> YK> 'my-class that matches on values val-a and val-b for slots a and b?
> >>
> >> yep, it is reasonable if you have relatively low number of objects
> >> in returned by (get-instances-by-value 'my-class 'slot-b val-b) query.
> >> if number of objects is significant and you get a slowdown because
> >> of this, you might want to optimize this. a trivial thing is to try it
> >> symmertrically with slot-a -- whatever returns less objects is better.
> >> less trivial optimizations would be to work on lower-level -- via
> >> map-inverted-index (to avoid allocating whole list but instead test
> >> objects
> >> one by one) or even cursor API (this way you can retrieve oids rather
> >> than objects, which should be faster, and also iterating both indices at
> >> once might be a significant benefit if values are not uniformly
> >> distributed).
> >>
> >> but the most optimal way doing this in case of high number of objects
> >> in both slot-a and slot-b queries would be building and using
> multi-column
> >> index. unfortunately, Elephant does not help you with it -- either
> you'll
> >> have
> >> to serialize slot tuples into a string (e.g (format nil "~a_~a" slot-a
> >> slot-b)),
> >> or reorganize your data to use a custom index structure (like btree of
> >> btrees).
> >>
> >> there are also backend-specific considerations. for postmodern
> >>
> >>  (intersection (get-instances-by-value 'my-class 'slot-b val-b)
> >>                    (get-instances-by-value 'my-class 'slot-a val-a))
> >>
> >> would be much faster then testing objects one by one, for BDB -- i doubt
> >> so.
> >>
> >> LP> Use MAP-CLASS, this will considerably speed up the query.
> >>
> >> how is that? using at least one index is much better than using no index
> >> at
> >> all.
> >>
> >>
> >>
> >>
> >> _______________________________________________
> >> elephant-devel site list
> >> elephant-devel at common-lisp.net
> >> http://common-lisp.net/mailman/listinfo/elephant-devel
> >
> >
> > _______________________________________________
> > elephant-devel site list
> > elephant-devel at common-lisp.net
> > http://common-lisp.net/mailman/listinfo/elephant-devel
> >
>
>
>
> --
> Robert Synnott
> http://myblog.rsynnott.com
> MSN: rsynnott at gmail.com
> Jabber: rsynnott at gmail.com
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20090114/25117a23/attachment.html>

From yarek.kowalik at gmail.com  Wed Jan 14 20:17:05 2009
From: yarek.kowalik at gmail.com (Yarek Kowalik)
Date: Wed, 14 Jan 2009 12:17:05 -0800
Subject: [elephant-devel] Querying for objects on two slots
In-Reply-To: <ecb948b30901141207r157d77eau2ded79b7f2ab41ad@mail.gmail.com>
References: <ecb948b30901052236t4762d164p492ffac56e040152@mail.gmail.com>
	<gjvlkt$8hh$1@ger.gmane.org>
	<ecb948b30901131725t308514cj7540f75e5314aae2@mail.gmail.com>
	<24f203480901131751o78df72e8x18013802642ba7e2@mail.gmail.com>
	<ecb948b30901141207r157d77eau2ded79b7f2ab41ad@mail.gmail.com>
Message-ID: <ecb948b30901141217s4ecfcf98kfda75912ac1252be@mail.gmail.com>

Correction (missing single quotes):

(format t "~8,'0X ~8,0'X" slot-a-val slot-b-val)



On Wed, Jan 14, 2009 at 12:07 PM, Yarek Kowalik <yarek.kowalik at gmail.com>wrote:

> Out of curiosity, the OIDs in elephant are they 32 bit integers or 64?  I'm
> running on Ubuntu-64/SBCL-64.
>
> I'm thinking that my 'query' slot vale would be generated something like
> this (for 32 bit values):
>
> (format t "~8,0X ~8,0X" slot-a-val slot-b-val)
>
> Yarek
>
>
> On Tue, Jan 13, 2009 at 5:51 PM, Robert Synnott <rsynnott at gmail.com>wrote:
>
>> I don't believe the Postmodern backend, at least, allows search by conses.
>>
>> If you want sensible (ordered) results with a and b if a and/or b are
>> integers, by the way, you should zero-pad them; otherwise, say, '1 2'
>> comes after '1 12', which is probably not what you want
>> Rob
>>
>> 2009/1/14 Yarek Kowalik <yarek.kowalik at gmail.com>:
>> > When serializing tuples, is the string representation best:  You suggest
>> > using (format t "~A ~A" a b) - is that efficient enough?  what about
>> doing
>> > (cons a b) = is there a way to index and search for conses? Any other
>> ideas?
>> >
>> > Yarek
>> >
>> > On Tue, Jan 6, 2009 at 5:17 AM, Alex Mizrahi <killerstorm at newmail.ru>
>> wrote:
>> >>
>> >> YK> Is this a reasonable way of finding an object of type
>> >> YK> 'my-class that matches on values val-a and val-b for slots a and b?
>> >>
>> >> yep, it is reasonable if you have relatively low number of objects
>> >> in returned by (get-instances-by-value 'my-class 'slot-b val-b) query.
>> >> if number of objects is significant and you get a slowdown because
>> >> of this, you might want to optimize this. a trivial thing is to try it
>> >> symmertrically with slot-a -- whatever returns less objects is better.
>> >> less trivial optimizations would be to work on lower-level -- via
>> >> map-inverted-index (to avoid allocating whole list but instead test
>> >> objects
>> >> one by one) or even cursor API (this way you can retrieve oids rather
>> >> than objects, which should be faster, and also iterating both indices
>> at
>> >> once might be a significant benefit if values are not uniformly
>> >> distributed).
>> >>
>> >> but the most optimal way doing this in case of high number of objects
>> >> in both slot-a and slot-b queries would be building and using
>> multi-column
>> >> index. unfortunately, Elephant does not help you with it -- either
>> you'll
>> >> have
>> >> to serialize slot tuples into a string (e.g (format nil "~a_~a" slot-a
>> >> slot-b)),
>> >> or reorganize your data to use a custom index structure (like btree of
>> >> btrees).
>> >>
>> >> there are also backend-specific considerations. for postmodern
>> >>
>> >>  (intersection (get-instances-by-value 'my-class 'slot-b val-b)
>> >>                    (get-instances-by-value 'my-class 'slot-a val-a))
>> >>
>> >> would be much faster then testing objects one by one, for BDB -- i
>> doubt
>> >> so.
>> >>
>> >> LP> Use MAP-CLASS, this will considerably speed up the query.
>> >>
>> >> how is that? using at least one index is much better than using no
>> index
>> >> at
>> >> all.
>> >>
>> >>
>> >>
>> >>
>> >> _______________________________________________
>> >> elephant-devel site list
>> >> elephant-devel at common-lisp.net
>> >> http://common-lisp.net/mailman/listinfo/elephant-devel
>> >
>> >
>> > _______________________________________________
>> > elephant-devel site list
>> > elephant-devel at common-lisp.net
>> > http://common-lisp.net/mailman/listinfo/elephant-devel
>> >
>>
>>
>>
>> --
>> Robert Synnott
>> http://myblog.rsynnott.com
>> MSN: rsynnott at gmail.com
>> Jabber: rsynnott at gmail.com
>>
>> _______________________________________________
>> elephant-devel site list
>> elephant-devel at common-lisp.net
>> http://common-lisp.net/mailman/listinfo/elephant-devel
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20090114/b7ea59ae/attachment.html>

From eslick at media.mit.edu  Wed Jan 14 20:55:17 2009
From: eslick at media.mit.edu (Ian Eslick)
Date: Wed, 14 Jan 2009 15:55:17 -0500
Subject: [elephant-devel] Querying for objects on two slots
In-Reply-To: <ecb948b30901141207r157d77eau2ded79b7f2ab41ad@mail.gmail.com>
References: <ecb948b30901052236t4762d164p492ffac56e040152@mail.gmail.com>
	<gjvlkt$8hh$1@ger.gmane.org>
	<ecb948b30901131725t308514cj7540f75e5314aae2@mail.gmail.com>
	<24f203480901131751o78df72e8x18013802642ba7e2@mail.gmail.com>
	<ecb948b30901141207r157d77eau2ded79b7f2ab41ad@mail.gmail.com>
Message-ID: <DE1452B3-8E6F-40DF-BC63-EB2C056CC516@media.mit.edu>

Right now they're 32 bits by convention, but I'd like them to be 64- 
bits on a 64-bit machine eventually so far I don't think we're in  
danger of blowing out 32 bits of oids - you can compact OID values by  
doing a migration.  Actually it wouldn't be hard to replace this with  
a 'find next unused oid' given that we have an oid table and the  
beginnings of a GC algorithm now.

Ian

On Jan 14, 2009, at 3:07 PM, Yarek Kowalik wrote:

> Out of curiosity, the OIDs in elephant are they 32 bit integers or  
> 64?  I'm running on Ubuntu-64/SBCL-64.
>
> I'm thinking that my 'query' slot vale would be generated something  
> like this (for 32 bit values):
>
> (format t "~8,0X ~8,0X" slot-a-val slot-b-val)
>
> Yarek
>
> On Tue, Jan 13, 2009 at 5:51 PM, Robert Synnott <rsynnott at gmail.com>  
> wrote:
> I don't believe the Postmodern backend, at least, allows search by  
> conses.
>
> If you want sensible (ordered) results with a and b if a and/or b are
> integers, by the way, you should zero-pad them; otherwise, say, '1 2'
> comes after '1 12', which is probably not what you want
> Rob
>
> 2009/1/14 Yarek Kowalik <yarek.kowalik at gmail.com>:
> > When serializing tuples, is the string representation best:  You  
> suggest
> > using (format t "~A ~A" a b) - is that efficient enough?  what  
> about doing
> > (cons a b) = is there a way to index and search for conses? Any  
> other ideas?
> >
> > Yarek
> >
> > On Tue, Jan 6, 2009 at 5:17 AM, Alex Mizrahi  
> <killerstorm at newmail.ru> wrote:
> >>
> >> YK> Is this a reasonable way of finding an object of type
> >> YK> 'my-class that matches on values val-a and val-b for slots a  
> and b?
> >>
> >> yep, it is reasonable if you have relatively low number of objects
> >> in returned by (get-instances-by-value 'my-class 'slot-b val-b)  
> query.
> >> if number of objects is significant and you get a slowdown because
> >> of this, you might want to optimize this. a trivial thing is to  
> try it
> >> symmertrically with slot-a -- whatever returns less objects is  
> better.
> >> less trivial optimizations would be to work on lower-level -- via
> >> map-inverted-index (to avoid allocating whole list but instead test
> >> objects
> >> one by one) or even cursor API (this way you can retrieve oids  
> rather
> >> than objects, which should be faster, and also iterating both  
> indices at
> >> once might be a significant benefit if values are not uniformly
> >> distributed).
> >>
> >> but the most optimal way doing this in case of high number of  
> objects
> >> in both slot-a and slot-b queries would be building and using  
> multi-column
> >> index. unfortunately, Elephant does not help you with it --  
> either you'll
> >> have
> >> to serialize slot tuples into a string (e.g (format nil "~a_~a"  
> slot-a
> >> slot-b)),
> >> or reorganize your data to use a custom index structure (like  
> btree of
> >> btrees).
> >>
> >> there are also backend-specific considerations. for postmodern
> >>
> >>  (intersection (get-instances-by-value 'my-class 'slot-b val-b)
> >>                    (get-instances-by-value 'my-class 'slot-a val- 
> a))
> >>
> >> would be much faster then testing objects one by one, for BDB --  
> i doubt
> >> so.
> >>
> >> LP> Use MAP-CLASS, this will considerably speed up the query.
> >>
> >> how is that? using at least one index is much better than using  
> no index
> >> at
> >> all.
> >>
> >>
> >>
> >>
> >> _______________________________________________
> >> elephant-devel site list
> >> elephant-devel at common-lisp.net
> >> http://common-lisp.net/mailman/listinfo/elephant-devel
> >
> >
> > _______________________________________________
> > elephant-devel site list
> > elephant-devel at common-lisp.net
> > http://common-lisp.net/mailman/listinfo/elephant-devel
> >
>
>
>
> --
> Robert Synnott
> http://myblog.rsynnott.com
> MSN: rsynnott at gmail.com
> Jabber: rsynnott at gmail.com
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel




From alain.picard at accenture.com  Fri Jan 16 03:18:15 2009
From: alain.picard at accenture.com (alain.picard at accenture.com)
Date: Fri, 16 Jan 2009 11:18:15 +0800
Subject: [elephant-devel] CLSQL Store
In-Reply-To: <461C0B66B0E04C1EBCFFF01B7E1C7D18@killer>
References: <ecb948b30901021117t4733cbf0m83306c66a863c36d@mail.gmail.com><gjmef5$l64$1@ger.gmane.org><1F1F136C-D922-46F8-BAC3-B252D34BF8BC@media.mit.edu>
	<461C0B66B0E04C1EBCFFF01B7E1C7D18@killer>
Message-ID: <8457077CBD070B45AC1307D5265A81B2626C51@APAXM1114.dir.svc.accenture.com>

 

> -----Original Message-----
> From: Alex Mizrahi [mailto:killerstorm at newmail.ru] 
> 
> explanation on BDB's site says that if you, say, use Elephant 
> rather than BDB directly, and Elephant is open source, that 
> is enough to be eligible for a "free" BDB version.

I simply do not see how you can read it this way.
I'm reading
http://www.oracle.com/technology/software/products/berkeley-db/htdocs/li
censing.html and note this paragraph:

The our open source license permits you to use Berkeley DB, Berkeley DB
Java Edition or Berkeley DB XML at no charge under the condition that if
you use the software in an application you redistribute, the complete
source code for your application must be available and freely
redistributable under reasonable conditions. 

and

What must I release as open source?

Under the open source license, you must release the complete source code
for the application that uses Berkeley DB, Berkeley DB Java Edition or
Berkeley DB XML. You do not need to release the source code for
components that are generally installed on the operating system on which
your application runs, such as system header files or libraries

I can't see that saying that "elephant uses DBD", and my app uses
elephant, so I don't have to release the sources to my app is in
accordance with the above license.

I'm deeply interested in finding out where Oracle states those
"friendly terms" you are referring to.  Can you post a pointer?

Thanks,
         Alain Picard


This message is for the designated recipient only and may contain privileged, proprietary, or otherwise private information.  If you have received it in error, please notify the sender immediately and delete the original.  Any other use of the email by you is prohibited.



From killerstorm at newmail.ru  Fri Jan 16 09:51:13 2009
From: killerstorm at newmail.ru (Alex Mizrahi)
Date: Fri, 16 Jan 2009 11:51:13 +0200
Subject: [elephant-devel] CLSQL Store
References: <ecb948b30901021117t4733cbf0m83306c66a863c36d@mail.gmail.com><gjmef5$l64$1@ger.gmane.org><1F1F136C-D922-46F8-BAC3-B252D34BF8BC@media.mit.edu><461C0B66B0E04C1EBCFFF01B7E1C7D18@killer>
	<8457077CBD070B45AC1307D5265A81B2626C51@APAXM1114.dir.svc.accenture.com>
Message-ID: <gkplap$7ds$1@ger.gmane.org>

 ap> I'm deeply interested in finding out where Oracle states those
 ap> "friendly terms" you are referring to.  Can you post a pointer?

I meant the next paragraph in the same document:
----
Do I have to pay for a Berkeley DB license to use it in my Perl or Python 
scripts?

No, you may use the Berkeley DB open source license at no cost. The Berkeley 
DB open source license requires that software that uses Berkeley DB be 
freely redistributable. In the case of Perl or Python, that software is Perl 
or Python, and not your scripts. Any scripts you write are your property, 
including scripts that make use of Berkeley DB. None of the Perl, Python or 
Berkeley DB licenses place any restrictions on what you may do with them.
----

I don't think that CL is anyhow worse than Python or Perl, and situation 
seems to
 be pretty much the same. For a Perl, BDB connector is a standalone module, 
as I understand,
 for Python it is included in standard  distribution. You can easily make 
SBCL+Elephant bundle,
for example, and claim that it is this bundly which uses BDB, and not your 
scripts -- it
would be same logic as in Oracle's statement above.

Of course, it is a question what is qualified to be a "script" and what is 
"software" (and if it
makes sense to take such distinction at all). Probably Oracle/Sleepycat are 
thinking about
the software in C-centric way -- as of executable that is statically or 
dynamically linked to
libdb. In this case, text of Perl/Python/CL programs is not technically 
linked to a libdb,
and thus it should be free from license restrictions. 






From eslick at media.mit.edu  Fri Jan 16 13:51:50 2009
From: eslick at media.mit.edu (Ian Eslick)
Date: Fri, 16 Jan 2009 08:51:50 -0500
Subject: [elephant-devel] CLSQL Store
In-Reply-To: <gkplap$7ds$1@ger.gmane.org>
References: <ecb948b30901021117t4733cbf0m83306c66a863c36d@mail.gmail.com><gjmef5$l64$1@ger.gmane.org><1F1F136C-D922-46F8-BAC3-B252D34BF8BC@media.mit.edu><461C0B66B0E04C1EBCFFF01B7E1C7D18@killer>
	<8457077CBD070B45AC1307D5265A81B2626C51@APAXM1114.dir.svc.accenture.com>
	<gkplap$7ds$1@ger.gmane.org>
Message-ID: <9CA671F6-67EB-4EA0-A1B9-307685022DCD@media.mit.edu>

Running BDB behind a website doesn't count as a 'distribution' that  
requires disclosure of the website source.  The cgi discussion is a  
sub-case of this.

This wouldn't apply to selling website frameworks or desktop software,  
of course.

Of course if you are going to be making money it would be worthwhile  
to get a legal opinion on this.

Ian

Sent from my iPhone

On Jan 16, 2009, at 4:51 AM, "Alex Mizrahi" <killerstorm at newmail.ru>  
wrote:

> ap> I'm deeply interested in finding out where Oracle states those
> ap> "friendly terms" you are referring to.  Can you post a pointer?
>
> I meant the next paragraph in the same document:
> ----
> Do I have to pay for a Berkeley DB license to use it in my Perl or  
> Python
> scripts?
>
> No, you may use the Berkeley DB open source license at no cost. The  
> Berkeley
> DB open source license requires that software that uses Berkeley DB be
> freely redistributable. In the case of Perl or Python, that software  
> is Perl
> or Python, and not your scripts. Any scripts you write are your  
> property,
> including scripts that make use of Berkeley DB. None of the Perl,  
> Python or
> Berkeley DB licenses place any restrictions on what you may do with  
> them.
> ----
>
> I don't think that CL is anyhow worse than Python or Perl, and  
> situation
> seems to
> be pretty much the same. For a Perl, BDB connector is a standalone  
> module,
> as I understand,
> for Python it is included in standard  distribution. You can easily  
> make
> SBCL+Elephant bundle,
> for example, and claim that it is this bundly which uses BDB, and  
> not your
> scripts -- it
> would be same logic as in Oracle's statement above.
>
> Of course, it is a question what is qualified to be a "script" and  
> what is
> "software" (and if it
> makes sense to take such distinction at all). Probably Oracle/ 
> Sleepycat are
> thinking about
> the software in C-centric way -- as of executable that is statically  
> or
> dynamically linked to
> libdb. In this case, text of Perl/Python/CL programs is not  
> technically
> linked to a libdb,
> and thus it should be free from license restrictions.
>
>
>
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel



From larry at theclapp.org  Sun Jan 25 21:30:36 2009
From: larry at theclapp.org (Larry Clapp)
Date: Sun, 25 Jan 2009 16:30:36 -0500
Subject: [elephant-devel] Elephant won't compile under Lispworks for Linux
	5.1
Message-ID: <20090125213036.GA1718@cupid.theclapp.org>

This appears to be similar to the error reported back in April 08
(http://common-lisp.net/pipermail/elephant-devel/2008-April/003925.html).

Search below for "++++ Error".

-- Larry


CL-USER 1 > (asdf :elephant)
; loading system definition from /home/lmc/lisp/systems/elephant.asd into
; #<The ASDF0 package, 0/16 internal, 0/16 external>
; Loading text file /home/lmc/lisp/elephant/elephant-1.0/elephant.asd
; registering #<SYSTEM ELEPHANT 21FE099F> as ELEPHANT
;; Creating system COMMON-LISP-USER::ELEPHANT->SRC->UTILS
;; Creating system COMMON-LISP-USER::ELEPHANT->SRC->MEMUTIL
;; Creating system COMMON-LISP-USER::ELEPHANT->SRC->ELEPHANT
;; Creating system COMMON-LISP-USER::ELEPHANT->SRC
;; Creating system COMMON-LISP-USER::ELEPHANT
; loading system definition from /home/lmc/lisp/systems/cl-base64.asd into
; #<The ASDF0 package, 0/16 internal, 0/16 external>
; Loading text file /home/lmc/lisp/cl-base64/cl-base64.asd
; registering #<SYSTEM CL-BASE64 2215982B> as CL-BASE64
;; Creating system COMMON-LISP-USER::CL-BASE64
; registering #<SYSTEM CL-BASE64-TESTS 200C8253> as CL-BASE64-TESTS
;; Creating system COMMON-LISP-USER::CL-BASE64-TESTS
; loading system definition from /home/lmc/lisp/systems/uffi.asd into
; #<The ASDF0 package, 0/16 internal, 0/16 external>
; Loading text file /usr/share/common-lisp/source/uffi/uffi.asd
; registering #<SYSTEM UFFI 200F68EF> as UFFI
;; Creating system COMMON-LISP-USER::UFFI->SRC
;; Creating system COMMON-LISP-USER::UFFI
; Loading fasl file /usr/share/common-lisp/source/uffi/src/package.ufasl
; Loading fasl file /usr/share/common-lisp/source/uffi/src/primitives.ufasl
; Loading fasl file /usr/share/common-lisp/source/uffi/src/objects.ufasl
; Loading fasl file /usr/share/common-lisp/source/uffi/src/aggregates.ufasl
; Loading fasl file /usr/share/common-lisp/source/uffi/src/functions.ufasl
; Loading fasl file /usr/share/common-lisp/source/uffi/src/strings.ufasl
; Loading fasl file /usr/share/common-lisp/source/uffi/src/libraries.ufasl
; Loading fasl file /usr/share/common-lisp/source/uffi/src/os.ufasl
; Loading fasl file /home/lmc/lisp/cl-base64/package.ufasl
; Loading fasl file /home/lmc/lisp/cl-base64/encode.ufasl
; Loading fasl file /home/lmc/lisp/cl-base64/decode.ufasl
;;; Compiling file /home/lmc/lisp/elephant/elephant-1.0/src/utils/package.lisp ...
;;; Safety = 1, Speed = 3, Space = 1, Float = 1, Interruptible = 0
;;; Compilation speed = 0, Debug = 3, Fixnum safety = 3
;;; Source level debugging is on
;;; Source file recording is  on
;;; Cross referencing is on
; (TOP-LEVEL-FORM 0)
; (TOP-LEVEL-FORM 1)
; (DEFPACKAGE "ELEPHANT-UTILS")
; (TOP-LEVEL-FORM 0)
; Loading fasl file /home/lmc/lisp/elephant/elephant-1.0/src/utils/package.ufasl
;;; Compiling file /home/lmc/lisp/elephant/elephant-1.0/src/utils/convenience.lisp ...
;;; Safety = 1, Speed = 3, Space = 1, Float = 1, Interruptible = 0
;;; Compilation speed = 0, Debug = 3, Fixnum safety = 3
;;; Source level debugging is on
;;; Source file recording is  on
;;; Cross referencing is on
; (TOP-LEVEL-FORM 0)
; (TOP-LEVEL-FORM 1)
; ELEPHANT-UTILS:WITH-GENSYMS
; ELEPHANT-UTILS:DO-SUBSETS
; ELEPHANT-UTILS:SUBSETS
; ELEPHANT-UTILS:REMOVE-KEYWORDS
; ELEPHANT-UTILS::CONCAT-SEPARATED-STRINGS
; ELEPHANT-UTILS::APPEND-SUBLISTS
; ELEPHANT-UTILS:IFRET
; ELEPHANT-UTILS:AIF
; ELEPHANT-UTILS:AWHEN
; ELEPHANT-UTILS:MKLIST
; ELEPHANT-UTILS:REMOVE-INDEXED-ELEMENT-AND-ADJUST
; ELEPHANT-UTILS:CURRY
; (TOP-LEVEL-FORM 0)
; Loading fasl file /home/lmc/lisp/elephant/elephant-1.0/src/utils/convenience.ufasl
;;; Compiling file /home/lmc/lisp/elephant/elephant-1.0/src/utils/locks.lisp ...
;;; Safety = 1, Speed = 3, Space = 1, Float = 1, Interruptible = 0
;;; Compilation speed = 0, Debug = 3, Fixnum safety = 3
;;; Source level debugging is on
;;; Source file recording is  on
;;; Cross referencing is on
; (TOP-LEVEL-FORM 0)
; (TOP-LEVEL-FORM 1)
; ELEPHANT-UTILS:ELE-MAKE-LOCK
; ELEPHANT-UTILS:ELE-WITH-LOCK
; ELEPHANT-UTILS:ELE-MAKE-FAST-LOCK
; ELEPHANT-UTILS:ELE-WITH-FAST-LOCK
; ELEPHANT-UTILS:ELE-THREAD-HASH-KEY
; (TOP-LEVEL-FORM 0)
; Loading fasl file /home/lmc/lisp/elephant/elephant-1.0/src/utils/locks.ufasl
;;; Compiling file /home/lmc/lisp/elephant/elephant-1.0/src/utils/os.lisp ...
;;; Safety = 1, Speed = 3, Space = 1, Float = 1, Interruptible = 0
;;; Compilation speed = 0, Debug = 3, Fixnum safety = 3
;;; Source level debugging is on
;;; Source file recording is  on
;;; Cross referencing is on
; (TOP-LEVEL-FORM 0)
; (TOP-LEVEL-FORM 1)
; ELEPHANT-UTILS::IN-DIRECTORY
; ELEPHANT-UTILS:LAUNCH-BACKGROUND-PROGRAM
;;;*** Warning in ELEPHANT-UTILS:KILL-BACKGROUND-PROGRAM: PROCESS-HANDLE is bound but not referenced
; ELEPHANT-UTILS:KILL-BACKGROUND-PROGRAM
; ELEPHANT-UTILS:CREATE-TEMP-FILENAME
; ELEPHANT-UTILS:CREATE-TEMP-DIRNAME
; ELEPHANT-UTILS:COPY-DIRECTORY
; ELEPHANT-UTILS:COPY-FILE-TO-DIR
; ELEPHANT-UTILS:COPY-FILE
; (TOP-LEVEL-FORM 0)
Warning: COMPILE-FILE warned while performing #<ASDF:COMPILE-OP NIL 21A3A02F> on
         #<ASDF:CL-SOURCE-FILE "os" 22000E8B>.
; Loading fasl file /home/lmc/lisp/elephant/elephant-1.0/src/utils/os.ufasl
Attempting to load libmemutil.so...
Loaded /home/lmc/lisp/elephant/elephant-1.0/src/memutil/libmemutil.so
;;; Compiling file /home/lmc/lisp/elephant/elephant-1.0/src/memutil/memutil.lisp ...
;;; Safety = 1, Speed = 3, Space = 1, Float = 1, Interruptible = 0
;;; Compilation speed = 0, Debug = 3, Fixnum safety = 3
;;; Source level debugging is on
;;; Source file recording is  on
;;; Cross referencing is on
; (TOP-LEVEL-FORM 0)
; (DEFPACKAGE "ELEPHANT-MEMUTIL")
; (TOP-LEVEL-FORM 2)
;;;*** Warning in (DEFTYPE ELEPHANT-MEMUTIL:POINTER-INT): (DEFTYPE ELEPHANT-MEMUTIL:POINTER-INT) defined more than once in /home/lmc/lisp/elephant/elephant-1.0/src/memutil/memutil.lisp.
; (DEFTYPE ELEPHANT-MEMUTIL:POINTER-INT)
;;;*** Warning in (DEFTYPE ELEPHANT-MEMUTIL:POINTER-VOID): (DEFTYPE ELEPHANT-MEMUTIL:POINTER-VOID) defined more than once in /home/lmc/lisp/elephant/elephant-1.0/src/memutil/memutil.lisp.
; (DEFTYPE ELEPHANT-MEMUTIL:POINTER-VOID)
; (FLI:DEFINE-FOREIGN-TYPE ELEPHANT-MEMUTIL:ARRAY-OR-POINTER-CHAR)
;;;*** Warning in (DEFTYPE ELEPHANT-MEMUTIL:ARRAY-OR-POINTER-CHAR): (DEFTYPE ELEPHANT-MEMUTIL:ARRAY-OR-POINTER-CHAR) defined more than once in /home/lmc/lisp/elephant/elephant-1.0/src/memutil/memutil.lisp.
; (DEFTYPE ELEPHANT-MEMUTIL:ARRAY-OR-POINTER-CHAR)
; (SUBFUNCTION ELEPHANT-MEMUTIL::COPY-BUFS (FLI:DEFINE-FOREIGN-FUNCTION ELEPHANT-MEMUTIL::COPY-BUFS))
; (FLI:DEFINE-FOREIGN-FUNCTION ELEPHANT-MEMUTIL::COPY-BUFS)
; (FLI:DEFINE-FOREIGN-FUNCTION ELEPHANT-MEMUTIL::COPY-BUFS)
; (DEFVAR ELEPHANT-MEMUTIL:+NULL-VOID+)
; (DEFVAR ELEPHANT-MEMUTIL:+NULL-CHAR+)
; (DEFVAR ELEPHANT-MEMUTIL::*BUFFER-STREAMS*)
; (DEFVAR ELEPHANT-MEMUTIL::*BUFFER-STREAMS-LOCK*)
; (SUBFUNCTION (DEFSETF ELEPHANT-MEMUTIL:BUFFER-STREAM-LENGTH) (DEFSTRUCT ELEPHANT-MEMUTIL:BUFFER-STREAM))
; (SUBFUNCTION (DEFSETF ELEPHANT-MEMUTIL::BUFFER-STREAM-POSITION) (DEFSTRUCT ELEPHANT-MEMUTIL:BUFFER-STREAM))
; (SUBFUNCTION (DEFSETF ELEPHANT-MEMUTIL:BUFFER-STREAM-SIZE) (DEFSTRUCT ELEPHANT-MEMUTIL:BUFFER-STREAM))
; (SUBFUNCTION (DEFSETF ELEPHANT-MEMUTIL:BUFFER-STREAM-BUFFER) (DEFSTRUCT ELEPHANT-MEMUTIL:BUFFER-STREAM))
; (SUBFUNCTION ELEPHANT-MEMUTIL:MAKE-BUFFER-STREAM (DEFSTRUCT ELEPHANT-MEMUTIL:BUFFER-STREAM))
; ELEPHANT-MEMUTIL::GRAB-BUFFER-STREAM
; ELEPHANT-MEMUTIL::RETURN-BUFFER-STREAM
; ELEPHANT-MEMUTIL:WITH-BUFFER-STREAMS
; (SUBFUNCTION ELEPHANT-MEMUTIL::READ-INT32 (FLI:DEFINE-FOREIGN-FUNCTION ELEPHANT-MEMUTIL::READ-INT32))
; (FLI:DEFINE-FOREIGN-FUNCTION ELEPHANT-MEMUTIL::READ-INT32)
; (FLI:DEFINE-FOREIGN-FUNCTION ELEPHANT-MEMUTIL::READ-INT32)
; (SUBFUNCTION ELEPHANT-MEMUTIL::READ-FIXNUM32 (FLI:DEFINE-FOREIGN-FUNCTION ELEPHANT-MEMUTIL::READ-FIXNUM32))
; (FLI:DEFINE-FOREIGN-FUNCTION ELEPHANT-MEMUTIL::READ-FIXNUM32)
; (FLI:DEFINE-FOREIGN-FUNCTION ELEPHANT-MEMUTIL::READ-FIXNUM32)
; (SUBFUNCTION ELEPHANT-MEMUTIL::READ-UINT32 (FLI:DEFINE-FOREIGN-FUNCTION ELEPHANT-MEMUTIL::READ-UINT32))
; (FLI:DEFINE-FOREIGN-FUNCTION ELEPHANT-MEMUTIL::READ-UINT32)
; (FLI:DEFINE-FOREIGN-FUNCTION ELEPHANT-MEMUTIL::READ-UINT32)
; (SUBFUNCTION ELEPHANT-MEMUTIL::READ-INT64 (FLI:DEFINE-FOREIGN-FUNCTION ELEPHANT-MEMUTIL::READ-INT64))
; (FLI:DEFINE-FOREIGN-FUNCTION ELEPHANT-MEMUTIL::READ-INT64)
; (FLI:DEFINE-FOREIGN-FUNCTION ELEPHANT-MEMUTIL::READ-INT64)
; (SUBFUNCTION ELEPHANT-MEMUTIL::READ-UINT64 (FLI:DEFINE-FOREIGN-FUNCTION ELEPHANT-MEMUTIL::READ-UINT64))
; (FLI:DEFINE-FOREIGN-FUNCTION ELEPHANT-MEMUTIL::READ-UINT64)
; (FLI:DEFINE-FOREIGN-FUNCTION ELEPHANT-MEMUTIL::READ-UINT64)
; (SUBFUNCTION ELEPHANT-MEMUTIL::READ-FLOAT (FLI:DEFINE-FOREIGN-FUNCTION ELEPHANT-MEMUTIL::READ-FLOAT))
; (FLI:DEFINE-FOREIGN-FUNCTION ELEPHANT-MEMUTIL::READ-FLOAT)
; (FLI:DEFINE-FOREIGN-FUNCTION ELEPHANT-MEMUTIL::READ-FLOAT)
; (SUBFUNCTION ELEPHANT-MEMUTIL::READ-DOUBLE (FLI:DEFINE-FOREIGN-FUNCTION ELEPHANT-MEMUTIL::READ-DOUBLE))
; (FLI:DEFINE-FOREIGN-FUNCTION ELEPHANT-MEMUTIL::READ-DOUBLE)
; (FLI:DEFINE-FOREIGN-FUNCTION ELEPHANT-MEMUTIL::READ-DOUBLE)
; (SUBFUNCTION ELEPHANT-MEMUTIL::WRITE-INT32 (FLI:DEFINE-FOREIGN-FUNCTION ELEPHANT-MEMUTIL::WRITE-INT32))
; (FLI:DEFINE-FOREIGN-FUNCTION ELEPHANT-MEMUTIL::WRITE-INT32)
; (FLI:DEFINE-FOREIGN-FUNCTION ELEPHANT-MEMUTIL::WRITE-INT32)
; (SUBFUNCTION ELEPHANT-MEMUTIL::WRITE-FIXNUM32 (FLI:DEFINE-FOREIGN-FUNCTION ELEPHANT-MEMUTIL::WRITE-FIXNUM32))
; (FLI:DEFINE-FOREIGN-FUNCTION ELEPHANT-MEMUTIL::WRITE-FIXNUM32)
; (FLI:DEFINE-FOREIGN-FUNCTION ELEPHANT-MEMUTIL::WRITE-FIXNUM32)
; (SUBFUNCTION ELEPHANT-MEMUTIL::WRITE-UINT32 (FLI:DEFINE-FOREIGN-FUNCTION ELEPHANT-MEMUTIL::WRITE-UINT32))
; (FLI:DEFINE-FOREIGN-FUNCTION ELEPHANT-MEMUTIL::WRITE-UINT32)
; (FLI:DEFINE-FOREIGN-FUNCTION ELEPHANT-MEMUTIL::WRITE-UINT32)
; (SUBFUNCTION ELEPHANT-MEMUTIL::WRITE-INT64 (FLI:DEFINE-FOREIGN-FUNCTION ELEPHANT-MEMUTIL::WRITE-INT64))
; (FLI:DEFINE-FOREIGN-FUNCTION ELEPHANT-MEMUTIL::WRITE-INT64)
; (FLI:DEFINE-FOREIGN-FUNCTION ELEPHANT-MEMUTIL::WRITE-INT64)
; (SUBFUNCTION ELEPHANT-MEMUTIL::WRITE-FIXNUM64 (FLI:DEFINE-FOREIGN-FUNCTION ELEPHANT-MEMUTIL::WRITE-FIXNUM64))
; (FLI:DEFINE-FOREIGN-FUNCTION ELEPHANT-MEMUTIL::WRITE-FIXNUM64)
; (FLI:DEFINE-FOREIGN-FUNCTION ELEPHANT-MEMUTIL::WRITE-FIXNUM64)
; (SUBFUNCTION ELEPHANT-MEMUTIL::WRITE-UINT64 (FLI:DEFINE-FOREIGN-FUNCTION ELEPHANT-MEMUTIL::WRITE-UINT64))
; (FLI:DEFINE-FOREIGN-FUNCTION ELEPHANT-MEMUTIL::WRITE-UINT64)
; (FLI:DEFINE-FOREIGN-FUNCTION ELEPHANT-MEMUTIL::WRITE-UINT64)
; (SUBFUNCTION ELEPHANT-MEMUTIL::WRITE-FLOAT (FLI:DEFINE-FOREIGN-FUNCTION ELEPHANT-MEMUTIL::WRITE-FLOAT))
; (FLI:DEFINE-FOREIGN-FUNCTION ELEPHANT-MEMUTIL::WRITE-FLOAT)
; (FLI:DEFINE-FOREIGN-FUNCTION ELEPHANT-MEMUTIL::WRITE-FLOAT)
; (SUBFUNCTION ELEPHANT-MEMUTIL::WRITE-DOUBLE (FLI:DEFINE-FOREIGN-FUNCTION ELEPHANT-MEMUTIL::WRITE-DOUBLE))
; (FLI:DEFINE-FOREIGN-FUNCTION ELEPHANT-MEMUTIL::WRITE-DOUBLE)
; (FLI:DEFINE-FOREIGN-FUNCTION ELEPHANT-MEMUTIL::WRITE-DOUBLE)
; (SUBFUNCTION ELEPHANT-MEMUTIL::OFFSET-CHAR-POINTER (FLI:DEFINE-FOREIGN-FUNCTION ELEPHANT-MEMUTIL::OFFSET-CHAR-POINTER))
; (FLI:DEFINE-FOREIGN-FUNCTION ELEPHANT-MEMUTIL::OFFSET-CHAR-POINTER)
; (FLI:DEFINE-FOREIGN-FUNCTION ELEPHANT-MEMUTIL::OFFSET-CHAR-POINTER)
; ELEPHANT-MEMUTIL:BYTE-LENGTH
; (SUBFUNCTION ELEPHANT-MEMUTIL::COPY-STR-TO-BUF (FLI:DEFINE-FOREIGN-FUNCTION ELEPHANT-MEMUTIL::COPY-STR-TO-BUF))
; (FLI:DEFINE-FOREIGN-FUNCTION ELEPHANT-MEMUTIL::COPY-STR-TO-BUF)
; (FLI:DEFINE-FOREIGN-FUNCTION ELEPHANT-MEMUTIL::COPY-STR-TO-BUF)
; ELEPHANT-MEMUTIL::PROCESS-STRUCT-SLOT-DEFS
; ELEPHANT-MEMUTIL::WITH-STRUCT-SLOTS
;;;*** Warning in ELEPHANT-MEMUTIL:RESIZE-BUFFER-STREAM: Ignoring type declaration with illegal type (ELEPHANT-MEMUTIL::ALIEN (* ELEPHANT-MEMUTIL::UNSIGNED-CHAR))
; ELEPHANT-MEMUTIL:RESIZE-BUFFER-STREAM
;;;*** Warning in ELEPHANT-MEMUTIL:RESIZE-BUFFER-STREAM-NO-COPY: Ignoring type declaration with illegal type (ELEPHANT-MEMUTIL::ALIEN (* ELEPHANT-MEMUTIL::UNSIGNED-CHAR))
;;;*** Warning in ELEPHANT-MEMUTIL:RESIZE-BUFFER-STREAM-NO-COPY: symbol-macro SIZE is bound but not referenced
; ELEPHANT-MEMUTIL:RESIZE-BUFFER-STREAM-NO-COPY
; ELEPHANT-MEMUTIL:RESET-BUFFER-STREAM
;;;*** Warning in ELEPHANT-MEMUTIL:BUFFER-WRITE-BYTE: Ignoring type declaration with illegal type (ELEPHANT-MEMUTIL::ALIEN (* ELEPHANT-MEMUTIL::UNSIGNED-CHAR))
; ELEPHANT-MEMUTIL:BUFFER-WRITE-BYTE
; ELEPHANT-MEMUTIL:BUFFER-WRITE-INT32
;;;*** Warning in ELEPHANT-MEMUTIL:BUFFER-WRITE-FIXNUM32: Ignoring type declaration with illegal type (ELEPHANT-MEMUTIL::ALIEN (* ELEPHANT-MEMUTIL::UNSIGNED-CHAR))
; ELEPHANT-MEMUTIL:BUFFER-WRITE-FIXNUM32
;;;*** Warning in ELEPHANT-MEMUTIL:BUFFER-WRITE-UINT32: Ignoring type declaration with illegal type (ELEPHANT-MEMUTIL::ALIEN (* ELEPHANT-MEMUTIL::UNSIGNED-CHAR))
; ELEPHANT-MEMUTIL:BUFFER-WRITE-UINT32
;;;*** Warning in ELEPHANT-MEMUTIL:BUFFER-WRITE-INT64: Ignoring type declaration with illegal type (ELEPHANT-MEMUTIL::ALIEN (* ELEPHANT-MEMUTIL::UNSIGNED-CHAR))
; ELEPHANT-MEMUTIL:BUFFER-WRITE-INT64
;;;*** Warning in ELEPHANT-MEMUTIL:BUFFER-WRITE-FIXNUM64: Ignoring type declaration with illegal type (ELEPHANT-MEMUTIL::ALIEN (* ELEPHANT-MEMUTIL::UNSIGNED-CHAR))
; ELEPHANT-MEMUTIL:BUFFER-WRITE-FIXNUM64
;;;*** Warning in ELEPHANT-MEMUTIL:BUFFER-WRITE-UINT64: Ignoring type declaration with illegal type (ELEPHANT-MEMUTIL::ALIEN (* ELEPHANT-MEMUTIL::UNSIGNED-CHAR))
; ELEPHANT-MEMUTIL:BUFFER-WRITE-UINT64
;;;*** Warning in ELEPHANT-MEMUTIL:BUFFER-WRITE-FLOAT: Ignoring type declaration with illegal type (ELEPHANT-MEMUTIL::ALIEN (* ELEPHANT-MEMUTIL::UNSIGNED-CHAR))
; ELEPHANT-MEMUTIL:BUFFER-WRITE-FLOAT
;;;*** Warning in ELEPHANT-MEMUTIL:BUFFER-WRITE-DOUBLE: Ignoring type declaration with illegal type (ELEPHANT-MEMUTIL::ALIEN (* ELEPHANT-MEMUTIL::UNSIGNED-CHAR))
; ELEPHANT-MEMUTIL:BUFFER-WRITE-DOUBLE
;;;*** Warning in ELEPHANT-MEMUTIL:BUFFER-WRITE-STRING: Ignoring type declaration with illegal type (ELEPHANT-MEMUTIL::ALIEN (* ELEPHANT-MEMUTIL::UNSIGNED-CHAR))
; ELEPHANT-MEMUTIL:BUFFER-WRITE-STRING
; ELEPHANT-MEMUTIL:BUFFER-READ-BYTE
; ELEPHANT-MEMUTIL::BUFFER-READ-BYTE-VECTOR
; ELEPHANT-MEMUTIL::BUFFER-WRITE-BYTE-VECTOR
; ELEPHANT-MEMUTIL:BUFFER-READ-TO-ARRAY-OFFSET
; ELEPHANT-MEMUTIL:BUFFER-WRITE-FROM-ARRAY-OFFSET
; ELEPHANT-MEMUTIL:BUFFER-WRITE-OID
; ELEPHANT-MEMUTIL:BUFFER-READ-OID
; ELEPHANT-MEMUTIL:BUFFER-READ-INT
; ELEPHANT-MEMUTIL:BUFFER-READ-FIXNUM
; ELEPHANT-MEMUTIL:BUFFER-WRITE-INT
; ELEPHANT-MEMUTIL:BUFFER-READ-UINT
; ELEPHANT-MEMUTIL:BUFFER-WRITE-UINT
; (DEFCONSTANT ELEPHANT-MEMUTIL::+2^32+)
; (DEFCONSTANT ELEPHANT-MEMUTIL::+2^64+)
; ELEPHANT-MEMUTIL:BUFFER-READ-FIXNUM32
; ELEPHANT-MEMUTIL:BUFFER-READ-INT32
; ELEPHANT-MEMUTIL:BUFFER-READ-UINT32
; ELEPHANT-MEMUTIL:BUFFER-READ-FIXNUM64
; ELEPHANT-MEMUTIL:BUFFER-READ-INT64
; ELEPHANT-MEMUTIL:BUFFER-READ-UINT64
; ELEPHANT-MEMUTIL:BUFFER-READ-FLOAT
; ELEPHANT-MEMUTIL:BUFFER-READ-DOUBLE
; ELEPHANT-MEMUTIL:BUFFER-READ-UCS1-STRING
; ELEPHANT-MEMUTIL:BUFFER-READ-UCS2-STRING
; (DEFPARAMETER ELEPHANT-MEMUTIL::+LITTLE-ENDIAN+)
; ELEPHANT-MEMUTIL:LITTLE-ENDIAN-P
; (TOP-LEVEL-FORM 0)
Warning: COMPILE-FILE warned while performing #<ASDF:COMPILE-OP NIL 21A3A02F> on
         #<ASDF:CL-SOURCE-FILE "memutil" 22000D0B>.
Warning: COMPILE-FILE failed while performing #<ASDF:COMPILE-OP NIL 21A3A02F> on
         #<ASDF:CL-SOURCE-FILE "memutil" 22000D0B>.
; Loading fasl file /home/lmc/lisp/elephant/elephant-1.0/src/memutil/memutil.ufasl
;;; Compiling file /home/lmc/lisp/elephant/elephant-1.0/src/elephant/package.lisp ...
;;; Safety = 1, Speed = 3, Space = 1, Float = 1, Interruptible = 0
;;; Compilation speed = 0, Debug = 3, Fixnum safety = 3
;;; Source level debugging is on
;;; Source file recording is  on
;;; Cross referencing is on
; (TOP-LEVEL-FORM 0)
; (TOP-LEVEL-FORM 1)
; (DEFPACKAGE "ELEPHANT")
; (TOP-LEVEL-FORM 3)
; (DEFPACKAGE "ELEPHANT-USER")
; (TOP-LEVEL-FORM 0)
; Loading fasl file /home/lmc/lisp/elephant/elephant-1.0/src/elephant/package.ufasl
;;; Compiling file /home/lmc/lisp/elephant/elephant-1.0/src/elephant/variables.lisp ...
;;; Safety = 1, Speed = 3, Space = 1, Float = 1, Interruptible = 0
;;; Compilation speed = 0, Debug = 3, Fixnum safety = 3
;;; Source level debugging is on
;;; Source file recording is  on
;;; Cross referencing is on
; (TOP-LEVEL-FORM 0)
; (TOP-LEVEL-FORM 1)
; (DEFVAR ELEPHANT::*ELEPHANT-CODE-VERSION*)
; (DEFVAR ELEPHANT::*ELEPHANT-CODE-VERSION-INT*)
; (DEFVAR ELEPHANT::*ELEPHANT-UNMARKED-CODE-VERSION*)
; (DEFVAR ELEPHANT::*ELEPHANT-PROPERTIES-LABEL*)
; (DEFVAR ELEPHANT::*USER-CONFIGURABLE-PARAMETERS*)
; (DEFVAR ELEPHANT::*ENABLE-MULTI-STORE-INDEXING*)
; (DEFCONSTANT ELEPHANT::+NONE+)
; (DEFCONSTANT ELEPHANT::+TXN+)
; (DEFCONSTANT ELEPHANT::+CHECKOUT+)
; (DEFPARAMETER ELEPHANT::*CACHED-INSTANCE-DEFAULT-MODE*)
; (DEFVAR ELEPHANT::*CIRCULARITY-INITIAL-HASH-SIZE*)
; (DEFPARAMETER ELEPHANT::*MAP-USING-DEGREE2*)
; (DEFVAR ELEPHANT::*BERKELEY-DB-CACHESIZE*)
; (DEFVAR ELEPHANT::*BERKELEY-DB-MAX-LOCKS*)
; (DEFVAR ELEPHANT::*BERKELEY-DB-MAX-OBJECTS*)
; (DEFVAR ELEPHANT::*DEFAULT-MVCC*)
; (DEFVAR ELEPHANT:*STORE-CONTROLLER*)
; (DEFVAR ELEPHANT::*CURRENT-TRANSACTION*)
; (DEFVAR ELEPHANT:*DEFAULT-RETRIES*)
; (DEFVAR ELEPHANT::*SERIALIZER-MARK-LIST*)
; (DEFVAR ELEPHANT::*SERIALIZER-INST-LIST*)
; (DEFPARAMETER ELEPHANT:*WARN-ON-MANUAL-CLASS-FINALIZATION*)
; (DEFPARAMETER ELEPHANT::*MIGRATE-MESSAGES*)
; (DEFPARAMETER ELEPHANT::*MIGRATE-VERBOSE*)
; (DEFPARAMETER ELEPHANT:*WARN-WHEN-DROPPING-PERSISTENT-SLOTS*)
; (DEFPARAMETER ELEPHANT::*RETURN-NULL-ON-MISSING-INSTANCE*)
; ELEPHANT:WITH-INHIBITED-WARNINGS
; (TOP-LEVEL-FORM 29)
; (TOP-LEVEL-FORM 0)
; Loading fasl file /home/lmc/lisp/elephant/elephant-1.0/src/elephant/variables.ufasl
;;; Compiling file /home/lmc/lisp/elephant/elephant-1.0/src/elephant/transactions.lisp ...
;;; Safety = 1, Speed = 3, Space = 1, Float = 1, Interruptible = 0
;;; Compilation speed = 0, Debug = 3, Fixnum safety = 3
;;; Source level debugging is on
;;; Source file recording is  on
;;; Cross referencing is on
; (TOP-LEVEL-FORM 0)
; (TOP-LEVEL-FORM 1)
; (DEFGENERIC ELEPHANT::EXECUTE-TRANSACTION)
; ELEPHANT::MAKE-TRANSACTION-RECORD
; ELEPHANT::TRANSACTION-STORE
; ELEPHANT::TRANSACTION-OBJECT
; ELEPHANT::TRANSACTION-PRIOR
; ELEPHANT::TRANSACTION-OBJECT-P
; ELEPHANT::PRIOR-OWNED-TXN
; ELEPHANT::OWNED-TXN-P
; (SUBFUNCTION (DEFCLASS ELEPHANT:TRANSACTION-RETRY-COUNT-EXCEEDED) (DEFINE-CONDITION ELEPHANT:TRANSACTION-RETRY-COUNT-EXCEEDED))
; (DEFINE-CONDITION ELEPHANT:TRANSACTION-RETRY-COUNT-EXCEEDED)
; ELEPHANT:WITH-TRANSACTION
; ELEPHANT:ENSURE-TRANSACTION
; ELEPHANT::WITH-BATCH-TRANSACTION
; (DEFGENERIC ELEPHANT:CONTROLLER-START-TRANSACTION)
; (DEFGENERIC ELEPHANT:CONTROLLER-COMMIT-TRANSACTION)
; (DEFGENERIC ELEPHANT:CONTROLLER-ABORT-TRANSACTION)
; (TOP-LEVEL-FORM 0)
; Loading fasl file /home/lmc/lisp/elephant/elephant-1.0/src/elephant/transactions.ufasl
;;; Compiling file /home/lmc/lisp/elephant/elephant-1.0/src/elephant/schemas.lisp ...
;;; Safety = 1, Speed = 3, Space = 1, Float = 1, Interruptible = 0
;;; Compilation speed = 0, Debug = 3, Fixnum safety = 3
;;; Source level debugging is on
;;; Source file recording is  on
;;; Cross referencing is on
; (TOP-LEVEL-FORM 0)
; (TOP-LEVEL-FORM 1)
; (DEFCLASS ELEPHANT::SCHEMA)
; (METHOD PRINT-OBJECT (ELEPHANT::SCHEMA T))
; (SUBFUNCTION (DEFSETF ELEPHANT::SLOT-REC-ARGS) (DEFSTRUCT ELEPHANT::SLOT-REC))
; (SUBFUNCTION (DEFSETF ELEPHANT::SLOT-REC-NAME) (DEFSTRUCT ELEPHANT::SLOT-REC))
; (SUBFUNCTION (DEFSETF ELEPHANT::SLOT-REC-TYPE) (DEFSTRUCT ELEPHANT::SLOT-REC))
; (SUBFUNCTION ELEPHANT::MAKE-SLOT-REC (DEFSTRUCT ELEPHANT::SLOT-REC))
; (SUBFUNCTION ELEPHANT::MAKE-SLOT-REC (DEFSTRUCT ELEPHANT::SLOT-REC))
; ELEPHANT::SLOT-REC-EQ
; ELEPHANT::INDEXED-SLOT-REC-EQ
; ELEPHANT::GET-SLOT-RECS-BY-TYPE
; ELEPHANT::COMPUTE-SCHEMA
; ELEPHANT::COMPUTE-TRANSIENT-SCHEMA
; (DEFPARAMETER ELEPHANT::*SLOT-DEF-TYPE-TAGS*)
; ELEPHANT::COMPUTE-SLOT-RECS
; ELEPHANT::COMPUTE-TRANSIENT-SLOT-RECS
; (METHOD ELEPHANT::COMPUTE-SLOT-RECS-BY-TYPE (T T T))
; (METHOD ELEPHANT::COMPUTE-SLOT-RECS-BY-TYPE ((EQL :INDEXED) T T))
; (DEFCLASS ELEPHANT::DB-SCHEMA)
; (METHOD PRINT-OBJECT (ELEPHANT::DB-SCHEMA T))
; ELEPHANT::MAKE-DB-SCHEMA
; ELEPHANT::LOGICAL-COPY-SCHEMA
; ELEPHANT::COPY-SCHEMA
; (METHOD ELEPHANT::MATCH-SCHEMAS (NULL T))
; (METHOD ELEPHANT::MATCH-SCHEMAS (T NULL))
; (METHOD ELEPHANT::MATCH-SCHEMAS (ELEPHANT::SCHEMA ELEPHANT::SCHEMA))
; ELEPHANT::SYMBOL<
; ELEPHANT::SORTED-SLOTS
; (METHOD ELEPHANT::SCHEMA-DIFF (T T))
; ELEPHANT::DIFF-TYPE
; ELEPHANT::DIFF-RECS
; (METHOD ELEPHANT::DEFAULT-CLASS-CONSTRUCTOR (ELEPHANT::DB-SCHEMA))
; ELEPHANT::SLOT-DEFS-FROM-SCHEMA
; ELEPHANT::COMPUTE-READER
; ELEPHANT::COMPUTE-WRITER
; ELEPHANT::DEFAULT-TEMPLATE
; ELEPHANT::DUMP-SCHEMA
; ELEPHANT::DUMP-SLOTS
; (TOP-LEVEL-FORM 0)
; Loading fasl file /home/lmc/lisp/elephant/elephant-1.0/src/elephant/schemas.ufasl
;;; Compiling file /home/lmc/lisp/elephant/elephant-1.0/src/elephant/metaclasses.lisp ...
;;; Safety = 1, Speed = 3, Space = 1, Float = 1, Interruptible = 0
;;; Compilation speed = 0, Debug = 3, Fixnum safety = 3
;;; Source level debugging is on
;;; Source file recording is  on
;;; Cross referencing is on
; (TOP-LEVEL-FORM 0)
; (TOP-LEVEL-FORM 1)
; (DEFCLASS ELEPHANT:PERSISTENT)
; (METHOD PRINT-OBJECT (ELEPHANT:PERSISTENT T))
; (DEFCLASS ELEPHANT:PERSISTENT-METACLASS)
; (METHOD ELEPHANT::GET-CLASS-SCHEMA (T))
; (METHOD ELEPHANT::SET-CLASS-SCHEMA (T T))
; (DEFSETF ELEPHANT::GET-CLASS-SCHEMA)
; (METHOD ELEPHANT::GET-STORE-SCHEMAS (T))
; (METHOD ELEPHANT::SET-STORE-SCHEMAS (T T))
; (DEFSETF ELEPHANT::GET-STORE-SCHEMAS)
; (METHOD ELEPHANT::GET-CLASS-INDEXING (T))
; (DEFSETF ELEPHANT::GET-CLASS-INDEXING)
; (METHOD ELEPHANT::GET-CACHE-STYLE (T))
; (DEFSETF ELEPHANT::GET-CACHE-STYLE)
; (METHOD ELEPHANT::HAS-CLASS-SCHEMA-P (ELEPHANT:PERSISTENT-METACLASS))
; (METHOD ELEPHANT::HAS-CLASS-CONTROLLER-SCHEMA-P (T ELEPHANT:PERSISTENT-METACLASS))
; (METHOD ELEPHANT::GET-CLASS-CONTROLLER-SCHEMA (T ELEPHANT:PERSISTENT-METACLASS))
; (METHOD ELEPHANT::ADD-CLASS-CONTROLLER-SCHEMA (T ELEPHANT:PERSISTENT-METACLASS T))
; (METHOD ELEPHANT::REMOVE-CLASS-CONTROLLER-SCHEMA (T ELEPHANT:PERSISTENT-METACLASS))
; (METHOD ELEPHANT::CLASS-INDEXING-ENABLED-P (ELEPHANT:PERSISTENT-METACLASS))
; ELEPHANT::MIGRATE-CLASS-INDEX-P
; ELEPHANT:DEFPCLASS
; ELEPHANT::ADD-PERSISTENT-METACLASS-ARGUMENT
; (DEFCLASS ELEPHANT::PERSISTENT-SLOT-DEFINITION)
; (DEFCLASS ELEPHANT::PERSISTENT-DIRECT-SLOT-DEFINITION)
; (DEFCLASS ELEPHANT::PERSISTENT-EFFECTIVE-SLOT-DEFINITION)
; (DEFGENERIC ELEPHANT::PERSISTENT-P)
; ELEPHANT::PERSISTENT-SLOT-DEFS
; ELEPHANT::PERSISTENT-SLOT-NAMES
; ELEPHANT::ALL-PERSISTENT-SLOT-NAMES
; ELEPHANT::ALL-SINGLE-VALUED-SLOT-DEFS
; (DEFCLASS ELEPHANT::CACHED-SLOT-DEFINITION)
; (DEFCLASS ELEPHANT::CACHED-DIRECT-SLOT-DEFINITION)
; (DEFCLASS ELEPHANT::CACHED-EFFECTIVE-SLOT-DEFINITION)
; ELEPHANT::CACHED-SLOT-DEFS
; ELEPHANT::CACHED-SLOT-NAMES
; (DEFCLASS ELEPHANT::TRANSIENT-SLOT-DEFINITION)
; (DEFCLASS ELEPHANT::TRANSIENT-DIRECT-SLOT-DEFINITION)
; (DEFCLASS ELEPHANT::TRANSIENT-EFFECTIVE-SLOT-DEFINITION)
; (DEFGENERIC ELEPHANT::TRANSIENT-P)
; ELEPHANT::ENSURE-TRANSIENT-CHAIN
; ELEPHANT::TRANSIENT-SLOT-DEFS
; ELEPHANT::TRANSIENT-SLOT-NAMES
; (DEFCLASS ELEPHANT::INDEXED-SLOT-DEFINITION)
; (DEFCLASS ELEPHANT::INDEXED-DIRECT-SLOT-DEFINITION)
; (DEFCLASS ELEPHANT::INDEXED-EFFECTIVE-SLOT-DEFINITION)
; (METHOD ELEPHANT::INDEXED-P (T))
; (METHOD ELEPHANT::GET-SLOT-DEF-INDEX (ELEPHANT::INDEXED-EFFECTIVE-SLOT-DEFINITION T))
; (METHOD ELEPHANT::ADD-SLOT-DEF-INDEX (T ELEPHANT::INDEXED-EFFECTIVE-SLOT-DEFINITION T))
; (METHOD ELEPHANT::CLEAR-SLOT-DEF-INDEX (ELEPHANT::INDEXED-EFFECTIVE-SLOT-DEFINITION T))
; (METHOD ELEPHANT::INDEXED-SLOT-DEFS (T))
; (METHOD ELEPHANT::INDEXED-SLOT-NAMES (T))
; (DEFCLASS ELEPHANT::DERIVED-INDEX-SLOT-DEFINITION)
; (DEFCLASS ELEPHANT::DERIVED-INDEX-DIRECT-SLOT-DEFINITION)
; (DEFCLASS ELEPHANT::DERIVED-INDEX-EFFECTIVE-SLOT-DEFINITION)
; (METHOD ELEPHANT::DERIVED-INDEX-SLOT-DEFS (T))
; (METHOD ELEPHANT::DERIVED-INDEX-SLOT-NAMES (T))
; ELEPHANT::COMPILE-DERIVED-FN
; ELEPHANT::GEN-DERIVED-SYM-WRAPPER
; ELEPHANT::GEN-DERIVED-FN-WRAPPER
; (DEFCLASS ELEPHANT::SET-VALUED-SLOT-DEFINITION)
; (DEFCLASS ELEPHANT::SET-VALUED-DIRECT-SLOT-DEFINITION)
; (DEFCLASS ELEPHANT::SET-VALUED-EFFECTIVE-SLOT-DEFINITION)
; ELEPHANT::SET-VALUED-SLOT-DEFS
; ELEPHANT::SET-VALUED-SLOT-NAMES
; (DEFCLASS ELEPHANT::ASSOCIATION-SLOT-DEFINITION)
; (DEFCLASS ELEPHANT::ASSOCIATION-DIRECT-SLOT-DEFINITION)
; (DEFCLASS ELEPHANT::ASSOCIATION-EFFECTIVE-SLOT-DEFINITION)
; (METHOD INITIALIZE-INSTANCE :AFTER (ELEPHANT::ASSOCIATION-EFFECTIVE-SLOT-DEFINITION))
; ELEPHANT::ASSOCIATION-END-P
; ELEPHANT::ASSOCIATION-SLOT-DEFS
; ELEPHANT::ASSOCIATION-SLOT-NAMES
; ELEPHANT::ASSOCIATION-END-SLOT-NAMES
; ELEPHANT::GET-ASSOCIATION-SLOT-INDEX
; (METHOD ELEPHANT::GET-SLOT-DEF-INDEX (ELEPHANT::ASSOCIATION-EFFECTIVE-SLOT-DEFINITION T))
; ELEPHANT::ADD-ASSOCIATION-SLOT-INDEX
; ELEPHANT::REMOVE-ASSOCIATION-SLOT-INDEX
; (METHOD VALIDATE-SUPERCLASS (ELEPHANT:PERSISTENT-METACLASS STANDARD-CLASS))
; (METHOD VALIDATE-SUPERCLASS (STANDARD-CLASS ELEPHANT:PERSISTENT-METACLASS))
; (DEFGENERIC ELEPHANT::DATABASE-ALLOCATION-P)
; (METHOD SLOT-DEFINITION-ALLOCATION (ELEPHANT::PERSISTENT-SLOT-DEFINITION))
; ELEPHANT::BIND-STANDARD-INIT-ARGUMENTS
; (METHOD CLOS:DIRECT-SLOT-DEFINITION-CLASS (ELEPHANT:PERSISTENT-METACLASS))
; (METHOD CLOS:EFFECTIVE-SLOT-DEFINITION-CLASS (ELEPHANT:PERSISTENT-METACLASS))
; (METHOD CLOS::COMPUTE-EFFECTIVE-SLOT-DEFINITION-INITARGS (ELEPHANT:PERSISTENT-METACLASS T T))
; ELEPHANT::FIND-CLASS-FOR-DIRECT-SLOT
; ELEPHANT::FIND-DIRECT-SLOT-DEF-BY-NAME
; ELEPHANT::FIND-SLOT-DEF-BY-NAME
; (METHOD ELEPHANT::FIND-SLOT-DEFS-BY-TYPE (ELEPHANT:PERSISTENT-METACLASS T))
; (METHOD ELEPHANT::FIND-SLOT-DEF-NAMES-BY-TYPE (ELEPHANT:PERSISTENT-METACLASS T))
; ELEPHANT::COUNT-TRUE
; (METHOD (SETF SLOT-DEFINITION-ALLOCATION) (T ELEPHANT::PERSISTENT-SLOT-DEFINITION))
; (TOP-LEVEL-FORM 0)
; Loading fasl file /home/lmc/lisp/elephant/elephant-1.0/src/elephant/metaclasses.ufasl
;;; Compiling file /home/lmc/lisp/elephant/elephant-1.0/src/elephant/classes.lisp ...
;;; Safety = 1, Speed = 3, Space = 1, Float = 1, Interruptible = 0
;;; Compilation speed = 0, Debug = 3, Fixnum safety = 3
;;; Source level debugging is on
;;; Source file recording is  on
;;; Cross referencing is on
; (TOP-LEVEL-FORM 0)
; (TOP-LEVEL-FORM 1)
; (DEFVAR ELEPHANT::*DEBUG-SI*)
; (METHOD INITIALIZE-INSTANCE :BEFORE (ELEPHANT:PERSISTENT))
; ELEPHANT::INITIAL-PERSISTENT-SETUP
; ELEPHANT::REGISTER-NEW-INSTANCE
; ELEPHANT::CHECK-VALID-STORE-CONTROLLER
; (DEFCLASS ELEPHANT:PERSISTENT-COLLECTION)
; (DEFCLASS ELEPHANT:PERSISTENT-OBJECT)

**++++ Error in (DEFCLASS ELEPHANT::CACHEABLE-PERSISTENT-OBJECT): 
  Invalid allocation type INSTANCE for slot-definition-allocation
; (METHOD SHARED-INITIALIZE :AROUND (ELEPHANT:PERSISTENT-METACLASS T))
; ELEPHANT::ENSURE-CLASS-INHERITS-FROM
; ELEPHANT::SUPERCLASS-MEMBER-P
; ELEPHANT::HAS-CACHED-SLOT-SPECIFICATION
; (METHOD CLOS:FINALIZE-INHERITANCE :AFTER (ELEPHANT:PERSISTENT-METACLASS))
; ELEPHANT::COMPUTE-DERIVED-INDEX-TRIGGERS
; (METHOD REINITIALIZE-INSTANCE :AROUND (ELEPHANT:PERSISTENT-METACLASS))
; ELEPHANT::BIND-SLOT-DEFS
; ELEPHANT::COMPUTE-BINDINGS
; (METHOD INITIALIZE-INSTANCE :AROUND (ELEPHANT:PERSISTENT-OBJECT))
; (METHOD SHARED-INITIALIZE :AROUND (ELEPHANT:PERSISTENT-OBJECT T))
;;;*** Warning in (METHOD SHARED-INITIALIZE :AROUND (ELEPHANT::CACHEABLE-PERSISTENT-OBJECT T)): Ignoring type declaration with illegal type ELEPHANT::CACHEABLE-PERSISTENT-OBJECT
; (METHOD SHARED-INITIALIZE :AROUND (ELEPHANT::CACHEABLE-PERSISTENT-OBJECT T))
; ELEPHANT::INITIALIZE-PERSISTENT-SLOTS
; ELEPHANT::INITIALIZE-SET-SLOTS
; ELEPHANT::INITIALIZE-FROM-INITARG
; ELEPHANT::GET-INIT-SLOTNAMES
; (SUBFUNCTION (DEFCLASS ELEPHANT:DROPPING-PERSISTENT-SLOT-DATA) (DEFINE-CONDITION ELEPHANT:DROPPING-PERSISTENT-SLOT-DATA))
; (DEFINE-CONDITION ELEPHANT:DROPPING-PERSISTENT-SLOT-DATA)
; (DEFINE-CONDITION ELEPHANT:DROPPING-PERSISTENT-SLOT-DATA)
; ELEPHANT::WARN-ABOUT-DROPPED-SLOTS
; (METHOD ELEPHANT::RECREATE-INSTANCE-USING-CLASS (T))
; (DEFGENERIC ELEPHANT::RECREATE-INSTANCE)
; (METHOD ELEPHANT::RECREATE-INSTANCE (ELEPHANT:PERSISTENT-OBJECT))
; (METHOD ELEPHANT::RECREATE-INSTANCE (ELEPHANT:PERSISTENT-COLLECTION))
; (METHOD UPDATE-INSTANCE-FOR-REDEFINED-CLASS :AROUND (ELEPHANT:PERSISTENT-OBJECT T T T))
; (METHOD CHANGE-CLASS :BEFORE (ELEPHANT:PERSISTENT STANDARD-CLASS))
; (METHOD UPDATE-INSTANCE-FOR-DIFFERENT-CLASS :AFTER (ELEPHANT:PERSISTENT-OBJECT ELEPHANT:PERSISTENT-OBJECT))
; (METHOD CHANGE-CLASS :BEFORE (STANDARD-OBJECT ELEPHANT:PERSISTENT-METACLASS))
; (METHOD CLOS:SLOT-VALUE-USING-CLASS (ELEPHANT:PERSISTENT-METACLASS ELEPHANT:PERSISTENT-OBJECT ELEPHANT::PERSISTENT-SLOT-DEFINITION))
; (METHOD (SETF CLOS:SLOT-VALUE-USING-CLASS) (T ELEPHANT:PERSISTENT-METACLASS ELEPHANT:PERSISTENT-OBJECT ELEPHANT::PERSISTENT-SLOT-DEFINITION))
; (METHOD CLOS:SLOT-BOUNDP-USING-CLASS (ELEPHANT:PERSISTENT-METACLASS ELEPHANT:PERSISTENT-OBJECT ELEPHANT::PERSISTENT-SLOT-DEFINITION))
; (METHOD CLOS:SLOT-BOUNDP-USING-CLASS (ELEPHANT:PERSISTENT-METACLASS ELEPHANT:PERSISTENT-OBJECT SYMBOL))
; (METHOD CLOS:SLOT-MAKUNBOUND-USING-CLASS (ELEPHANT:PERSISTENT-METACLASS ELEPHANT:PERSISTENT-OBJECT ELEPHANT::PERSISTENT-SLOT-DEFINITION))
; ELEPHANT::VALID-PERSISTENT-REFERENCE-P
; (SUBFUNCTION (DEFCLASS ELEPHANT:CROSS-REFERENCE-ERROR) (DEFINE-CONDITION ELEPHANT:CROSS-REFERENCE-ERROR))
; (DEFINE-CONDITION ELEPHANT:CROSS-REFERENCE-ERROR)
; (DEFINE-CONDITION ELEPHANT:CROSS-REFERENCE-ERROR)
; ELEPHANT::SIGNAL-CROSS-REFERENCE-ERROR
; (METHOD CLOS:SLOT-VALUE-USING-CLASS (ELEPHANT:PERSISTENT-METACLASS ELEPHANT:PERSISTENT-OBJECT T))
; (METHOD (SETF CLOS:SLOT-VALUE-USING-CLASS) (T ELEPHANT:PERSISTENT-METACLASS ELEPHANT:PERSISTENT-OBJECT T))
; (METHOD CLOS:SLOT-MAKUNBOUND-USING-CLASS (ELEPHANT:PERSISTENT-METACLASS ELEPHANT:PERSISTENT-OBJECT T))
; (TOP-LEVEL-FORM 0)
; *** 1 error detected, no fasl file produced.
Warning: COMPILE-FILE warned while performing #<ASDF:COMPILE-OP NIL 21A3A02F> on
         #<ASDF:CL-SOURCE-FILE "classes" 21A4D99B>.
Warning: COMPILE-FILE failed while performing #<ASDF:COMPILE-OP NIL 21A3A02F> on
         #<ASDF:CL-SOURCE-FILE "classes" 21A4D99B>.
Error: erred while invoking #<ASDF:COMPILE-OP NIL 21A3A02F> on
       #<ASDF:CL-SOURCE-FILE "classes" 21A4D99B>
  1 (continue) Retry performing #<ASDF:COMPILE-OP NIL 21A3A02F> on
               #<ASDF:CL-SOURCE-FILE "classes" 21A4D99B>.
  2 Continue, treating #<ASDF:COMPILE-OP NIL 21A3A02F> on #<ASDF:CL-SOURCE-FILE "classes" 21A4D99B>
    as having been successful.
  3 (abort) Return to level 0.
  4 Return to top loop level 0.

Type :b for backtrace, :c <option number> to proceed,  or :? for other options

CL-USER 2 : 1 > 



From alain.picard at accenture.com  Tue Jan 27 02:08:54 2009
From: alain.picard at accenture.com (alain.picard at accenture.com)
Date: Tue, 27 Jan 2009 10:08:54 +0800
Subject: [elephant-devel] CLSQL Store
In-Reply-To: <9CA671F6-67EB-4EA0-A1B9-307685022DCD@media.mit.edu>
References: <ecb948b30901021117t4733cbf0m83306c66a863c36d@mail.gmail.com><gjmef5$l64$1@ger.gmane.org><1F1F136C-D922-46F8-BAC3-B252D34BF8BC@media.mit.edu><461C0B66B0E04C1EBCFFF01B7E1C7D18@killer><8457077CBD070B45AC1307D5265A81B2626C51@APAXM1114.dir.svc.accenture.com><gkplap$7ds$1@ger.gmane.org>
	<9CA671F6-67EB-4EA0-A1B9-307685022DCD@media.mit.edu>
Message-ID: <8457077CBD070B45AC1307D5265A81B266E69F@APAXM1114.dir.svc.accenture.com>

 

> -----Original Message-----
> From: Ian Eslick [mailto:eslick at media.mit.edu] 
> Running BDB behind a website doesn't count as a 
> 'distribution' that requires disclosure of the website 
> source.  The cgi discussion is a sub-case of this.
> 
> This wouldn't apply to selling website frameworks or desktop 
> software, of course.
> 
> Of course if you are going to be making money it would be 
> worthwhile to get a legal opinion on this.
> > I meant the next paragraph in the same document:
> > ----
> > Do I have to pay for a Berkeley DB license to use it in my Perl or 
> > Python scripts?
> >
> > I don't think that CL is anyhow worse than Python or Perl, and 
> > situation seems to be pretty much the same. For a Perl, BDB 
> connector 
> > is a standalone module, as I understand, for Python it is 
> included in 
> > standard  distribution. You can easily make
> > SBCL+Elephant bundle,
> > for example, and claim that it is this bundly which uses 
> BDB, and not 
> > your scripts -- it would be same logic as in Oracle's 
> statement above.
> >
> > Of course, it is a question what is qualified to be a "script" and 
> > what is "software" (and if it makes sense to take such 
> distinction at 
> > all).

OK.  I'm interested to hear you guys think this way.
So you think if I package the software as a SBCL+LLGPL
utils/BDB+BerkeleyDB,
I could redistribute that "core" as open source, and run
my "script" (i.e. my website) "on top of" that.

I think speaking to lawyers would still not really resolve the
issue; the aim isn't simply to be prepared to win a lawsuit;
the aim is to never get sued in the first place.  For that to
happen, I'd be more interested in the actual intent of the license
holder (i.e. Oracle), so they'd have to be contacted to bless
this interpretation.

Anyway, thanks for the response!  This is all highly relevant!

  --alain



This message is for the designated recipient only and may contain privileged, proprietary, or otherwise private information.  If you have received it in error, please notify the sender immediately and delete the original.  Any other use of the email by you is prohibited.



From eslick at media.mit.edu  Tue Jan 27 04:00:25 2009
From: eslick at media.mit.edu (Ian Eslick)
Date: Mon, 26 Jan 2009 23:00:25 -0500
Subject: [elephant-devel] CLSQL Store
In-Reply-To: <8457077CBD070B45AC1307D5265A81B266E69F@APAXM1114.dir.svc.accenture.com>
References: <ecb948b30901021117t4733cbf0m83306c66a863c36d@mail.gmail.com><gjmef5$l64$1@ger.gmane.org><1F1F136C-D922-46F8-BAC3-B252D34BF8BC@media.mit.edu><461C0B66B0E04C1EBCFFF01B7E1C7D18@killer><8457077CBD070B45AC1307D5265A81B2626C51@APAXM1114.dir.svc.accenture.com><gkplap$7ds$1@ger.gmane.org>
	<9CA671F6-67EB-4EA0-A1B9-307685022DCD@media.mit.edu>
	<8457077CBD070B45AC1307D5265A81B266E69F@APAXM1114.dir.svc.accenture.com>
Message-ID: <D7D805B9-8619-4393-92F9-A2A055F38F25@media.mit.edu>

Well, this is advice is just about worth the bandwidth used to  
download it...

Elephant and Lisp are widely available open source platforms, just  
like Python.  If you are running an application on top of that  
platform, such as a networked service, I'm fairly certain you are as  
in the clear with Lisp as with Python.  So this implies you can look  
at commercial models that use Python+BDB or Ruby+BDB and use that as a  
guiding precedent.

Think about the spirit of the business model the license is intended  
to support.  This is invaluable in guiding your interpretation.  If  
you are selling software that requires that someone else download the  
platform and use it, your financial success now depends on someone  
duplicating and running a parallel instance of BDB.  This strikes me  
as a value for which the developers of BDB should be compensated.  My  
guess is that the difference between ("buy my scripts" and download  
Elephant vs. "here is a shrink wrapped solution") is academic and  
these are identical in the eyes of any reasonable business model.

However it is difficult to make this model work in the case of web  
servers or persistent object systems.  How do you charge for a small  
number of instances of BDB running a large web service?  It isn't  
worth charging a 'per-instance' fee and it's very hard to value or  
charge for some 'volume of use'.  I've seen other embedded software  
vendors try to charge for value and it rarely works, so better to  
focus on the businesses where $ is tied to volume.

Cheers,
Ian

PS - A lawyer is the only person who can competently, and legally,  
give you an opinion as to whether your use does or does not fall under  
the license terms and thus whether you will or will not ever get sued.

On Jan 26, 2009, at 9:08 PM, <alain.picard at accenture.com> <alain.picard at accenture.com 
 > wrote:

>
>
>> -----Original Message-----
>> From: Ian Eslick [mailto:eslick at media.mit.edu]
>> Running BDB behind a website doesn't count as a
>> 'distribution' that requires disclosure of the website
>> source.  The cgi discussion is a sub-case of this.
>>
>> This wouldn't apply to selling website frameworks or desktop
>> software, of course.
>>
>> Of course if you are going to be making money it would be
>> worthwhile to get a legal opinion on this.
>>> I meant the next paragraph in the same document:
>>> ----
>>> Do I have to pay for a Berkeley DB license to use it in my Perl or
>>> Python scripts?
>>>
>>> I don't think that CL is anyhow worse than Python or Perl, and
>>> situation seems to be pretty much the same. For a Perl, BDB
>> connector
>>> is a standalone module, as I understand, for Python it is
>> included in
>>> standard  distribution. You can easily make
>>> SBCL+Elephant bundle,
>>> for example, and claim that it is this bundly which uses
>> BDB, and not
>>> your scripts -- it would be same logic as in Oracle's
>> statement above.
>>>
>>> Of course, it is a question what is qualified to be a "script" and
>>> what is "software" (and if it makes sense to take such
>> distinction at
>>> all).
>
> OK.  I'm interested to hear you guys think this way.
> So you think if I package the software as a SBCL+LLGPL
> utils/BDB+BerkeleyDB,
> I could redistribute that "core" as open source, and run
> my "script" (i.e. my website) "on top of" that.
>
> I think speaking to lawyers would still not really resolve the
> issue; the aim isn't simply to be prepared to win a lawsuit;
> the aim is to never get sued in the first place.  For that to
> happen, I'd be more interested in the actual intent of the license
> holder (i.e. Oracle), so they'd have to be contacted to bless
> this interpretation.
>
> Anyway, thanks for the response!  This is all highly relevant!
>
>  --alain
>
>
>
> This message is for the designated recipient only and may contain  
> privileged, proprietary, or otherwise private information.  If you  
> have received it in error, please notify the sender immediately and  
> delete the original.  Any other use of the email by you is prohibited.
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel




From alain.picard at accenture.com  Tue Jan 27 05:02:02 2009
From: alain.picard at accenture.com (alain.picard at accenture.com)
Date: Tue, 27 Jan 2009 13:02:02 +0800
Subject: [elephant-devel] CLSQL Store
In-Reply-To: <D7D805B9-8619-4393-92F9-A2A055F38F25@media.mit.edu>
References: <ecb948b30901021117t4733cbf0m83306c66a863c36d@mail.gmail.com><gjmef5$l64$1@ger.gmane.org><1F1F136C-D922-46F8-BAC3-B252D34BF8BC@media.mit.edu><461C0B66B0E04C1EBCFFF01B7E1C7D18@killer><8457077CBD070B45AC1307D5265A81B2626C51@APAXM1114.dir.svc.accenture.com><gkplap$7ds$1@ger.gmane.org><9CA671F6-67EB-4EA0-A1B9-307685022DCD@media.mit.edu><8457077CBD070B45AC1307D5265A81B266E69F@APAXM1114.dir.svc.accenture.com>
	<D7D805B9-8619-4393-92F9-A2A055F38F25@media.mit.edu>
Message-ID: <8457077CBD070B45AC1307D5265A81B266E701@APAXM1114.dir.svc.accenture.com>

 

> -----Original Message-----
> From: Ian Eslick [mailto:eslick at media.mit.edu] 
> 
> Well, this is advice is just about worth the bandwidth used 
> to download it...

Duly noted.

> 
> Elephant and Lisp are widely available open source platforms, 
> just like Python.  If you are running an application on top 
> of that platform, such as a networked service, I'm fairly 
> certain you are as in the clear with Lisp as with Python.  So 
> this implies you can look at commercial models that use 
> Python+BDB or Ruby+BDB and use that as a guiding precedent.

Okay.  And such precedent exists --- i.e. there are, today,
several web businesses using python or ruby with BDB as a back end
using it's open source license?  That would be an interesting data
point.
 
> PS - A lawyer is the only person who can competently, and 
> legally, give you an opinion as to whether your use does or 
> does not fall under the license terms and thus whether you 
> will or will not ever get sued.

Absolutely.  And I can tell you that they'll NEVER guarantee
that you won't get sued --- I've spoken to quite a few lawyers
in my time. :-)  I've been told "you can be sued at any time
by anyone for anything.  Doesn't mean they'll win, of course."

That's why knowing the intent of the license holder is so critical.

Thanks for your help, Ian!


This message is for the designated recipient only and may contain privileged, proprietary, or otherwise private information.  If you have received it in error, please notify the sender immediately and delete the original.  Any other use of the email by you is prohibited.



From killerstorm at newmail.ru  Tue Jan 27 10:53:17 2009
From: killerstorm at newmail.ru (Alex Mizrahi)
Date: Tue, 27 Jan 2009 12:53:17 +0200
Subject: [elephant-devel] CLSQL Store
References: <ecb948b30901021117t4733cbf0m83306c66a863c36d@mail.gmail.com><gjmef5$l64$1@ger.gmane.org><1F1F136C-D922-46F8-BAC3-B252D34BF8BC@media.mit.edu><461C0B66B0E04C1EBCFFF01B7E1C7D18@killer><8457077CBD070B45AC1307D5265A81B2626C51@APAXM1114.dir.svc.accenture.com><gkplap$7ds$1@ger.gmane.org><9CA671F6-67EB-4EA0-A1B9-307685022DCD@media.mit.edu>
	<8457077CBD070B45AC1307D5265A81B266E69F@APAXM1114.dir.svc.accenture.com>
Message-ID: <glmp30$jde$1@ger.gmane.org>

 AP> I could redistribute that "core" as open source, and run
 AP> my "script" (i.e. my website) "on top of" that.

If you're using it only in your website, I think you don't even
need this open source mumbo-jumbo, as Oracle's license
only restricts _redistribution_ but not use, and
redistribution is a well defined thing -- I doubt that anybody
will consider it redistribution if you do not create copies
and pass them to third parties. 






From rsynnott at gmail.com  Tue Jan 27 11:35:37 2009
From: rsynnott at gmail.com (Robert Synnott)
Date: Tue, 27 Jan 2009 11:35:37 +0000
Subject: [elephant-devel] CLSQL Store
In-Reply-To: <glmp30$jde$1@ger.gmane.org>
References: <ecb948b30901021117t4733cbf0m83306c66a863c36d@mail.gmail.com>
	<gjmef5$l64$1@ger.gmane.org>
	<1F1F136C-D922-46F8-BAC3-B252D34BF8BC@media.mit.edu>
	<461C0B66B0E04C1EBCFFF01B7E1C7D18@killer>
	<8457077CBD070B45AC1307D5265A81B2626C51@APAXM1114.dir.svc.accenture.com>
	<gkplap$7ds$1@ger.gmane.org>
	<9CA671F6-67EB-4EA0-A1B9-307685022DCD@media.mit.edu>
	<8457077CBD070B45AC1307D5265A81B266E69F@APAXM1114.dir.svc.accenture.com>
	<glmp30$jde$1@ger.gmane.org>
Message-ID: <24f203480901270335j49c9e1acxd86d2a713a952981@mail.gmail.com>

2009/1/27 Alex Mizrahi <killerstorm at newmail.ru>:
>  AP> I could redistribute that "core" as open source, and run
>  AP> my "script" (i.e. my website) "on top of" that.
>
> If you're using it only in your website, I think you don't even
> need this open source mumbo-jumbo, as Oracle's license
> only restricts _redistribution_ but not use, and
> redistribution is a well defined thing -- I doubt that anybody
> will consider it redistribution if you do not create copies
> and pass them to third parties.
>
>
>
Yes, using it on your website should probably be okay; the AGPL
prohibits this, but use of GPL software in this way is common.
Rob

-- 
Robert Synnott
http://myblog.rsynnott.com
MSN: rsynnott at gmail.com
Jabber: rsynnott at gmail.com



From develgenius at gmail.com  Tue Jan 27 16:00:15 2009
From: develgenius at gmail.com (John)
Date: Tue, 27 Jan 2009 11:00:15 -0500
Subject: [elephant-devel] Learning about Elephant
Message-ID: <e6c974240901270800l11e2b308qf0fb2561782c6f86@mail.gmail.com>

Hi all,

I've recently come across Elephant and have been reading the manual (even
though it claims to be a bit stale). My approach is to read the manual and
then dig into the code to learn more details or discrepancies not
documented/updated on manual. In so far, I have a few questions I wonder if
anyone could answer. I know that most everyone says not to worry about
performance until you really need to, and I agree with that. However, having
in mind realistic potentials or limitations of the framework may allow you
to pre-plan or even decide if Elephant will be a good fit. My particular
interest is Elephant with BDB.

1) Every read/write of pclass slots is done directly from/to the database,
so no in-memory "copy" exists (unless some sort of transient cache slot
model is used). This is good. However, is Elephant "intelligent" enough so
that if you attempt to setf a slot value with the same value stored in the
slot, Elephant will "avoid" the writing to the database, since the value
hasn't really changed? If that's not the out-of-the-box behavior, I pressume
that it would be relatively easy to add this functionality using MOP. I also
assume that doing this will require another read before the write in order
to compare the value to be written, but then again, Elephant claims that
reads are much cheaper than writes.

2) According to Oracle: "Berkeley DB includes support for building highly
available applications based on replication..." (
http://www.oracle.com/technology/documentation/berkeley-db/db/ref/rep/intro.html)
So, I assume that in the event that a single machine becomes unable to
handle the load of a busy application, BDB supports replication to multiple
machines with near-instant replication effects (performance is inversly
proportional to replication speed). So, the question is, can Elephant
accommodate to this model? If I read correctly, the manual states that
elephant maintains a weak hash of persistent objects. So, if BDB is deployed
in a distributed model and Elephant is running in each separate machine, we
could in theory "trust" that the data read/written from/to disk will be all
in sync across all machines (as long as database IO on same object/slot
occurs at a frequency greater than the replication rate). The question I
have is with the weak hash. If a write is made in one machine, the data on
disk is updated across all machines. However, the weak hash remains stale in
the machines were the data was only replicated to. Is this an actual problem
or does Elephant use the weak hash "intelligently" to recover in the event
that the weak hash becomes "unexpectedly" stale?

3) I come from a RDBMS world, so I'm still learning the modalities of
connected objects vs just related rows. So, reading the tutorials you
describe a friends model using PSets. So, imagine a concept similar to
Facebook in terms of a friends database. I have millions of people created
in the system and they all create their list of friends. Some people may
have "few" or no friends while others could have hundreds of thousands of
friends (e.g. Pres. Obama). Are PSets the correct way to model this for
larger number of objects or is there a more appropriate methodology
recommended in Elephant? Obviously the idea behind this is so that you could
perform manipulations on these "friends" relatively easy, such as add/remove
friends or perform global queries as to list all friends of people who are
friends of Pres. Obama. There are references on the list about a query
system being worked on and some vanilla version being available, but
independently of that, I think my question is more related to the object
model implementation. Maybe I'm wrong.

Anyway, thank you for your help in advanced. Look forward to hearing back
from anyone soon and keep learning more about Elephant.

Thanks,
JD
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20090127/02ff0142/attachment.html>

From eslick at media.mit.edu  Tue Jan 27 18:32:26 2009
From: eslick at media.mit.edu (Ian Eslick)
Date: Tue, 27 Jan 2009 13:32:26 -0500
Subject: [elephant-devel] Learning about Elephant
In-Reply-To: <e6c974240901270800l11e2b308qf0fb2561782c6f86@mail.gmail.com>
References: <e6c974240901270800l11e2b308qf0fb2561782c6f86@mail.gmail.com>
Message-ID: <5D2242B8-1A86-4C89-96DD-3311A6EA2392@media.mit.edu>


On Jan 27, 2009, at 11:00 AM, John wrote:
>
> 1) Every read/write of pclass slots is done directly from/to the  
> database, so no in-memory "copy" exists (unless some sort of  
> transient cache slot model is used). This is good. However, is  
> Elephant "intelligent" enough so that if you attempt to setf a slot  
> value with the same value stored in the slot, Elephant will "avoid"  
> the writing to the database, since the value hasn't really changed?  
> If that's not the out-of-the-box behavior, I pressume that it would  
> be relatively easy to add this functionality using MOP. I also  
> assume that doing this will require another read before the write in  
> order to compare the value to be written, but then again, Elephant  
> claims that reads are much cheaper than writes.
>

Such a thing would be possible, but you would need to have a local  
copy of the value in memory and this could create more problems than  
it solves.  It's also unclear to me how common this case.  I don't  
write the same value to the same slot very often!

> 2) According to Oracle: "Berkeley DB includes support for building  
> highly available applications based on replication..." (http://www.oracle.com/technology/documentation/berkeley-db/db/ref/rep/intro.html 
> ) So, I assume that in the event that a single machine becomes  
> unable to handle the load of a busy application, BDB supports  
> replication to multiple machines with near-instant replication  
> effects (performance is inversly proportional to replication speed).  
> So, the question is, can Elephant accommodate to this model? If I  
> read correctly, the manual states that elephant maintains a weak  
> hash of persistent objects. So, if BDB is deployed in a distributed  
> model and Elephant is running in each separate machine, we could in  
> theory "trust" that the data read/written from/to disk will be all  
> in sync across all machines (as long as database IO on same object/ 
> slot occurs at a frequency greater than the replication rate). The  
> question I have is with the weak hash. If a write is made in one  
> machine, the data on disk is updated across all machines. However,  
> the weak hash remains stale in the machines were the data was only  
> replicated to. Is this an actual problem or does Elephant use the  
> weak hash "intelligently" to recover in the event that the weak hash  
> becomes "unexpectedly" stale?

The weak hash is simply used to speed up the 'recreation' of a  
persistent reference when it is deserialized from the underlying  
storage medium (BDB in this case) so fits into this model.  I've  
looked several times at building-in the support to enable a full  
replication model for BDB, but it's a fair bit of work to deal with  
the single-master requirement for replication and distributed  
transactions for global coherence. The write performance and  
contention performance may decline noticeably, but conflict-free read  
performance should be the same as in the non-replicated case.  I don't  
fully understand all these issues yet and am prioritizing a lisp-only  
Prevalence style backend in my development roadmap.

> 3) I come from a RDBMS world, so I'm still learning the modalities  
> of connected objects vs just related rows. So, reading the tutorials  
> you describe a friends model using PSets. So, imagine a concept  
> similar to Facebook in terms of a friends database. I have millions  
> of people created in the system and they all create their list of  
> friends. Some people may have "few" or no friends while others could  
> have hundreds of thousands of friends (e.g. Pres. Obama). Are PSets  
> the correct way to model this for larger number of objects or is  
> there a more appropriate methodology recommended in Elephant?  
> Obviously the idea behind this is so that you could perform  
> manipulations on these "friends" relatively easy, such as add/remove  
> friends or perform global queries as to list all friends of people  
> who are friends of Pres. Obama. There are references on the list  
> about a query system being worked on and some vanilla version being  
> available, but independently of that, I think my question is more  
> related to the object model implementation. Maybe I'm wrong.

Psets today are a convenience API around the BTree that can be  
overridden by data stores later for more efficient implementation  
(e.g. hash vs. tree) as appropriate for that store.  BTrees are cheap  
for BDB to create and use so use of them at any size is fine.  You  
might want to wrap an abstraction around the BTree directly since the  
pset API is an un-ordered set.  If you want to do range extraction or  
set intersection you'll want the btree's support for ordered objects.

The query system currently doesn't support psets, although it should  
eventually.

Slot-associations currently don't deal with reflexive references  
(associations among instances of a single class), but that would also  
address the 'friends-of' problem.

I hope you enjoy elephant!  If you want to add/implement or spec out  
some of these changes you are thinking about, I'd be happy to support  
you.

Cheers,
Ian

> Anyway, thank you for your help in advanced. Look forward to hearing  
> back from anyone soon and keep learning more about Elephant.
>
> Thanks,
> JD
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel




From eslick at media.mit.edu  Tue Jan 27 20:22:53 2009
From: eslick at media.mit.edu (Ian Eslick)
Date: Tue, 27 Jan 2009 15:22:53 -0500
Subject: [elephant-devel] Learning about Elephant
In-Reply-To: <5D2242B8-1A86-4C89-96DD-3311A6EA2392@media.mit.edu>
References: <e6c974240901270800l11e2b308qf0fb2561782c6f86@mail.gmail.com>
	<5D2242B8-1A86-4C89-96DD-3311A6EA2392@media.mit.edu>
Message-ID: <D387C483-7C97-4452-A42B-91C862BC499A@media.mit.edu>

Slot associations appear to work reflexively, by the way.

(defpclass self-assoc ()
	    ((friends-of :accessor friends-of
		     	 :associate (self-assoc friended-me)
		         :many-to-many t)
	     (friended-me :accessor friended-me
		       :associate (self-assoc friends-of)
		       :many-to-many t)))

This simply says I have an association among self-assoc instances.   
Any (setf (friends-of person1) person2) says that person2 is a friend- 
of person1.  Automatically, you get the inverse mapping which is that  
person1 friended person2 (assuming these associations are directional).

If they are not directional, then you can do this:
(defpclass person ()
   ((friends :accessor friends-of
	:associate (person friends) :many-to-many t)))

Now if you (setf (friends-of person1) person2), you will get:

(friends-of person1) => person2
(friends-of person2) => person1

This is a non-directional relationship between two entities so there's  
no need for two slots.

This all seems to work well in the current Alpha 2.

Ian

On Jan 27, 2009, at 1:32 PM, Ian Eslick wrote:

>
> On Jan 27, 2009, at 11:00 AM, John wrote:
>>
>> 1) Every read/write of pclass slots is done directly from/to the
>> database, so no in-memory "copy" exists (unless some sort of
>> transient cache slot model is used). This is good. However, is
>> Elephant "intelligent" enough so that if you attempt to setf a slot
>> value with the same value stored in the slot, Elephant will "avoid"
>> the writing to the database, since the value hasn't really changed?
>> If that's not the out-of-the-box behavior, I pressume that it would
>> be relatively easy to add this functionality using MOP. I also
>> assume that doing this will require another read before the write in
>> order to compare the value to be written, but then again, Elephant
>> claims that reads are much cheaper than writes.
>>
>
> Such a thing would be possible, but you would need to have a local
> copy of the value in memory and this could create more problems than
> it solves.  It's also unclear to me how common this case.  I don't
> write the same value to the same slot very often!
>
>> 2) According to Oracle: "Berkeley DB includes support for building
>> highly available applications based on replication..." (http://www.oracle.com/technology/documentation/berkeley-db/db/ref/rep/intro.html
>> ) So, I assume that in the event that a single machine becomes
>> unable to handle the load of a busy application, BDB supports
>> replication to multiple machines with near-instant replication
>> effects (performance is inversly proportional to replication speed).
>> So, the question is, can Elephant accommodate to this model? If I
>> read correctly, the manual states that elephant maintains a weak
>> hash of persistent objects. So, if BDB is deployed in a distributed
>> model and Elephant is running in each separate machine, we could in
>> theory "trust" that the data read/written from/to disk will be all
>> in sync across all machines (as long as database IO on same object/
>> slot occurs at a frequency greater than the replication rate). The
>> question I have is with the weak hash. If a write is made in one
>> machine, the data on disk is updated across all machines. However,
>> the weak hash remains stale in the machines were the data was only
>> replicated to. Is this an actual problem or does Elephant use the
>> weak hash "intelligently" to recover in the event that the weak hash
>> becomes "unexpectedly" stale?
>
> The weak hash is simply used to speed up the 'recreation' of a
> persistent reference when it is deserialized from the underlying
> storage medium (BDB in this case) so fits into this model.  I've
> looked several times at building-in the support to enable a full
> replication model for BDB, but it's a fair bit of work to deal with
> the single-master requirement for replication and distributed
> transactions for global coherence. The write performance and
> contention performance may decline noticeably, but conflict-free read
> performance should be the same as in the non-replicated case.  I don't
> fully understand all these issues yet and am prioritizing a lisp-only
> Prevalence style backend in my development roadmap.
>
>> 3) I come from a RDBMS world, so I'm still learning the modalities
>> of connected objects vs just related rows. So, reading the tutorials
>> you describe a friends model using PSets. So, imagine a concept
>> similar to Facebook in terms of a friends database. I have millions
>> of people created in the system and they all create their list of
>> friends. Some people may have "few" or no friends while others could
>> have hundreds of thousands of friends (e.g. Pres. Obama). Are PSets
>> the correct way to model this for larger number of objects or is
>> there a more appropriate methodology recommended in Elephant?
>> Obviously the idea behind this is so that you could perform
>> manipulations on these "friends" relatively easy, such as add/remove
>> friends or perform global queries as to list all friends of people
>> who are friends of Pres. Obama. There are references on the list
>> about a query system being worked on and some vanilla version being
>> available, but independently of that, I think my question is more
>> related to the object model implementation. Maybe I'm wrong.
>
> Psets today are a convenience API around the BTree that can be
> overridden by data stores later for more efficient implementation
> (e.g. hash vs. tree) as appropriate for that store.  BTrees are cheap
> for BDB to create and use so use of them at any size is fine.  You
> might want to wrap an abstraction around the BTree directly since the
> pset API is an un-ordered set.  If you want to do range extraction or
> set intersection you'll want the btree's support for ordered objects.
>
> The query system currently doesn't support psets, although it should
> eventually.
>
> Slot-associations currently don't deal with reflexive references
> (associations among instances of a single class), but that would also
> address the 'friends-of' problem.
>
> I hope you enjoy elephant!  If you want to add/implement or spec out
> some of these changes you are thinking about, I'd be happy to support
> you.
>
> Cheers,
> Ian
>
>> Anyway, thank you for your help in advanced. Look forward to hearing
>> back from anyone soon and keep learning more about Elephant.
>>
>> Thanks,
>> JD
>> _______________________________________________
>> elephant-devel site list
>> elephant-devel at common-lisp.net
>> http://common-lisp.net/mailman/listinfo/elephant-devel
>
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel




From develgenius at gmail.com  Tue Jan 27 19:59:14 2009
From: develgenius at gmail.com (John)
Date: Tue, 27 Jan 2009 14:59:14 -0500
Subject: [elephant-devel] Learning about Elephant
In-Reply-To: <5D2242B8-1A86-4C89-96DD-3311A6EA2392@media.mit.edu>
References: <e6c974240901270800l11e2b308qf0fb2561782c6f86@mail.gmail.com>
	<5D2242B8-1A86-4C89-96DD-3311A6EA2392@media.mit.edu>
Message-ID: <e6c974240901271159w7ec7bb37v3dfc1098e7991021@mail.gmail.com>

Hi Ian,

Thank you for your prompt response. Some comments below

On Tue, Jan 27, 2009 at 1:32 PM, Ian Eslick <eslick at media.mit.edu> wrote:

>
> Such a thing would be possible, but you would need to have a local
> copy of the value in memory and this could create more problems than
> it solves.  It's also unclear to me how common this case.  I don't
> write the same value to the same slot very often!


My question came from thinking of web applications. A user chooses to edit a
record. After s/he submits the form, I wouldn't want to incur the DB write
"costs" unless the data was actually changed by the user.


> The weak hash is simply used to speed up the 'recreation' of a
> persistent reference when it is deserialized from the underlying
> storage medium (BDB in this case) so fits into this model.  I've
> looked several times at building-in the support to enable a full
> replication model for BDB, but it's a fair bit of work to deal with
> the single-master requirement for replication and distributed
> transactions for global coherence. The write performance and
> contention performance may decline noticeably, but conflict-free read
> performance should be the same as in the non-replicated case.  I don't
> fully understand all these issues yet and am prioritizing a lisp-only
> Prevalence style backend in my development roadmap.


I'm not too familiar with BDB replication myself and was just googling
around. After reading that document in more detail and your explanation,
makes me realize that replication or the distributed BDB model is not such a
simple task. Since I don't think that my application(s) would really
out-scale the single machine Elephant/BDB deployment, I would certainly not
want to encourage investing time in this and rather concentrate on a
lisp-only prevalence  backend. Maybe one that covers these issues in the
future.


> Psets today are a convenience API around the BTree that can be
> overridden by data stores later for more efficient implementation
> (e.g. hash vs. tree) as appropriate for that store.  BTrees are cheap
> for BDB to create and use so use of them at any size is fine.  You
> might want to wrap an abstraction around the BTree directly since the
> pset API is an un-ordered set.  If you want to do range extraction or
> set intersection you'll want the btree's support for ordered objects.
>
> The query system currently doesn't support psets, although it should
> eventually.
>
> Slot-associations currently don't deal with reflexive references
> (associations among instances of a single class), but that would also
> address the 'friends-of' problem.


I haven't looked too much into the code, but your response seems positive
overall. I haven't read anything about slot-associations in the manual, so I
guess I'll have to learn about them in the code and the tests themselves.

I hope you enjoy elephant!  If you want to add/implement or spec out
> some of these changes you are thinking about, I'd be happy to support
> you.


I'll see if there is anything I can contribute. I first need to familiarize
myself more with it and start getting my feet wet.

Thank you again and best regards,
JD
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20090127/2d11a78d/attachment.html>

From develgenius at gmail.com  Tue Jan 27 22:06:26 2009
From: develgenius at gmail.com (John)
Date: Tue, 27 Jan 2009 17:06:26 -0500
Subject: [elephant-devel] Learning about Elephant
In-Reply-To: <D387C483-7C97-4452-A42B-91C862BC499A@media.mit.edu>
References: <e6c974240901270800l11e2b308qf0fb2561782c6f86@mail.gmail.com>
	<5D2242B8-1A86-4C89-96DD-3311A6EA2392@media.mit.edu>
	<D387C483-7C97-4452-A42B-91C862BC499A@media.mit.edu>
Message-ID: <e6c974240901271406s1cb8462ct58cb52ab320ccc2@mail.gmail.com>

Thanks for the clear example. However, maybe the idioms are confusing me,
but many-to-many tells me that I should be able to do something like:

(setf (friends-of person1) person2)
(setf (friends-of person1) person3)
(setf (friends-of person1) person4)
(setf (friends-of person1) person5)

(setf (friends-of person2) person3)

such that:

(friends-of person1) => (person2 person3 person4 person5)
(friends-of person2) => (person1 person3)
(friends-of person3) => (person1 person2)
(friends-of person4) => person1
(friends-of person5) => person1

Would that be the expected behavior?

Best regards,
JD

On Tue, Jan 27, 2009 at 3:22 PM, Ian Eslick <eslick at media.mit.edu> wrote:

> Slot associations appear to work reflexively, by the way.
>
> (defpclass self-assoc ()
>            ((friends-of :accessor friends-of
>                         :associate (self-assoc friended-me)
>                         :many-to-many t)
>             (friended-me :accessor friended-me
>                       :associate (self-assoc friends-of)
>                       :many-to-many t)))
>
> This simply says I have an association among self-assoc instances.
> Any (setf (friends-of person1) person2) says that person2 is a friend-
> of person1.  Automatically, you get the inverse mapping which is that
> person1 friended person2 (assuming these associations are directional).
>
> If they are not directional, then you can do this:
> (defpclass person ()
>   ((friends :accessor friends-of
>        :associate (person friends) :many-to-many t)))
>
> Now if you (setf (friends-of person1) person2), you will get:
>
> (friends-of person1) => person2
> (friends-of person2) => person1
>
> This is a non-directional relationship between two entities so there's
> no need for two slots.
>
> This all seems to work well in the current Alpha 2.
>
> Ian
>
> On Jan 27, 2009, at 1:32 PM, Ian Eslick wrote:
>
> >
> > On Jan 27, 2009, at 11:00 AM, John wrote:
> >>
> >> 1) Every read/write of pclass slots is done directly from/to the
> >> database, so no in-memory "copy" exists (unless some sort of
> >> transient cache slot model is used). This is good. However, is
> >> Elephant "intelligent" enough so that if you attempt to setf a slot
> >> value with the same value stored in the slot, Elephant will "avoid"
> >> the writing to the database, since the value hasn't really changed?
> >> If that's not the out-of-the-box behavior, I pressume that it would
> >> be relatively easy to add this functionality using MOP. I also
> >> assume that doing this will require another read before the write in
> >> order to compare the value to be written, but then again, Elephant
> >> claims that reads are much cheaper than writes.
> >>
> >
> > Such a thing would be possible, but you would need to have a local
> > copy of the value in memory and this could create more problems than
> > it solves.  It's also unclear to me how common this case.  I don't
> > write the same value to the same slot very often!
> >
> >> 2) According to Oracle: "Berkeley DB includes support for building
> >> highly available applications based on replication..." (
> http://www.oracle.com/technology/documentation/berkeley-db/db/ref/rep/intro.html
> >> ) So, I assume that in the event that a single machine becomes
> >> unable to handle the load of a busy application, BDB supports
> >> replication to multiple machines with near-instant replication
> >> effects (performance is inversly proportional to replication speed).
> >> So, the question is, can Elephant accommodate to this model? If I
> >> read correctly, the manual states that elephant maintains a weak
> >> hash of persistent objects. So, if BDB is deployed in a distributed
> >> model and Elephant is running in each separate machine, we could in
> >> theory "trust" that the data read/written from/to disk will be all
> >> in sync across all machines (as long as database IO on same object/
> >> slot occurs at a frequency greater than the replication rate). The
> >> question I have is with the weak hash. If a write is made in one
> >> machine, the data on disk is updated across all machines. However,
> >> the weak hash remains stale in the machines were the data was only
> >> replicated to. Is this an actual problem or does Elephant use the
> >> weak hash "intelligently" to recover in the event that the weak hash
> >> becomes "unexpectedly" stale?
> >
> > The weak hash is simply used to speed up the 'recreation' of a
> > persistent reference when it is deserialized from the underlying
> > storage medium (BDB in this case) so fits into this model.  I've
> > looked several times at building-in the support to enable a full
> > replication model for BDB, but it's a fair bit of work to deal with
> > the single-master requirement for replication and distributed
> > transactions for global coherence. The write performance and
> > contention performance may decline noticeably, but conflict-free read
> > performance should be the same as in the non-replicated case.  I don't
> > fully understand all these issues yet and am prioritizing a lisp-only
> > Prevalence style backend in my development roadmap.
> >
> >> 3) I come from a RDBMS world, so I'm still learning the modalities
> >> of connected objects vs just related rows. So, reading the tutorials
> >> you describe a friends model using PSets. So, imagine a concept
> >> similar to Facebook in terms of a friends database. I have millions
> >> of people created in the system and they all create their list of
> >> friends. Some people may have "few" or no friends while others could
> >> have hundreds of thousands of friends (e.g. Pres. Obama). Are PSets
> >> the correct way to model this for larger number of objects or is
> >> there a more appropriate methodology recommended in Elephant?
> >> Obviously the idea behind this is so that you could perform
> >> manipulations on these "friends" relatively easy, such as add/remove
> >> friends or perform global queries as to list all friends of people
> >> who are friends of Pres. Obama. There are references on the list
> >> about a query system being worked on and some vanilla version being
> >> available, but independently of that, I think my question is more
> >> related to the object model implementation. Maybe I'm wrong.
> >
> > Psets today are a convenience API around the BTree that can be
> > overridden by data stores later for more efficient implementation
> > (e.g. hash vs. tree) as appropriate for that store.  BTrees are cheap
> > for BDB to create and use so use of them at any size is fine.  You
> > might want to wrap an abstraction around the BTree directly since the
> > pset API is an un-ordered set.  If you want to do range extraction or
> > set intersection you'll want the btree's support for ordered objects.
> >
> > The query system currently doesn't support psets, although it should
> > eventually.
> >
> > Slot-associations currently don't deal with reflexive references
> > (associations among instances of a single class), but that would also
> > address the 'friends-of' problem.
> >
> > I hope you enjoy elephant!  If you want to add/implement or spec out
> > some of these changes you are thinking about, I'd be happy to support
> > you.
> >
> > Cheers,
> > Ian
> >
> >> Anyway, thank you for your help in advanced. Look forward to hearing
> >> back from anyone soon and keep learning more about Elephant.
> >>
> >> Thanks,
> >> JD
> >> _______________________________________________
> >> elephant-devel site list
> >> elephant-devel at common-lisp.net
> >> http://common-lisp.net/mailman/listinfo/elephant-devel
> >
> >
> > _______________________________________________
> > elephant-devel site list
> > elephant-devel at common-lisp.net
> > http://common-lisp.net/mailman/listinfo/elephant-devel
>
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20090127/17bed66e/attachment.html>

From franks-muc at web.de  Tue Jan 27 23:04:06 2009
From: franks-muc at web.de (Frank Schorr)
Date: Wed, 28 Jan 2009 00:04:06 +0100
Subject: [elephant-devel] elephant-1.0 on Lispworks+win32; bdb; postmodern
Message-ID: <1535991424@web.de>

installed elephant-1.0 on Lispworks+win32;
tested bdb and postmodern

1. in file gc.lisp defgeneric walk-heap defines a method on base-string.
   error: error base-string is not a class
   I deleted this method.

2. arnesi must be loaded...
   error: package lexical not found.
   I replaced all occurences in file lexenv.lisp as follows (not knowing exactly what I was doing)
   lexical::environmet --> environment::environment
   lexical::functions  --> compiler::functions
   lexical::variables  --> compiler::variables
   
   this makes fiveAM compile and load (and test itself ?)

3. test base-strings in testserializer.lisp generates this error upon asdf:load-op:
   Saw character U+0080 with syntax type invalid.
   I deleted this test



4. BDB-tests

   error in elephant-db-path
   (ARRAY NIL) is an illegal type specifier.
   arg directory comes in as a string.
   I modified the code so that (merge-pathnames directory (make-pathname :name "%ELEPHANT"))
   is executed.

   A new bdb is created in the directory

5. The condition #<ELEPHANT-DESERIALIZATION-ERROR 22141CE8> occurred

Call to DESERIALIZE (offset 362)
  ELEPHANT::BS       : #S(ELEPHANT-MEMUTIL:BUFFER-STREAM :BUFFER #<Pointer to type (:UNSIGNED :CHAR) = #x00941F78> :SIZE 45 :POSITION 2843 :LENGTH 80)
  ELEPHANT::SC       : #<BDB-STORE-CONTROLLER c:\lisp\libraries\elephant-1.0\tests\testdb\>
  ELEPHANT::OID-ONLY : NIL
  DBG::G             : #<WIN32::EXCEPTION 21667B6B>
  ELEPHANT::E        : #<WIN32::EXCEPTION 21667B6B>

Call to (METHOD ELEPHANT::PERSISTENT-SLOT-READER (DB-BDB::BDB-STORE-CONTROLLER T T)) (offset 180)

Call to CLOS::METHOD-COMBINATION-TEMPLATE (offset 45)

Call to (METHOD CLOS::DYNAMIC-SLOT-VALUE-USING-SLOTD (T T)) (offset 120)
  CLOS::INSTANCE : #<BDB-INDEXED-BTREE oid:-3>
  CLOS::SLOTD    : #<ELEPHANT::PERSISTENT-EFFECTIVE-SLOT-DEFINITION DB-BDB::INDICES 2226987F>

Call to (METHOD SHARED-INITIALIZE :AFTER (DB-BDB::BDB-INDEXED-BTREE T)) (offset 14)

Call to CLOS::METHOD-COMBINATION-TEMPLATE (offset 152)

Call to (HARLEQUIN-COMMON-LISP:SUBFUNCTION CALL-NEXT-METHOD (METHOD SHARED-INITIALIZE :AROUND (PERSISTENT-OBJECT T))) (offset 201)

Call to (METHOD SHARED-INITIALIZE :AROUND (PERSISTENT-OBJECT T)) (offset 499)
  ELEPHANT::INSTANCE                       : #<BDB-INDEXED-BTREE oid:-3>
  ELEPHANT::SLOT-NAMES                     : T
  ELEPHANT::INITARGS                       : (:FROM-OID -3 :SC #<BDB-STORE-CONTROLLER c:\lisp\libraries\elephant-1.0\tests\testdb\> :INDICES #<EQL Hash Table{0} 21666587>)
  ELEPHANT::FROM-OID                       : -3
  CLOS::.ISL.                              : :DONT-KNOW
  DBG::G                                   : (#<Closure CLOS::METHOD-COMBINATION-TEMPLATE 2165B392>)
  DBG::G                                   : #<PERSISTENT-METACLASS DB-BDB::BDB-INDEXED-BTREE 222217A7>
  ELEPHANT::TRANSIENT-SLOTS                : (ELEPHANT::OID ELEPHANT::SPEC DB-BDB::INDICES-CACHE)
  ELEPHANT::CACHED-SLOTS                   : NIL
  ELEPHANT::INDEXED-SLOTS                  : NIL
  ELEPHANT::DERIVED-SLOTS                  : NIL
  ELEPHANT::ASSOCIATION-END-SLOTS          : NIL
  ELEPHANT::PERSISTENT-SLOTS               : (DB-BDB::INDICES)
  CLASS                                    : #<PERSISTENT-METACLASS DB-BDB::BDB-INDEXED-BTREE 222217A7>
  ELEPHANT::PERSISTENT-INITIALIZABLE-SLOTS : (DB-BDB::INDICES)
  ELEPHANT::SET-SLOTS                      : NIL

Call to CLOS::METHOD-COMBINATION-TEMPLATE (offset 45)

Call to CLOS::METHOD-COMBINATION-TEMPLATE (offset 79)

Call to CLOS::NEXT-METHOD-CALL-2 (offset 87)
  CONS               : (METHOD INITIALIZE-INSTANCE :AROUND (PERSISTENT-OBJECT))
  CLOS::NEXT-METHODS : (#<Closure CLOS::METHOD-COMBINATION-TEMPLATE 2165B3C2>)
  #:G174431          : #<BDB-INDEXED-BTREE oid:-3>
  #:G174432          : (:FROM-OID -3 :SC #<BDB-STORE-CONTROLLER c:\lisp\libraries\elephant-1.0\tests\testdb\> :INDICES #<EQL Hash Table{0} 21666587>)

Call to CLOS::METHOD-COMBINATION-TEMPLATE (offset 42)

Call to CLOS::MAKE-INSTANCE-FROM-CLASS-1 (offset 452)
  CLASS          : #<PERSISTENT-METACLASS DB-BDB::BDB-INDEXED-BTREE 222217A7>
  CLOS::INITARGS : (:FROM-OID -3 :SC #<BDB-STORE-CONTROLLER c:\lisp\libraries\elephant-1.0\tests\testdb\> :INDICES #<EQL Hash Table{0} 21666587>)

Call to (HARLEQUIN-COMMON-LISP:SUBFUNCTION 2 (METHOD ELEPHANT::OPEN-CONTROLLER (DB-BDB::BDB-STORE-CONTROLLER))) (offset 175)

Call to (METHOD ELEPHANT::EXECUTE-TRANSACTION (DB-BDB::BDB-STORE-CONTROLLER T)) (offset 741)
  DB-BDB::SC                      : #<BDB-STORE-CONTROLLER c:\lisp\libraries\elephant-1.0\tests\testdb\>
  DB-BDB::TXN-FN                  : #<Closure ((METHOD ELEPHANT::OPEN-CONTROLLER (DB-BDB::BDB-STORE-CONTROLLER)) . 2) 2165B1FA>
  DBG::G                          : :DONT-KNOW
  DB-BDB::TRANSACTION             : :DONT-KNOW
  DB-BDB::PARENT                  : NIL
  DB-BDB::ENVIRONMENT             : NIL
  DB-BDB::RETRIES                 : 200
  DB-BDB::DEGREE-2                : NIL
  DB-BDB::READ-UNCOMMITTED        : NIL
  DB-BDB::TXN-NOSYNC              : NIL
  DB-BDB::TXN-NOWAIT              : NIL
  DB-BDB::TXN-SYNC                : NIL
  DB-BDB::SNAPSHOT                : NIL
  DB-BDB::INHIBIT-ROLLBACK-FN     : NIL
  CLOS::.ISL.                     : :DONT-KNOW
  DB-BDB::ENV                     : #<Pointer to type :VOID = #x016C4500>
  COUNT                           : 0
  DB-BDB::SUCCESS                 : NIL
  DB-BDB::TXN                     : #<Pointer to type :VOID = #x016C6C90>
  DB-BDB::RESULT                  : NIL
  ELEPHANT::*CURRENT-TRANSACTION* : (#<BDB-STORE-CONTROLLER c:\lisp\libraries\elephant-1.0\tests\testdb\> #<Pointer to type :VOID = #x016C6C90> NIL)
  *STORE-CONTROLLER*              : #<BDB-STORE-CONTROLLER c:\lisp\libraries\elephant-1.0\tests\testdb\>
  DBG::G                          : #<Closure ((METHOD ELEPHANT::EXECUTE-TRANSACTION (DB-BDB::BDB-STORE-CONTROLLER T)) . 2) 2187D8B2>
  CONDITIONS::*HANDLER-CLUSTERS*  : (((CONDITION . #<Closure # 2187D8B2>)) ((WARNING . EDITOR::EDITOR-EVALUATION-WARNING)) ((END-OF-FILE . #<Closure # 2216AB22>) (READER-ERROR . #<Closure # 2187DD82>)))

Call to (METHOD ELEPHANT::OPEN-CONTROLLER (DB-BDB::BDB-STORE-CONTROLLER)) (offset 2211)

Call to CLOS::METHOD-COMBINATION-TEMPLATE (offset 62)

Call to OPEN-STORE (offset 66)

Call to DO-BACKEND-TESTS (offset 106)
  SPEC               : (:BDB "c:\\lisp\\libraries\\elephant-1.0\\tests\\testdb\\")
  *STORE-CONTROLLER* : NIL

Call to EVAL (offset 113)
  EXP : (DO-BACKEND-TESTS)



6. turning now to postmodern:


The condition #<ELEPHANT-DESERIALIZATION-ERROR 2009F808> occurred
  1 (continue) retry db-postmodern transaction
  2 DB-POSTMODERN::ABORT-TRANSACTION
  3 (abort) Abort job 8 :(BIND-STANDARD-STREAMS-AND-EXECUTE #<EDITOR::EDITOR-OUTPUT-STREAM #<EDITOR:BUFFER Background Output> 2190405F> (FUNCALL-BACKGROUND-JOB-AUX #<CAPI:EDITOR-PANE CAPI:EDITOR-PANE  2198348B> BACKGROUND-REGION-EVAL (# # # # T NIL NIL)))

Type :b for backtrace, :c <option number> to proceed,  or :? for other options

ELE-TESTS 1 : 1 > :b :verbose
Call to DESERIALIZE (offset 362)
  ELEPHANT::BS       : #S(ELEPHANT-MEMUTIL:BUFFER-STREAM :BUFFER #<Pointer to type (:UNSIGNED :CHAR) = #x0039FBD8> :SIZE 45 :POSITION 2844 :LENGTH 80)
  ELEPHANT::SC       : #<POSTMODERN-STORE-CONTROLLER (POSTGRESQL 127.0.0.1 elepm postgres )>
  ELEPHANT::OID-ONLY : NIL
  DBG::G             : #<CONDITIONS::COERCE-TO-CONDITION-ERROR 2009C9F3>
  ELEPHANT::E        : #<CONDITIONS::COERCE-TO-CONDITION-ERROR 2009C9F3>

Call to DB-POSTMODERN::DESERIALIZE-FROM-DATABASE (offset 63)
  DB-POSTMODERN::X         : #(17 0 0 0 0 13 9 3 0 0 0 69 81 76 9 11 0 0 0 67 79 77 77 79 78 45 76 73 83 80 4 0 0 0 64 4 0 0 128 63 ...)
  DB-POSTMODERN::SC        : #<POSTMODERN-STORE-CONTROLLER (POSTGRESQL 127.0.0.1 elepm postgres )>
  DB-POSTMODERN::OIDS-ONLY : NIL
  DB-POSTMODERN::OTHER     : #S(ELEPHANT-MEMUTIL:BUFFER-STREAM :BUFFER #<Pointer to type (:UNSIGNED :CHAR) = #x0039FBD8> :SIZE 45 :POSITION 2844 :LENGTH 80)

Call to (METHOD DB-POSTMODERN::INTERNAL-GET-VALUE (T DB-POSTMODERN::PM-BTREE)) (offset 369)
  DB-POSTMODERN::KEY          : "3 INDICES"
  DB-POSTMODERN::BT           : #<DB-POSTMODERN::PM-BTREE db-table:slots 217BB097>
  CLOS::.ISL.                 : :DONT-KNOW
  DB-POSTMODERN::VALUE        : NIL
  DB-POSTMODERN::EXISTS-P     : NIL
  DB-POSTMODERN::*SC*         : #<POSTMODERN-STORE-CONTROLLER (POSTGRESQL 127.0.0.1 elepm postgres )>
  DB-POSTMODERN::*CONNECTION* : #<CL-POSTGRES:DATABASE-CONNECTION 217BE883>
  DB-POSTMODERN::RESULT       : #(17 0 0 0 0 13 9 3 0 0 0 69 81 76 9 11 0 0 0 67 79 77 77 79 78 45 76 73 83 80 4 0 0 0 64 4 0 0 128 63 ...)

Call to (METHOD GET-VALUE (T DB-POSTMODERN::PM-BTREE)) (offset 20)
  DB-POSTMODERN::KEY : "3 INDICES"
  DB-POSTMODERN::BT  : #<DB-POSTMODERN::PM-BTREE db-table:slots 217BB097>
  CLOS::.ISL.        : :DONT-KNOW

Call to (METHOD ELEPHANT::PERSISTENT-SLOT-READER (DB-POSTMODERN::POSTMODERN-STORE-CONTROLLER T T)) (offset 78)
  DB-POSTMODERN::SC        : #<POSTMODERN-STORE-CONTROLLER (POSTGRESQL 127.0.0.1 elepm postgres )>
  DB-POSTMODERN::INSTANCE  : #<DB-POSTMODERN::PM-INDEXED-BTREE db-table:schemas 217F8A77>
  DB-POSTMODERN::NAME      : DB-POSTMODERN::INDICES
  DB-POSTMODERN::OIDS-ONLY : :DONT-KNOW
  CLOS::.ISL.              : :DONT-KNOW
  DB-POSTMODERN::*SC*      : #<POSTMODERN-STORE-CONTROLLER (POSTGRESQL 127.0.0.1 elepm postgres )>

Call to CLOS::METHOD-COMBINATION-TEMPLATE (offset 45)

Call to (METHOD CLOS::DYNAMIC-SLOT-VALUE-USING-SLOTD (T T)) (offset 120)
  CLOS::INSTANCE : #<DB-POSTMODERN::PM-INDEXED-BTREE db-table:schemas 217F8A77>
  CLOS::SLOTD    : #<ELEPHANT::PERSISTENT-EFFECTIVE-SLOT-DEFINITION DB-POSTMODERN::INDICES 22346B0B>

Call to (METHOD GET-INDEX (DB-POSTMODERN::PM-INDEXED-BTREE T)) (offset 16)
  DB-POSTMODERN::BT         : #<DB-POSTMODERN::PM-INDEXED-BTREE db-table:schemas 217F8A77>
  DB-POSTMODERN::INDEX-NAME : ELEPHANT::BY-NAME
  CLOS::.ISL.               : :DONT-KNOW

Call to (METHOD ELEPHANT::ENSURE-INDEX (INDEXED-BTREE T)) (offset 27)

Call to (HARLEQUIN-COMMON-LISP:SUBFUNCTION 2 (METHOD ELEPHANT::OPEN-CONTROLLER :AFTER (STORE-CONTROLLER))) (offset 65)

Call to DB-POSTMODERN::EXECUTE-TRANSACTION-ONE-TRY (offset 169)
  DB-POSTMODERN::SC                : #<POSTMODERN-STORE-CONTROLLER (POSTGRESQL 127.0.0.1 elepm postgres )>
  DB-POSTMODERN::TXN-FN            : #<Closure ((METHOD ELEPHANT::OPEN-CONTROLLER :AFTER (STORE-CONTROLLER)) . 2) 217F216A>
  DB-POSTMODERN::ALWAYS-ROLLBACK   : NIL
  DB-POSTMODERN::TRAN              : #<POSTMODERN::TRANSACTION-HANDLE 217F70BB>
  DB-POSTMODERN::COMMITED          : NIL
  DB-POSTMODERN::*TXN-VALUE-CACHE* : NIL

Call to (METHOD ELEPHANT::EXECUTE-TRANSACTION (DB-POSTMODERN::POSTMODERN-STORE-CONTROLLER T)) (offset 889)
  DB-POSTMODERN::SC               : #<POSTMODERN-STORE-CONTROLLER (POSTGRESQL 127.0.0.1 elepm postgres )>
  DB-POSTMODERN::TXN-FN           : #<Closure ((METHOD ELEPHANT::OPEN-CONTROLLER :AFTER (STORE-CONTROLLER)) . 2) 217F216A>
  DBG::G                          : :DONT-KNOW
  DB-POSTMODERN::ALWAYS-ROLLBACK  : NIL
  DB-POSTMODERN::RETRY-CLEANUP-FN : NIL
  DB-POSTMODERN::RETRIES          : 200
  CLOS::.ISL.                     : :DONT-KNOW
  POSTMODERN:*DATABASE*           : #<CL-POSTGRES:DATABASE-CONNECTION 217BE883>
  DB-POSTMODERN::TRY              : 200
  DBG::G                          : #<Closure ((METHOD ELEPHANT::EXECUTE-TRANSACTION (DB-POSTMODERN::POSTMODERN-STORE-CONTROLLER T)) . 5) 217F20AA>
  DBG::G                          : #<Function ((METHOD ELEPHANT::EXECUTE-TRANSACTION (DB-POSTMODERN::POSTMODERN-STORE-CONTROLLER T)) . 4) 222D653A>
  DBG::G                          : #<Closure ((METHOD ELEPHANT::EXECUTE-TRANSACTION (DB-POSTMODERN::POSTMODERN-STORE-CONTROLLER T)) . 3) 217F20C2>
  CONDITIONS::*RESTART-CLUSTERS*  : ((#<RESTART ABORT 2188E04F>) (#<RESTART ABORT 2188E2B7>) (#<RESTART DB-POSTMODERN::RETRY-TRANSACTION 2009F7AB> #<RESTART DB-POSTMODERN::ABORT-TRANSACTION 2009F77B>) (#<RESTART ABORT 2009F73F>) (#<RESTART ABORT 21E43D1F>))
  DBG::G                          : #<Function ((METHOD ELEPHANT::EXECUTE-TRANSACTION (DB-POSTMODERN::POSTMODERN-STORE-CONTROLLER T)) . 8) 222D6E7A>
  CONDITIONS::*HANDLER-CLUSTERS*  : (((CL-POSTGRES:DATABASE-ERROR . #<Function # 222D6E7A>)) ((WARNING . EDITOR::EDITOR-EVALUATION-WARNING)) ((END-OF-FILE . #<Closure # 217F2062>) (READER-ERROR . #<Closure # 2188F032>)))

Call to CLOS::METHOD-COMBINATION-TEMPLATE (offset 146)

Call to CLOS::NEXT-METHOD-CALL-2 (offset 87)
  CONS               : (METHOD ELEPHANT::OPEN-CONTROLLER :AROUND (DB-POSTMODERN::POSTMODERN-STORE-CONTROLLER))
  CLOS::NEXT-METHODS : (#<Closure CLOS::METHOD-COMBINATION-TEMPLATE 217F14C2>)
  #:G174431          : #<POSTMODERN-STORE-CONTROLLER (POSTGRESQL 127.0.0.1 elepm postgres )>
  #:G174432          : NIL

Call to (METHOD ELEPHANT::OPEN-CONTROLLER :AROUND (DB-POSTMODERN::POSTMODERN-STORE-CONTROLLER)) (offset 61)
  DB-POSTMODERN::SC          : #<POSTMODERN-STORE-CONTROLLER (POSTGRESQL 127.0.0.1 elepm postgres )>
  DBG::G                     : NIL
  CLOS::.ISL.                : :DONT-KNOW
  DBG::G                     : (#<Closure CLOS::METHOD-COMBINATION-TEMPLATE 217F14C2>)
  DB-POSTMODERN:*CACHE-MODE* : NIL

Call to CLOS::METHOD-COMBINATION-TEMPLATE (offset 42)

Call to OPEN-STORE (offset 66)

Call to DO-BACKEND-TESTS (offset 106)
  SPEC               : (:POSTMODERN (:POSTGRESQL "127.0.0.1" "elepm" "postgres" ""))
  *STORE-CONTROLLER* : NIL

Call to EVAL (offset 113)
  EXP : (DO-BACKEND-TESTS)

Call to EDITOR::EDITOR-EVAL (offset 177)
  EDITOR:BUFFER  : :DONT-KNOW
  EDITOR::SEXP   : (DO-BACKEND-TESTS)
  EDITOR::ERRORP : T

eof
__________________________________________________________________
Deutschlands gr??te Online-Videothek schenkt Ihnen 12.000 Videos!*
http://entertainment.web.de/de/entertainment/maxdome/index.html




From eslick at media.mit.edu  Wed Jan 28 05:26:55 2009
From: eslick at media.mit.edu (Ian Eslick)
Date: Wed, 28 Jan 2009 00:26:55 -0500
Subject: [elephant-devel] Learning about Elephant
In-Reply-To: <e6c974240901271159w7ec7bb37v3dfc1098e7991021@mail.gmail.com>
References: <e6c974240901270800l11e2b308qf0fb2561782c6f86@mail.gmail.com>
	<5D2242B8-1A86-4C89-96DD-3311A6EA2392@media.mit.edu>
	<e6c974240901271159w7ec7bb37v3dfc1098e7991021@mail.gmail.com>
Message-ID: <F87217F2-26CB-44B6-97B5-54047EBF7876@media.mit.edu>

>
Hi John,

Your mileage may vary, but I'm really not concerned about users  
causing significant performance degradation by doing lots of null  
edits.  If that is happening, it's probably a user interface design  
problem.  The trivial solution is to write a jscript function that  
sets a hidden input value if the contents of any of the inputs are  
changed.  You can inhibit the POST entirely if that's the case, or  
allow the server to skip processing it.  This should cut out a huge %  
of your overall null edits (if you don't have some weird community  
with a high percentage of no-javascript users) and it's only a few  
lines of code.

> I'm not too familiar with BDB replication myself and was just  
> googling around. After reading that document in more detail and your  
> explanation, makes me realize that replication or the distributed  
> BDB model is not such a simple task. Since I don't think that my  
> application(s) would really out-scale the single machine Elephant/ 
> BDB deployment, I would certainly not want to encourage investing  
> time in this and rather concentrate on a lisp-only prevalence   
> backend. Maybe one that covers these issues in the future.

We should all be so lucky as to blow out a single server!  If you have  
that much traffic, you can usually afford some reworking to scale if  
your application is amenable to it.  Often you can get pretty far just  
by doing some optimizations, offloading static file to a second  
server, smart caching, some restructuring and performance  
optimizations, transaction tuning, etc.  There is almost always room  
for significant optimization in a new system.

>
> I haven't looked too much into the code, but your response seems  
> positive overall. I haven't read anything about slot-associations in  
> the manual, so I guess I'll have to learn about them in the code and  
> the tests themselves.
>

Check out the elephant-devel mailing list, associations are described  
several times there.  I also provided an example in a recent e-mail.

The API is straightforward so between the new metaclass slot initargs  
(src/elephant/metaclass.lisp) and the association interface (src/ 
elephant/associations.lisp) you should be able to work with them.

Regards,
Ian



From eslick at media.mit.edu  Wed Jan 28 05:32:38 2009
From: eslick at media.mit.edu (Ian Eslick)
Date: Wed, 28 Jan 2009 00:32:38 -0500
Subject: [elephant-devel] Learning about Elephant
In-Reply-To: <e6c974240901271406s1cb8462ct58cb52ab320ccc2@mail.gmail.com>
References: <e6c974240901270800l11e2b308qf0fb2561782c6f86@mail.gmail.com>
	<5D2242B8-1A86-4C89-96DD-3311A6EA2392@media.mit.edu>
	<D387C483-7C97-4452-A42B-91C862BC499A@media.mit.edu>
	<e6c974240901271406s1cb8462ct58cb52ab320ccc2@mail.gmail.com>
Message-ID: <A25DD755-A080-41F0-8F56-61B2DEBC54D6@media.mit.edu>

That would be correct if you added (setf (friends-of person3) person2)

On Jan 27, 2009, at 5:06 PM, John wrote:

> Thanks for the clear example. However, maybe the idioms are  
> confusing me, but many-to-many tells me that I should be able to do  
> something like:
>
> (setf (friends-of person1) person2)
> (setf (friends-of person1) person3)
> (setf (friends-of person1) person4)
> (setf (friends-of person1) person5)
>
> (setf (friends-of person2) person3)
>
> such that:
>
> (friends-of person1) => (person2 person3 person4 person5)
> (friends-of person2) => (person1 person3)
> (friends-of person3) => (person1 person2)
> (friends-of person4) => person1
> (friends-of person5) => person1
>
> Would that be the expected behavior?
>
> Best regards,
> JD
>
> On Tue, Jan 27, 2009 at 3:22 PM, Ian Eslick <eslick at media.mit.edu>  
> wrote:
> Slot associations appear to work reflexively, by the way.
>
> (defpclass self-assoc ()
>            ((friends-of :accessor friends-of
>                         :associate (self-assoc friended-me)
>                         :many-to-many t)
>             (friended-me :accessor friended-me
>                       :associate (self-assoc friends-of)
>                       :many-to-many t)))
>
> This simply says I have an association among self-assoc instances.
> Any (setf (friends-of person1) person2) says that person2 is a friend-
> of person1.  Automatically, you get the inverse mapping which is that
> person1 friended person2 (assuming these associations are  
> directional).
>
> If they are not directional, then you can do this:
> (defpclass person ()
>   ((friends :accessor friends-of
>        :associate (person friends) :many-to-many t)))
>
> Now if you (setf (friends-of person1) person2), you will get:
>
> (friends-of person1) => person2
> (friends-of person2) => person1
>
> This is a non-directional relationship between two entities so there's
> no need for two slots.
>
> This all seems to work well in the current Alpha 2.
>
> Ian
>
> On Jan 27, 2009, at 1:32 PM, Ian Eslick wrote:
>
> >
> > On Jan 27, 2009, at 11:00 AM, John wrote:
> >>
> >> 1) Every read/write of pclass slots is done directly from/to the
> >> database, so no in-memory "copy" exists (unless some sort of
> >> transient cache slot model is used). This is good. However, is
> >> Elephant "intelligent" enough so that if you attempt to setf a slot
> >> value with the same value stored in the slot, Elephant will "avoid"
> >> the writing to the database, since the value hasn't really changed?
> >> If that's not the out-of-the-box behavior, I pressume that it would
> >> be relatively easy to add this functionality using MOP. I also
> >> assume that doing this will require another read before the write  
> in
> >> order to compare the value to be written, but then again, Elephant
> >> claims that reads are much cheaper than writes.
> >>
> >
> > Such a thing would be possible, but you would need to have a local
> > copy of the value in memory and this could create more problems than
> > it solves.  It's also unclear to me how common this case.  I don't
> > write the same value to the same slot very often!
> >
> >> 2) According to Oracle: "Berkeley DB includes support for building
> >> highly available applications based on replication..." (http://www.oracle.com/technology/documentation/berkeley-db/db/ref/rep/intro.html
> >> ) So, I assume that in the event that a single machine becomes
> >> unable to handle the load of a busy application, BDB supports
> >> replication to multiple machines with near-instant replication
> >> effects (performance is inversly proportional to replication  
> speed).
> >> So, the question is, can Elephant accommodate to this model? If I
> >> read correctly, the manual states that elephant maintains a weak
> >> hash of persistent objects. So, if BDB is deployed in a distributed
> >> model and Elephant is running in each separate machine, we could in
> >> theory "trust" that the data read/written from/to disk will be all
> >> in sync across all machines (as long as database IO on same object/
> >> slot occurs at a frequency greater than the replication rate). The
> >> question I have is with the weak hash. If a write is made in one
> >> machine, the data on disk is updated across all machines. However,
> >> the weak hash remains stale in the machines were the data was only
> >> replicated to. Is this an actual problem or does Elephant use the
> >> weak hash "intelligently" to recover in the event that the weak  
> hash
> >> becomes "unexpectedly" stale?
> >
> > The weak hash is simply used to speed up the 'recreation' of a
> > persistent reference when it is deserialized from the underlying
> > storage medium (BDB in this case) so fits into this model.  I've
> > looked several times at building-in the support to enable a full
> > replication model for BDB, but it's a fair bit of work to deal with
> > the single-master requirement for replication and distributed
> > transactions for global coherence. The write performance and
> > contention performance may decline noticeably, but conflict-free  
> read
> > performance should be the same as in the non-replicated case.  I  
> don't
> > fully understand all these issues yet and am prioritizing a lisp- 
> only
> > Prevalence style backend in my development roadmap.
> >
> >> 3) I come from a RDBMS world, so I'm still learning the modalities
> >> of connected objects vs just related rows. So, reading the  
> tutorials
> >> you describe a friends model using PSets. So, imagine a concept
> >> similar to Facebook in terms of a friends database. I have millions
> >> of people created in the system and they all create their list of
> >> friends. Some people may have "few" or no friends while others  
> could
> >> have hundreds of thousands of friends (e.g. Pres. Obama). Are PSets
> >> the correct way to model this for larger number of objects or is
> >> there a more appropriate methodology recommended in Elephant?
> >> Obviously the idea behind this is so that you could perform
> >> manipulations on these "friends" relatively easy, such as add/ 
> remove
> >> friends or perform global queries as to list all friends of people
> >> who are friends of Pres. Obama. There are references on the list
> >> about a query system being worked on and some vanilla version being
> >> available, but independently of that, I think my question is more
> >> related to the object model implementation. Maybe I'm wrong.
> >
> > Psets today are a convenience API around the BTree that can be
> > overridden by data stores later for more efficient implementation
> > (e.g. hash vs. tree) as appropriate for that store.  BTrees are  
> cheap
> > for BDB to create and use so use of them at any size is fine.  You
> > might want to wrap an abstraction around the BTree directly since  
> the
> > pset API is an un-ordered set.  If you want to do range extraction  
> or
> > set intersection you'll want the btree's support for ordered  
> objects.
> >
> > The query system currently doesn't support psets, although it should
> > eventually.
> >
> > Slot-associations currently don't deal with reflexive references
> > (associations among instances of a single class), but that would  
> also
> > address the 'friends-of' problem.
> >
> > I hope you enjoy elephant!  If you want to add/implement or spec out
> > some of these changes you are thinking about, I'd be happy to  
> support
> > you.
> >
> > Cheers,
> > Ian
> >
> >> Anyway, thank you for your help in advanced. Look forward to  
> hearing
> >> back from anyone soon and keep learning more about Elephant.
> >>
> >> Thanks,
> >> JD
> >> _______________________________________________
> >> elephant-devel site list
> >> elephant-devel at common-lisp.net
> >> http://common-lisp.net/mailman/listinfo/elephant-devel
> >
> >
> > _______________________________________________
> > elephant-devel site list
> > elephant-devel at common-lisp.net
> > http://common-lisp.net/mailman/listinfo/elephant-devel
>
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel




From eslick at media.mit.edu  Wed Jan 28 05:37:49 2009
From: eslick at media.mit.edu (Ian Eslick)
Date: Wed, 28 Jan 2009 00:37:49 -0500
Subject: [elephant-devel] elephant-1.0 on Lispworks+win32; bdb;
	postmodern
In-Reply-To: <1535991424@web.de>
References: <1535991424@web.de>
Message-ID: <476E724B-DBA4-4ED7-978E-9EFB7B1B8E4B@media.mit.edu>

Frank,

Thank you!  This is great feedback.

I was seeing a deserialization error recently as well and believed I  
had fixed it.  I made some optimizations to the serializer  
implementation (src/memutil) that may cause problems for Lispworks but  
were fixed for others like SBCL, Clozure, etc.

Did this happen on the first open-controller or somewhere in the  
middle of the tests?

You should post the fiveAM bug, if that is indeed what it is, to the  
appropriate list.

I'll try to get to this in the next few days.

Ian


On Jan 27, 2009, at 6:04 PM, Frank Schorr wrote:

> installed elephant-1.0 on Lispworks+win32;
> tested bdb and postmodern
>
> 1. in file gc.lisp defgeneric walk-heap defines a method on base- 
> string.
>   error: error base-string is not a class
>   I deleted this method.
>
> 2. arnesi must be loaded...
>   error: package lexical not found.
>   I replaced all occurences in file lexenv.lisp as follows (not  
> knowing exactly what I was doing)
>   lexical::environmet --> environment::environment
>   lexical::functions  --> compiler::functions
>   lexical::variables  --> compiler::variables
>
>   this makes fiveAM compile and load (and test itself ?)
>
> 3. test base-strings in testserializer.lisp generates this error  
> upon asdf:load-op:
>   Saw character U+0080 with syntax type invalid.
>   I deleted this test
>
>
>
> 4. BDB-tests
>
>   error in elephant-db-path
>   (ARRAY NIL) is an illegal type specifier.
>   arg directory comes in as a string.
>   I modified the code so that (merge-pathnames directory (make- 
> pathname :name "%ELEPHANT"))
>   is executed.
>
>   A new bdb is created in the directory
>
> 5. The condition #<ELEPHANT-DESERIALIZATION-ERROR 22141CE8> occurred
>
> Call to DESERIALIZE (offset 362)
>  ELEPHANT::BS       : #S(ELEPHANT-MEMUTIL:BUFFER-STREAM :BUFFER  
> #<Pointer to type (:UNSIGNED :CHAR) = #x00941F78> :SIZE 45 :POSITION  
> 2843 :LENGTH 80)
>  ELEPHANT::SC       : #<BDB-STORE-CONTROLLER c:\lisp\libraries 
> \elephant-1.0\tests\testdb\>
>  ELEPHANT::OID-ONLY : NIL
>  DBG::G             : #<WIN32::EXCEPTION 21667B6B>
>  ELEPHANT::E        : #<WIN32::EXCEPTION 21667B6B>
>
> Call to (METHOD ELEPHANT::PERSISTENT-SLOT-READER (DB-BDB::BDB-STORE- 
> CONTROLLER T T)) (offset 180)
>
> Call to CLOS::METHOD-COMBINATION-TEMPLATE (offset 45)
>
> Call to (METHOD CLOS::DYNAMIC-SLOT-VALUE-USING-SLOTD (T T)) (offset  
> 120)
>  CLOS::INSTANCE : #<BDB-INDEXED-BTREE oid:-3>
>  CLOS::SLOTD    : #<ELEPHANT::PERSISTENT-EFFECTIVE-SLOT-DEFINITION  
> DB-BDB::INDICES 2226987F>
>
> Call to (METHOD SHARED-INITIALIZE :AFTER (DB-BDB::BDB-INDEXED-BTREE  
> T)) (offset 14)
>
> Call to CLOS::METHOD-COMBINATION-TEMPLATE (offset 152)
>
> Call to (HARLEQUIN-COMMON-LISP:SUBFUNCTION CALL-NEXT-METHOD (METHOD  
> SHARED-INITIALIZE :AROUND (PERSISTENT-OBJECT T))) (offset 201)
>
> Call to (METHOD SHARED-INITIALIZE :AROUND (PERSISTENT-OBJECT T))  
> (offset 499)
>  ELEPHANT::INSTANCE                       : #<BDB-INDEXED-BTREE  
> oid:-3>
>  ELEPHANT::SLOT-NAMES                     : T
>  ELEPHANT::INITARGS                       : (:FROM-OID -3 :SC #<BDB- 
> STORE-CONTROLLER c:\lisp\libraries\elephant-1.0\tests\testdb 
> \> :INDICES #<EQL Hash Table{0} 21666587>)
>  ELEPHANT::FROM-OID                       : -3
>  CLOS::.ISL.                              : :DONT-KNOW
>  DBG::G                                   : (#<Closure CLOS::METHOD- 
> COMBINATION-TEMPLATE 2165B392>)
>  DBG::G                                   : #<PERSISTENT-METACLASS  
> DB-BDB::BDB-INDEXED-BTREE 222217A7>
>  ELEPHANT::TRANSIENT-SLOTS                : (ELEPHANT::OID  
> ELEPHANT::SPEC DB-BDB::INDICES-CACHE)
>  ELEPHANT::CACHED-SLOTS                   : NIL
>  ELEPHANT::INDEXED-SLOTS                  : NIL
>  ELEPHANT::DERIVED-SLOTS                  : NIL
>  ELEPHANT::ASSOCIATION-END-SLOTS          : NIL
>  ELEPHANT::PERSISTENT-SLOTS               : (DB-BDB::INDICES)
>  CLASS                                    : #<PERSISTENT-METACLASS  
> DB-BDB::BDB-INDEXED-BTREE 222217A7>
>  ELEPHANT::PERSISTENT-INITIALIZABLE-SLOTS : (DB-BDB::INDICES)
>  ELEPHANT::SET-SLOTS                      : NIL
>
> Call to CLOS::METHOD-COMBINATION-TEMPLATE (offset 45)
>
> Call to CLOS::METHOD-COMBINATION-TEMPLATE (offset 79)
>
> Call to CLOS::NEXT-METHOD-CALL-2 (offset 87)
>  CONS               : (METHOD INITIALIZE-INSTANCE :AROUND  
> (PERSISTENT-OBJECT))
>  CLOS::NEXT-METHODS : (#<Closure CLOS::METHOD-COMBINATION-TEMPLATE  
> 2165B3C2>)
>  #:G174431          : #<BDB-INDEXED-BTREE oid:-3>
>  #:G174432          : (:FROM-OID -3 :SC #<BDB-STORE-CONTROLLER c: 
> \lisp\libraries\elephant-1.0\tests\testdb\> :INDICES #<EQL Hash  
> Table{0} 21666587>)
>
> Call to CLOS::METHOD-COMBINATION-TEMPLATE (offset 42)
>
> Call to CLOS::MAKE-INSTANCE-FROM-CLASS-1 (offset 452)
>  CLASS          : #<PERSISTENT-METACLASS DB-BDB::BDB-INDEXED-BTREE  
> 222217A7>
>  CLOS::INITARGS : (:FROM-OID -3 :SC #<BDB-STORE-CONTROLLER c:\lisp 
> \libraries\elephant-1.0\tests\testdb\> :INDICES #<EQL Hash Table{0}  
> 21666587>)
>
> Call to (HARLEQUIN-COMMON-LISP:SUBFUNCTION 2 (METHOD ELEPHANT::OPEN- 
> CONTROLLER (DB-BDB::BDB-STORE-CONTROLLER))) (offset 175)
>
> Call to (METHOD ELEPHANT::EXECUTE-TRANSACTION (DB-BDB::BDB-STORE- 
> CONTROLLER T)) (offset 741)
>  DB-BDB::SC                      : #<BDB-STORE-CONTROLLER c:\lisp 
> \libraries\elephant-1.0\tests\testdb\>
>  DB-BDB::TXN-FN                  : #<Closure ((METHOD ELEPHANT::OPEN- 
> CONTROLLER (DB-BDB::BDB-STORE-CONTROLLER)) . 2) 2165B1FA>
>  DBG::G                          : :DONT-KNOW
>  DB-BDB::TRANSACTION             : :DONT-KNOW
>  DB-BDB::PARENT                  : NIL
>  DB-BDB::ENVIRONMENT             : NIL
>  DB-BDB::RETRIES                 : 200
>  DB-BDB::DEGREE-2                : NIL
>  DB-BDB::READ-UNCOMMITTED        : NIL
>  DB-BDB::TXN-NOSYNC              : NIL
>  DB-BDB::TXN-NOWAIT              : NIL
>  DB-BDB::TXN-SYNC                : NIL
>  DB-BDB::SNAPSHOT                : NIL
>  DB-BDB::INHIBIT-ROLLBACK-FN     : NIL
>  CLOS::.ISL.                     : :DONT-KNOW
>  DB-BDB::ENV                     : #<Pointer to type :VOID =  
> #x016C4500>
>  COUNT                           : 0
>  DB-BDB::SUCCESS                 : NIL
>  DB-BDB::TXN                     : #<Pointer to type :VOID =  
> #x016C6C90>
>  DB-BDB::RESULT                  : NIL
>  ELEPHANT::*CURRENT-TRANSACTION* : (#<BDB-STORE-CONTROLLER c:\lisp 
> \libraries\elephant-1.0\tests\testdb\> #<Pointer to type :VOID =  
> #x016C6C90> NIL)
>  *STORE-CONTROLLER*              : #<BDB-STORE-CONTROLLER c:\lisp 
> \libraries\elephant-1.0\tests\testdb\>
>  DBG::G                          : #<Closure ((METHOD  
> ELEPHANT::EXECUTE-TRANSACTION (DB-BDB::BDB-STORE-CONTROLLER T)) . 2)  
> 2187D8B2>
>  CONDITIONS::*HANDLER-CLUSTERS*  : (((CONDITION . #<Closure #  
> 2187D8B2>)) ((WARNING . EDITOR::EDITOR-EVALUATION-WARNING)) ((END-OF- 
> FILE . #<Closure # 2216AB22>) (READER-ERROR . #<Closure # 2187DD82>)))
>
> Call to (METHOD ELEPHANT::OPEN-CONTROLLER (DB-BDB::BDB-STORE- 
> CONTROLLER)) (offset 2211)
>
> Call to CLOS::METHOD-COMBINATION-TEMPLATE (offset 62)
>
> Call to OPEN-STORE (offset 66)
>
> Call to DO-BACKEND-TESTS (offset 106)
>  SPEC               : (:BDB "c:\\lisp\\libraries\\elephant-1.0\\tests 
> \\testdb\\")
>  *STORE-CONTROLLER* : NIL
>
> Call to EVAL (offset 113)
>  EXP : (DO-BACKEND-TESTS)
>
>
>
> 6. turning now to postmodern:
>
>
> The condition #<ELEPHANT-DESERIALIZATION-ERROR 2009F808> occurred
>  1 (continue) retry db-postmodern transaction
>  2 DB-POSTMODERN::ABORT-TRANSACTION
>  3 (abort) Abort job 8 :(BIND-STANDARD-STREAMS-AND-EXECUTE  
> #<EDITOR::EDITOR-OUTPUT-STREAM #<EDITOR:BUFFER Background Output>  
> 2190405F> (FUNCALL-BACKGROUND-JOB-AUX #<CAPI:EDITOR-PANE CAPI:EDITOR- 
> PANE  2198348B> BACKGROUND-REGION-EVAL (# # # # T NIL NIL)))
>
> Type :b for backtrace, :c <option number> to proceed,  or :? for  
> other options
>
> ELE-TESTS 1 : 1 > :b :verbose
> Call to DESERIALIZE (offset 362)
>  ELEPHANT::BS       : #S(ELEPHANT-MEMUTIL:BUFFER-STREAM :BUFFER  
> #<Pointer to type (:UNSIGNED :CHAR) = #x0039FBD8> :SIZE 45 :POSITION  
> 2844 :LENGTH 80)
>  ELEPHANT::SC       : #<POSTMODERN-STORE-CONTROLLER (POSTGRESQL  
> 127.0.0.1 elepm postgres )>
>  ELEPHANT::OID-ONLY : NIL
>  DBG::G             : #<CONDITIONS::COERCE-TO-CONDITION-ERROR  
> 2009C9F3>
>  ELEPHANT::E        : #<CONDITIONS::COERCE-TO-CONDITION-ERROR  
> 2009C9F3>
>
> Call to DB-POSTMODERN::DESERIALIZE-FROM-DATABASE (offset 63)
>  DB-POSTMODERN::X         : #(17 0 0 0 0 13 9 3 0 0 0 69 81 76 9 11  
> 0 0 0 67 79 77 77 79 78 45 76 73 83 80 4 0 0 0 64 4 0 0 128 63 ...)
>  DB-POSTMODERN::SC        : #<POSTMODERN-STORE-CONTROLLER  
> (POSTGRESQL 127.0.0.1 elepm postgres )>
>  DB-POSTMODERN::OIDS-ONLY : NIL
>  DB-POSTMODERN::OTHER     : #S(ELEPHANT-MEMUTIL:BUFFER- 
> STREAM :BUFFER #<Pointer to type (:UNSIGNED :CHAR) =  
> #x0039FBD8> :SIZE 45 :POSITION 2844 :LENGTH 80)
>
> Call to (METHOD DB-POSTMODERN::INTERNAL-GET-VALUE (T DB- 
> POSTMODERN::PM-BTREE)) (offset 369)
>  DB-POSTMODERN::KEY          : "3 INDICES"
>  DB-POSTMODERN::BT           : #<DB-POSTMODERN::PM-BTREE db- 
> table:slots 217BB097>
>  CLOS::.ISL.                 : :DONT-KNOW
>  DB-POSTMODERN::VALUE        : NIL
>  DB-POSTMODERN::EXISTS-P     : NIL
>  DB-POSTMODERN::*SC*         : #<POSTMODERN-STORE-CONTROLLER  
> (POSTGRESQL 127.0.0.1 elepm postgres )>
>  DB-POSTMODERN::*CONNECTION* : #<CL-POSTGRES:DATABASE-CONNECTION  
> 217BE883>
>  DB-POSTMODERN::RESULT       : #(17 0 0 0 0 13 9 3 0 0 0 69 81 76 9  
> 11 0 0 0 67 79 77 77 79 78 45 76 73 83 80 4 0 0 0 64 4 0 0 128 63 ...)
>
> Call to (METHOD GET-VALUE (T DB-POSTMODERN::PM-BTREE)) (offset 20)
>  DB-POSTMODERN::KEY : "3 INDICES"
>  DB-POSTMODERN::BT  : #<DB-POSTMODERN::PM-BTREE db-table:slots  
> 217BB097>
>  CLOS::.ISL.        : :DONT-KNOW
>
> Call to (METHOD ELEPHANT::PERSISTENT-SLOT-READER (DB- 
> POSTMODERN::POSTMODERN-STORE-CONTROLLER T T)) (offset 78)
>  DB-POSTMODERN::SC        : #<POSTMODERN-STORE-CONTROLLER  
> (POSTGRESQL 127.0.0.1 elepm postgres )>
>  DB-POSTMODERN::INSTANCE  : #<DB-POSTMODERN::PM-INDEXED-BTREE db- 
> table:schemas 217F8A77>
>  DB-POSTMODERN::NAME      : DB-POSTMODERN::INDICES
>  DB-POSTMODERN::OIDS-ONLY : :DONT-KNOW
>  CLOS::.ISL.              : :DONT-KNOW
>  DB-POSTMODERN::*SC*      : #<POSTMODERN-STORE-CONTROLLER  
> (POSTGRESQL 127.0.0.1 elepm postgres )>
>
> Call to CLOS::METHOD-COMBINATION-TEMPLATE (offset 45)
>
> Call to (METHOD CLOS::DYNAMIC-SLOT-VALUE-USING-SLOTD (T T)) (offset  
> 120)
>  CLOS::INSTANCE : #<DB-POSTMODERN::PM-INDEXED-BTREE db-table:schemas  
> 217F8A77>
>  CLOS::SLOTD    : #<ELEPHANT::PERSISTENT-EFFECTIVE-SLOT-DEFINITION  
> DB-POSTMODERN::INDICES 22346B0B>
>
> Call to (METHOD GET-INDEX (DB-POSTMODERN::PM-INDEXED-BTREE T))  
> (offset 16)
>  DB-POSTMODERN::BT         : #<DB-POSTMODERN::PM-INDEXED-BTREE db- 
> table:schemas 217F8A77>
>  DB-POSTMODERN::INDEX-NAME : ELEPHANT::BY-NAME
>  CLOS::.ISL.               : :DONT-KNOW
>
> Call to (METHOD ELEPHANT::ENSURE-INDEX (INDEXED-BTREE T)) (offset 27)
>
> Call to (HARLEQUIN-COMMON-LISP:SUBFUNCTION 2 (METHOD ELEPHANT::OPEN- 
> CONTROLLER :AFTER (STORE-CONTROLLER))) (offset 65)
>
> Call to DB-POSTMODERN::EXECUTE-TRANSACTION-ONE-TRY (offset 169)
>  DB-POSTMODERN::SC                : #<POSTMODERN-STORE-CONTROLLER  
> (POSTGRESQL 127.0.0.1 elepm postgres )>
>  DB-POSTMODERN::TXN-FN            : #<Closure ((METHOD  
> ELEPHANT::OPEN-CONTROLLER :AFTER (STORE-CONTROLLER)) . 2) 217F216A>
>  DB-POSTMODERN::ALWAYS-ROLLBACK   : NIL
>  DB-POSTMODERN::TRAN              : #<POSTMODERN::TRANSACTION-HANDLE  
> 217F70BB>
>  DB-POSTMODERN::COMMITED          : NIL
>  DB-POSTMODERN::*TXN-VALUE-CACHE* : NIL
>
> Call to (METHOD ELEPHANT::EXECUTE-TRANSACTION (DB- 
> POSTMODERN::POSTMODERN-STORE-CONTROLLER T)) (offset 889)
>  DB-POSTMODERN::SC               : #<POSTMODERN-STORE-CONTROLLER  
> (POSTGRESQL 127.0.0.1 elepm postgres )>
>  DB-POSTMODERN::TXN-FN           : #<Closure ((METHOD ELEPHANT::OPEN- 
> CONTROLLER :AFTER (STORE-CONTROLLER)) . 2) 217F216A>
>  DBG::G                          : :DONT-KNOW
>  DB-POSTMODERN::ALWAYS-ROLLBACK  : NIL
>  DB-POSTMODERN::RETRY-CLEANUP-FN : NIL
>  DB-POSTMODERN::RETRIES          : 200
>  CLOS::.ISL.                     : :DONT-KNOW
>  POSTMODERN:*DATABASE*           : #<CL-POSTGRES:DATABASE-CONNECTION  
> 217BE883>
>  DB-POSTMODERN::TRY              : 200
>  DBG::G                          : #<Closure ((METHOD  
> ELEPHANT::EXECUTE-TRANSACTION (DB-POSTMODERN::POSTMODERN-STORE- 
> CONTROLLER T)) . 5) 217F20AA>
>  DBG::G                          : #<Function ((METHOD  
> ELEPHANT::EXECUTE-TRANSACTION (DB-POSTMODERN::POSTMODERN-STORE- 
> CONTROLLER T)) . 4) 222D653A>
>  DBG::G                          : #<Closure ((METHOD  
> ELEPHANT::EXECUTE-TRANSACTION (DB-POSTMODERN::POSTMODERN-STORE- 
> CONTROLLER T)) . 3) 217F20C2>
>  CONDITIONS::*RESTART-CLUSTERS*  : ((#<RESTART ABORT 2188E04F>)  
> (#<RESTART ABORT 2188E2B7>) (#<RESTART DB-POSTMODERN::RETRY- 
> TRANSACTION 2009F7AB> #<RESTART DB-POSTMODERN::ABORT-TRANSACTION  
> 2009F77B>) (#<RESTART ABORT 2009F73F>) (#<RESTART ABORT 21E43D1F>))
>  DBG::G                          : #<Function ((METHOD  
> ELEPHANT::EXECUTE-TRANSACTION (DB-POSTMODERN::POSTMODERN-STORE- 
> CONTROLLER T)) . 8) 222D6E7A>
>  CONDITIONS::*HANDLER-CLUSTERS*  : (((CL-POSTGRES:DATABASE-ERROR .  
> #<Function # 222D6E7A>)) ((WARNING . EDITOR::EDITOR-EVALUATION- 
> WARNING)) ((END-OF-FILE . #<Closure # 217F2062>) (READER-ERROR .  
> #<Closure # 2188F032>)))
>
> Call to CLOS::METHOD-COMBINATION-TEMPLATE (offset 146)
>
> Call to CLOS::NEXT-METHOD-CALL-2 (offset 87)
>  CONS               : (METHOD ELEPHANT::OPEN-CONTROLLER :AROUND (DB- 
> POSTMODERN::POSTMODERN-STORE-CONTROLLER))
>  CLOS::NEXT-METHODS : (#<Closure CLOS::METHOD-COMBINATION-TEMPLATE  
> 217F14C2>)
>  #:G174431          : #<POSTMODERN-STORE-CONTROLLER (POSTGRESQL  
> 127.0.0.1 elepm postgres )>
>  #:G174432          : NIL
>
> Call to (METHOD ELEPHANT::OPEN-CONTROLLER :AROUND (DB- 
> POSTMODERN::POSTMODERN-STORE-CONTROLLER)) (offset 61)
>  DB-POSTMODERN::SC          : #<POSTMODERN-STORE-CONTROLLER  
> (POSTGRESQL 127.0.0.1 elepm postgres )>
>  DBG::G                     : NIL
>  CLOS::.ISL.                : :DONT-KNOW
>  DBG::G                     : (#<Closure CLOS::METHOD-COMBINATION- 
> TEMPLATE 217F14C2>)
>  DB-POSTMODERN:*CACHE-MODE* : NIL
>
> Call to CLOS::METHOD-COMBINATION-TEMPLATE (offset 42)
>
> Call to OPEN-STORE (offset 66)
>
> Call to DO-BACKEND-TESTS (offset 106)
>  SPEC               : (:POSTMODERN (:POSTGRESQL "127.0.0.1" "elepm"  
> "postgres" ""))
>  *STORE-CONTROLLER* : NIL
>
> Call to EVAL (offset 113)
>  EXP : (DO-BACKEND-TESTS)
>
> Call to EDITOR::EDITOR-EVAL (offset 177)
>  EDITOR:BUFFER  : :DONT-KNOW
>  EDITOR::SEXP   : (DO-BACKEND-TESTS)
>  EDITOR::ERRORP : T
>
> eof
> __________________________________________________________________
> Deutschlands gr??te Online-Videothek schenkt Ihnen 12.000 Videos!*
> http://entertainment.web.de/de/entertainment/maxdome/index.html
>
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel




From develgenius at gmail.com  Wed Jan 28 15:02:20 2009
From: develgenius at gmail.com (John)
Date: Wed, 28 Jan 2009 10:02:20 -0500
Subject: [elephant-devel] Learning about Elephant
In-Reply-To: <F87217F2-26CB-44B6-97B5-54047EBF7876@media.mit.edu>
References: <e6c974240901270800l11e2b308qf0fb2561782c6f86@mail.gmail.com>
	<5D2242B8-1A86-4C89-96DD-3311A6EA2392@media.mit.edu>
	<e6c974240901271159w7ec7bb37v3dfc1098e7991021@mail.gmail.com>
	<F87217F2-26CB-44B6-97B5-54047EBF7876@media.mit.edu>
Message-ID: <e6c974240901280702h65e9dd55m6c503c5715f1900a@mail.gmail.com>

On Wed, Jan 28, 2009 at 12:26 AM, Ian Eslick <eslick at media.mit.edu> wrote:

> >
> Hi John,
>
> Your mileage may vary, but I'm really not concerned about users
> causing significant performance degradation by doing lots of null
> edits.  If that is happening, it's probably a user interface design
> problem.  The trivial solution is to write a jscript function that
> sets a hidden input value if the contents of any of the inputs are
> changed.  You can inhibit the POST entirely if that's the case, or
> allow the server to skip processing it.  This should cut out a huge %
> of your overall null edits (if you don't have some weird community
> with a high percentage of no-javascript users) and it's only a few
> lines of code.


Agree.  Maybe I can just compare the values of the POST variables against
the slot values and if anything changed, then proceed with the
update/transaction. I'm not too much of a fan on relying on javascript for
that type of validation since people can "hack" that easily (don't ask... it
will be a public application so you never know what type of users we'll get)

We should all be so lucky as to blow out a single server!  If you have
> that much traffic, you can usually afford some reworking to scale if
> your application is amenable to it.  Often you can get pretty far just
> by doing some optimizations, offloading static file to a second
> server, smart caching, some restructuring and performance
> optimizations, transaction tuning, etc.  There is almost always room
> for significant optimization in a new system.


Correct. And that's why premature optimization is almost always more of a
waste of time. I was only being a dreamer and imagining that our application
will revolutionize the Internet and will blow the sucks off Facebook :D


> Check out the elephant-devel mailing list, associations are described
> several times there.  I also provided an example in a recent e-mail.
>
> The API is straightforward so between the new metaclass slot initargs
> (src/elephant/metaclass.lisp) and the association interface (src/
> elephant/associations.lisp) you should be able to work with them.
>
> Regards,
> Ian


Thank  you again. Will do.

Best regards,
JD
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20090128/75513b5e/attachment.html>

From develgenius at gmail.com  Wed Jan 28 16:19:26 2009
From: develgenius at gmail.com (John)
Date: Wed, 28 Jan 2009 11:19:26 -0500
Subject: [elephant-devel] Learning about Elephant
In-Reply-To: <A25DD755-A080-41F0-8F56-61B2DEBC54D6@media.mit.edu>
References: <e6c974240901270800l11e2b308qf0fb2561782c6f86@mail.gmail.com>
	<5D2242B8-1A86-4C89-96DD-3311A6EA2392@media.mit.edu>
	<D387C483-7C97-4452-A42B-91C862BC499A@media.mit.edu>
	<e6c974240901271406s1cb8462ct58cb52ab320ccc2@mail.gmail.com>
	<A25DD755-A080-41F0-8F56-61B2DEBC54D6@media.mit.edu>
Message-ID: <e6c974240901280819r20f6e575s8d30e3c5c31c5b40@mail.gmail.com>

Hi Ian,

I guess there is nothing better than trying it out :)

ELE-TESTS> (defpclass person ()
             ((friends :accessor friends-of :associate (person friends)
:many-to-many t)))
#<PERSISTENT-METACLASS PERSON>
ELE-TESTS> (defparameter person1 (make-instance 'person))
PERSON1
ELE-TESTS> (defparameter person2 (make-instance 'person))
PERSON2
ELE-TESTS> (defparameter person3 (make-instance 'person))
PERSON3
ELE-TESTS> (defparameter person4 (make-instance 'person))
PERSON4
ELE-TESTS> (defparameter person5 (make-instance 'person))
PERSON5
ELE-TESTS> (setf (friends-of person1) person2)
#<PERSON oid:3>
ELE-TESTS> (setf (friends-of person1) person3)
#<PERSON oid:4>
ELE-TESTS> (setf (friends-of person1) person4)
#<PERSON oid:5>
ELE-TESTS> (setf (friends-of person1) person5)
#<PERSON oid:6>
ELE-TESTS> (setf (friends-of person2) person3)
#<PERSON oid:4>
ELE-TESTS> (friends-of person1)
(#<PERSON oid:3> #<PERSON oid:4> #<PERSON oid:5> #<PERSON oid:6>)
ELE-TESTS> (friends-of person2)
(#<PERSON oid:2> #<PERSON oid:4>)
ELE-TESTS> (friends-of person3)
(#<PERSON oid:2> #<PERSON oid:3>)
ELE-TESTS> (friends-of person4)
(#<PERSON oid:2>)
ELE-TESTS> (friends-of person5)
(#<PERSON oid:2>)
ELE-TESTS> (remove-association person1 'friends person2)
NIL
ELE-TESTS> (friends-of person1)
(#<PERSON oid:4> #<PERSON oid:5> #<PERSON oid:6>)

So, it does seem to behave as I suspected.

Now, I think the last few questions I have on this subject before delving
further are

1) What data structure do associations use to store the list of associated
objects? My little understanding of update-association-end implies that the
association slot creates some sort of indexed-btree.

2) Are they storing a reference to the objects themselves or just the OIDs.
It seems to me that it stores a reference on that "indexed-btree". However,
this (setf (get-value (oid target) index) (oid instance)) in
update-association-end seems to imply that it stores the key/value using the
OID.

3) Can the list be indexed in any way or do we basically need to "manually"
sort it to our liking once we retrieve the associations. If my assumptions
above are correct, the associations are indexed based on OID sequence.

As a general question in reference to #2 above, does Elephant always use the
OIDs to store references of objects? I read on the list that a while back
someone (I think it was you) posted something along the lines that we
shouldn't use OIDs as unique identifiers because OIDs could change. I
suppose during migrations, when objects are re-created, the OIDs change. Is
that correct? Do the migrations take care of updating all these OID
references?

Now, a separate modeling question arises :) Imagine a many-to-many
relationship where the relationship needs to store additional information.
For example, imagine a membership club. Person1 becomes a member of Club1 on
1/1/09 and is paid thru 3/31/09. We have a class for person and a class for
club, but when we store the association between the two, we would like to
store the "effective-date" and the "termination-date" of the membership
(association). How would you approach that?

Thanks again,
JD

On Wed, Jan 28, 2009 at 12:32 AM, Ian Eslick <eslick at media.mit.edu> wrote:

> That would be correct if you added (setf (friends-of person3) person2)
>
> On Jan 27, 2009, at 5:06 PM, John wrote:
>
> > Thanks for the clear example. However, maybe the idioms are
> > confusing me, but many-to-many tells me that I should be able to do
> > something like:
> >
> > (setf (friends-of person1) person2)
> > (setf (friends-of person1) person3)
> > (setf (friends-of person1) person4)
> > (setf (friends-of person1) person5)
> >
> > (setf (friends-of person2) person3)
> >
> > such that:
> >
> > (friends-of person1) => (person2 person3 person4 person5)
> > (friends-of person2) => (person1 person3)
> > (friends-of person3) => (person1 person2)
> > (friends-of person4) => person1
> > (friends-of person5) => person1
> >
> > Would that be the expected behavior?
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20090128/acc7a06a/attachment.html>

From franks-muc at web.de  Wed Jan 28 19:08:37 2009
From: franks-muc at web.de (Frank Schorr)
Date: Wed, 28 Jan 2009 20:08:37 +0100
Subject: [elephant-devel]
 =?iso-8859-15?q?elephant-1=2E0_on_Lispworks+win3?=
 =?iso-8859-15?q?2=3B_bdb=3B_postmodern?=
Message-ID: <1537352222@web.de>

> -----Urspr?ngliche Nachricht-----
> Von: "Ian Eslick" <eslick at media.mit.edu>

> 
> Did this happen on the first open-controller or somewhere in the  
> middle of the tests?
> 

I can't tell. 
But I tried this:

(asdf:operate 'asdf:load-op :elephant)
(asdf:operate 'asdf:load-op :ele-bdb)
(setf *db-spec* '(:BDB "c:/temp/testdb/"))
(ele:open-store *db-spec*)

[The bdb files are created in the directory. Hope this helps]

Error: The slot ELEPHANT::SCHEMA-NAME-INDEX is unbound in the object #<BDB-STORE-CONTROLLER c:/temp/testdb/> (an instance of class #<STANDARD-CLASS DB-BDB::BDB-STORE-CONTROLLER 21F0DF4F>).
  1 (continue) Try reading slot ELEPHANT::SCHEMA-NAME-INDEX again.
  2 Specify a value to use this time for slot ELEPHANT::SCHEMA-NAME-INDEX.
  3 Specify a value to set slot ELEPHANT::SCHEMA-NAME-INDEX to.
  4 (abort) Abort job 4 :(BIND-STANDARD-STREAMS-AND-EXECUTE #<EDITOR::EDITOR-OUTPUT-STREAM #<EDITOR:BUFFER Background Output> 21CE1B7F> (FUNCALL-BACKGROUND-JOB-AUX #<CAPI:EDITOR-PANE CAPI:EDITOR-PANE  217334BB> BACKGROUND-REGION-EVAL (# # # # T NIL NIL)))

Type :b for backtrace, :c <option number> to proceed,  or :? for other options

CL-USER 1 : 1 > :b :verbose
Call to (METHOD SLOT-UNBOUND (T T T)) (offset 512)
  CLASS           : #<STANDARD-CLASS DB-BDB::BDB-STORE-CONTROLLER 21F0DF4F>
  CLOS::INSTANCE  : #<BDB-STORE-CONTROLLER c:/temp/testdb/>
  CLOS::SLOT-NAME : ELEPHANT::SCHEMA-NAME-INDEX

Call to CLOS::CALL-SLOT-UNBOUND-FUNCTION (offset 21)
  CLASS           : #<STANDARD-CLASS DB-BDB::BDB-STORE-CONTROLLER 21F0DF4F>
  CLOS::INSTANCE  : #<BDB-STORE-CONTROLLER c:/temp/testdb/>
  CLOS::SLOT-NAME : ELEPHANT::SCHEMA-NAME-INDEX

Call to ELEPHANT::GET-DB-SCHEMAS (offset 31)

Call to ELEPHANT::GET-CURRENT-DB-SCHEMA (offset 23)

Call to (METHOD ELEPHANT::LOOKUP-SCHEMA (ELEPHANT:STORE-CONTROLLER ELEPHANT:PERSISTENT-METACLASS)) (offset 91)

Call to (METHOD CLOS:FINALIZE-INHERITANCE :AFTER (ELEPHANT:PERSISTENT-METACLASS)) (offset 361)
  ELEPHANT::INSTANCE   : #<ELEPHANT:PERSISTENT-METACLASS ELEPHANT:PERSISTENT-OBJECT 21344DAF>
  CLOS::.ISL.          : #(#(#(1477 (CLOS::DEPENDENTS CLOS::NAME CLOS:DEFAULT-INITARGS CLOS::DIRECT-DEFAULT-INITARGS CLOS::DIRECT-SLOTS CLOS::DIRECT-SUBCLASSES CLOS::DIRECT-SUPERCLASSES CLOS::PRECEDENCE-LIST CLOS::EFFECTIVE-SLOTS CLOS::NO-INSTANCE-SLOTS CLOS::PROTOTYPE CLOS::DIRECT-METHODS CLOS::WRAPPER CLOS::CONSTRUCTORS CLOS::DOCUMENTATION-SLOT CLOS::PLIST CLOS::POTENTIAL-INITARGS CLOS::MAKE-INSTANCE-FLAGS ELEPHANT::%CLASS-SCHEMA ELEPHANT::%STORE-SCHEMAS ELEPHANT::%CLASS-INDEXING ELEPHANT::%CACHE-STYLE) NIL #<STANDARD-CLASS ELEPHANT:PERSISTENT-METACLASS 21352493> STANDARD-CLASS) #(20) NIL NIL) ELEPHANT::%CLASS-INDEXING 2 0)
  CLOS::.PV.           : #(20)
  ELEPHANT::OLD-SCHEMA : NIL
  ELEPHANT::NEW-SCHEMA : #<CLASS-SCHEMA PERSISTENT-OBJECT>

Call to CLOS::METHOD-COMBINATION-TEMPLATE (offset 140)

Call to CLOS::ENSURE-CLASS-FINALIZED (offset 150)
  CLASS : #<ELEPHANT:PERSISTENT-METACLASS ELEPHANT:PERSISTENT-OBJECT 21344DAF>

Call to CLOS::ENSURE-CLASS-FINALIZED (offset 336)
  CLASS : #<ELEPHANT:PERSISTENT-METACLASS DB-BDB::BDB-INDEXED-BTREE 21E77043>

Call to (METHOD CLOS:DEFAULT-INITARGS (CLASS T)) (offset 27)
  CLASS          : #<ELEPHANT:PERSISTENT-METACLASS DB-BDB::BDB-INDEXED-BTREE 21E77043>
  CLOS::INITARGS : (:FROM-OID -3 :SC #<BDB-STORE-CONTROLLER c:/temp/testdb/> :INDICES #<EQL Hash Table{0} 2173A1AF>)

Call to CLOS::MAKE-INSTANCE-FROM-CLASS-1 (offset 386)
  CLASS          : #<ELEPHANT:PERSISTENT-METACLASS DB-BDB::BDB-INDEXED-BTREE 21E77043>
  CLOS::INITARGS : (:FROM-OID -3 :SC #<BDB-STORE-CONTROLLER c:/temp/testdb/> :INDICES #<EQL Hash Table{0} 2173A1AF>)

Call to (SUBFUNCTION 2 (METHOD ELEPHANT::OPEN-CONTROLLER (DB-BDB::BDB-STORE-CONTROLLER))) (offset 175)

Call to (METHOD ELEPHANT::EXECUTE-TRANSACTION (DB-BDB::BDB-STORE-CONTROLLER T)) (offset 741)
  DB-BDB::SC                      : #<BDB-STORE-CONTROLLER c:/temp/testdb/>
  DB-BDB::TXN-FN                  : #<Closure ((METHOD ELEPHANT::OPEN-CONTROLLER (DB-BDB::BDB-STORE-CONTROLLER)) . 2) 21E8DD72>
  DBG::G                          : :DONT-KNOW
  DB-BDB::TRANSACTION             : :DONT-KNOW
  DB-BDB::PARENT                  : NIL
  DB-BDB::ENVIRONMENT             : NIL
  DB-BDB::RETRIES                 : 200
  DB-BDB::DEGREE-2                : NIL
  DB-BDB::READ-UNCOMMITTED        : NIL
  DB-BDB::TXN-NOSYNC              : NIL
  DB-BDB::TXN-NOWAIT              : NIL
  DB-BDB::TXN-SYNC                : NIL
  DB-BDB::SNAPSHOT                : NIL
  DB-BDB::INHIBIT-ROLLBACK-FN     : NIL
  CLOS::.ISL.                     : :DONT-KNOW
  DB-BDB::ENV                     : #<Pointer to type :VOID = #x00F04500>
  COUNT                           : 0
  DB-BDB::SUCCESS                 : NIL
  DB-BDB::TXN                     : #<Pointer to type :VOID = #x00F06B58>
  DB-BDB::RESULT                  : NIL
  ELEPHANT::*CURRENT-TRANSACTION* : (#<BDB-STORE-CONTROLLER c:/temp/testdb/> #<Pointer to type :VOID = #x00F06B58> NIL)
  ELEPHANT:*STORE-CONTROLLER*     : #<BDB-STORE-CONTROLLER c:/temp/testdb/>
  DBG::G                          : #<Closure ((METHOD ELEPHANT::EXECUTE-TRANSACTION (DB-BDB::BDB-STORE-CONTROLLER T)) . 2) 2187F16A>
  CONDITIONS::*HANDLER-CLUSTERS*  : (((CONDITION . #<Closure # 2187F16A>)) ((WARNING . EDITOR::EDITOR-EVALUATION-WARNING)) ((END-OF-FILE . #<Closure # 21E2EC3A>) (READER-ERROR . #<Closure # 2187F62A>)))

Call to (METHOD ELEPHANT::OPEN-CONTROLLER (DB-BDB::BDB-STORE-CONTROLLER)) (offset 2211)

Call to CLOS::METHOD-COMBINATION-TEMPLATE (offset 62)

Call to ELEPHANT:OPEN-STORE (offset 66)

Call to EVAL (offset 113)
  EXP : (ELEPHANT:OPEN-STORE *DB-SPEC*)

Call to EDITOR::EDITOR-EVAL (offset 177)
  EDITOR:BUFFER  : :DONT-KNOW
  EDITOR::SEXP   : (ELEPHANT:OPEN-STORE *DB-SPEC*)
  EDITOR::ERRORP : T

Call to (SUBFUNCTION 3 (SUBFUNCTION 1 EDITOR::REGION-LISP-EVAL)) (offset 26)
  EDITOR::FORM  : (ELEPHANT:OPEN-STORE *DB-SPEC*)
  EDITOR:BUFFER : #<EDITOR:BUFFER work.lisp>

Call to (SUBFUNCTION 1 EDITOR::REGION-LISP-EVAL) (offset 624)
  #:BUFFER41963          : #<EDITOR:BUFFER work.lisp>
  EDITOR:BUFFER          : #<EDITOR:BUFFER work.lisp>
  PRINT                  : T
  EDITOR::START          : #<EDITOR::POINT "work.lisp" 0 offset 4163 21E34127>
  EDITOR::END            : #<EDITOR::POINT "work.lisp" 0 offset 4189 21E3414F>
  DBG::|block-catch-tag| : (NIL)
  DBG::|block-catch-tag| : (NIL)

Call to EDITOR::WITH-COMPILATION-ENVIRONMENT-AT-POINT-FN (offset 266)
  EDITOR::POINT         : #<EDITOR::POINT "work.lisp" 0 offset 4163 21E34127>
  EDITOR::START-MESSAGE : "Evaluating..."
  EDITOR::END-MESSAGE   : "Finished evaluating"
  EDITOR::BODY-FN       : #<Closure (EDITOR::REGION-LISP-EVAL . 1) 21E2EC22>

Call to EDITOR::REGION-LISP-EVAL (offset 204)
  EDITOR:BUFFER : #<EDITOR:BUFFER work.lisp>
  EDITOR::START : #<EDITOR::POINT "work.lisp" 0 offset 4163 21E34127>
  EDITOR::END   : #<EDITOR::POINT "work.lisp" 0 offset 4189 21E3414F>
  PRINT         : T

Call to EDITOR::BACKGROUND-REGION-EVAL (offset 103)
  EDITOR::WINDOW         : #<EDITOR::WM-WINDOW "work.lisp" 216DE40F>
  EDITOR:BUFFER          : #<EDITOR:BUFFER work.lisp>
  EDITOR::START          : #<EDITOR::POINT "work.lisp" 0 offset 4163 21E34127>
  EDITOR::END            : #<EDITOR::POINT "work.lisp" 0 offset 4189 21E3414F>
  PRINT                  : T
  EDITOR::WARNINGS       : NIL
  EDITOR::AFTER-FUNCTION : NIL

Call to CAPI::FUNCALL-BACKGROUND-JOB-AUX (offset 50)
  CAPI::PANE : #<CAPI:EDITOR-PANE CAPI:EDITOR-PANE  217334BB>
  FUNCTION   : EDITOR::BACKGROUND-REGION-EVAL
  CAPI::ARGS : (#<EDITOR::WM-WINDOW "work.lisp" 216DE40F> #<EDITOR:BUFFER work.lisp> #<EDITOR::POINT "work.lisp" 0 offset 4163 21E34127> #<EDITOR::POINT "work.lisp" 0 offset 4189 21E3414F> T NIL NIL)

Call to CAPI::BIND-STANDARD-STREAMS-AND-EXECUTE (offset 204)
  STREAM          : #<EDITOR::EDITOR-OUTPUT-STREAM #<EDITOR:BUFFER Background Output> 21CE1B7F>
  CAPI::FUNC-ARGS : (CAPI::FUNCALL-BACKGROUND-JOB-AUX #<CAPI:EDITOR-PANE CAPI:EDITOR-PANE  217334BB> EDITOR::BACKGROUND-REGION-EVAL (#<EDITOR::WM-WINDOW "work.lisp" 216DE40F> #<EDITOR:BUFFER work.lisp> #<EDITOR::POINT "work.lisp" 0 offset 4163 21E34127> #<EDITOR::POINT "work.lisp" 0 offset 4189 21E3414F> T NIL NIL))

Call to MP::BACKGROUND-EXECUTE-LOOP (offset 629)

Call to (SUBFUNCTION MP::PROCESS-SG-FUNCTION MP::INITIALIZE-PROCESS-STACK) (offset 301)
  MP::IGNORED : :DONT-KNOW
  VALUES      : NIL
  MP:PROCESS  : #<MP:PROCESS Name "Background execute 2" Priority 50000000 State "Running">
  MP::SYMBOLS : NIL


CL-USER 2 : 1 > 

_______________________________________________________________________
NUR NOCH BIS 31.01.! WEB.DE FreeDSL - Telefonanschluss + DSL
f?r nur 16,37 EURO/mtl.!* http://dsl.web.de/?ac=OM.AD.AD008K13805B7069a




From yarek.kowalik at gmail.com  Wed Jan 28 20:33:42 2009
From: yarek.kowalik at gmail.com (Yarek Kowalik)
Date: Wed, 28 Jan 2009 12:33:42 -0800
Subject: [elephant-devel] Given the OID,
	can one obtain the object's slot value without recreating the
	object?
Message-ID: <ecb948b30901281233h7cb688afka0aa89a494a72ff8@mail.gmail.com>

Given the OID, Is there a way to obtain an object slot value without
recreating the object?  I want to reacreate the object only when a specific
value is present.

I am aware that I could use map-inverse-index to search for value, but it
does not do exacty what I want:  I want to get all the objects with slot 'a
matching "value-a", but filtered on another slot, say slot 'b, with matching
"value-b".

Yarek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20090128/dfae03ec/attachment.html>

From eslick at media.mit.edu  Thu Jan 29 21:12:04 2009
From: eslick at media.mit.edu (Ian Eslick)
Date: Thu, 29 Jan 2009 16:12:04 -0500
Subject: [elephant-devel] Given the OID,
	can one obtain the object's slot value without recreating the
	object?
In-Reply-To: <ecb948b30901281233h7cb688afka0aa89a494a72ff8@mail.gmail.com>
References: <ecb948b30901281233h7cb688afka0aa89a494a72ff8@mail.gmail.com>
Message-ID: <0DCDA978-BDC6-4C26-A7B4-76E621CCFF09@media.mit.edu>

Use map-inverse-index :oid t to get the list of oids matching  
parameter A
Use map-inverse-index :oid t to get the list of oids matching  
parameter B
Take the intersection of the OIDs
Recreate your objects.

And in fact this is exactly what the query system does and should just  
to the right thing for you already (though with some extra temporary  
memory overhead at the moment).  I'm blanking on the syntax, but it  
looks something like this:

(select ((a object))
   (eq (slot1 a) 'foo)
   (eq (slot2 a) 'bar))

Ian


On Jan 28, 2009, at 3:33 PM, Yarek Kowalik wrote:

> Given the OID, Is there a way to obtain an object slot value without  
> recreating the object?  I want to reacreate the object only when a  
> specific value is present.
>
> I am aware that I could use map-inverse-index to search for value,  
> but it  does not do exacty what I want:  I want to get all the  
> objects with slot 'a matching "value-a", but filtered on another  
> slot, say slot 'b, with matching "value-b".
>
> Yarek
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel