From stesch at no-spoon.de Thu Feb 3 12:30:05 2005 From: stesch at no-spoon.de (Stefan Scholl) Date: Thu, 3 Feb 2005 13:30:05 +0100 Subject: [elephant-devel] Accessing persistant instances Message-ID: <20050203123005.GC12099@parsec.no-spoon.de> Hi! A little question to help me understand something: NOTES says that "the current setup use one big btree table for all slots of all instances of all classes, keyed on OID + Slot ID". Does this mean that accessing an instance doesn't read in the whole object? Just the slots I read with the corresponding accessor method? E.g.: I have a class with 3 slots small1, small2, and big. Now I iterate over one complete btree with map-btree and just print small1 and small2. Will slot big be read anyway? Regards, Stefan From ben at medianstrip.net Thu Feb 3 17:24:05 2005 From: ben at medianstrip.net (Ben) Date: Thu, 3 Feb 2005 12:24:05 -0500 (EST) Subject: [elephant-devel] Accessing persistant instances In-Reply-To: <20050203123005.GC12099@parsec.no-spoon.de> References: <20050203123005.GC12099@parsec.no-spoon.de> Message-ID: <20050203122337.N47682@contarex.medianstrip.net> slot big will not be read unless you access it. hope this answers your question. take care, B On Thu, 3 Feb 2005, Stefan Scholl wrote: > Hi! > > A little question to help me understand something: > > NOTES says that "the current setup use one big btree table for > all slots of all instances of all classes, keyed on OID + Slot > ID". > > Does this mean that accessing an instance doesn't read in the > whole object? Just the slots I read with the corresponding > accessor method? > > E.g.: I have a class with 3 slots small1, small2, and big. Now I > iterate over one complete btree with map-btree and just print > small1 and small2. Will slot big be read anyway? > > > Regards, > Stefan > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel > From stesch at no-spoon.de Thu Feb 3 18:29:09 2005 From: stesch at no-spoon.de (Stefan Scholl) Date: Thu, 3 Feb 2005 19:29:09 +0100 Subject: [elephant-devel] Accessing persistant instances In-Reply-To: <20050203122337.N47682@contarex.medianstrip.net> References: <20050203123005.GC12099@parsec.no-spoon.de> <20050203122337.N47682@contarex.medianstrip.net> Message-ID: <20050203182909.GE12099@parsec.no-spoon.de> On 2005-02-03 12:24:05, Ben wrote: > On Thu, 3 Feb 2005, Stefan Scholl wrote: > >E.g.: I have a class with 3 slots small1, small2, and big. Now I > >iterate over one complete btree with map-btree and just print > >small1 and small2. Will slot big be read anyway? > > slot big will not be read unless you access it. > > hope this answers your question. Yes, it does. Thank you! One of Daniel Barlow's points against OO interfaces for RDBMS was that all columns get selected when you "get" an object: http://ww.telent.net/diary/2003/6/#21.51860 And this point is valid for any other kind of DB. Nice to see that Elephant gets this right. Regards, Stefan From acristin.misc at gmail.com Wed Feb 9 05:15:47 2005 From: acristin.misc at gmail.com (Andy Misc Cristina) Date: Tue, 08 Feb 2005 23:15:47 -0600 Subject: [elephant-devel] Elephant on win32 with VS.NET and acl70 Message-ID: <42099C83.3000508@gmail.com> Hi, I am trying to use elephant(0.2.1) in windows XP with allegro 7.0. I have built db-4.2.52NC and the libsleepycat.c files in VS .NET 2k3, according to the instructions in the install guide. However, allegro errors out on loading libsleepycat. Are there any known issues with my version of visual studio? I'm not sure what information is relevant here, but if someone could tell me what would be useful, I'll be happy to provide it. Thanks for any help, Andy From mega at hotpop.com Fri Feb 11 21:48:13 2005 From: mega at hotpop.com (Gabor Melis) Date: Fri, 11 Feb 2005 22:48:13 +0100 Subject: [elephant-devel] running out of locks Message-ID: <200502112248.13633.mega@hotpop.com> I have started performance testing my application for which a few thousand users were needed: (defclass user () ((login :accessor user-login :initarg :login :initform "") (login2 :accessor user-login2 :initarg :login :initform "") (login3 :accessor user-login3 :initarg :login :initform "")) (:metaclass persistent-metaclass)) (defun add-users-to-root () (with-transaction () (dotimes (i 10000) (add-to-root i (make-instance 'user))))) Then I got an error from next-counter: Berkeley DB error: Unknown error: -5 [Condition of type SLEEPYCAT:DB-ERROR] Restarts: 0: [ABORT] Abort handling SLIME request. 1: [DESTROY-THREAD] Destroy this thread (6135) Backtrace: 0: (SLEEPYCAT::NEXT-COUNTER # # # # 12 # 16) 1: ((INITIALIZE-INSTANCE :BEFORE (ELEPHANT:PERSISTENT)) # # # NIL) 2: ("#'(LAMBDA (SB-PCL::.PV-CELL. SB-PCL::.NEXT-METHOD-CALL. SB-PCL::.ARG0. ...) (DECLARE #) ...)" # # # NIL) 3: ((MAKE-INSTANCE (CLASS)) # # # NIL) 4: (ELETEST::ADD-USERS-TO-ROOT) It turned out the cause of the error was put in next_counter failing with "Berkeley DB error: Cannot allocate memory" that in turn was caused by running out of locks. It was verified db_stat: [eletestdb]$ db4.2_stat -c 60 Last allocated locker ID. 2147M Current maximum unused locker ID. 9 Number of lock modes. 1000 Maximum number of locks possible. 1000 Maximum number of lockers possible. 1000 Maximum number of lock objects possible. 1098 Number of current locks. 1101 Maximum number of locks at any one time. 10 Number of current lockers. 11 Maximum number of lockers at any one time. 599 Number of current lock objects. 600 Maximum number of lock objects at any one time. 196238 Total number of locks requested. 188556 Total number of locks released. 0 Total number of lock requests failing because DB_LOCK_NOWAIT was set. 0 Total number of locks not immediately available due to conflicts. 0 Number of deadlocks. 0 Lock timeout value. 0 Number of locks that have timed out. 0 Transaction timeout value. 0 Number of transactions that have timed out. 360KB The size of the lock region.. 0 The number of region locks granted after waiting. 315390 The number of region locks granted without waiting. Adding the users without a transaction works but slow. Adding the same number of users within a transaction usually produces different "Maximum number of locks at any one time.". Is this seemingly non-deterministic accumulation of locks expected? Thanks, G?bor From ben at medianstrip.net Sat Feb 12 00:58:56 2005 From: ben at medianstrip.net (Ben) Date: Fri, 11 Feb 2005 19:58:56 -0500 (EST) Subject: [elephant-devel] running out of locks In-Reply-To: <200502112248.13633.mega@hotpop.com> References: <200502112248.13633.mega@hotpop.com> Message-ID: <20050211193640.K9331@contarex.medianstrip.net> ouch. it looks like my "poor man's counters" are really POOR. (i thought i had debugged all the lock issues.....) it's a nasty c function which was an interim until 4.3 came out.....now that 4.3 is out i'll upgrade. it looks like it is nigh time to upgrade elephant. this weekend i will (at least) upgrade it to use sleepycat 4.3 and the new sequences, which should solve this problem. hopefully i'll be able to get andrew to fix the MOP stuff, and maybe we'll get to some other things. by the way, the poor man's counters c function returns error values which are not in concordance with the usual sleepycat stuff. that error indicates the transaction failed after 100 tries. it looks like from your debugging that indeed it's the locks which are the problem. thanks for the heads up. i'll keep you informed. take care, B On Fri, 11 Feb 2005, Gabor Melis wrote: > I have started performance testing my application for which a few thousand > users were needed: > > > > (defclass user () > ((login :accessor user-login :initarg :login :initform "") > (login2 :accessor user-login2 :initarg :login :initform "") > (login3 :accessor user-login3 :initarg :login :initform "")) > (:metaclass persistent-metaclass)) > > (defun add-users-to-root () > (with-transaction () > (dotimes (i 10000) > (add-to-root i (make-instance 'user))))) > > > > Then I got an error from next-counter: > > > > Berkeley DB error: Unknown error: -5 > [Condition of type SLEEPYCAT:DB-ERROR] > > Restarts: > 0: [ABORT] Abort handling SLIME request. > 1: [DESTROY-THREAD] Destroy this thread (6135) > > Backtrace: > 0: (SLEEPYCAT::NEXT-COUNTER # #X082797E0 :TYPE (* T)> # #X08285648 :TYPE (* T)> # #X08282CF0 :TYPE (* T)> # #X08279490 :TYPE (* (SIGNED 8))> 12 # #X08277BD0 :TYPE (* (SIGNED 8))> 16) > 1: ((INITIALIZE-INSTANCE :BEFORE (ELEPHANT:PERSISTENT)) # argument> # # NIL) > 2: ("#'(LAMBDA (SB-PCL::.PV-CELL. SB-PCL::.NEXT-METHOD-CALL. > SB-PCL::.ARG0. ...) (DECLARE #) ...)" # # argument> # NIL) > 3: ((MAKE-INSTANCE (CLASS)) # # > # NIL) > 4: (ELETEST::ADD-USERS-TO-ROOT) > > > > It turned out the cause of the error was put in next_counter failing with > "Berkeley DB error: Cannot allocate memory" that in turn was caused by > running out of locks. It was verified db_stat: > > [eletestdb]$ db4.2_stat -c > 60 Last allocated locker ID. > 2147M Current maximum unused locker ID. > 9 Number of lock modes. > 1000 Maximum number of locks possible. > 1000 Maximum number of lockers possible. > 1000 Maximum number of lock objects possible. > 1098 Number of current locks. > 1101 Maximum number of locks at any one time. > 10 Number of current lockers. > 11 Maximum number of lockers at any one time. > 599 Number of current lock objects. > 600 Maximum number of lock objects at any one time. > 196238 Total number of locks requested. > 188556 Total number of locks released. > 0 Total number of lock requests failing because DB_LOCK_NOWAIT was set. > 0 Total number of locks not immediately available due to conflicts. > 0 Number of deadlocks. > 0 Lock timeout value. > 0 Number of locks that have timed out. > 0 Transaction timeout value. > 0 Number of transactions that have timed out. > 360KB The size of the lock region.. > 0 The number of region locks granted after waiting. > 315390 The number of region locks granted without waiting. > > Adding the users without a transaction works but slow. Adding the same number > of users within a transaction usually produces different "Maximum number of > locks at any one time.". Is this seemingly non-deterministic accumulation of > locks expected? > > Thanks, G?bor > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel > From ben at medianstrip.net Sat Feb 12 20:45:10 2005 From: ben at medianstrip.net (Ben) Date: Sat, 12 Feb 2005 15:45:10 -0500 (EST) Subject: [elephant-devel] running out of locks In-Reply-To: <200502121338.06201.mega@hotpop.com> References: <200502112248.13633.mega@hotpop.com> <20050211193640.K9331@contarex.medianstrip.net> <200502121338.06201.mega@hotpop.com> Message-ID: <20050212144437.R569@contarex.medianstrip.net> On Sat, 12 Feb 2005, [utf-8] G??bor Melis wrote: > A related problem - although not with elephant - is using cursors to iterate > over more than "Maximum number of locks possible" elements in a transaction. > Runs out of locks predictably using up 1 lock per object. If there is no > enclosing transaction it uses 1 lock max. Looking at > http://www.sleepycat.com/docs/ref/lock/max.html it sounds normal :-(. They > are going to support cursors releasing locks for passed objects, but until > then I'll iterate with no enclosing transaction: where did you find info on releasing locks on passed objects? > the only question is how to make sure there is no enclosing transaction? I'm > thinking maybe something like this would do: > > (defmacro without-transaction (() &body) > `(run-elephant-thread (lambda () , at body))) the current transaction is held in the *current-transaction* special. i think if you can do something like (defmacro without-transaction (&body) (let ((*current-transaction* +NULL-VOID+)) `, at body)) and the stuff in body should be not in a transaction. (i don't remember my backquote syntax.....) caveat: there is some hair with the way with-transaction works, so i can't 100% guarantee nesting with and withouts will be totally kosher. it is supposed to work though so if you run into problems with this let me know. take care, B From mega at hotpop.com Sun Feb 13 09:31:53 2005 From: mega at hotpop.com (=?utf-8?q?G=C3=A1bor_Melis?=) Date: Sun, 13 Feb 2005 10:31:53 +0100 Subject: [elephant-devel] running out of locks In-Reply-To: <20050212144437.R569@contarex.medianstrip.net> References: <200502112248.13633.mega@hotpop.com> <200502121338.06201.mega@hotpop.com> <20050212144437.R569@contarex.medianstrip.net> Message-ID: <200502131031.53821.mega@hotpop.com> On Saturday 12 February 2005 21:45, Ben wrote: > where did you find info on releasing locks on passed objects? It is in the same thread I sent a link for, search for 'behind'. > the current transaction is held in the *current-transaction* special. > i think if you can do something like > > (defmacro without-transaction (&body) > (let ((*current-transaction* +NULL-VOID+)) > `, at body)) > > and the stuff in body should be not in a transaction. (i don't > remember my backquote syntax.....) > > caveat: there is some hair with the way with-transaction works, so i > can't 100% guarantee nesting with and withouts will be totally kosher. > it is supposed to work though so if you run into problems with this > let me know. Thanks, G?bor > > take care, B From garry.steedman at gmail.com Mon Feb 14 03:52:16 2005 From: garry.steedman at gmail.com (garry.steedman at gmail.com) Date: Mon, 14 Feb 2005 03:52:16 +0000 (UTC) Subject: [elephant-devel] Problems installing elephant on gentoo linux/CMUCL Message-ID: Hello all, I was wondering if anyone's managed to get elephant working on gentoo linux/CMUCL? If yes, any pointers would be appreciated as I've tried the ebuild and a source tarball, but without any luck (my install box has all the required dependencies: ASDF, UFFI, libpthreads, Berkeley 4.2 - at least all the obvious ones!) I include the CMUCL backtrace below just in case anyone has any pointers... cheers, garry # to the generic function #. But the method has fewer required arguments than the generic function. [Condition of type KERNEL:SIMPLE-PROGRAM-ERROR] Restarts: 0: [CONTINUE] Return NIL from load of #p"/usr/lib/common-lisp/cmucl/elephant/src/metaclasses.x86f". 1: [RETRY] Retry performing # on #. 2: [ACCEPT] Continue, treating # on # as having been successful. 3: [ABORT] Abort handling SLIME request. 4: [ABORT] Return to Top-Level. Backtrace: 0: (PCL::CHECK-METHOD-ARG-INFO # #S(PCL::ARG-INFO :ARG-INFO-LAMBDA-LIST (PCL::CLASS PCL::INITARGS) :ARG-INFO-PRECEDENCE (0 1) :ARG-INFO-METATYPES (PCL:STANDARD-INSTANCE T) :ARG-INFO-NUMBER-OPTIONAL 0 :ARG-INFO-KEY/REST-P NIL ...) #) 1: (PCL::SET-ARG-INFO # :NEW-METHOD # :LAMBDA-LIST ...) 2: (PCL::REAL-ADD-METHOD # # NIL) 3: (PCL::LOAD-DEFMETHOD-INTERNAL STANDARD-METHOD PCL:DIRECT-SLOT-DEFINITION-CLASS NIL (ELEPHANT:PERSISTENT-METACLASS) ...) 4: (C::DO-CALL # 16 17 4 ...) 5: (COMMON-LISP::FOP-FUNCALL-FOR-EFFECT) 6: (COMMON-LISP::LOAD-GROUP #) 7: (COMMON-LISP::FASLOAD #) 8: (COMMON-LISP::INTERNAL-LOAD #p"/usr/lib/common-lisp/cmucl/elephant/src/metaclasses.x86f" #p"/usr/lib/common-lisp/cmucl/elephant/src/metaclasses.x86f" :ERROR :BINARY) 9: (COMMON-LISP::INTERNAL-LOAD #p"/usr/lib/common-lisp/cmucl/elephant/src/metaclasses.x86f" #p"/usr/lib/common-lisp/cmucl/elephant/src/metaclasses.x86f" :ERROR NIL) 10: (LOAD #p"/usr/lib/common-lisp/cmucl/elephant/src/metaclasses.x86f" :VERBOSE NIL :PRINT ...) 11: ("DEFMETHOD PERFORM (LOAD-OP CL-SOURCE-FILE)" #<#1=unused-arg> #<#1#> # #) 12: ("LAMBDA (PCL::EFFECTIVE-METHOD-GENSYM-0 PCL::EFFECTIVE-METHOD-GENSYM-1 PCL::EFFECTIVE-METHOD-GENSYM-2)" #<#1=unused-arg> #<#1#> # #) 13: (#:G0) 14: (ASDF:OPERATE ASDF:LOAD-OP :ELEPHANT) 15: (ASDF:OPERATE 2 ASDF:LOAD-OP :ELEPHANT)[:EXTERNAL] From ben at medianstrip.net Mon Feb 14 18:42:45 2005 From: ben at medianstrip.net (Ben) Date: Mon, 14 Feb 2005 13:42:45 -0500 (EST) Subject: [elephant-devel] Problems installing elephant on gentoo linux/CMUCL In-Reply-To: References: Message-ID: <20050214134009.V22431@contarex.medianstrip.net> what version of CMUCL are you using? what patches? because it looks like recently CMUCL and SBCL received patches to their MOP implementations, which seem to have affected elephant. note we're working on upgrading elephant at the moment, we hope to have a new testable version soon. (CMUCL / Linux is definitely a target -- I use CMUCL / FreeBSD myself.) MOP is a fuzzy area in most implementations.....a moving target! take care, B On Mon, 14 Feb 2005 garry.steedman at gmail.com wrote: > Hello all, > > I was wondering if anyone's managed to get elephant working on gentoo > linux/CMUCL? If yes, any pointers would be appreciated as I've tried the > ebuild and a source tarball, but without any luck (my install box has > all the required dependencies: ASDF, UFFI, libpthreads, Berkeley 4.2 - at > least all the obvious ones!) > > I include the CMUCL backtrace below just in case anyone has any pointers... > > cheers, > > garry > > > > # (ELEPHANT:PERSISTENT-METACLASS) > {480E3ACD}> to the generic function > # {28196C51}>. > But the method has fewer required arguments than the generic function. > [Condition of type KERNEL:SIMPLE-PROGRAM-ERROR] > > Restarts: > 0: [CONTINUE] Return NIL from load of > #p"/usr/lib/common-lisp/cmucl/elephant/src/metaclasses.x86f". > 1: [RETRY] Retry performing # on > #. > 2: [ACCEPT] Continue, treating # on > # as having been successful. > 3: [ABORT] Abort handling SLIME request. > 4: [ABORT] Return to Top-Level. > > Backtrace: > 0: (PCL::CHECK-METHOD-ARG-INFO # PCL:DIRECT-SLOT-DEFINITION-CLASS (3) {28196C51}> #S(PCL::ARG-INFO > :ARG-INFO-LAMBDA-LIST (PCL::CLASS PCL::INITARGS) :ARG-INFO-PRECEDENCE (0 1) > :ARG-INFO-METATYPES (PCL:STANDARD-INSTANCE T) :ARG-INFO-NUMBER-OPTIONAL 0 > :ARG-INFO-KEY/REST-P NIL ...) # PCL:DIRECT-SLOT-DEFINITION-CLASS (ELEPHANT:PERSISTENT-METACLASS) {480E3ACD}>) > 1: (PCL::SET-ARG-INFO # PCL:DIRECT-SLOT-DEFINITION-CLASS (3) {28196C51}> :NEW-METHOD > # (ELEPHANT:PERSISTENT-METACLASS) {480E3ACD}> :LAMBDA-LIST ...) > 2: (PCL::REAL-ADD-METHOD # PCL:DIRECT-SLOT-DEFINITION-CLASS (3) {28196C51}> # PCL:DIRECT-SLOT-DEFINITION-CLASS (ELEPHANT:PERSISTENT-METACLASS) {480E3ACD}> > NIL) > 3: (PCL::LOAD-DEFMETHOD-INTERNAL STANDARD-METHOD > PCL:DIRECT-SLOT-DEFINITION-CLASS NIL (ELEPHANT:PERSISTENT-METACLASS) ...) > 4: (C::DO-CALL # 16 17 4 ...) > 5: (COMMON-LISP::FOP-FUNCALL-FOR-EFFECT) > 6: (COMMON-LISP::LOAD-GROUP # "/usr/lib/common-lisp/cmucl/elephant/src/metaclasses.x86f">) > 7: (COMMON-LISP::FASLOAD # "/usr/lib/common-lisp/cmucl/elephant/src/metaclasses.x86f">) > 8: (COMMON-LISP::INTERNAL-LOAD > #p"/usr/lib/common-lisp/cmucl/elephant/src/metaclasses.x86f" > #p"/usr/lib/common-lisp/cmucl/elephant/src/metaclasses.x86f" :ERROR :BINARY) > 9: (COMMON-LISP::INTERNAL-LOAD > #p"/usr/lib/common-lisp/cmucl/elephant/src/metaclasses.x86f" > #p"/usr/lib/common-lisp/cmucl/elephant/src/metaclasses.x86f" :ERROR NIL) > 10: (LOAD #p"/usr/lib/common-lisp/cmucl/elephant/src/metaclasses.x86f" > :VERBOSE NIL :PRINT ...) > 11: ("DEFMETHOD PERFORM (LOAD-OP CL-SOURCE-FILE)" #<#1=unused-arg> #<#1#> > # # {4874D8FD}>) > 12: ("LAMBDA (PCL::EFFECTIVE-METHOD-GENSYM-0 PCL::EFFECTIVE-METHOD-GENSYM-1 > PCL::EFFECTIVE-METHOD-GENSYM-2)" #<#1=unused-arg> #<#1#> # {487FB1CD}> #) > 13: (#:G0) > 14: (ASDF:OPERATE ASDF:LOAD-OP :ELEPHANT) > 15: (ASDF:OPERATE 2 ASDF:LOAD-OP :ELEPHANT)[:EXTERNAL] > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel > From ben at medianstrip.net Mon Feb 14 18:51:19 2005 From: ben at medianstrip.net (Ben) Date: Mon, 14 Feb 2005 13:51:19 -0500 (EST) Subject: [elephant-devel] Elephant on win32 with VS.NET and acl70 In-Reply-To: <42099C83.3000508@gmail.com> References: <42099C83.3000508@gmail.com> Message-ID: <20050214134943.N22431@contarex.medianstrip.net> whoops, this got lost in my email spool. can you send me a backtrace / error log? i haven't had a chance to test with allegro 7.0 yet, nor the latest VS .NET stuff. i'm not sure when i'll get access to a windows box next, but it's on my list of things to do. take care, B On Tue, 8 Feb 2005, Andy Misc Cristina wrote: > Hi, > > I am trying to use elephant(0.2.1) in windows XP with allegro 7.0. I have > built db-4.2.52NC and the libsleepycat.c files in VS .NET 2k3, according to > the instructions in the install guide. However, allegro errors out on > loading libsleepycat. Are there any known issues with my version of visual > studio? I'm not sure what information is relevant here, but if someone could > tell me what would be useful, I'll be happy to provide it. > Thanks for any help, > Andy > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel > From blumberg at math.uchicago.edu Mon Feb 14 20:34:39 2005 From: blumberg at math.uchicago.edu (Andrew Blumberg) Date: Mon, 14 Feb 2005 14:34:39 -0600 (CST) Subject: [elephant-devel] Problems installing elephant on gentoo linux/CMUCL In-Reply-To: References: Message-ID: hey garry, so i'm afraid you're going to have to upgrade to cmucl 19a. in 18e, the definition of pcl:direct-slot-definition-class has signature (class initargs) rather than (class &rest initargs). this noncompliance was fixed subsequently. i guess you could also dike out the &rest in the elephant source, but i suspect there are other lurking pcl problems in the older implementation which may cause unhappiness. - andrew On Mon, 14 Feb 2005 garry.steedman at gmail.com wrote: > Hello all, > > I was wondering if anyone's managed to get elephant working on gentoo > linux/CMUCL? If yes, any pointers would be appreciated as I've tried the > ebuild and a source tarball, but without any luck (my install box has > all the required dependencies: ASDF, UFFI, libpthreads, Berkeley 4.2 - at > least all the obvious ones!) > > I include the CMUCL backtrace below just in case anyone has any pointers... > > cheers, > > garry > > > > # (ELEPHANT:PERSISTENT-METACLASS) > {480E3ACD}> to the generic function > # {28196C51}>. > But the method has fewer required arguments than the generic function. > [Condition of type KERNEL:SIMPLE-PROGRAM-ERROR] > > Restarts: > 0: [CONTINUE] Return NIL from load of > #p"/usr/lib/common-lisp/cmucl/elephant/src/metaclasses.x86f". > 1: [RETRY] Retry performing # on > #. > 2: [ACCEPT] Continue, treating # on > # as having been successful. > 3: [ABORT] Abort handling SLIME request. > 4: [ABORT] Return to Top-Level. > > Backtrace: > 0: (PCL::CHECK-METHOD-ARG-INFO # PCL:DIRECT-SLOT-DEFINITION-CLASS (3) {28196C51}> #S(PCL::ARG-INFO > :ARG-INFO-LAMBDA-LIST (PCL::CLASS PCL::INITARGS) :ARG-INFO-PRECEDENCE (0 1) > :ARG-INFO-METATYPES (PCL:STANDARD-INSTANCE T) :ARG-INFO-NUMBER-OPTIONAL 0 > :ARG-INFO-KEY/REST-P NIL ...) # PCL:DIRECT-SLOT-DEFINITION-CLASS (ELEPHANT:PERSISTENT-METACLASS) {480E3ACD}>) > 1: (PCL::SET-ARG-INFO # PCL:DIRECT-SLOT-DEFINITION-CLASS (3) {28196C51}> :NEW-METHOD > # (ELEPHANT:PERSISTENT-METACLASS) {480E3ACD}> :LAMBDA-LIST ...) > 2: (PCL::REAL-ADD-METHOD # PCL:DIRECT-SLOT-DEFINITION-CLASS (3) {28196C51}> # PCL:DIRECT-SLOT-DEFINITION-CLASS (ELEPHANT:PERSISTENT-METACLASS) {480E3ACD}> > NIL) > 3: (PCL::LOAD-DEFMETHOD-INTERNAL STANDARD-METHOD > PCL:DIRECT-SLOT-DEFINITION-CLASS NIL (ELEPHANT:PERSISTENT-METACLASS) ...) > 4: (C::DO-CALL # 16 17 4 ...) > 5: (COMMON-LISP::FOP-FUNCALL-FOR-EFFECT) > 6: (COMMON-LISP::LOAD-GROUP # "/usr/lib/common-lisp/cmucl/elephant/src/metaclasses.x86f">) > 7: (COMMON-LISP::FASLOAD # "/usr/lib/common-lisp/cmucl/elephant/src/metaclasses.x86f">) > 8: (COMMON-LISP::INTERNAL-LOAD > #p"/usr/lib/common-lisp/cmucl/elephant/src/metaclasses.x86f" > #p"/usr/lib/common-lisp/cmucl/elephant/src/metaclasses.x86f" :ERROR :BINARY) > 9: (COMMON-LISP::INTERNAL-LOAD > #p"/usr/lib/common-lisp/cmucl/elephant/src/metaclasses.x86f" > #p"/usr/lib/common-lisp/cmucl/elephant/src/metaclasses.x86f" :ERROR NIL) > 10: (LOAD #p"/usr/lib/common-lisp/cmucl/elephant/src/metaclasses.x86f" > :VERBOSE NIL :PRINT ...) > 11: ("DEFMETHOD PERFORM (LOAD-OP CL-SOURCE-FILE)" #<#1=unused-arg> #<#1#> > # # {4874D8FD}>) > 12: ("LAMBDA (PCL::EFFECTIVE-METHOD-GENSYM-0 PCL::EFFECTIVE-METHOD-GENSYM-1 > PCL::EFFECTIVE-METHOD-GENSYM-2)" #<#1=unused-arg> #<#1#> # {487FB1CD}> #) > 13: (#:G0) > 14: (ASDF:OPERATE ASDF:LOAD-OP :ELEPHANT) > 15: (ASDF:OPERATE 2 ASDF:LOAD-OP :ELEPHANT)[:EXTERNAL] > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel > From blumberg at math.uchicago.edu Mon Feb 14 19:57:20 2005 From: blumberg at math.uchicago.edu (Andrew Blumberg) Date: Mon, 14 Feb 2005 13:57:20 -0600 (CST) Subject: [elephant-devel] Problems installing elephant on gentoo linux/CMUCL In-Reply-To: <20050214134009.V22431@contarex.medianstrip.net> References: <20050214134009.V22431@contarex.medianstrip.net> Message-ID: hi garry, so direct-slot-definition-class should (according to the spec) have the signature ((foo class) &rest initargs), and our specialization of that method in the elephant sources conforms to this. your backtrace below indicates that your cmucl seems to disagree. . . in addition to indicating the version information about cmucl, it would be extremely helpful if you could find out what argument pattern that pcl:direct-slot-definition-class thinks it wants in your setup. i'd be sort of surprised if the new patches borked this somehow; i'm guessing maybe you have an older pcl implementation. - andrew On Mon, 14 Feb 2005, Ben wrote: > what version of CMUCL are you using? what patches? because it looks > like recently CMUCL and SBCL received patches to their MOP > implementations, which seem to have affected elephant. note we're > working on upgrading elephant at the moment, we hope to have a new > testable version soon. (CMUCL / Linux is definitely a target -- I use > CMUCL / FreeBSD myself.) > > MOP is a fuzzy area in most implementations.....a moving target! > > take care, B > > On Mon, 14 Feb 2005 garry.steedman at gmail.com wrote: > >> Hello all, >> >> I was wondering if anyone's managed to get elephant working on gentoo >> linux/CMUCL? If yes, any pointers would be appreciated as I've tried the >> ebuild and a source tarball, but without any luck (my install box has >> all the required dependencies: ASDF, UFFI, libpthreads, Berkeley 4.2 - at >> least all the obvious ones!) >> >> I include the CMUCL backtrace below just in case anyone has any pointers... >> >> cheers, >> >> garry >> >> >> >> #> (ELEPHANT:PERSISTENT-METACLASS) >> {480E3ACD}> to the generic function >> #> {28196C51}>. >> But the method has fewer required arguments than the generic function. >> [Condition of type KERNEL:SIMPLE-PROGRAM-ERROR] >> >> Restarts: >> 0: [CONTINUE] Return NIL from load of >> #p"/usr/lib/common-lisp/cmucl/elephant/src/metaclasses.x86f". >> 1: [RETRY] Retry performing # on >> #. >> 2: [ACCEPT] Continue, treating # on >> # as having been successful. >> 3: [ABORT] Abort handling SLIME request. >> 4: [ABORT] Return to Top-Level. >> >> Backtrace: >> 0: (PCL::CHECK-METHOD-ARG-INFO #> PCL:DIRECT-SLOT-DEFINITION-CLASS (3) {28196C51}> #S(PCL::ARG-INFO >> :ARG-INFO-LAMBDA-LIST (PCL::CLASS PCL::INITARGS) :ARG-INFO-PRECEDENCE (0 1) >> :ARG-INFO-METATYPES (PCL:STANDARD-INSTANCE T) :ARG-INFO-NUMBER-OPTIONAL 0 >> :ARG-INFO-KEY/REST-P NIL ...) #> PCL:DIRECT-SLOT-DEFINITION-CLASS (ELEPHANT:PERSISTENT-METACLASS) >> {480E3ACD}>) >> 1: (PCL::SET-ARG-INFO #> PCL:DIRECT-SLOT-DEFINITION-CLASS (3) {28196C51}> :NEW-METHOD >> #> (ELEPHANT:PERSISTENT-METACLASS) {480E3ACD}> :LAMBDA-LIST ...) >> 2: (PCL::REAL-ADD-METHOD #> PCL:DIRECT-SLOT-DEFINITION-CLASS (3) {28196C51}> #> PCL:DIRECT-SLOT-DEFINITION-CLASS (ELEPHANT:PERSISTENT-METACLASS) >> {480E3ACD}> NIL) >> 3: (PCL::LOAD-DEFMETHOD-INTERNAL STANDARD-METHOD >> PCL:DIRECT-SLOT-DEFINITION-CLASS NIL (ELEPHANT:PERSISTENT-METACLASS) ...) >> 4: (C::DO-CALL # 16 17 4 ...) >> 5: (COMMON-LISP::FOP-FUNCALL-FOR-EFFECT) >> 6: (COMMON-LISP::LOAD-GROUP #> "/usr/lib/common-lisp/cmucl/elephant/src/metaclasses.x86f">) >> 7: (COMMON-LISP::FASLOAD #> "/usr/lib/common-lisp/cmucl/elephant/src/metaclasses.x86f">) >> 8: (COMMON-LISP::INTERNAL-LOAD >> #p"/usr/lib/common-lisp/cmucl/elephant/src/metaclasses.x86f" >> #p"/usr/lib/common-lisp/cmucl/elephant/src/metaclasses.x86f" :ERROR >> :BINARY) >> 9: (COMMON-LISP::INTERNAL-LOAD >> #p"/usr/lib/common-lisp/cmucl/elephant/src/metaclasses.x86f" >> #p"/usr/lib/common-lisp/cmucl/elephant/src/metaclasses.x86f" :ERROR NIL) >> 10: (LOAD #p"/usr/lib/common-lisp/cmucl/elephant/src/metaclasses.x86f" >> :VERBOSE NIL :PRINT ...) >> 11: ("DEFMETHOD PERFORM (LOAD-OP CL-SOURCE-FILE)" #<#1=unused-arg> #<#1#> >> # #> {4874D8FD}>) >> 12: ("LAMBDA (PCL::EFFECTIVE-METHOD-GENSYM-0 >> PCL::EFFECTIVE-METHOD-GENSYM-1 PCL::EFFECTIVE-METHOD-GENSYM-2)" >> #<#1=unused-arg> #<#1#> # >> #) >> 13: (#:G0) >> 14: (ASDF:OPERATE ASDF:LOAD-OP :ELEPHANT) >> 15: (ASDF:OPERATE 2 ASDF:LOAD-OP :ELEPHANT)[:EXTERNAL] >> _______________________________________________ >> elephant-devel site list >> elephant-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/elephant-devel >> > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel > From acristin.misc at gmail.com Tue Feb 15 00:56:36 2005 From: acristin.misc at gmail.com (Andy Misc Cristina) Date: Mon, 14 Feb 2005 18:56:36 -0600 Subject: [elephant-devel] Elephant on win32 with VS.NET and acl70 In-Reply-To: <20050214134943.N22431@contarex.medianstrip.net> References: <42099C83.3000508@gmail.com> <20050214134943.N22431@contarex.medianstrip.net> Message-ID: <421148C4.3080303@gmail.com> Sure, although I have had some more success. First I learned that case sensitivity in common lisp isn't such a great idea, so I'll be using the alisp more than the mlisp in the future. Second, I learned that just putting the dlls in the elephant package directory will make things load. Now I get a completely different error that I will try to debug tonight, but I'll go ahead and put it here too, in case its something someone has dealt with before. ; While compiling (:TOP-LEVEL-FORM "sleepycat.lisp" 4131): Error: `(:BTREE 1)' is not of the expected type `NUMBER' [condition type: TYPE-ERROR] [1] SLEEPYCAT(3): :current (DEF-ENUM (DEF-ENUM DBTYPE ('(:BTREE 1) :HASH :QUEUE :RECNO :UNKNOWN)) #) I'm thinking this is probably a uffi to allegro issue, but who knows? I sure don't :-) Andy Ben wrote: > whoops, this got lost in my email spool. > > can you send me a backtrace / error log? > > i haven't had a chance to test with allegro 7.0 yet, nor the latest VS > .NET stuff. i'm not sure when i'll get access to a windows box next, > but it's on my list of things to do. > > take care, B > > On Tue, 8 Feb 2005, Andy Misc Cristina wrote: > >> Hi, >> >> I am trying to use elephant(0.2.1) in windows XP with allegro 7.0. >> I have built db-4.2.52NC and the libsleepycat.c files in VS .NET 2k3, >> according to the instructions in the install guide. However, allegro >> errors out on loading libsleepycat. Are there any known issues with >> my version of visual studio? I'm not sure what information is >> relevant here, but if someone could tell me what would be useful, >> I'll be happy to provide it. Thanks for any help, >> Andy >> _______________________________________________ >> elephant-devel site list >> elephant-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/elephant-devel >> > From garry.steedman at gmail.com Mon Feb 14 20:41:58 2005 From: garry.steedman at gmail.com (garry.steedman at gmail.com) Date: Mon, 14 Feb 2005 20:41:58 +0000 (UTC) Subject: [elephant-devel] Problems installing elephant on gentoo linux/CMUCL In-Reply-To: <20050214134009.V22431@contarex.medianstrip.net> References: <20050214134009.V22431@contarex.medianstrip.net> Message-ID: Hi Ben, CMUCL 18e + CLOS 18e is what I appear to be running... any idea what the timeline is for yr next release & whether this problem will be fixed? cheers, garry On Mon, 14 Feb 2005, Ben wrote: > what version of CMUCL are you using? what patches? because it looks > like recently CMUCL and SBCL received patches to their MOP > implementations, which seem to have affected elephant. note we're > working on upgrading elephant at the moment, we hope to have a new > testable version soon. (CMUCL / Linux is definitely a target -- I use > CMUCL / FreeBSD myself.) > > MOP is a fuzzy area in most implementations.....a moving target! > > take care, B > > On Mon, 14 Feb 2005 garry.steedman at gmail.com wrote: > >> Hello all, >> >> I was wondering if anyone's managed to get elephant working on gentoo >> linux/CMUCL? If yes, any pointers would be appreciated as I've tried the >> ebuild and a source tarball, but without any luck (my install box has >> all the required dependencies: ASDF, UFFI, libpthreads, Berkeley 4.2 - at >> least all the obvious ones!) >> >> I include the CMUCL backtrace below just in case anyone has any >> pointers... >> >> cheers, >> >> garry >> >> >> >> #> (ELEPHANT:PERSISTENT-METACLASS) >> {480E3ACD}> to the generic function >> #> {28196C51}>. >> But the method has fewer required arguments than the generic function. >> [Condition of type KERNEL:SIMPLE-PROGRAM-ERROR] >> >> Restarts: >> 0: [CONTINUE] Return NIL from load of >> #p"/usr/lib/common-lisp/cmucl/elephant/src/metaclasses.x86f". >> 1: [RETRY] Retry performing # on >> #. >> 2: [ACCEPT] Continue, treating # on >> # as having been >> successful. >> 3: [ABORT] Abort handling SLIME request. >> 4: [ABORT] Return to Top-Level. >> >> Backtrace: >> 0: (PCL::CHECK-METHOD-ARG-INFO #> PCL:DIRECT-SLOT-DEFINITION-CLASS (3) {28196C51}> #S(PCL::ARG-INFO >>> ARG-INFO-LAMBDA-LIST (PCL::CLASS PCL::INITARGS) :ARG-INFO-PRECEDENCE (0 >>> 1) ARG-INFO-METATYPES (PCL:STANDARD-INSTANCE T) >>> :ARG-INFO-NUMBER-OPTIONAL 0 ARG-INFO-KEY/REST-P NIL ...) >>> #> PCL:DIRECT-SLOT-DEFINITION-CLASS (ELEPHANT:PERSISTENT-METACLASS) >> {480E3ACD}>) >> 1: (PCL::SET-ARG-INFO #> PCL:DIRECT-SLOT-DEFINITION-CLASS (3) {28196C51}> :NEW-METHOD >> #> (ELEPHANT:PERSISTENT-METACLASS) {480E3ACD}> :LAMBDA-LIST ...) >> 2: (PCL::REAL-ADD-METHOD #>> DIRECT-SLOT-DEFINITION-CLASS (3) {28196C51}> #>> DIRECT-SLOT-DEFINITION-CLASS (ELEPHANT:PERSISTENT-METACLASS) >>> {480E3ACD}> >> NIL) >> 3: (PCL::LOAD-DEFMETHOD-INTERNAL STANDARD-METHOD >> PCL:DIRECT-SLOT-DEFINITION-CLASS NIL (ELEPHANT:PERSISTENT-METACLASS) ...) >> 4: (C::DO-CALL # 16 17 4 ...) >> 5: (COMMON-LISP::FOP-FUNCALL-FOR-EFFECT) >> 6: (COMMON-LISP::LOAD-GROUP #> "/usr/lib/common-lisp/cmucl/elephant/src/metaclasses.x86f">) >> 7: (COMMON-LISP::FASLOAD #> "/usr/lib/common-lisp/cmucl/elephant/src/metaclasses.x86f">) >> 8: (COMMON-LISP::INTERNAL-LOAD >>> p"/usr/lib/common-lisp/cmucl/elephant/src/metaclasses.x86f" >>> p"/usr/lib/common-lisp/cmucl/elephant/src/metaclasses.x86f" :ERROR >>> :BINARY) >> 9: (COMMON-LISP::INTERNAL-LOAD >>> p"/usr/lib/common-lisp/cmucl/elephant/src/metaclasses.x86f" >>> p"/usr/lib/common-lisp/cmucl/elephant/src/metaclasses.x86f" :ERROR NIL) >> 10: (LOAD #p"/usr/lib/common-lisp/cmucl/elephant/src/metaclasses.x86f" >>> VERBOSE NIL :PRINT ...) >> 11: ("DEFMETHOD PERFORM (LOAD-OP CL-SOURCE-FILE)" #<#1=unused-arg> #<#1#> >> # #> {4874D8FD}>) >> 12: ("LAMBDA (PCL::EFFECTIVE-METHOD-GENSYM-0 >> PCL::EFFECTIVE-METHOD-GENSYM-1 PCL::EFFECTIVE-METHOD-GENSYM-2)" >> #<#1=unused-arg> #<#1#> # >> #) >> 13: (#:G0) >> 14: (ASDF:OPERATE ASDF:LOAD-OP :ELEPHANT) >> 15: (ASDF:OPERATE 2 ASDF:LOAD-OP :ELEPHANT)[:EXTERNAL] >> _______________________________________________ >> elephant-devel site list >> elephant-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/elephant-devel >> > > From mega at hotpop.com Tue Feb 15 11:35:22 2005 From: mega at hotpop.com (=?utf-8?q?G=C3=A1bor_Melis?=) Date: Tue, 15 Feb 2005 12:35:22 +0100 Subject: [elephant-devel] secondary indeces In-Reply-To: <20041123121954.E79429@contarex.medianstrip.net> References: <04Nov23.114842cet.336250@fwall.essnet.se> <20041123121954.E79429@contarex.medianstrip.net> Message-ID: <200502151235.22812.mega@hotpop.com> On Tuesday 23 November 2004 18:29, Ben wrote: > i don't know how to handle this in any general fashion, because > objects don't know when they are being stored in a btree with a > secondary index. it's great you bring up this point, i'll definitely > think about how to do it. one might consider registering mutation > functions on object insert which would update secondary indices. > however it seems nigh impossible to do this right. a persisted object > may be loaded without the container btree loaded, in which case > elephant has no idea that some (not loaded) secondary index needs to > be updated. i will look into this issue. > > if you want to mutate the secondary index by hand, i think on the > store-controller object there are two slots which are handles to the > secondary index. one is opened as a secondary index, the other opened > as a raw btree. you can update the raw one manually, but be careful! For the record, I now use this to manage my indices from 'user space': (defmacro with-reindexing ((key btree) &body body) "Force update of indices by removing and readding KEY." (with-gensyms (value k b) `(with-transaction () (let* ((,k ,key) (,b ,btree) (,value (get-value ,k ,b))) (remove-kv ,k ,b) (unwind-protect (progn , at body) (setf (get-value ,k ,b) ,value)))))) In the scenario with a user class having an activation-code slot that is the key of the secondary index it works like this: (defmethod (setf user-activation-code) :around (value (user user)) (with-reindexed-user (user) (call-next-method))) With-reindexed-user is just a convenience macro on top of with-reindexing that looks up the key and btree for the user object. Not a pretty sight, but it is the best I could come up with and it is buried in the bowels of the business model where no client code ventures. G?bor From garry.steedman at gmail.com Tue Feb 15 01:34:13 2005 From: garry.steedman at gmail.com (garry.steedman at gmail.com) Date: Tue, 15 Feb 2005 01:34:13 +0000 (UTC) Subject: [elephant-devel] Problems installing elephant on gentoo linux/CMUCL In-Reply-To: References: Message-ID: Hi Andrew + Ben, thanks for that: I will try to get 19a running and try installing elephant again... cheers, garry On Mon, 14 Feb 2005, Andrew Blumberg wrote: > hey garry, > > so i'm afraid you're going to have to upgrade to cmucl 19a. in 18e, > the definition of pcl:direct-slot-definition-class has signature (class > initargs) rather than (class &rest initargs). this noncompliance was fixed > subsequently. i guess you could also dike out the &rest in the elephant > source, but i suspect there are other lurking pcl problems in the older > implementation which may cause unhappiness. > > - andrew > > On Mon, 14 Feb 2005 garry.steedman at gmail.com wrote: > >> Hello all, >> >> I was wondering if anyone's managed to get elephant working on gentoo >> linux/CMUCL? If yes, any pointers would be appreciated as I've tried the >> ebuild and a source tarball, but without any luck (my install box has >> all the required dependencies: ASDF, UFFI, libpthreads, Berkeley 4.2 - at >> least all the obvious ones!) >> >> I include the CMUCL backtrace below just in case anyone has any >> pointers... >> >> cheers, >> >> garry >> >> >> >> #> (ELEPHANT:PERSISTENT-METACLASS) >> {480E3ACD}> to the generic function >> #> {28196C51}>. >> But the method has fewer required arguments than the generic function. >> [Condition of type KERNEL:SIMPLE-PROGRAM-ERROR] >> >> Restarts: >> 0: [CONTINUE] Return NIL from load of >> #p"/usr/lib/common-lisp/cmucl/elephant/src/metaclasses.x86f". >> 1: [RETRY] Retry performing # on >> #. >> 2: [ACCEPT] Continue, treating # on >> # as having been >> successful. >> 3: [ABORT] Abort handling SLIME request. >> 4: [ABORT] Return to Top-Level. >> >> Backtrace: >> 0: (PCL::CHECK-METHOD-ARG-INFO #> PCL:DIRECT-SLOT-DEFINITION-CLASS (3) {28196C51}> #S(PCL::ARG-INFO >>> ARG-INFO-LAMBDA-LIST (PCL::CLASS PCL::INITARGS) :ARG-INFO-PRECEDENCE (0 >>> 1) ARG-INFO-METATYPES (PCL:STANDARD-INSTANCE T) >>> :ARG-INFO-NUMBER-OPTIONAL 0 ARG-INFO-KEY/REST-P NIL ...) >>> #> PCL:DIRECT-SLOT-DEFINITION-CLASS (ELEPHANT:PERSISTENT-METACLASS) >> {480E3ACD}>) >> 1: (PCL::SET-ARG-INFO #> PCL:DIRECT-SLOT-DEFINITION-CLASS (3) {28196C51}> :NEW-METHOD >> #> (ELEPHANT:PERSISTENT-METACLASS) {480E3ACD}> :LAMBDA-LIST ...) >> 2: (PCL::REAL-ADD-METHOD #>> DIRECT-SLOT-DEFINITION-CLASS (3) {28196C51}> #>> DIRECT-SLOT-DEFINITION-CLASS (ELEPHANT:PERSISTENT-METACLASS) >>> {480E3ACD}> >> NIL) >> 3: (PCL::LOAD-DEFMETHOD-INTERNAL STANDARD-METHOD >> PCL:DIRECT-SLOT-DEFINITION-CLASS NIL (ELEPHANT:PERSISTENT-METACLASS) ...) >> 4: (C::DO-CALL # 16 17 4 ...) >> 5: (COMMON-LISP::FOP-FUNCALL-FOR-EFFECT) >> 6: (COMMON-LISP::LOAD-GROUP #> "/usr/lib/common-lisp/cmucl/elephant/src/metaclasses.x86f">) >> 7: (COMMON-LISP::FASLOAD #> "/usr/lib/common-lisp/cmucl/elephant/src/metaclasses.x86f">) >> 8: (COMMON-LISP::INTERNAL-LOAD >>> p"/usr/lib/common-lisp/cmucl/elephant/src/metaclasses.x86f" >>> p"/usr/lib/common-lisp/cmucl/elephant/src/metaclasses.x86f" :ERROR >>> :BINARY) >> 9: (COMMON-LISP::INTERNAL-LOAD >>> p"/usr/lib/common-lisp/cmucl/elephant/src/metaclasses.x86f" >>> p"/usr/lib/common-lisp/cmucl/elephant/src/metaclasses.x86f" :ERROR NIL) >> 10: (LOAD #p"/usr/lib/common-lisp/cmucl/elephant/src/metaclasses.x86f" >>> VERBOSE NIL :PRINT ...) >> 11: ("DEFMETHOD PERFORM (LOAD-OP CL-SOURCE-FILE)" #<#1=unused-arg> #<#1#> >> # #> {4874D8FD}>) >> 12: ("LAMBDA (PCL::EFFECTIVE-METHOD-GENSYM-0 >> PCL::EFFECTIVE-METHOD-GENSYM-1 PCL::EFFECTIVE-METHOD-GENSYM-2)" >> #<#1=unused-arg> #<#1#> # >> #) >> 13: (#:G0) >> 14: (ASDF:OPERATE ASDF:LOAD-OP :ELEPHANT) >> 15: (ASDF:OPERATE 2 ASDF:LOAD-OP :ELEPHANT)[:EXTERNAL] >> _______________________________________________ >> elephant-devel site list >> elephant-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/elephant-devel >> > > From aml at gia.ist.utl.pt Fri Feb 18 14:58:25 2005 From: aml at gia.ist.utl.pt (Antonio Menezes Leitao) Date: Fri, 18 Feb 2005 14:58:25 +0000 Subject: [elephant-devel] Berkeley DB error: Unknown error: -5 Message-ID: <87brahap9q.fsf@gia.ist.utl.pt> Hi, Hi. Anyone has experienced a "Berkeley DB error: Unknown error: -5"? It is caused by ele::next-oid calling sleepycat::next-counter. Any quick solution? Thanks in advance, Ant?nio Leit?o. From ben at medianstrip.net Fri Feb 18 22:20:08 2005 From: ben at medianstrip.net (Ben) Date: Fri, 18 Feb 2005 17:20:08 -0500 (EST) Subject: [elephant-devel] running out of locks In-Reply-To: <200502181415.03338.mega@hotpop.com> References: <200502112248.13633.mega@hotpop.com> <200502131031.53821.mega@hotpop.com> <20050217174259.J86399@contarex.medianstrip.net> <200502181415.03338.mega@hotpop.com> Message-ID: <20050218171904.B1858@contarex.medianstrip.net> have you tried using degree-2 cursors iterating over a btree? on adding lots of users: is there a reason why they have to be added all in one transaction? can you break up the transaction into smaller chunks? B On Fri, 18 Feb 2005, [utf-8] G??bor Melis wrote: > On Thursday 17 February 2005 23:44, you wrote: >> here's the tarball, let me know if you have problems - B > > Played a bit with it and it is mostly fine: > > - db4.3: OK > - sbcl + unicode: Woohoo > - degree 2 isolation: Cool > - sequences: don't know. My tests of adding many users still consume a wildly > varying number of locks. Maybe it's something else. > > G > From ben at medianstrip.net Fri Feb 18 22:21:29 2005 From: ben at medianstrip.net (Ben) Date: Fri, 18 Feb 2005 17:21:29 -0500 (EST) Subject: [elephant-devel] running out of locks In-Reply-To: <200502181702.31393.mega@hotpop.com> References: <200502112248.13633.mega@hotpop.com> <200502131031.53821.mega@hotpop.com> <20050217174259.J86399@contarex.medianstrip.net> <200502181702.31393.mega@hotpop.com> Message-ID: <20050218172013.G1858@contarex.medianstrip.net> that is in some ways intentional, as i didn't want to cons for the "most often used case." (multiple-value-bind conses.) i suppose the right answer is to provide a "with-multiplve-values-transaction" macro. should be easy to create from the original one. i'll look into this. take care, B On Fri, 18 Feb 2005, [utf-8] G??bor Melis wrote: > On Thursday 17 February 2005 23:44, you wrote: >> here's the tarball, let me know if you have problems - B > > oh, and the with-transaction multiple value bug is still there > From ben at medianstrip.net Fri Feb 18 22:24:13 2005 From: ben at medianstrip.net (Ben) Date: Fri, 18 Feb 2005 17:24:13 -0500 (EST) Subject: [elephant-devel] running out of locks In-Reply-To: <200502182229.45406.mega@hotpop.com> References: <200502112248.13633.mega@hotpop.com> <200502131031.53821.mega@hotpop.com> <20050217174259.J86399@contarex.medianstrip.net> <200502182229.45406.mega@hotpop.com> Message-ID: <20050218172141.B1858@contarex.medianstrip.net> that's a good idea. i am going to add a "degree-2" keyword to map-btree, and use it on add-index. i'll send you a patch as soon as my connection gets better. if you could test it that would be great. take care, B On Fri, 18 Feb 2005, [utf-8] G??bor Melis wrote: > On Thursday 17 February 2005 23:44, you wrote: >> here's the tarball, let me know if you have problems - B > > I got the feedback fever. > > add-index with populate maps the btree and runs out of locks here :-) > degree-2? > From ben at medianstrip.net Fri Feb 18 22:33:54 2005 From: ben at medianstrip.net (Ben) Date: Fri, 18 Feb 2005 17:33:54 -0500 (EST) Subject: [elephant-devel] Berkeley DB error: Unknown error: -5 In-Reply-To: <87brahap9q.fsf@gia.ist.utl.pt> References: <87brahap9q.fsf@gia.ist.utl.pt> Message-ID: <20050218173254.O3493@contarex.medianstrip.net> it looks like you're running into the same problem gabor melis did. we're in the process of cutting a new release which should fix your problem, probably by this weekend. if you can't wait, i can send you an untested tarball. take care, B On Fri, 18 Feb 2005, Antonio Menezes Leitao wrote: > Hi, > > Hi. Anyone has experienced a "Berkeley DB error: Unknown error: -5"? > > It is caused by ele::next-oid calling sleepycat::next-counter. > > Any quick solution? > > Thanks in advance, > > Ant?nio Leit?o. > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel > From ben at medianstrip.net Fri Feb 18 22:32:44 2005 From: ben at medianstrip.net (Ben) Date: Fri, 18 Feb 2005 17:32:44 -0500 (EST) Subject: [elephant-devel] running out of locks In-Reply-To: <200502182229.45406.mega@hotpop.com> References: <200502112248.13633.mega@hotpop.com> <200502131031.53821.mega@hotpop.com> <20050217174259.J86399@contarex.medianstrip.net> <200502182229.45406.mega@hotpop.com> Message-ID: <20050218173141.U3493@contarex.medianstrip.net> try this one? B On Fri, 18 Feb 2005, [utf-8] G?bor Melis wrote: > On Thursday 17 February 2005 23:44, you wrote: >> here's the tarball, let me know if you have problems - B > > I got the feedback fever. > > add-index with populate maps the btree and runs out of locks here :-) > degree-2? > -------------- next part -------------- ;;; -*- Mode: Lisp; Syntax: ANSI-Common-Lisp; Base: 10 -*- ;;; ;;; collections.lisp -- view Berkeley DBs as Lisp collections ;;; ;;; Initial version 8/26/2004 by Ben Lee ;;; ;;; ;;; part of ;;; ;;; Elephant: an object-oriented database for Common Lisp ;;; ;;; Copyright (c) 2004 by Andrew Blumberg and Ben Lee ;;; ;;; ;;; This program is released under the following license ;;; ("GPL"). For differenct licensing terms, contact the ;;; copyright holders. ;;; ;;; This program is free software; you can redistribute it ;;; and/or modify it under the terms of the GNU General ;;; Public License as published by the Free Software ;;; Foundation; either version 2 of the License, or (at ;;; your option) any later version. ;;; ;;; This program is distributed in the hope that it will be ;;; useful, but WITHOUT ANY WARRANTY; without even the ;;; implied warranty of MERCHANTABILITY or FITNESS FOR A ;;; PARTICULAR PURPOSE. See the GNU General Public License ;;; for more details. ;;; ;;; The GNU General Public License can be found in the file ;;; LICENSE which should have been distributed with this ;;; code. It can also be found at ;;; ;;; http://www.opensource.org/licenses/gpl-license.php ;;; ;;; You should have received a copy of the GNU General ;;; Public License along with this program; if not, write ;;; to the Free Software Foundation, Inc., 59 Temple Place, ;;; Suite 330, Boston, MA 02111-1307 USA ;;; (in-package "ELEPHANT") ;;; collection types ;;; we're slot-less (defclass persistent-collection (persistent) () (:documentation "Abstract superclass of all collection types.")) ;;; btree access (defclass btree (persistent-collection) () (:documentation "A hash-table like interface to a BTree, which stores things in a semi-ordered fashion.")) (defgeneric get-value (key bt) (:documentation "Get a value from a Btree.")) (defgeneric (setf get-value) (value key bt) (:documentation "Put a key / value pair into a BTree.")) (defgeneric remove-kv (key bt) (:documentation "Remove a key / value pair from a BTree.")) (defmethod get-value (key (bt btree)) (declare (optimize (speed 3))) (with-buffer-streams (key-buf value-buf) (buffer-write-int (oid bt) key-buf) (serialize key key-buf) (let ((buf (db-get-key-buffered (controller-btrees *store-controller*) key-buf value-buf))) (if buf (values (deserialize buf) T) (values nil nil))))) (defmethod (setf get-value) (value key (bt btree)) (declare (optimize (speed 3))) (with-buffer-streams (key-buf value-buf) (buffer-write-int (oid bt) key-buf) (serialize key key-buf) (serialize value value-buf) (db-put-buffered (controller-btrees *store-controller*) key-buf value-buf :auto-commit *auto-commit*) value)) (defmethod remove-kv (key (bt btree)) (declare (optimize (speed 3))) (with-buffer-streams (key-buf) (buffer-write-int (oid bt) key-buf) (serialize key key-buf) (db-delete-buffered (controller-btrees *store-controller*) key-buf :auto-commit *auto-commit*))) ;; Secondary indices (defclass indexed-btree (btree) ((indices :accessor indices :initform (make-hash-table)) (indices-cache :accessor indices-cache :initform (make-hash-table) :transient t)) (:metaclass persistent-metaclass) (:documentation "A BTree which supports secondary indices.")) (defmethod shared-initialize :after ((instance indexed-btree) slot-names &rest rest) (declare (ignore slot-names rest)) (setf (indices-cache instance) (indices instance))) (defgeneric add-index (bt &key index-name key-form populate) (:documentation "Add a secondary index. The indices are stored in an eq hash-table, so the index-name should be a symbol. key-form should be a symbol naming a function, or a list which defines a lambda -- actual functions aren't supported. The function should take 3 arguments: the secondary DB, primary key and value, and return two values: a boolean indicating whether to index this key / value, and the secondary key if so. If populate = t it will fill in secondary keys for existing primary entries (may be expensive!)")) (defgeneric get-index (bt index-name) (:documentation "Get a named index.")) (defgeneric remove-index (bt index-name) (:documentation "Remove a named index.")) (defmethod add-index ((bt indexed-btree) &key index-name key-form populate) (if (and (not (null index-name)) (symbolp index-name) (or (symbolp key-form) (listp key-form))) (let ((indices (indices bt)) (index (make-instance 'btree-index :primary bt :key-form key-form))) (setf (gethash index-name (indices-cache bt)) index) (setf (gethash index-name indices) index) (setf (indices bt) indices) (when populate (let ((key-fn (key-fn index))) (with-buffer-streams (primary-buf secondary-buf) (with-transaction () (map-btree #'(lambda (k v) (multiple-value-bind (index? secondary-key) (funcall key-fn index k v) (when index? (buffer-write-int (oid bt) primary-buf) (serialize k primary-buf) (buffer-write-int (oid index) secondary-buf) (serialize secondary-key secondary-buf) ;; should silently do nothing if ;; the key/value already exists (db-put-buffered (controller-indices *store-controller*) secondary-buf primary-buf) (reset-buffer-stream primary-buf) (reset-buffer-stream secondary-buf)))) bt :degree-2 t))))) index) (error "Invalid index initargs!"))) (defmethod get-index ((bt indexed-btree) index-name) (gethash index-name (indices-cache bt))) (defmethod remove-index ((bt indexed-btree) index-name) (remhash index-name (indices-cache bt)) (let ((indices (indices bt))) (remhash index-name indices) (setf (indices bt) indices))) (defmethod (setf get-value) (value key (bt indexed-btree)) "Set a key / value pair, and update secondary indices." (declare (optimize (speed 3))) (let ((indices (indices-cache bt))) (with-buffer-streams (key-buf value-buf secondary-buf) (buffer-write-int (oid bt) key-buf) (serialize key key-buf) (serialize value value-buf) (with-transaction () (db-put-buffered (controller-btrees *store-controller*) key-buf value-buf) (loop for index being the hash-value of indices do (multiple-value-bind (index? secondary-key) (funcall (key-fn index) index key value) (when index? (buffer-write-int (oid index) secondary-buf) (serialize secondary-key secondary-buf) ;; should silently do nothing if the key/value already ;; exists (db-put-buffered (controller-indices *store-controller*) secondary-buf key-buf) (reset-buffer-stream secondary-buf)))) value)))) (defmethod remove-kv (key (bt indexed-btree)) "Remove a key / value pair, and update secondary indices." (declare (optimize (speed 3))) (with-buffer-streams (key-buf secondary-buf) (buffer-write-int (oid bt) key-buf) (serialize key key-buf) (with-transaction () (let ((value (get-value key bt))) (when value (let ((indices (indices-cache bt))) (loop for index being the hash-value of indices do (multiple-value-bind (index? secondary-key) (funcall (key-fn index) index key value) (when index? (buffer-write-int (oid index) secondary-buf) (serialize secondary-key secondary-buf) ;; need to remove kv pairs with a cursor! -- ;; this is a C performance hack (sleepycat::db-delete-kv-buffered (controller-indices *store-controller*) secondary-buf key-buf) (reset-buffer-stream secondary-buf)))) (db-delete-buffered (controller-btrees *store-controller*) key-buf))))))) (defclass btree-index (btree) ((primary :type indexed-btree :reader primary :initarg :primary) (key-form :reader key-form :initarg :key-form) (key-fn :type function :accessor key-fn :transient t)) (:metaclass persistent-metaclass) (:documentation "Secondary index to an indexed-btree.")) (defmethod shared-initialize :after ((instance btree-index) slot-names &rest rest) (declare (ignore slot-names rest)) (let ((key-form (key-form instance))) (if (and (symbolp key-form) (fboundp key-form)) (setf (key-fn instance) (fdefinition key-form)) (setf (key-fn instance) (compile nil key-form))))) (defmethod get-value (key (bt btree-index)) "Get the value in the primary DB from a secondary key." (declare (optimize (speed 3))) (with-buffer-streams (key-buf value-buf) (buffer-write-int (oid bt) key-buf) (serialize key key-buf) (let ((buf (db-get-key-buffered (controller-indices-assoc *store-controller*) key-buf value-buf))) (if buf (values (deserialize buf) T) (values nil nil))))) (defmethod (setf get-value) (value key (bt btree-index)) "Puts are not allowed on secondary indices. Try adding to the primary." (declare (ignore value key bt)) (error "Puts are forbidden on secondary indices. Try adding to the primary.")) (defgeneric get-primary-key (key bt) (:documentation "Get the primary key from a secondary key.")) (defmethod get-primary-key (key (bt btree-index)) (declare (optimize (speed 3))) (with-buffer-streams (key-buf value-buf) (buffer-write-int (oid bt) key-buf) (serialize key key-buf) (let ((buf (db-get-key-buffered (controller-indices *store-controller*) key-buf value-buf))) (if buf (let ((oid (buffer-read-fixnum buf))) (values (deserialize buf) oid)) (values nil nil))))) (defmethod remove-kv (key (bt btree-index)) "Remove a key / value from the PRIMARY by a secondary lookup, updating ALL other secondary indices." (declare (optimize (speed 3))) (remove-kv (get-primary-key key bt) (primary bt))) ;; Cursor operations (defclass cursor () ((handle :accessor cursor-handle :initarg :handle) (oid :accessor cursor-oid :type fixnum :initarg :oid) (initialized-p :accessor cursor-initialized-p :type boolean :initform nil :initarg :initialized-p) (btree :accessor cursor-btree :initarg :btree)) (:documentation "A cursor for traversing (primary) BTrees.")) (defgeneric make-cursor (bt &key degree-2 dirty-read) (:documentation "Construct a cursor for traversing BTrees.")) (defgeneric cursor-close (cursor) (:documentation "Close the cursor. Make sure to close cursors before the enclosing transaction is closed!")) (defgeneric cursor-duplicate (cursor) (:documentation "Duplicate a cursor.")) (defgeneric cursor-current (cursor) (:documentation "Get the key / value at the cursor position. Returns has-pair key value, where has-pair is a boolean indicating there was a pair.")) (defgeneric cursor-first (cursor) (:documentation "Move the cursor to the beginning of the BTree, returning has-pair key value.")) (defgeneric cursor-last (cursor) (:documentation "Move the cursor to the end of the BTree, returning has-pair key value.")) (defgeneric cursor-next (cursor) (:documentation "Advance the cursor, returning has-pair key value.")) (defgeneric cursor-prev (cursor) (:documentation "Move the cursor back, returning has-pair key value.")) (defgeneric cursor-set (cursor key) (:documentation "Move the cursor to a particular key, returning has-pair key value.")) (defgeneric cursor-set-range (cursor key) (:documentation "Move the cursor to the first key-value pair with key greater or equal to the key argument, according to the lisp sorter. Returns has-pair key value.")) (defgeneric cursor-get-both (cursor key value) (:documentation "Moves the cursor to a particular key / value pair, returning has-pair key value.")) (defgeneric cursor-get-both-range (cursor key value) (:documentation "Moves the cursor to the first key / value pair with key equal to the key argument and value greater or equal to the value argument. Not really useful for us since primaries don't have duplicates. Returns has-pair key value.")) (defgeneric cursor-delete (cursor) (:documentation "Delete by cursor. The cursor is at an invalid position after a successful delete.")) (defgeneric cursor-put (cursor value &key key) (:documentation "Put by cursor. Currently doesn't properly move the cursor.")) (defmethod make-cursor ((bt btree) &key degree-2 dirty-read) "Make a cursor from a btree." (declare (optimize (speed 3))) (make-instance 'cursor :btree bt :handle (db-cursor (controller-btrees *store-controller*) :degree-2 degree-2 :dirty-read dirty-read) :oid (oid bt))) (defmacro with-btree-cursor ((var bt &key degree-2 dirty-read) &body body) "Macro which opens a named cursor on a BTree (primary or not), evaluates the forms, then closes the cursor." `(let ((,var (make-cursor ,bt :degree-2 ,degree-2 :dirty-read ,dirty-read))) (unwind-protect (progn , at body) (cursor-close ,var)))) (defun map-btree (fn bt &key degree-2 dirty-read) "Like maphash." (with-btree-cursor (curs bt :degree-2 degree-2 :dirty-read dirty-read) (loop (multiple-value-bind (more k v) (cursor-next curs) (unless more (return nil)) (funcall fn k v))))) (defmethod cursor-close ((cursor cursor)) (declare (optimize (speed 3))) (db-cursor-close (cursor-handle cursor)) (setf (cursor-initialized-p cursor) nil)) (defmethod cursor-duplicate ((cursor cursor)) (declare (optimize (speed 3))) (make-instance (type-of cursor) :initialized-p (cursor-initialized-p cursor) :oid (cursor-oid cursor) :handle (db-cursor-duplicate (cursor-handle cursor) :position (cursor-initialized-p cursor)))) (defmethod cursor-current ((cursor cursor)) (declare (optimize (speed 3))) (when (cursor-initialized-p cursor) (with-buffer-streams (key-buf value-buf) (multiple-value-bind (key val) (db-cursor-move-buffered (cursor-handle cursor) key-buf value-buf :current t) (if (and key (= (buffer-read-int key) (cursor-oid cursor))) (progn (setf (cursor-initialized-p cursor) t) (values t (deserialize key) (deserialize val))) (setf (cursor-initialized-p cursor) nil)))))) (defmethod cursor-first ((cursor cursor)) (declare (optimize (speed 3))) (with-buffer-streams (key-buf value-buf) (buffer-write-int (cursor-oid cursor) key-buf) (multiple-value-bind (key val) (db-cursor-set-buffered (cursor-handle cursor) key-buf value-buf :set-range t) (if (and key (= (buffer-read-int key) (cursor-oid cursor))) (progn (setf (cursor-initialized-p cursor) t) (values t (deserialize key) (deserialize val))) (setf (cursor-initialized-p cursor) nil))))) ;;A bit of a hack..... (defmethod cursor-last ((cursor cursor)) (declare (optimize (speed 3))) (with-buffer-streams (key-buf value-buf) (buffer-write-int (+ (cursor-oid cursor) 1) key-buf) (if (db-cursor-set-buffered (cursor-handle cursor) key-buf value-buf :set-range t) (progn (reset-buffer-stream key-buf) (reset-buffer-stream value-buf) (multiple-value-bind (key val) (db-cursor-move-buffered (cursor-handle cursor) key-buf value-buf :prev t) (if (and key (= (buffer-read-int key) (cursor-oid cursor))) (progn (setf (cursor-initialized-p cursor) t) (values t (deserialize key) (deserialize val))) (setf (cursor-initialized-p cursor) nil)))) (multiple-value-bind (key val) (db-cursor-move-buffered (cursor-handle cursor) key-buf value-buf :last t) (if (and key (= (buffer-read-int key) (cursor-oid cursor))) (progn (setf (cursor-initialized-p cursor) t) (values t (deserialize key) (deserialize val))) (setf (cursor-initialized-p cursor) nil)))))) (defmethod cursor-next ((cursor cursor)) (declare (optimize (speed 3))) (if (cursor-initialized-p cursor) (with-buffer-streams (key-buf value-buf) (multiple-value-bind (key val) (db-cursor-move-buffered (cursor-handle cursor) key-buf value-buf :next t) (if (and key (= (buffer-read-int key) (cursor-oid cursor))) (values t (deserialize key) (deserialize val)) (setf (cursor-initialized-p cursor) nil)))) (cursor-first cursor))) (defmethod cursor-prev ((cursor cursor)) (declare (optimize (speed 3))) (if (cursor-initialized-p cursor) (with-buffer-streams (key-buf value-buf) (multiple-value-bind (key val) (db-cursor-move-buffered (cursor-handle cursor) key-buf value-buf :prev t) (if (and key (= (buffer-read-int key) (cursor-oid cursor))) (values t (deserialize key) (deserialize val)) (setf (cursor-initialized-p cursor) nil)))) (cursor-last cursor))) (defmethod cursor-set ((cursor cursor) key) (declare (optimize (speed 3))) (with-buffer-streams (key-buf value-buf) (buffer-write-int (cursor-oid cursor) key-buf) (serialize key key-buf) (multiple-value-bind (k val) (db-cursor-set-buffered (cursor-handle cursor) key-buf value-buf :set t) (if k (progn (setf (cursor-initialized-p cursor) t) (values t key (deserialize val))) (setf (cursor-initialized-p cursor) nil))))) (defmethod cursor-set-range ((cursor cursor) key) (declare (optimize (speed 3))) (with-buffer-streams (key-buf value-buf) (buffer-write-int (cursor-oid cursor) key-buf) (serialize key key-buf) (multiple-value-bind (k val) (db-cursor-set-buffered (cursor-handle cursor) key-buf value-buf :set-range t) (if (and k (= (buffer-read-int k) (cursor-oid cursor))) (progn (setf (cursor-initialized-p cursor) t) (values t (deserialize k) (deserialize val))) (setf (cursor-initialized-p cursor) nil))))) (defmethod cursor-get-both ((cursor cursor) key value) (declare (optimize (speed 3))) (with-buffer-streams (key-buf value-buf) (buffer-write-int (cursor-oid cursor) key-buf) (serialize key key-buf) (serialize value value-buf) (multiple-value-bind (k v) (db-cursor-get-both-buffered (cursor-handle cursor) key-buf value-buf :get-both t) (declare (ignore v)) (if k (progn (setf (cursor-initialized-p cursor) t) (values t key value)) (setf (cursor-initialized-p cursor) nil))))) (defmethod cursor-get-both-range ((cursor cursor) key value) (declare (optimize (speed 3))) (with-buffer-streams (key-buf value-buf) (buffer-write-int (cursor-oid cursor) key-buf) (serialize key key-buf) (serialize value value-buf) (multiple-value-bind (k v) (db-cursor-get-both-buffered (cursor-handle cursor) key-buf value-buf :get-both-range t) (if k (progn (setf (cursor-initialized-p cursor) t) (values t key (deserialize v))) (setf (cursor-initialized-p cursor) nil))))) (defmethod cursor-delete ((cursor cursor)) (declare (optimize (speed 3))) (if (cursor-initialized-p cursor) (with-buffer-streams (key-buf value-buf) (multiple-value-bind (key val) (db-cursor-move-buffered (cursor-handle cursor) key-buf value-buf :current t) (declare (ignore val)) (when (and key (= (buffer-read-int key) (cursor-oid cursor))) ;; in case of a secondary index this should delete everything ;; as specified by the BDB docs. (remove-kv (deserialize key) (cursor-btree cursor))) (setf (cursor-initialized-p cursor) nil))) (error "Can't delete with uninitialized cursor!"))) (defmethod cursor-put ((cursor cursor) value &key (key nil key-specified-p)) "Put by cursor. Not particularly useful since primaries don't support duplicates. Currently doesn't properly move the cursor." (declare (optimize (speed 3))) (if key-specified-p (setf (get-value key (cursor-btree cursor)) value) (if (cursor-initialized-p cursor) (with-buffer-streams (key-buf value-buf) (multiple-value-bind (k v) (db-cursor-move-buffered (cursor-handle cursor) key-buf value-buf :current t) (declare (ignore v)) (if (and k (= (buffer-read-int k) (cursor-oid cursor))) (setf (get-value (deserialize k) (cursor-btree cursor)) value) (setf (cursor-initialized-p cursor) nil)))) (error "Can't put with uninitialized cursor!")))) ;; Secondary cursors (defclass secondary-cursor (cursor) () (:documentation "Cursor for traversing secondary indices.")) (defgeneric cursor-pcurrent (cursor) (:documentation "Returns has-tuple / secondary key / value / primary key at the current position.")) (defgeneric cursor-pfirst (cursor) (:documentation "Moves the key to the beginning of the secondary index. Returns has-tuple / secondary key / value / primary key.")) (defgeneric cursor-plast (cursor) (:documentation "Moves the key to the end of the secondary index. Returns has-tuple / secondary key / value / primary key.")) (defgeneric cursor-pnext (cursor) (:documentation "Advances the cursor. Returns has-tuple / secondary key / value / primary key.")) (defgeneric cursor-pprev (cursor) (:documentation "Moves the cursor back. Returns has-tuple / secondary key / value / primary key.")) (defgeneric cursor-pset (cursor key) (:documentation "Moves the cursor to a particular key. Returns has-tuple / secondary key / value / primary key.")) (defgeneric cursor-pset-range (cursor key) (:documentation "Move the cursor to the first key-value pair with key greater or equal to the key argument, according to the lisp sorter. Returns has-pair secondary key value primary key.")) (defgeneric cursor-pget-both (cursor key value) (:documentation "Moves the cursor to a particular secondary key / primary key pair. Returns has-tuple / secondary key / value / primary key.")) (defgeneric cursor-pget-both-range (cursor key value) (:documentation "Moves the cursor to a the first secondary key / primary key pair, with secondary key equal to the key argument, and primary key greater or equal to the pkey argument. Returns has-tuple / secondary key / value / primary key.")) (defgeneric cursor-next-dup (cursor) (:documentation "Move to the next duplicate element (with the same key.) Returns has-pair key value.")) (defgeneric cursor-next-nodup (cursor) (:documentation "Move to the next non-duplicate element (with different key.) Returns has-pair key value.")) (defgeneric cursor-prev-nodup (cursor) (:documentation "Move to the previous non-duplicate element (with different key.) Returns has-pair key value.")) (defgeneric cursor-pnext-dup (cursor) (:documentation "Move to the next duplicate element (with the same key.) Returns has-tuple / secondary key / value / primary key.")) (defgeneric cursor-pnext-nodup (cursor) (:documentation "Move to the next non-duplicate element (with different key.) Returns has-tuple / secondary key / value / primary key.")) (defgeneric cursor-pprev-nodup (cursor) (:documentation "Move to the previous non-duplicate element (with different key.) Returns has-tuple / secondary key / value / primary key.")) (defmethod make-cursor ((bt btree-index) &key degree-2 dirty-read) "Make a secondary-cursor from a secondary index." (declare (optimize (speed 3))) (make-instance 'secondary-cursor :btree bt :handle (db-cursor (controller-indices-assoc *store-controller*) :degree-2 degree-2 :dirty-read dirty-read) :oid (oid bt))) (defmethod cursor-pcurrent ((cursor secondary-cursor)) (declare (optimize (speed 3))) (when (cursor-initialized-p cursor) (with-buffer-streams (key-buf pkey-buf value-buf) (multiple-value-bind (key pkey val) (db-cursor-pmove-buffered (cursor-handle cursor) key-buf pkey-buf value-buf :current t) (if (and key (= (buffer-read-int key) (cursor-oid cursor))) (progn (setf (cursor-initialized-p cursor) t) (values t (deserialize key) (deserialize val) (progn (buffer-read-int pkey) (deserialize pkey)))) (setf (cursor-initialized-p cursor) nil)))))) (defmethod cursor-pfirst ((cursor secondary-cursor)) (declare (optimize (speed 3))) (with-buffer-streams (key-buf pkey-buf value-buf) (buffer-write-int (cursor-oid cursor) key-buf) (multiple-value-bind (key pkey val) (db-cursor-pset-buffered (cursor-handle cursor) key-buf pkey-buf value-buf :set-range t) (if (and key (= (buffer-read-int key) (cursor-oid cursor))) (progn (setf (cursor-initialized-p cursor) t) (values t (deserialize key) (deserialize val) (progn (buffer-read-int pkey) (deserialize pkey)))) (setf (cursor-initialized-p cursor) nil))))) ;;A bit of a hack..... (defmethod cursor-plast ((cursor secondary-cursor)) (declare (optimize (speed 3))) (with-buffer-streams (key-buf pkey-buf value-buf) (buffer-write-int (+ (cursor-oid cursor) 1) key-buf) (if (db-cursor-set-buffered (cursor-handle cursor) key-buf value-buf :set-range t) (progn (reset-buffer-stream key-buf) (reset-buffer-stream value-buf) (multiple-value-bind (key pkey val) (db-cursor-pmove-buffered (cursor-handle cursor) key-buf pkey-buf value-buf :prev t) (if (and key (= (buffer-read-int key) (cursor-oid cursor))) (progn (setf (cursor-initialized-p cursor) t) (values t (deserialize key) (deserialize val) (progn (buffer-read-int pkey) (deserialize pkey)))) (setf (cursor-initialized-p cursor) nil)))) (multiple-value-bind (key pkey val) (db-cursor-pmove-buffered (cursor-handle cursor) key-buf pkey-buf value-buf :last t) (if (and key (= (buffer-read-int key) (cursor-oid cursor))) (progn (setf (cursor-initialized-p cursor) t) (values t (deserialize key) (deserialize val) (progn (buffer-read-int pkey) (deserialize pkey)))) (setf (cursor-initialized-p cursor) nil)))))) (defmethod cursor-pnext ((cursor secondary-cursor)) (declare (optimize (speed 3))) (if (cursor-initialized-p cursor) (with-buffer-streams (key-buf pkey-buf value-buf) (multiple-value-bind (key pkey val) (db-cursor-pmove-buffered (cursor-handle cursor) key-buf pkey-buf value-buf :next t) (if (and key (= (buffer-read-int key) (cursor-oid cursor))) (values t (deserialize key) (deserialize val) (progn (buffer-read-int pkey) (deserialize pkey))) (setf (cursor-initialized-p cursor) nil)))) (cursor-pfirst cursor))) (defmethod cursor-pprev ((cursor secondary-cursor)) (declare (optimize (speed 3))) (if (cursor-initialized-p cursor) (with-buffer-streams (key-buf pkey-buf value-buf) (multiple-value-bind (key pkey val) (db-cursor-pmove-buffered (cursor-handle cursor) key-buf pkey-buf value-buf :prev t) (if (and key (= (buffer-read-int key) (cursor-oid cursor))) (values t (deserialize key) (deserialize val) (progn (buffer-read-int pkey) (deserialize pkey))) (setf (cursor-initialized-p cursor) nil)))) (cursor-plast cursor))) (defmethod cursor-pset ((cursor secondary-cursor) key) (declare (optimize (speed 3))) (with-buffer-streams (key-buf pkey-buf value-buf) (buffer-write-int (cursor-oid cursor) key-buf) (serialize key key-buf) (multiple-value-bind (k pkey val) (db-cursor-pset-buffered (cursor-handle cursor) key-buf pkey-buf value-buf :set t) (if k (progn (setf (cursor-initialized-p cursor) t) (values t key (deserialize val) (progn (buffer-read-int pkey) (deserialize pkey)))) (setf (cursor-initialized-p cursor) nil))))) (defmethod cursor-pset-range ((cursor secondary-cursor) key) (declare (optimize (speed 3))) (with-buffer-streams (key-buf pkey-buf value-buf) (buffer-write-int (cursor-oid cursor) key-buf) (serialize key key-buf) (multiple-value-bind (k pkey val) (db-cursor-pset-buffered (cursor-handle cursor) key-buf pkey-buf value-buf :set-range t) (if (and k (= (buffer-read-int k) (cursor-oid cursor))) (progn (setf (cursor-initialized-p cursor) t) (values t (deserialize k) (deserialize val) (progn (buffer-read-int pkey) (deserialize pkey)))) (setf (cursor-initialized-p cursor) nil))))) (defmethod cursor-pget-both ((cursor secondary-cursor) key pkey) (declare (optimize (speed 3))) (with-buffer-streams (key-buf pkey-buf value-buf) (let ((primary-oid (oid (primary (cursor-btree cursor))))) (buffer-write-int (cursor-oid cursor) key-buf) (serialize key key-buf) (buffer-write-int primary-oid pkey-buf) (serialize pkey pkey-buf) (multiple-value-bind (k p val) (db-cursor-pget-both-buffered (cursor-handle cursor) key-buf pkey-buf value-buf :get-both t) (declare (ignore p)) (if k (progn (setf (cursor-initialized-p cursor) t) (values t key (deserialize val) pkey)) (setf (cursor-initialized-p cursor) nil)))))) (defmethod cursor-pget-both-range ((cursor secondary-cursor) key pkey) (declare (optimize (speed 3))) (with-buffer-streams (key-buf pkey-buf value-buf) (let ((primary-oid (oid (primary (cursor-btree cursor))))) (buffer-write-int (cursor-oid cursor) key-buf) (serialize key key-buf) (buffer-write-int primary-oid pkey-buf) (serialize pkey pkey-buf) (multiple-value-bind (k p val) (db-cursor-pget-both-buffered (cursor-handle cursor) key-buf pkey-buf value-buf :get-both-range t) (if k (progn (setf (cursor-initialized-p cursor) t) (values t key (deserialize val) (progn (buffer-read-int p) (deserialize p)))) (setf (cursor-initialized-p cursor) nil)))))) (defmethod cursor-delete ((cursor secondary-cursor)) "Delete by cursor: deletes ALL secondary indices." (declare (optimize (speed 3))) (if (cursor-initialized-p cursor) (with-buffer-streams (key-buf pkey-buf value-buf) (multiple-value-bind (key pkey val) (db-cursor-pmove-buffered (cursor-handle cursor) key-buf pkey-buf value-buf :current t) (declare (ignore val)) (when (and key (= (buffer-read-int key) (cursor-oid cursor)) (= (buffer-read-int pkey) (oid (primary (cursor-btree cursor))))) (remove-kv (deserialize pkey) (primary (cursor-btree cursor)))) (setf (cursor-initialized-p cursor) nil))) (error "Can't delete with uninitialized cursor!"))) (defmethod cursor-get-both ((cursor secondary-cursor) key value) "cursor-get-both not implemented for secondary indices. Use cursor-pget-both." (declare (ignore cursor key value)) (error "cursor-get-both not implemented on secondary indices. Use cursor-pget-both.")) (defmethod cursor-get-both-range ((cursor secondary-cursor) key value) "cursor-get-both-range not implemented for secondary indices. Use cursor-pget-both-range." (declare (ignore cursor key value)) (error "cursor-get-both-range not implemented on secondary indices. Use cursor-pget-both-range.")) (defmethod cursor-put ((cursor secondary-cursor) value &rest rest) "Puts are forbidden on secondary indices. Try adding to the primary." (declare (ignore rest value cursor)) (error "Puts are forbidden on secondary indices. Try adding to the primary.")) (defmethod cursor-next-dup ((cursor secondary-cursor)) (declare (optimize (speed 3))) (when (cursor-initialized-p cursor) (with-buffer-streams (key-buf value-buf) (multiple-value-bind (key val) (db-cursor-move-buffered (cursor-handle cursor) key-buf value-buf :next-dup t) (if (and key (= (buffer-read-int key) (cursor-oid cursor))) (values t (deserialize key) (deserialize val)) (setf (cursor-initialized-p cursor) nil)))))) (defmethod cursor-next-nodup ((cursor secondary-cursor)) (declare (optimize (speed 3))) (if (cursor-initialized-p cursor) (with-buffer-streams (key-buf value-buf) (multiple-value-bind (key val) (db-cursor-move-buffered (cursor-handle cursor) key-buf value-buf :next-nodup t) (if (and key (= (buffer-read-int key) (cursor-oid cursor))) (values t (deserialize key) (deserialize val)) (setf (cursor-initialized-p cursor) nil)))) (cursor-first cursor))) (defmethod cursor-prev-nodup ((cursor secondary-cursor)) (declare (optimize (speed 3))) (if (cursor-initialized-p cursor) (with-buffer-streams (key-buf value-buf) (multiple-value-bind (key val) (db-cursor-move-buffered (cursor-handle cursor) key-buf value-buf :prev-nodup t) (if (and key (= (buffer-read-int key) (cursor-oid cursor))) (values t (deserialize key) (deserialize val)) (setf (cursor-initialized-p cursor) nil)))) (cursor-last cursor))) (defmethod cursor-pnext-dup ((cursor secondary-cursor)) (declare (optimize (speed 3))) (when (cursor-initialized-p cursor) (with-buffer-streams (key-buf pkey-buf value-buf) (multiple-value-bind (key pkey val) (db-cursor-pmove-buffered (cursor-handle cursor) key-buf pkey-buf value-buf :next-dup t) (if (and key (= (buffer-read-int key) (cursor-oid cursor))) (values t (deserialize key) (deserialize val) (progn (buffer-read-int pkey) (deserialize pkey))) (setf (cursor-initialized-p cursor) nil)))))) (defmethod cursor-pnext-nodup ((cursor secondary-cursor)) (declare (optimize (speed 3))) (if (cursor-initialized-p cursor) (with-buffer-streams (key-buf pkey-buf value-buf) (multiple-value-bind (key pkey val) (db-cursor-pmove-buffered (cursor-handle cursor) key-buf pkey-buf value-buf :next-nodup t) (if (and key (= (buffer-read-int key) (cursor-oid cursor))) (values t (deserialize key) (deserialize val) (progn (buffer-read-int pkey) (deserialize pkey))) (setf (cursor-initialized-p cursor) nil)))) (cursor-pfirst cursor))) (defmethod cursor-pprev-nodup ((cursor secondary-cursor)) (declare (optimize (speed 3))) (if (cursor-initialized-p cursor) (with-buffer-streams (key-buf pkey-buf value-buf) (multiple-value-bind (key pkey val) (db-cursor-pmove-buffered (cursor-handle cursor) key-buf pkey-buf value-buf :prev-nodup t) (if (and key (= (buffer-read-int key) (cursor-oid cursor))) (values t (deserialize key) (deserialize val) (progn (buffer-read-int pkey) (deserialize pkey))) (setf (cursor-initialized-p cursor) nil)))) (cursor-plast cursor))) From mega at hotpop.com Fri Feb 18 23:34:40 2005 From: mega at hotpop.com (=?utf-8?q?G=C3=A1bor_Melis?=) Date: Sat, 19 Feb 2005 00:34:40 +0100 Subject: [elephant-devel] running out of locks In-Reply-To: <20050218173141.U3493@contarex.medianstrip.net> References: <200502112248.13633.mega@hotpop.com> <200502182229.45406.mega@hotpop.com> <20050218173141.U3493@contarex.medianstrip.net> Message-ID: <200502190034.41066.mega@hotpop.com> On Friday 18 February 2005 23:32, Ben wrote: > try this one? Oops. It doesn't work, because degree-2 does not help with the huge write it is doing. > > B > > On Fri, 18 Feb 2005, [utf-8] G?bor Melis wrote: > > On Thursday 17 February 2005 23:44, you wrote: > >> here's the tarball, let me know if you have problems - B > > > > I got the feedback fever. > > > > add-index with populate maps the btree and runs out of locks here :-) > > degree-2? From mega at hotpop.com Fri Feb 18 15:23:05 2005 From: mega at hotpop.com (=?iso-8859-1?q?G=E1bor_Melis?=) Date: Fri, 18 Feb 2005 16:23:05 +0100 Subject: [elephant-devel] Berkeley DB error: Unknown error: -5 In-Reply-To: <87brahap9q.fsf@gia.ist.utl.pt> References: <87brahap9q.fsf@gia.ist.utl.pt> Message-ID: <200502181623.05461.mega@hotpop.com> On Friday 18 February 2005 15:58, Antonio Menezes Leitao wrote: > Hi, > > Hi. Anyone has experienced a "Berkeley DB error: Unknown error: -5"? Yes. See my message 'running out of locks'. You may try raising the lock related parameters: max lockers, lock objects, locks. Elephant does not support this part of the api but placing a DB_CONFIG file there should help. > > It is caused by ele::next-oid calling sleepycat::next-counter. > > Any quick solution? > > Thanks in advance, > > Ant?nio Leit?o. > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From aml at gia.ist.utl.pt Sat Feb 19 09:35:38 2005 From: aml at gia.ist.utl.pt (Antonio Menezes Leitao) Date: Sat, 19 Feb 2005 09:35:38 +0000 Subject: [elephant-devel] Berkeley DB error: Unknown error: -5 In-Reply-To: <20050218173254.O3493@contarex.medianstrip.net> (ben@medianstrip.net's message of "Fri, 18 Feb 2005 17:33:54 -0500 (EST)") References: <87brahap9q.fsf@gia.ist.utl.pt> <20050218173254.O3493@contarex.medianstrip.net> Message-ID: <87wtt4c2ol.fsf@gia.ist.utl.pt> Hi, Ben writes: > it looks like you're running into the same problem gabor melis did. > we're in the process of cutting a new release which should fix your > problem, probably by this weekend. if you can't wait, i can send you > an untested tarball. Fortunately, Gabor's solution solved my problem. Everything is working fine, now. BTW, will that new release also solve the problems that prevent elephant from running on recent SBCL versions (the ones with unicode support)? Best regards, Ant?nio Leit?o. From ben at medianstrip.net Sat Feb 19 17:16:59 2005 From: ben at medianstrip.net (Ben) Date: Sat, 19 Feb 2005 12:16:59 -0500 (EST) Subject: [elephant-devel] Berkeley DB error: Unknown error: -5 In-Reply-To: <87wtt4c2ol.fsf@gia.ist.utl.pt> References: <87brahap9q.fsf@gia.ist.utl.pt> <20050218173254.O3493@contarex.medianstrip.net> <87wtt4c2ol.fsf@gia.ist.utl.pt> Message-ID: <20050219121622.H93456@contarex.medianstrip.net> yes. i can send you a tarball if you want. B On Sat, 19 Feb 2005, Antonio Menezes Leitao wrote: > Hi, > > Ben writes: > >> it looks like you're running into the same problem gabor melis did. >> we're in the process of cutting a new release which should fix your >> problem, probably by this weekend. if you can't wait, i can send you >> an untested tarball. > > Fortunately, Gabor's solution solved my problem. Everything is > working fine, now. > > BTW, will that new release also solve the problems that prevent > elephant from running on recent SBCL versions (the ones with unicode > support)? > > Best regards, > > Ant?nio Leit?o. > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel > From mega at hotpop.com Sat Feb 19 17:41:43 2005 From: mega at hotpop.com (=?utf-8?q?G=C3=A1bor_Melis?=) Date: Sat, 19 Feb 2005 18:41:43 +0100 Subject: [elephant-devel] db gc In-Reply-To: <20041118135658.E91422@contarex.medianstrip.net> References: <04Nov18.154346cet.336116@fwall.essnet.se> <20041118135658.E91422@contarex.medianstrip.net> Message-ID: <200502191841.44119.mega@hotpop.com> On Thursday 18 November 2004 20:02, Ben wrote: > Writing the GC is long overdue. There are some technical issues with > this which have yet to be solved, actually. In the first pass, it > will probably require taking the store off-line and running a separate > gc program on it -- that shouldn't be too hard. it may be possible to > write an online collector but as of yet i don't know how to do it. i > expect the gc will come with the next release (in a month or so -- > after i'm done teaching this quarter!) I started hacking on the gc. It works by replacing deserialize with a similar function that records the oids instead of calling get-cached-instance and reading all slots, key-value pairs in persistent classes and btrees. (defmethod walk-persistent ((btree btree)) (map-btree (lambda (key value) (declare (ignore key value))) btree :degree-2 t)) (defmethod walk-persistent ((obj persistent)) (let ((class (class-of obj)) (persistent-effective-slot-definition-class (find-class 'persistent-effective-slot-definition))) (loop for slot-definition in (class-slots class) when (eq (class-of slot-definition) persistent-effective-slot-definition-class) do (slot-value-using-class class obj slot-definition)))) (defun elephant-gc (&optional (sc *store-controller*)) (let ((old-oids (make-hash-table)) (new-oids (make-hash-table))) (flet ((marker (controller oid class) (declare (ignore controller)) (unless (gethash oid old-oids) (setf (gethash oid new-oids) class)))) (with-marking-deserialize (#'marker) ;; mark the root (setf (gethash -1 old-oids) (class-name (class-of (controller-root sc)))) (walk-persistent (controller-root sc)) ;; (loop while (< 0 (hash-table-count new-oids)) do (maphash (lambda (oid class) (walk-persistent (make-instance class :from-oid oid)) (setf (gethash oid old-oids) class)) new-oids) (clrhash new-oids)))) ;; now OLD-OIDS contains the oids of all reachable objects (maphash (lambda (oid class) (format t "~S ~S~%" oid class)) old-oids) )) It seems to detect live objects OK. The next step is to iterate through controller-db and controller-btrees and delete records that have keys starting with a non-alive oid, right? Controller-indices and controller-indices-assoc can be left alone, I hope. What are those technical issues you mentioned above? Can references to a persistent object have different class names (maybe due to a change-class)? G From mega at hotpop.com Fri Feb 18 22:51:57 2005 From: mega at hotpop.com (=?utf-8?q?G=C3=A1bor_Melis?=) Date: Fri, 18 Feb 2005 23:51:57 +0100 Subject: [elephant-devel] running out of locks In-Reply-To: <20050218171904.B1858@contarex.medianstrip.net> References: <200502112248.13633.mega@hotpop.com> <200502181415.03338.mega@hotpop.com> <20050218171904.B1858@contarex.medianstrip.net> Message-ID: <200502182351.57449.mega@hotpop.com> On Friday 18 February 2005 23:20, Ben wrote: > have you tried using degree-2 cursors iterating over a btree? I have. They work as advertised. I've already converted my without-transactions to degree-2s. > on adding lots of users: is there a reason why they have to be added > all in one transaction? can you break up the transaction into smaller > chunks? It's just a setup for performance/stress test. There is no reason other than impatience to have them in the same transaction. However, I feel uneasy about the varying number of locks needed. I don't understand it and wonder if there are similar lock spots in the application. From walter at pelissero.de Mon Feb 21 13:23:25 2005 From: walter at pelissero.de (Walter C. Pelissero) Date: Mon, 21 Feb 2005 14:23:25 +0100 (CET) Subject: [elephant-devel] is initform evaluated? Message-ID: <200502211323.j1LDNPOM003576@zaphod.home.loc> Apparently the initform slot argument gets evaluated somewhere during the definition of a persistent class. Here is an example. (defclass foo () ((x :initform (error "default x") :initarg :x)) (:metaclass ele:persistent-metaclass)) If you evaluate this class definition you get an error, which is not what I expect until I actually make an instance of that class. -- walter pelissero http://www.pelissero.de From blumberg at math.uchicago.edu Mon Feb 21 16:59:14 2005 From: blumberg at math.uchicago.edu (Andrew Blumberg) Date: Mon, 21 Feb 2005 10:59:14 -0600 (CST) Subject: [elephant-devel] is initform evaluated? In-Reply-To: <200502211323.j1LDNPOM003576@zaphod.home.loc> References: <200502211323.j1LDNPOM003576@zaphod.home.loc> Message-ID: hey walter, this is a consequence of the fact that in the current release, persistent slots are actually handed as class slots and so the initialization semantics are wrong. in the new release, we've moved to using a nonstandard allocation type :database, which solves this sort of problem. the new release should be out "any day now", but if you're in a hurry we could send you patches sooner, depending on your platform. regards, andrew On Mon, 21 Feb 2005, Walter C. Pelissero wrote: > Apparently the initform slot argument gets evaluated somewhere during > the definition of a persistent class. > > Here is an example. > > (defclass foo () > ((x :initform (error "default x") > :initarg :x)) > (:metaclass ele:persistent-metaclass)) > > If you evaluate this class definition you get an error, which is not > what I expect until I actually make an instance of that class. > > From walter at pelissero.de Wed Feb 23 16:44:46 2005 From: walter at pelissero.de (Walter C. Pelissero) Date: Wed, 23 Feb 2005 17:44:46 +0100 Subject: [elephant-devel] Cannot allocate memory Message-ID: <16924.45822.723694.56778@zaphod.home.loc> Looks as I've been hitting some DB limit but I haven't found out which one. At the 376th iteration of a stress test I got a sleepycat:db-error (Berkeley DB error: Cannot allocate memory). Each test iteration is wrapped in a transaction, so I really don't understand what is consuming Sleepycat DB resources. If I had to guess from "db_stat -c" I'd say I'm running out of locks: 360% db_stat-4.2 -c 18 Last allocated locker ID. 2147M Current maximum unused locker ID. 9 Number of lock modes. 1000 Maximum number of locks possible. 1000 Maximum number of lockers possible. 1000 Maximum number of lock objects possible. 1004 Number of current locks. 1005 Maximum number of locks at any one time. 10 Number of current lockers. 11 Maximum number of lockers at any one time. 576 Number of current lock objects. 897 Maximum number of lock objects at any one time. 5305968 Total number of locks requested. 5296499 Total number of locks released. 0 Total number of lock requests failing because DB_LOCK_NOWAIT was set. 0 Total number of locks not immediately available due to conflicts. 0 Number of deadlocks. 0 Lock timeout value. 0 Number of locks that have timed out. 0 Transaction timeout value. 0 Number of transactions that have timed out. 360KB The size of the lock region.. 0 The number of region locks granted after waiting. 5331227 The number of region locks granted without waiting. Is that right? If so, how am I supposed to limit the number of locks held by my code/elephant? -- walter pelissero http://www.pelissero.de From robertlread at austin.rr.com Wed Feb 23 18:27:25 2005 From: robertlread at austin.rr.com (Robert L. Read) Date: Wed, 23 Feb 2005 12:27:25 -0600 Subject: [elephant-devel] Cannot allocate memory In-Reply-To: <16924.45822.723694.56778@zaphod.home.loc> References: <16924.45822.723694.56778@zaphod.home.loc> Message-ID: <1109183245.2024.1.camel@localhost.localdomain> Forgive an uneducated and probably irrelevant response, but I got the same error when I was accidentally opening a separate connection in each of my tests. I was creating several dozens or hundreds of store controllers, I think. When I solved that problem, I don't thnk this has reoccurred for me. Forgive my not being able to be more specific. On Wed, 2005-02-23 at 17:44 +0100, Walter C. Pelissero wrote: > Looks as I've been hitting some DB limit but I haven't found out which > one. > > At the 376th iteration of a stress test I got a sleepycat:db-error > (Berkeley DB error: Cannot allocate memory). Each test iteration is > wrapped in a transaction, so I really don't understand what is > consuming Sleepycat DB resources. > > If I had to guess from "db_stat -c" I'd say I'm running out of locks: > > 360% db_stat-4.2 -c > 18 Last allocated locker ID. > 2147M Current maximum unused locker ID. > 9 Number of lock modes. > 1000 Maximum number of locks possible. > 1000 Maximum number of lockers possible. > 1000 Maximum number of lock objects possible. > 1004 Number of current locks. > 1005 Maximum number of locks at any one time. > 10 Number of current lockers. > 11 Maximum number of lockers at any one time. > 576 Number of current lock objects. > 897 Maximum number of lock objects at any one time. > 5305968 Total number of locks requested. > 5296499 Total number of locks released. > 0 Total number of lock requests failing because DB_LOCK_NOWAIT was set. > 0 Total number of locks not immediately available due to conflicts. > 0 Number of deadlocks. > 0 Lock timeout value. > 0 Number of locks that have timed out. > 0 Transaction timeout value. > 0 Number of transactions that have timed out. > 360KB The size of the lock region.. > 0 The number of region locks granted after waiting. > 5331227 The number of region locks granted without waiting. > > Is that right? If so, how am I supposed to limit the number of locks > held by my code/elephant? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From walter at pelissero.de Wed Feb 23 18:29:13 2005 From: walter at pelissero.de (Walter C. Pelissero) Date: Wed, 23 Feb 2005 19:29:13 +0100 Subject: [elephant-devel] Cannot allocate memory In-Reply-To: <1109183245.2024.1.camel@localhost.localdomain> References: <16924.45822.723694.56778@zaphod.home.loc> <1109183245.2024.1.camel@localhost.localdomain> Message-ID: <16924.52089.174040.207276@zaphod.home.loc> Robert L. Read writes: > Forgive an uneducated and probably irrelevant response, but I got > the same error when I was accidentally opening a separate > connection in each of my tests. I was creating several dozens or > hundreds of store controllers, I think. When I solved that > problem, I don't thnk this has reoccurred for me. Thanks for the prompt answer, but no, it must be something else as open-store is called only once. I traced it, just to be sure. -- walter pelissero http://www.pelissero.de From ben at medianstrip.net Wed Feb 23 23:39:03 2005 From: ben at medianstrip.net (Ben) Date: Wed, 23 Feb 2005 18:39:03 -0500 (EST) Subject: [elephant-devel] Cannot allocate memory In-Reply-To: <1109183245.2024.1.camel@localhost.localdomain> References: <16924.45822.723694.56778@zaphod.home.loc> <1109183245.2024.1.camel@localhost.localdomain> Message-ID: <20050223183206.S84096@contarex.medianstrip.net> Hello Kind Users, Sorry for the radio silence but Andrew and I have both been swamped with work. 1) I will check in soon work which makes elephant work with SBCL and unicode, fixes various MOP problems, upgrades to sleepycat 4.3, using the new sequences and degree-2 isolation. this should fix some (but not all) of the lock problems. 2) Eventually this will get rolled into a real release. 3) I am happy to try to help squash bugs et al but time is short for me right now. I'm happy to accept patches as well. Your efforts testing and debugging would be much appreciated. About this lock issue: Gabor Melis has also reported issues with locks. I'm not quite sure what it is. Can you send your test code or describe the sorts of things it is doing? Does the issue remain if you close and reopen the store inbetween iterations of the test? I've discovered myself some very strange SBCL gc behavior. After some intense db action sometimes my SBCL goes into a very prolonged GC (so it appears). Let me know if you run into this problem, or if it some artifact of FreeBSD / SBCL. Take care, B On Wed, 23 Feb 2005, Robert L. Read wrote: > > Forgive an uneducated and probably irrelevant response, but I got the > same error when I > was accidentally opening a separate connection in each of my tests. I > was creating several dozens > or hundreds of store controllers, I think. When I solved that problem, > I don't thnk this has > reoccurred for me. > > Forgive my not being able to be more specific. > > > On Wed, 2005-02-23 at 17:44 +0100, Walter C. Pelissero wrote: > >> Looks as I've been hitting some DB limit but I haven't found out which >> one. >> >> At the 376th iteration of a stress test I got a sleepycat:db-error >> (Berkeley DB error: Cannot allocate memory). Each test iteration is >> wrapped in a transaction, so I really don't understand what is >> consuming Sleepycat DB resources. >> >> If I had to guess from "db_stat -c" I'd say I'm running out of locks: >> >> 360% db_stat-4.2 -c >> 18 Last allocated locker ID. >> 2147M Current maximum unused locker ID. >> 9 Number of lock modes. >> 1000 Maximum number of locks possible. >> 1000 Maximum number of lockers possible. >> 1000 Maximum number of lock objects possible. >> 1004 Number of current locks. >> 1005 Maximum number of locks at any one time. >> 10 Number of current lockers. >> 11 Maximum number of lockers at any one time. >> 576 Number of current lock objects. >> 897 Maximum number of lock objects at any one time. >> 5305968 Total number of locks requested. >> 5296499 Total number of locks released. >> 0 Total number of lock requests failing because DB_LOCK_NOWAIT was set. >> 0 Total number of locks not immediately available due to conflicts. >> 0 Number of deadlocks. >> 0 Lock timeout value. >> 0 Number of locks that have timed out. >> 0 Transaction timeout value. >> 0 Number of transactions that have timed out. >> 360KB The size of the lock region.. >> 0 The number of region locks granted after waiting. >> 5331227 The number of region locks granted without waiting. >> >> Is that right? If so, how am I supposed to limit the number of locks >> held by my code/elephant? >> > From ben at medianstrip.net Thu Feb 24 01:11:33 2005 From: ben at medianstrip.net (Ben) Date: Wed, 23 Feb 2005 20:11:33 -0500 (EST) Subject: [elephant-devel] Cannot allocate memory In-Reply-To: <16924.52089.174040.207276@zaphod.home.loc> References: <16924.45822.723694.56778@zaphod.home.loc> <1109183245.2024.1.camel@localhost.localdomain> <16924.52089.174040.207276@zaphod.home.loc> Message-ID: <20050223201059.U94251@contarex.medianstrip.net> I've commited the new changes. Walter -- can you try your tests with the new stuff? (you'll need to upgrade to sleepycat 4.3, and do a fresh cvs checkout.) B On Wed, 23 Feb 2005, Walter C. Pelissero wrote: > Robert L. Read writes: > > Forgive an uneducated and probably irrelevant response, but I got > > the same error when I was accidentally opening a separate > > connection in each of my tests. I was creating several dozens or > > hundreds of store controllers, I think. When I solved that > > problem, I don't thnk this has reoccurred for me. > > Thanks for the prompt answer, but no, it must be something else as > open-store is called only once. I traced it, just to be sure. > > -- > walter pelissero > http://www.pelissero.de > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel > From ben at medianstrip.net Thu Feb 24 01:22:19 2005 From: ben at medianstrip.net (Ben) Date: Wed, 23 Feb 2005 20:22:19 -0500 (EST) Subject: [elephant-devel] Cannot allocate memory In-Reply-To: <16924.52089.174040.207276@zaphod.home.loc> References: <16924.45822.723694.56778@zaphod.home.loc> <1109183245.2024.1.camel@localhost.localdomain> <16924.52089.174040.207276@zaphod.home.loc> Message-ID: <20050223202202.W95003@contarex.medianstrip.net> somehow i messed up on my cvs checkin. please hold off for a bit until i fix it. B On Wed, 23 Feb 2005, Walter C. Pelissero wrote: > Robert L. Read writes: > > Forgive an uneducated and probably irrelevant response, but I got > > the same error when I was accidentally opening a separate > > connection in each of my tests. I was creating several dozens or > > hundreds of store controllers, I think. When I solved that > > problem, I don't thnk this has reoccurred for me. > > Thanks for the prompt answer, but no, it must be something else as > open-store is called only once. I traced it, just to be sure. > > -- > walter pelissero > http://www.pelissero.de > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel > From ben at medianstrip.net Thu Feb 24 01:32:52 2005 From: ben at medianstrip.net (Ben) Date: Wed, 23 Feb 2005 20:32:52 -0500 (EST) Subject: [elephant-devel] Cannot allocate memory In-Reply-To: <16924.52089.174040.207276@zaphod.home.loc> References: <16924.45822.723694.56778@zaphod.home.loc> <1109183245.2024.1.camel@localhost.localdomain> <16924.52089.174040.207276@zaphod.home.loc> Message-ID: <20050223203213.X95764@contarex.medianstrip.net> It looks like the checkin went OK. Maybe I should sleep more before doing the checkins next time... B On Wed, 23 Feb 2005, Walter C. Pelissero wrote: > Robert L. Read writes: > > Forgive an uneducated and probably irrelevant response, but I got > > the same error when I was accidentally opening a separate > > connection in each of my tests. I was creating several dozens or > > hundreds of store controllers, I think. When I solved that > > problem, I don't thnk this has reoccurred for me. > > Thanks for the prompt answer, but no, it must be something else as > open-store is called only once. I traced it, just to be sure. > > -- > walter pelissero > http://www.pelissero.de > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel > From walter at pelissero.de Sun Feb 27 11:45:27 2005 From: walter at pelissero.de (Walter C. Pelissero) Date: Sun, 27 Feb 2005 12:45:27 +0100 Subject: [elephant-devel] Cannot allocate memory In-Reply-To: <20050223201059.U94251@contarex.medianstrip.net> References: <16924.45822.723694.56778@zaphod.home.loc> <1109183245.2024.1.camel@localhost.localdomain> <16924.52089.174040.207276@zaphod.home.loc> <20050223201059.U94251@contarex.medianstrip.net> Message-ID: <16929.45783.532769.431106@zaphod.home.loc> Ben writes: > I've commited the new changes. Walter -- can you try your tests with > the new stuff? (you'll need to upgrade to sleepycat 4.3, and do a > fresh cvs checkout.) I downloaded the latest elephant and installed db-4.3. The problem didn't go away but I've been able to isolate it. My program, in certain situations, starts a complete scan of the database (summing the value of a class slot). This is done within a transaction that somehow exausts Sleepycat resources. Here is how to reproduce the problem. I run several times the following function with increasing START values. (Just iterating more times in the loop would trigger the DB error, so I tried to avoid it.) (defun load-elephant (start) (be btree (ensure-btree "test") (ele:with-transaction () (loop with end = (+ start 100000) for i from start by 1 below end do (setf (ele:get-value i btree) i))))) After having, say, one million items, I run the following function, which doesn't do anything special beside counting the items in the btree. (defun crash-elephant () (ele:with-transaction () (elephant:with-btree-cursor (btree (ensure-btree "test")) (loop with items = 0 for (ok key value) = (multiple-value-list (ele:cursor-first btree)) then (multiple-value-list (ele:cursor-next btree)) while ok do (incf items) finally (return items))))) This should yield the DB error. -- walter pelissero http://www.pelissero.de From ben at medianstrip.net Mon Feb 28 06:28:42 2005 From: ben at medianstrip.net (Ben) Date: Mon, 28 Feb 2005 01:28:42 -0500 (EST) Subject: [elephant-devel] Cannot allocate memory In-Reply-To: <16929.45783.532769.431106@zaphod.home.loc> References: <16924.45822.723694.56778@zaphod.home.loc> <1109183245.2024.1.camel@localhost.localdomain> <16924.52089.174040.207276@zaphod.home.loc> <20050223201059.U94251@contarex.medianstrip.net> <16929.45783.532769.431106@zaphod.home.loc> Message-ID: <20050228012730.I76312@contarex.medianstrip.net> Have you tried using degree-2 cursors / transactions on your read? sleepycat by default uses serializable cursors / transactions. this means that during the course of a read you're guaranteed that the stuff you've seen won't change. this, as you can imagine, requires a lot of locks. degree-2 relaxes that condition, you might consider it. B On Sun, 27 Feb 2005, Walter C. Pelissero wrote: > Ben writes: > > I've commited the new changes. Walter -- can you try your tests with > > the new stuff? (you'll need to upgrade to sleepycat 4.3, and do a > > fresh cvs checkout.) > > I downloaded the latest elephant and installed db-4.3. The problem > didn't go away but I've been able to isolate it. My program, in > certain situations, starts a complete scan of the database (summing > the value of a class slot). This is done within a transaction that > somehow exausts Sleepycat resources. Here is how to reproduce the > problem. > > I run several times the following function with increasing START > values. (Just iterating more times in the loop would trigger the DB > error, so I tried to avoid it.) > > (defun load-elephant (start) > (be btree (ensure-btree "test") > (ele:with-transaction () > (loop > with end = (+ start 100000) > for i from start by 1 below end > do (setf (ele:get-value i btree) i))))) > > After having, say, one million items, I run the following function, > which doesn't do anything special beside counting the items in the > btree. > > (defun crash-elephant () > (ele:with-transaction () > (elephant:with-btree-cursor (btree (ensure-btree "test")) > (loop > with items = 0 > for (ok key value) = (multiple-value-list (ele:cursor-first btree)) > then (multiple-value-list (ele:cursor-next btree)) > while ok > do (incf items) > finally (return items))))) > > This should yield the DB error. > > -- > walter pelissero > http://www.pelissero.de > From walter at pelissero.de Mon Feb 28 10:35:05 2005 From: walter at pelissero.de (Walter C. Pelissero) Date: Mon, 28 Feb 2005 11:35:05 +0100 Subject: [elephant-devel] Cannot allocate memory In-Reply-To: <20050228012730.I76312@contarex.medianstrip.net> References: <16924.45822.723694.56778@zaphod.home.loc> <1109183245.2024.1.camel@localhost.localdomain> <16924.52089.174040.207276@zaphod.home.loc> <20050223201059.U94251@contarex.medianstrip.net> <16929.45783.532769.431106@zaphod.home.loc> <20050228012730.I76312@contarex.medianstrip.net> Message-ID: <16930.62425.493704.179335@zaphod.home.loc> Ben writes: > Have you tried using degree-2 cursors / transactions on your read? > sleepycat by default uses serializable cursors / transactions. this > means that during the course of a read you're guaranteed that the > stuff you've seen won't change. this, as you can imagine, requires a > lot of locks. degree-2 relaxes that condition, you might consider it. Not sure. The function that scans the DB just happens to be called from a transaction; it doesn't require a transaction itself. I'll try to find out whether degree-2 cursors will do. Thanks a lot. -- walter pelissero http://www.pelissero.de From ben at medianstrip.net Mon Feb 28 20:30:08 2005 From: ben at medianstrip.net (Ben) Date: Mon, 28 Feb 2005 15:30:08 -0500 (EST) Subject: [elephant-devel] Re: Elephant license In-Reply-To: References: Message-ID: <20050228113040.T25570@contarex.medianstrip.net> Edi, I'm forwarding this to the list, because it is a discussion users probably will want to see. The reason why I chose GPL was 1) it was compatible with sleepycat and 2) I wanted some kind of fair use clause. But since I use your code all the time, hey, it's only fair you can use mine! If you want something more formal, let me know. However, it appears that the sleepycat license is incompatible with your desires, unless you're willing to pay them. There are some potential solutions: Elephant is really 4 pieces: 1) a serializer for general types, 2) MOP stuff for CLOS types, 3) collection classes, and 4) a back end (currently FFI bindings to sleepycat.) 1) Conceivably one could swap out different back ends. I thought about trying to do this for SQLLite or Firebird. This would be a bit of work but it wouldn't be that hard, just time consuming. I think the sleepycat.lisp / berkeley-db.lisp files should be a good model. The only requirement for the back end is that they have sequences and be able to store byte arrays. However, if you want niceties like sorted btrees (esp in the presence of Unicode+bignums), there will be some work to get that to work. 2) Write an all-Lisp back end. There are many possible approaches here: transaction log in one file, ala ZODB FileStorage, or use the underlying filesystem btree and do something like ZODB DirectoryStorage, or recreate WOOD.....this seems hard to get right. I was originally going to do this when I started writing elephant, but decided against it, for that reason. 3) Pay sleepycat. I don't know how much they cost. Here's the caveat -- I don't have time to make these kinds of changes. But I'm certainly willing to advise. CLSQL appears to have some kind of MOPish framework. It has a decent license too right? take care, B On Mon, 28 Feb 2005, Edi Weitz wrote: > Hi! > > In the "LICENSE" file from Elephant it says > > "For differenct licensing terms, contact the copyright holders." > > so here I am... :) > > I've looked at Elephant and it seems to be nice. However, I noticed > that the license is the GPL and from my understanding this means that > if I write a Lisp application for a customer of mine which uses > Elephant then the whole application has to be under the GPL as well > which is not an option for me. > > My question is: Would you consider releasing Elephant under a > different license like LLGPL or BSD? And if not would you consider > offering a dual-license model where one can pay for the right not to > open source code based on Elephant? > > Also, do you have any experience with SleepyCat's licensing model? > Would I need to pay for a SleepyCat license as well or is that covered > by Elephant's open source license? (I'm sending the same question to > them but maybe you know this already.) > > My apologies if this has been dicussed before. I briefly searched the > mailing list archives but couldn't find anything related to this > question. > > Thanks for your help, > Edi Weitz. > > -- > Dr. Edmund Weitz > Hamburg > Germany >