From read at robertlread.net Wed Mar 1 18:58:25 2006 From: read at robertlread.net (Robert L. Read) Date: Wed, 01 Mar 2006 12:58:25 -0600 Subject: [elephant-devel] Committing some changes to the HEAD... Message-ID: <1141239505.4834.256.camel@localhost.localdomain> The following changes make me all green, even after repeated test runs, even on databases that are fresh, under SBCL 9.9 under Linux, under both SQL and BDB. In serializer.lisp, this code is needed: > ;; This circumlocution is necessitated by > ;; an apparent bug in SBCL 9.9 --- type-of sometimes > ;; does NOT return the "proper name" of the class as the > ;; CLHS says it should, but gives the class object itself, > ;; which cannot be directly serialized.... > (let ((tp (type-of frob))) > #+(or sbcl) > (if (not (symbolp tp)) > (setf tp (class-name (class-of frob)))) > (%serialize tp)) > ) Additionally, I had to change the idiom in the tests to handle the case that the persistent classes exist in the DB from a previous test run, but have not yet been even defined as classes in the LISP object space. For example, in indexing-reconnect-db: (when (class-indexedp-by-name 'idx-five) (defclass idx-five () ((slot1 :initarg :slot1 :initform 1 :accessor slot1 :index t) (slot2 :initarg :slot2 :initform 2 :accessor slot2) (slot3 :initarg :slot3 :initform 3 :accessor slot3 :index t)) (:metaclass persistent-metaclass)) (disable-class-indexing 'idx-five :errorp nil) (setf (find-class 'idx-five) nil)) (defclass idx-five () ((slot1 :initarg :slot1 :initform 1 :accessor slot1 :index t) (slot2 :initarg :slot2 :initform 2 :accessor slot2) (slot3 :initarg :slot3 :initform 3 :accessor slot3 :index t)) (:metaclass persistent-metaclass)) which is really hideous, handles the problem and makes the test correctly idempotent under all circumstances. I suspect Ian Eslick may have some way of simplifying this last bit; when he comes up for air from his current work we will discuss it. AFAIK, the latest release now awaits only addition tests on other (non SBCL/Linux) systems and the documentation of the class indexing that I have promised to Ian but not yet written. ---- Robert L. Read, PhD read &T robertlread.net Consider visiting Progressive Engineering: http://robertlread.net/pe In Austin: 912-8593 "Think globally, Act locally." -- RBF -------------- next part -------------- An HTML attachment was scrubbed... URL: From kykim_lists at mac.com Thu Mar 2 16:31:54 2006 From: kykim_lists at mac.com (Kevin Y. Kim) Date: Thu, 2 Mar 2006 11:31:54 -0500 Subject: [elephant-devel] Other BDBs? Message-ID: <6DF43F0C-CD00-499F-82DF-DBFCE208F3B0@mac.com> Admittedly, I don't know much about Sleepycat BDB (or BDBs in general), but is it possible to use Elephant against other BDBs (i.e. GDBM, NDBM, QDBM)? Just curious. Thanks, -kevin From read at robertlread.net Thu Mar 2 16:54:40 2006 From: read at robertlread.net (Robert L. Read) Date: Thu, 02 Mar 2006 10:54:40 -0600 Subject: [elephant-devel] Other BDBs? In-Reply-To: <6DF43F0C-CD00-499F-82DF-DBFCE208F3B0@mac.com> References: <6DF43F0C-CD00-499F-82DF-DBFCE208F3B0@mac.com> Message-ID: <1141318481.4834.274.camel@localhost.localdomain> Not at present. It might not be terribly difficult to add that functionality, but it wouldn't be trivial. I'm personally not too familiar with it, so I can't really predict how hard it would be. It would be a very valuable addition to the project, however. On Thu, 2006-03-02 at 11:31 -0500, Kevin Y. Kim wrote: > Admittedly, I don't know much about Sleepycat BDB (or BDBs in general), > but is it possible to use Elephant against other BDBs > (i.e. GDBM, NDBM, QDBM)? Just curious. > > Thanks, > -kevin > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From eslick at csail.mit.edu Thu Mar 2 16:56:16 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Thu, 02 Mar 2006 11:56:16 -0500 Subject: [elephant-devel] Other BDBs? In-Reply-To: <6DF43F0C-CD00-499F-82DF-DBFCE208F3B0@mac.com> References: <6DF43F0C-CD00-499F-82DF-DBFCE208F3B0@mac.com> Message-ID: <440723B0.5050804@csail.mit.edu> A quick acronym decomposition will help here: BDB = Berkeley DataBase Sleepycat = the company that commercialized Berkeley's database code. BDB is a library that implements on-disk BTrees, memory management, transactions, etc. It's possible to write a new backend to support other databases based on tables of various kinds but it's a non-trivial exercise, particularly to handle the foreign function calls to most libraries. The front-end of elephant provides support for a metaclass protocol and a BTree API to manage persistent lisp objects and tables. We do hope someday to have a native lisp backend. I hope that is helpful, Ian Kevin Y. Kim wrote: > Admittedly, I don't know much about Sleepycat BDB (or BDBs in general), > but is it possible to use Elephant against other BDBs > (i.e. GDBM, NDBM, QDBM)? Just curious. > > Thanks, > -kevin > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From whalliburton at gmail.com Thu Mar 2 22:11:48 2006 From: whalliburton at gmail.com (William Halliburton) Date: Thu, 2 Mar 2006 17:11:48 -0500 Subject: [elephant-devel] Newbee questions Message-ID: <4e7bd29e0603021411w4e23d9f8t2b61dfcd0600146b@mail.gmail.com> Starting to use elephant with bdb backend. Please tell me if the following explanation is correct. Each time I make-instance a class with a metaclass of :persistent-metaclass the class is placed into the database with a unique OID. I can retrieve this back with (make-instance 'class :with-oid OID). Now this object in not in the root. Must I place it into the root? How do I remove old objects? Thank you, William Halliburton -------------- next part -------------- An HTML attachment was scrubbed... URL: From read at robertlread.net Fri Mar 3 14:59:54 2006 From: read at robertlread.net (Robert L. Read) Date: Fri, 03 Mar 2006 08:59:54 -0600 Subject: [elephant-devel] Newbee questions In-Reply-To: <4e7bd29e0603021411w4e23d9f8t2b61dfcd0600146b@mail.gmail.com> References: <4e7bd29e0603021411w4e23d9f8t2b61dfcd0600146b@mail.gmail.com> Message-ID: <1141397995.4834.324.camel@localhost.localdomain> Dear William, Welcome. Are you using 0.5 or the latest version from CVS (which has not quite been formally released yet)? I think this changes slightly based on which version you are using. Under the 0.5: I think the "Tutorial" on the section Persistent Classes http://common-lisp.net/project/elephant/doc/Persistent- Classes.html#Persistent-Classes shows how to do what you want; I'm pretty sure that it is accurate, but haven't actually done it. It shows placing the whole object into the root; I honestly would have to go back to 0.5 to say whether this is really necessary or not. You can try my code snippet below under 0.5 and see if it works. (We have been working on what will be 0.6 for quite a while.) Under the most recent CVS version: Yes, each time you make-instance a class with a metaclass of :persistent-metaclass, the instance is placed in the database with a unique OID. If you store the OID (for example in the root), you can can then close the store-controller, exit lisp, restart LISP, reopen the store-controller, and then either do (make-instance 'my-persistent-class :from-oid OID), or, if you have indexed a slot of enables class indexing on the class, you could you (get-instance-by- class 'my-persistent-class) ELE-TESTS> (defclass my-persistent-class () ((slot1 :accessor slot1) (slot2 :accessor slot2)) (:metaclass persistent-metaclass)) # ELE-TESTS> (setq x (make-instance 'my-persistent-class)) ; loading system definition from /usr/local/share/lisp/ele-bdb.asd into ; # ; registering # as ELE-BDB STYLE-WARNING: implicitly creating new generic function BUILD-BTREE- INDEX # ELE-TESTS> (setq x (make-instance 'my-persistent-class)) # ELE-TESTS> (elephant::oid x) 14700 ELE-TESTS> (close-store) NIL ELE-TESTS> (open-store *default-spec*) # ELE-TESTS> (make-instance 'my-persistent-class :from-oid 14700) # ELE-TESTS> In the file classindex.lisp, you can find: (defgeneric get-instances-by-class (persistent-metaclass)) (defgeneric get-instances-by-value (persistent-metaclass slot-name value)) (defgeneric get-instances-by-range (persistent-metaclass slot-name start end)) which are very useful for this, and may mean you don't have to store the OID in the root if you "enable-class-indexing" on the class. On Thu, 2006-03-02 at 17:11 -0500, William Halliburton wrote: > > Starting to use elephant with bdb backend. > > Please tell me if the following explanation is correct. > > Each time I make-instance a class with a metaclass of :persistent- > metaclass the class is placed into the database with a unique OID. I > can retrieve this back with (make-instance 'class :with-oid OID). Now > this object in not in the root. Must I place it into the root? How do > I remove old objects? > > Thank you, > William Halliburton > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From evrim at core.gen.tr Fri Mar 3 18:52:46 2006 From: evrim at core.gen.tr (Evrim ULU) Date: Fri, 03 Mar 2006 20:52:46 +0200 Subject: [elephant-devel] Incompatibilities between different versions Message-ID: <4408907E.9030304@core.gen.tr> Hi, I was using the CVS version for 9.Feb.2006 and created a bdb data-store but it seems elephant code is re-structured and now, with the latest CVS, I am unable to use the old database since when I try to open the store it says "Unable to find class ELEPHANT::BDB-BTREE". I've seen that it's moved to a sleppycat specific package. I've also tried to migrate my old database to postgres via clsql and then back to bdb having newer version of elephant but failed again. So, my question is, is there a way to get around this migration problem. Kind Regards, Evrim. From read at robertlread.net Fri Mar 3 20:15:01 2006 From: read at robertlread.net (Robert L. Read) Date: Fri, 03 Mar 2006 14:15:01 -0600 Subject: [elephant-devel] Incompatibilities between different versions In-Reply-To: <4408907E.9030304@core.gen.tr> References: <4408907E.9030304@core.gen.tr> Message-ID: <1141416901.4834.382.camel@localhost.localdomain> Dear Evrim, Ian and I had identified this problem; I'm sorry that you tripped over it before we documented it or fixed it. You can't open a database less than or equal to 0.5 with the current cvs code. I don't know any very simple way around this problem; Ian and I were planning to address it with a fairly deep solution of assigning version numbers to the formats and then providing a specific upgrade mechanism. So, unfortunately, you options are: 1) Wait 3 or 4 weeks for Iain and I to fix it, 2) Figure it out yourself and give us the patch (good for us, perhaps bad for you...) 3) Continue use 0.5. Let me know what you decide to do or if we can help you more with this; we will definitely have to solve this problem before we make the 0.6 release. On Fri, 2006-03-03 at 20:52 +0200, Evrim ULU wrote: > Hi, > > I was using the CVS version for 9.Feb.2006 and created a bdb data-store > but it seems elephant code is re-structured and now, with the latest > CVS, I am unable to use the old database since when I try to open the > store it says "Unable to find class ELEPHANT::BDB-BTREE". I've seen that > it's moved to a sleppycat specific package. > > I've also tried to migrate my old database to postgres via clsql and > then back to bdb having newer version of elephant but failed again. > > So, my question is, is there a way to get around this migration problem. > > Kind Regards, > Evrim. > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From evrim at core.gen.tr Fri Mar 3 19:38:47 2006 From: evrim at core.gen.tr (Evrim ULU) Date: Fri, 03 Mar 2006 21:38:47 +0200 Subject: [elephant-devel] Incompatibilities between different versions In-Reply-To: <1141416901.4834.382.camel@localhost.localdomain> References: <4408907E.9030304@core.gen.tr> <1141416901.4834.382.camel@localhost.localdomain> Message-ID: <44089B47.4060400@core.gen.tr> Robert L. Read wrote: > > So, unfortunately, you options are: > 1) Wait 3 or 4 weeks for Iain and I to fix it, > 2) Figure it out yourself and give us the patch (good for us, perhaps > bad for you...) > 3) Continue use 0.5. > > Let me know what you decide to do or if we can help you more with > this; > we will definitely have to solve this problem before we make the 0.6 > release. Dear Robert, I've dropped most of the btrees from my code so, it won't be a problem next-time but i am curious if it will happen again or not. If you can state your thoughts about upgrading, it'll be great. Btw, i've tried to solve the problem for a week but unfortunately it seems i'm not familiar with elephant yet. I hope, in time, i'll know better and help you to develop it. Kind Regards, Evrim. From read at robertlread.net Fri Mar 3 21:03:32 2006 From: read at robertlread.net (Robert L. Read) Date: Fri, 03 Mar 2006 15:03:32 -0600 Subject: [elephant-devel] Incompatibilities between different versions In-Reply-To: <44089B47.4060400@core.gen.tr> References: <4408907E.9030304@core.gen.tr> <1141416901.4834.382.camel@localhost.localdomain> <44089B47.4060400@core.gen.tr> Message-ID: <1141419812.4834.401.camel@localhost.localdomain> Thank you, your help will be welcome. My thoughts are on upgrading are: I think Elephant is getting mature enough and possibly having enough users (though I have little hard evidence of this) that we need to start taking upgrading very seriously, and ensuring and testing that one can always go from a database create with version X to a database created with version X+N smoothly. By "smoothly", I do not mean that you can upgrade with out noticing a change or having to do something, but rather that there will be a well-defined process that you don't have to be an Elephant hacker to follow to get from X to X+N. Probably, this will be automated in most cases, and will be triggered upon attempting to make a connection. This policy allows Elephant to continue to improve in two ways: 1) We can improve the serializers, which are really efficiency-only improvements but none the less very important in terms of performance. The CL-SQL serializer in particular is obviously improvable. 2) We might make a data change which can be computed, but none the less requires the user to make a decision. For example, a new indexing method which is usually faster but sometimes slower becomes available. The user should make a constant decision whether to use the new indexing method or not. So, under this policy, the act of upgrading will in general take a little human interaction, generally in the form of answering some multiple-choice questions, and may require a "re-writing" of the database while it is quiescent (that is, off-line) to use a new serialization codec or somesuch. I think the Elephant project should make the promise (P): P: Any data you now have in an Elephant repository, will be movable somehow, to a repository at any higher version number. I invite discussion of this proposed policy. I don't really know how many people out there are using Elephant, and of them how many, if any, are using it in long-lived applications with important data. But I know that if we want that to occur in the future, we have to act in a manner supportive of it now. As far as code compatibility is concerned, I think it is acceptable to change the API in slight, and documented manners. For example, moving the packaging structure, which Ian did, probably breaks some code that uses a previous version and requires very minor code tweaks to upgrade to the latest version. I think this is acceptable now because: 1) most Elephant users seem to know as much about it as I do :->, and 2) there are still big improvements (such as Ian's reorganization) that we can make if we don't try to fix the API too early. Certainly, I personally want to use Elephant for long-lived, permanent data and would as a user demand promise P of a system like Elephant. On Fri, 2006-03-03 at 21:38 +0200, Evrim ULU wrote: > Dear Robert, > > I've dropped most of the btrees from my code so, it won't be a problem > next-time but i am curious if it will happen again or not. If you can > state your thoughts about upgrading, it'll be great. > > Btw, i've tried to solve the problem for a week but unfortunately it > seems i'm not familiar with elephant yet. I hope, in time, i'll know > better and help you to develop it. -------------- next part -------------- An HTML attachment was scrubbed... URL: From evrim at core.gen.tr Fri Mar 3 20:32:43 2006 From: evrim at core.gen.tr (Evrim ULU) Date: Fri, 03 Mar 2006 22:32:43 +0200 Subject: [elephant-devel] Incompatibilities between different versions In-Reply-To: <1141419812.4834.401.camel@localhost.localdomain> References: <4408907E.9030304@core.gen.tr> <1141416901.4834.382.camel@localhost.localdomain> <44089B47.4060400@core.gen.tr> <1141419812.4834.401.camel@localhost.localdomain> Message-ID: <4408A7EB.5000105@core.gen.tr> Robert L. Read wrote: > > Certainly, I personally want to use Elephant for long-lived, permanent > data and would > as a user demand promise P of a system like Elephant. Robert, Yes, i am trying to use elephant in a serious but *very-experimental* application. I would be glad if I can somehow upgrade btrees next time. I am positive about code tweaking until 1.0. In addition, I agree with all the ideas you have stated. I hope, i'll be helpful in elephant development. Kind Regards, Evrim. From aycan.irican at core.gen.tr Fri Mar 3 21:11:57 2006 From: aycan.irican at core.gen.tr (Aycan iRiCAN) Date: Fri, 03 Mar 2006 23:11:57 +0200 Subject: [elephant-devel] Question about mailing list Message-ID: <873bhz44iq.fsf@coretal.core.gen.tr> Hi, I think you can protect this page so we get less SP4M. You would better change the list options to protect our emails from malicious bots. http://common-lisp.net/cgi-bin/mailman/roster/elephant-devel Best Regards, -- Aycan iRiCAN C0R3 Computer Security Group http://www.core.gen.tr -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 188 bytes Desc: not available URL: From read at robertlread.net Fri Mar 3 21:53:00 2006 From: read at robertlread.net (Robert L. Read) Date: Fri, 03 Mar 2006 15:53:00 -0600 Subject: [elephant-devel] Question about mailing list In-Reply-To: <873bhz44iq.fsf@coretal.core.gen.tr> References: <873bhz44iq.fsf@coretal.core.gen.tr> Message-ID: <1141422780.4834.409.camel@localhost.localdomain> I have changed the settings so that only the members can see that list. Is that what you meant? I don't know what else might be changed; I'm sorry if anyone has gotten spammed due to my negligence about that. On Fri, 2006-03-03 at 23:11 +0200, Aycan iRiCAN wrote: > Hi, > > I think you can protect this page so we get less SP4M. You would > better change the list options to protect our emails from malicious > bots. > > http://common-lisp.net/cgi-bin/mailman/roster/elephant-devel > > Best Regards, > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From whalliburton at gmail.com Sat Mar 4 05:15:00 2006 From: whalliburton at gmail.com (William Halliburton) Date: Sat, 4 Mar 2006 00:15:00 -0500 Subject: [elephant-devel] Newbee questions In-Reply-To: <1141397995.4834.324.camel@localhost.localdomain> References: <4e7bd29e0603021411w4e23d9f8t2b61dfcd0600146b@mail.gmail.com> <1141397995.4834.324.camel@localhost.localdomain> Message-ID: <4e7bd29e0603032115l59304d6bv3a372b261267f728@mail.gmail.com> Thank you for the explanation. I was mostly looking to see how to GC the group. Looks like a future feture. William On 3/3/06, Robert L. Read wrote: > > Dear William, > Welcome. > > Are you using 0.5 or the latest version from CVS (which has not quite > been formally released yet)? > I think this changes slightly based on which version you are using. > > Under the 0.5: > > I think the "Tutorial" on the section Persistent Classes > > http://common-lisp.net/project/elephant/doc/Persistent-Classes.html#Persistent-Classes > shows how to do what you want; I'm pretty sure that it is accurate, but > haven't actually done it. > It shows placing the whole object into the root; I honestly would have to > go back to 0.5 to > say whether this is really necessary or not. You can try my code snippet > below under 0.5 > and see if it works. (We have been working on what will be 0.6 for quite > a while.) > > Under the most recent CVS version: > > Yes, each time you make-instance a class with a metaclass of > :persistent-metaclass, the instance > is placed in the database with a unique OID. If you store the OID (for > example in the root), you > can can then close the store-controller, exit lisp, restart LISP, reopen > the store-controller, and > then either do (make-instance 'my-persistent-class :from-oid OID), or, if > you have indexed a slot > of enables class indexing on the class, you could you > (get-instance-by-class 'my-persistent-class) > > ELE-TESTS> (defclass my-persistent-class () > ((slot1 :accessor slot1) > (slot2 :accessor slot2)) > (:metaclass persistent-metaclass)) > > # > ELE-TESTS> (setq x (make-instance 'my-persistent-class)) > ; loading system definition from /usr/local/share/lisp/ele-bdb.asd into > ; # > ; registering # as ELE-BDB > STYLE-WARNING: implicitly creating new generic function BUILD-BTREE-INDEX > # > ELE-TESTS> (setq x (make-instance 'my-persistent-class)) > # > ELE-TESTS> (elephant::oid x) > 14700 > ELE-TESTS> (close-store) > NIL > ELE-TESTS> (open-store *default-spec*) > # > ELE-TESTS> (make-instance 'my-persistent-class :from-oid 14700) > # > ELE-TESTS> > > In the file classindex.lisp, you can find: > > (defgeneric get-instances-by-class (persistent-metaclass)) > (defgeneric get-instances-by-value (persistent-metaclass slot-name value)) > (defgeneric get-instances-by-range (persistent-metaclass slot-name start > end)) > > which are very useful for this, and may mean you don't have to store the > OID in the root if > you "enable-class-indexing" on the class. > > > > > > > On Thu, 2006-03-02 at 17:11 -0500, William Halliburton wrote: > > > Starting to use elephant with bdb backend. > > Please tell me if the following explanation is correct. > > Each time I make-instance a class with a metaclass of > :persistent-metaclass the class is placed into the database with a unique > OID. I can retrieve this back with (make-instance 'class :with-oid OID). Now > this object in not in the root. Must I place it into the root? How do I > remove old objects? > > Thank you, > William Halliburton > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From whalliburton at gmail.com Sat Mar 4 05:15:38 2006 From: whalliburton at gmail.com (William Halliburton) Date: Sat, 4 Mar 2006 00:15:38 -0500 Subject: [elephant-devel] AMD64 problems Message-ID: <4e7bd29e0603032115m7b97a1cek34595bb0142ca32f@mail.gmail.com> Elephant does not seem to work on my AMD64 with SBCL 0.9.10 or 0.9.9. See the following trace. Elephant version 0.5.0. Same occurs for current CVS. Does anyone have similar problems? William Halliburton /usr/lib64/sbcl/src/code $ sbcl This is SBCL 0.9.10, an implementation of ANSI Common Lisp. More information about SBCL is available at . SBCL is free software, provided as is, with absolutely no warranty. It is mostly in the public domain; some portions are provided under BSD-style licenses. See the CREDITS and COPYING files in the distribution for more information. * *features* (:ASDF :CLC-OS-DEBIAN :COMMON-LISP-CONTROLLER :SB-THREAD :ANSI-CL :COMMON-LISP :SBCL :UNIX :SB-DOC :SB-PACKAGE-LOCKS :SB-UNICODE :SB-SOURCE-LOCATIONS :IEEE-FLOATING-POINT :X86-64 :ELF :LINUX :GENCGC :STACK-GROWS-DOWNWARD-NOT-UPWARD :C-STACK-IS-CONTROL-STACK :LINKAGE-TABLE :STACK-ALLOCATABLE-CLOSURES :ALIEN-CALLBACKS :OS-PROVIDES-DLOPEN :OS-PROVIDES-DLADDR :OS-PROVIDES-PUTWC) * (require 'elephant) ; loading system definition from /usr/share/common-lisp/systems/uffi.asd into ; # ; registering # as UFFI ; loading system definition from /usr/share/common-lisp/systems/cl- base64.asd ; into # ; registering # as CL-BASE64 ; registering # as CL-BASE64-TESTS ; loading system definition from /usr/share/common-lisp/systems/kmrcl.asd into ; # ; registering # as KMRCL ("SB-BSD-SOCKETS") * (require 'ele-bdb) NIL * (use-package :elephant) T * (defparameter *db-path* "/home/conrad/website/ucw/ele-test/db") *DB-PATH* * (open-store *db-path*) # * (get-from-root "foo1") NIL NIL * (defclass foo () ((name :accessor foo.name :initarg :name :initform "")) (:metaclass persistent-metaclass)) # * (add-to-root "foo1" (make-instance 'foo :name "asd")) # * (get-from-root "foo1") Argh! gc_find_freeish_pages failed (restart_page), nbytes=300647710752. -------------- next part -------------- An HTML attachment was scrubbed... URL: From read at robertlread.net Sat Mar 4 15:14:46 2006 From: read at robertlread.net (Robert L. Read) Date: Sat, 04 Mar 2006 09:14:46 -0600 Subject: [elephant-devel] Newbee questions In-Reply-To: <4e7bd29e0603032115l59304d6bv3a372b261267f728@mail.gmail.com> References: <4e7bd29e0603021411w4e23d9f8t2b61dfcd0600146b@mail.gmail.com> <1141397995.4834.324.camel@localhost.localdomain> <4e7bd29e0603032115l59304d6bv3a372b261267f728@mail.gmail.com> Message-ID: <1141485286.4834.413.camel@localhost.localdomain> If by GC you mean Garbage Collect, Ian has pointed out that migrate a store and migrating back represents a primitive form of generational garbage collection. On Sat, 2006-03-04 at 00:15 -0500, William Halliburton wrote: > Thank you for the explanation. I was mostly looking to see how to GC > the group. Looks like a future feture. > > William > > On 3/3/06, Robert L. Read wrote: > > Dear William, > Welcome. > > Are you using 0.5 or the latest version from CVS (which > has not quite been formally released yet)? > I think this changes slightly based on which version you are > using. > > Under the 0.5: > > I think the "Tutorial" on the section Persistent Classes > http://common-lisp.net/project/elephant/doc/Persistent- > Classes.html#Persistent-Classes > shows how to do what you want; I'm pretty sure that it is > accurate, but haven't actually done it. > It shows placing the whole object into the root; I honestly > would have to go back to 0.5 to > say whether this is really necessary or not. You can try my > code snippet below under 0.5 > and see if it works. (We have been working on what will be > 0.6 for quite a while.) > > Under the most recent CVS version: > > Yes, each time you make-instance a class with a metaclass > of :persistent-metaclass, the instance > is placed in the database with a unique OID. If you store the > OID (for example in the root), you > can can then close the store-controller, exit lisp, restart > LISP, reopen the store-controller, and > then either do (make-instance 'my-persistent-class :from-oid > OID), or, if you have indexed a slot > of enables class indexing on the class, you could you (get- > instance-by-class 'my-persistent-class) > > ELE-TESTS> (defclass my-persistent-class () > ((slot1 :accessor slot1) > (slot2 :accessor slot2)) > (:metaclass persistent-metaclass)) > > # > ELE-TESTS> (setq x (make-instance 'my-persistent-class)) > ; loading system definition from /usr/local/share/lisp/ele- > bdb.asd into > ; # > ; registering # as ELE-BDB > STYLE-WARNING: implicitly creating new generic function BUILD- > BTREE-INDEX > # > ELE-TESTS> (setq x (make-instance 'my-persistent-class)) > # > ELE-TESTS> (elephant::oid x) > 14700 > ELE-TESTS> (close-store) > NIL > ELE-TESTS> (open-store *default-spec*) > # > ELE-TESTS> (make-instance 'my-persistent-class :from-oid > 14700) > # > ELE-TESTS> > > In the file classindex.lisp, you can find: > > (defgeneric get-instances-by-class (persistent-metaclass)) > (defgeneric get-instances-by-value (persistent-metaclass slot- > name value)) > (defgeneric get-instances-by-range (persistent-metaclass slot- > name start end)) > > which are very useful for this, and may mean you don't have to > store the OID in the root if > you "enable-class-indexing" on the class. > > > > > > > On Thu, 2006-03-02 at 17:11 -0500, William Halliburton wrote: > > > Starting to use elephant with bdb backend. > > Please tell me if the following explanation is correct. > > Each time I make-instance a class with a metaclass > of :persistent-metaclass the class is placed into the database > with a unique OID. I can retrieve this back with (make- > instance 'class :with-oid OID). Now this object in not in the > root. Must I place it into the root? How do I remove old > objects? > > Thank you, > William Halliburton > > _______________________________________________ > elephant-devel site list > > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel > > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel > > > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From whalliburton at gmail.com Sun Mar 5 03:12:04 2006 From: whalliburton at gmail.com (William Halliburton) Date: Sat, 4 Mar 2006 22:12:04 -0500 Subject: [elephant-devel] Key dangers. Message-ID: <4e7bd29e0603041912h7c978a3w41372e0fb43e5c6b@mail.gmail.com> I am new so this is probably old news. Maybe need to place warning in manual under btrees. Not a bug, just a pain in the a... Been tracking a wierd behavior when I was using (get-value str *db*) and I could not get out what I put in. Tracked it down to the keys being of different string types. One of simple-base-string and the other of (simple-array character). (setf (get-value (namestring #p"asd") *db*) "foo") (get-value "asd" *db*) -> NIL Section 2.9 mentions that the BTrees hash on value via the serialization and i'm sure that the two string type serialize differently. Maybe a line like: "Hashing on strings of different types will result in separate hash entries." Thank you, William Halliburton -------------- next part -------------- An HTML attachment was scrubbed... URL: From eslick at csail.mit.edu Sun Mar 5 04:45:58 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Sat, 04 Mar 2006 23:45:58 -0500 Subject: [elephant-devel] Key dangers. In-Reply-To: <4e7bd29e0603041912h7c978a3w41372e0fb43e5c6b@mail.gmail.com> References: <4e7bd29e0603041912h7c978a3w41372e0fb43e5c6b@mail.gmail.com> Message-ID: <440A6D06.7070607@csail.mit.edu> William, What distribution are you using? There is an issue with unicode lisps that has resulted in these differences and it's possible that the system is not yet properly systematic. Everyone, I've been looking at a fix for 0.5.0 to 0.6.0 upgrades which is essentially a deprecated symbol map that will take symbols defined in the elephant package and move them to symbols defined in the sleepycat package. This would be invisible to users. I just haven't had a chance to test it yet. In general I agree with Robert's comments about upgrading. The current migration code would be easy to turn into a database serializer that would allow you to dump and reload an entire db in a reasonable way that was independent of a specific backend. I won't be able to get to this for awhile but I'm happy to engage in a discussion if someone would like to implement it. Regards, Ian William Halliburton wrote: > > I am new so this is probably old news. Maybe need to place warning in > manual under btrees. > > Not a bug, just a pain in the a... > > Been tracking a wierd behavior when I was using (get-value str *db*) > and I could not get out what I put in. > > Tracked it down to the keys being of different string types. One of > simple-base-string and the other of (simple-array character). > > (setf (get-value (namestring #p"asd") *db*) "foo") > > (get-value "asd" *db*) -> NIL > > Section 2.9 mentions that the BTrees hash on value via the > serialization and i'm sure that the two string type serialize > differently. Maybe a line like: > > "Hashing on strings of different types will result in separate hash > entries." > > Thank you, > William Halliburton > ------------------------------------------------------------------------ > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From whalliburton at gmail.com Sun Mar 5 05:25:06 2006 From: whalliburton at gmail.com (William Halliburton) Date: Sun, 5 Mar 2006 00:25:06 -0500 Subject: [elephant-devel] Key dangers. In-Reply-To: <440A6D06.7070607@csail.mit.edu> References: <4e7bd29e0603041912h7c978a3w41372e0fb43e5c6b@mail.gmail.com> <440A6D06.7070607@csail.mit.edu> Message-ID: <4e7bd29e0603042125r413aafd2sa8483bcbadba20c9@mail.gmail.com> Sorry, should have been more specific. SBCL 0.9.9 On 3/4/06, Ian Eslick wrote: > > William, > > What distribution are you using? There is an issue with unicode lisps > that has resulted in these differences and it's possible that the system > is not yet properly systematic. > > Everyone, > > I've been looking at a fix for 0.5.0 to 0.6.0 upgrades which is > essentially a deprecated symbol map that will take symbols defined in > the elephant package and move them to symbols defined in the sleepycat > package. This would be invisible to users. I just haven't had a chance > to test it yet. > > In general I agree with Robert's comments about upgrading. The current > migration code would be easy to turn into a database serializer that > would allow you to dump and reload an entire db in a reasonable way that > was independent of a specific backend. I won't be able to get to this > for awhile but I'm happy to engage in a discussion if someone would like > to implement it. > > Regards, > Ian > > William Halliburton wrote: > > > > I am new so this is probably old news. Maybe need to place warning in > > manual under btrees. > > > > Not a bug, just a pain in the a... > > > > Been tracking a wierd behavior when I was using (get-value str *db*) > > and I could not get out what I put in. > > > > Tracked it down to the keys being of different string types. One of > > simple-base-string and the other of (simple-array character). > > > > (setf (get-value (namestring #p"asd") *db*) "foo") > > > > (get-value "asd" *db*) -> NIL > > > > Section 2.9 mentions that the BTrees hash on value via the > > serialization and i'm sure that the two string type serialize > > differently. Maybe a line like: > > > > "Hashing on strings of different types will result in separate hash > > entries." > > > > Thank you, > > William Halliburton > > ------------------------------------------------------------------------ > > > > _______________________________________________ > > elephant-devel site list > > elephant-devel at common-lisp.net > > http://common-lisp.net/mailman/listinfo/elephant-devel > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aycan.irican at core.gen.tr Sun Mar 5 18:36:21 2006 From: aycan.irican at core.gen.tr (Aycan iRiCAN) Date: Sun, 05 Mar 2006 20:36:21 +0200 Subject: [elephant-devel] cl-store Message-ID: <873bhw20yi.fsf@core.gen.tr> A little question: Why elephant wrote their own serializers instead of reuse the cl-store package? Best Regards, -- Aycan iRiCAN C0R3 Computer Security Group http://www.core.gen.tr -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 188 bytes Desc: not available URL: From midfield at gmail.com Sun Mar 5 19:46:21 2006 From: midfield at gmail.com (Ben) Date: Sun, 5 Mar 2006 13:46:21 -0600 Subject: [elephant-devel] cl-store In-Reply-To: <873bhw20yi.fsf@core.gen.tr> References: <873bhw20yi.fsf@core.gen.tr> Message-ID: <9157df230603051146p613da1eei945537f9261186af@mail.gmail.com> i did check out cl-store. i liked it for the most part. i think my original reasons for not using it was because i wanted to serialize to native arrays for use with FFI. i think this rears it's ugly head with stuff like unicode. there may have been some other reasons i can't entirely remember. B On 3/5/06, Aycan iRiCAN wrote: > > A little question: > > Why elephant wrote their own serializers instead of reuse the cl-store > package? > > Best Regards, > > -- > Aycan iRiCAN > C0R3 Computer Security Group > http://www.core.gen.tr > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel > > > From kykim_lists at mac.com Mon Mar 6 23:25:51 2006 From: kykim_lists at mac.com (Kevin Y. Kim (Lists)) Date: Mon, 6 Mar 2006 18:25:51 -0500 Subject: [elephant-devel] Too many files open Message-ID: <7AED1645-D413-4147-8063-91A77CD3D5FD@mac.com> Hi, (Perhaps I'm going about this all wrong, but...) I'm trying to populate a large number of Sleepycat databases. I've created the necessary underlying directories. What I'm attempting is something like this: (defun populate-db (tree data-pairs) (mapcar #'(lambda (p) (setf (get-value (car p) tree) (cadr p))) data-pairs)) (mapcar #'(lambda (db) (with-open-store db (setf btree (make-btree) (add-to-root "tree" btree) (populate-db *btree* list-of-data-pairs))) list-of-databases) I have hundreds of databases, with each database having thousands of data-pairs. So, after loading up around 150 databases, I get an error from Sleepycat that I've got too many databases open. I *assume* that even though I've wrapped everything in the "with-open-store" call, the databases are not closing because I've got a reference to a btree. Is that correct? If so, is there an explicit way I can ensure that each database is closed before I move on to the next one? If anyone has alternate suggestions on how I should store this data, feel free to pass them along. Thanks, -kevin From read at robertlread.net Tue Mar 7 00:16:46 2006 From: read at robertlread.net (Robert L. Read) Date: Mon, 06 Mar 2006 18:16:46 -0600 Subject: [elephant-devel] Too many files open In-Reply-To: <7AED1645-D413-4147-8063-91A77CD3D5FD@mac.com> References: <7AED1645-D413-4147-8063-91A77CD3D5FD@mac.com> Message-ID: <1141690607.4834.468.camel@localhost.localdomain> Dear Kevin, I would not go so far as to see it is outside what you should expect of Elephant, but certainly if I had to store thousands of datapoints in conceptual groups numbering in the hundreds my first instinct would be to place them all in one database, and put them in different btrees there. Sleepycat should be perfectly happy with millions and millions of data objects. I vaguely remember running into a problem similar to what you are having. The idiom you are using, the macro "with-open-store", should be closing everything, and appears to have the code in place to do the right thing. You are using it correctly, at least conceptually. I cannot guess whether this is a bug in the way that we close things, or some bug in the Sleepycat library itself that will be harder to work around. A heroic programmer would take your code and turn it in to a test, and add the test the suite, so that even if there is some limit we can not easily work around, we can at least explore the limit. If you will post or send me an example of that, I will work it into the test suite. It will be good to make sure that both the Sleepycat and SQL implementations work correctly in this respect. I don't know anything about your needs or application, but you certainly should consider simply using one database, unless there is some obvious reason this is bad for you. (In my opinion, the issue of "data ownership" is the only one that typically drives you to split a datastore across separate databases: if client X things they own the data and want to be able to wipe it out and back it up independent of client Y, it is often easiest to use completely separate datastores. But there is not much of a performance reason to do so. On Mon, 2006-03-06 at 18:25 -0500, Kevin Y. Kim (Lists) wrote: > Hi, > > (Perhaps I'm going about this all wrong, but...) > > I'm trying to populate a large number of Sleepycat databases. > I've created the necessary underlying directories. > What I'm attempting is something like this: > > (defun populate-db (tree data-pairs) > (mapcar #'(lambda (p) (setf (get-value (car p) tree) (cadr p))) > data-pairs)) > > (mapcar #'(lambda (db) > (with-open-store db > (setf btree (make-btree) > (add-to-root "tree" btree) > (populate-db *btree* list-of-data-pairs))) > list-of-databases) > > I have hundreds of databases, with each database having thousands of > data-pairs. > > So, after loading up around 150 databases, I get an error from > Sleepycat that > I've got too many databases open. I *assume* that even though I've > wrapped > everything in the "with-open-store" call, the databases are not closing > because I've got a reference to a btree. Is that correct? If so, is > there an > explicit way I can ensure that each database is closed before I move > on to the > next one? > > If anyone has alternate suggestions on how I should store this data, > feel free to > pass them along. > > Thanks, > -kevin > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel ---- Robert L. Read, PhD read &T robertlread.net Consider visiting Progressive Engineering: http://robertlread.net/pe In Austin: 912-8593 "Think globally, Act locally." -- RBF -------------- next part -------------- An HTML attachment was scrubbed... URL: From eslick at csail.mit.edu Tue Mar 7 04:06:43 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Mon, 06 Mar 2006 23:06:43 -0500 Subject: [elephant-devel] Too many files open In-Reply-To: <1141690607.4834.468.camel@localhost.localdomain> References: <7AED1645-D413-4147-8063-91A77CD3D5FD@mac.com> <1141690607.4834.468.camel@localhost.localdomain> Message-ID: <440D06D3.7020303@csail.mit.edu> Another note on lots of databases. with-open-store is a pretty expensive operation, so unless you are only opening them infrequently and for batch operation, a fine-grained use of with-open-store is likely to cost you a great deal of performance. Ian Robert L. Read wrote: > Dear Kevin, > I would not go so far as to see it is outside what you should > expect of Elephant, but > certainly if I had to store thousands of datapoints in conceptual > groups numbering in the > hundreds my first instinct would be to place them all in one database, > and put them > in different btrees there. > Sleepycat should be perfectly happy with millions and millions of > data objects. > I vaguely remember running into a problem similar to what you are > having. > The idiom you are using, the macro "with-open-store", should be > closing everything, > and appears to have the code in place to do the right thing. You are > using it correctly, > at least conceptually. > I cannot guess whether this is a bug in the way that we close > things, or some bug > in the Sleepycat library itself that will be harder to work around. > A heroic programmer would take your code and turn it in to a test, > and add the test > the suite, so that even if there is some limit we can not easily work > around, we can > at least explore the limit. If you will post or send me an example of > that, I will > work it into the test suite. It will be good to make sure that both > the Sleepycat and > SQL implementations work correctly in this respect. > > I don't know anything about your needs or application, but you > certainly should consider > simply using one database, unless there is some obvious reason this is > bad for you. > (In my opinion, the issue of "data ownership" is the only one that > typically drives you > to split a datastore across separate databases: if client X things > they own the data > and want to be able to wipe it out and back it up independent of > client Y, it is often > easiest to use completely separate datastores. But there is not much > of a performance > reason to do so. > > > On Mon, 2006-03-06 at 18:25 -0500, Kevin Y. Kim (Lists) wrote: >> Hi, >> >> (Perhaps I'm going about this all wrong, but...) >> >> I'm trying to populate a large number of Sleepycat databases. >> I've created the necessary underlying directories. >> What I'm attempting is something like this: >> >> (defun populate-db (tree data-pairs) >> (mapcar #'(lambda (p) (setf (get-value (car p) tree) (cadr p))) >> data-pairs)) >> >> (mapcar #'(lambda (db) >> (with-open-store db >> (setf btree (make-btree) >> (add-to-root "tree" btree) >> (populate-db *btree* list-of-data-pairs))) >> list-of-databases) >> >> I have hundreds of databases, with each database having thousands of >> data-pairs. >> >> So, after loading up around 150 databases, I get an error from >> Sleepycat that >> I've got too many databases open. I *assume* that even though I've >> wrapped >> everything in the "with-open-store" call, the databases are not closing >> because I've got a reference to a btree. Is that correct? If so, is >> there an >> explicit way I can ensure that each database is closed before I move >> on to the >> next one? >> >> If anyone has alternate suggestions on how I should store this data, >> feel free to >> pass them along. >> >> Thanks, >> -kevin >> >> _______________________________________________ >> elephant-devel site list >> elephant-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/elephant-devel >> > ---- > Robert L. Read, PhD read &T > robertlread.net > Consider visiting Progressive Engineering: http://robertlread.net/pe > In Austin: 912-8593 "Think > globally, Act locally." -- RBF > > > ------------------------------------------------------------------------ > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From eslick at csail.mit.edu Tue Mar 7 19:12:01 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Tue, 07 Mar 2006 14:12:01 -0500 Subject: [elephant-devel] Test Message-ID: <440DDB01.7070302@csail.mit.edu> I haven't been able to test my little patch to controllers.lisp that should handle upgrading from 0.5.0 db's to 0.6.0 db's. Could someone run a quick test on my behalf? I'm happy to help guide a short debugging cycle (it's a trivial change - recognize deprecated classes and create the replacement class instead). A more robust solution would be to migrate and overload the way persistent classes are serialized for one store and deserialized for another. Probably we need to add a way to store simple database metadata (list structure as a registry with strings, symbols and numbers) that contains version and other information. Upon opening we can look up version specific upgrade code if the elephant version is higher than the database version and upgrade upon open, prompting the user for a target spec if upgrade-by-copy or in-place. The easiest way to do this is to reserve some keywords in the root by using a reserved value such as 'elephant::*elephant-metadata*. In fact I recommend we agree on a standard for this and ensure that 0.6.0 contains this metadata so we can at least recognize 0.6.0+ db's when they are opened. I apologizing if I'm repeating earlier discussions. I've been effectively offline for the past week and a half. Proposal: (add-to-root 'elephant::*elephant-metadata* '((version 0 6 0) ;; format (version major minor sub1 ... subN) (spec :bdb "foo") ;; format (spec ) (lisp . :sbcl))) ;; format (lisp . ) In fact we might do the following as well: 1) assert an error when add-to-root is used by the user to write elephant::*elephant-metadata* 2) create a simple interface around add-to-root which provides metadata access (defmethod get-metadata (tag &optional (sc *store-controller*)) (let ((entry (assoc tag (get-from-root *elephant-metadata*))) (when entry (cdr entry)))) (defmethod set-metadata (tag value &optional (sc *store-controller*)) (unless (reserved-metadata-tags tag) ...)) This way we make sure the user cannot shoot themselves in the foot by overwriting reserved metadata tags or the reserved root keyword. Migration should handle the reserved keyword specially by not migrating reserved tags but migrating any user-defined tags. This might be overkill as users may only need access to system-maintained metadata rather than yet another namespace for storing data as the root should function equally well... Thoughts? Ian From read at robertlread.net Tue Mar 7 20:00:07 2006 From: read at robertlread.net (Robert L. Read) Date: Tue, 07 Mar 2006 14:00:07 -0600 Subject: [elephant-devel] Test In-Reply-To: <440DDB01.7070302@csail.mit.edu> References: <440DDB01.7070302@csail.mit.edu> Message-ID: <1141761608.4834.480.camel@localhost.localdomain> This proposed solution is just what I was imagining (only better described.) Certainly, whether we use it or not, 0.6.0 should start attaching a version to a store. On Tue, 2006-03-07 at 14:12 -0500, Ian Eslick wrote: > Probably we need to add a way to store simple database metadata (list > structure as a registry with strings, symbols and numbers) that contains > version and other information. Upon opening we can look up version > specific upgrade code if the elephant version is higher than the > database version and upgrade upon open, prompting the user for a target > spec if upgrade-by-copy or in-place. The easiest way to do this is to > reserve some keywords in the root by using a reserved value such as > 'elephant::*elephant-metadata*. > > In fact I recommend we agree on a standard for this and ensure that > 0.6.0 contains this metadata so we can at least recognize 0.6.0+ db's > when they are opened. I apologizing if I'm repeating earlier > discussions. I've been effectively offline for the past week and a half. > > Proposal: > > (add-to-root 'elephant::*elephant-metadata* > '((version 0 6 0) ;; format (version major minor sub1 ... subN) > (spec :bdb "foo") ;; format (spec ) > (lisp . :sbcl))) ;; format (lisp . ) > > In fact we might do the following as well: > 1) assert an error when add-to-root is used by the user to write > elephant::*elephant-metadata* > 2) create a simple interface around add-to-root which provides metadata > access > > (defmethod get-metadata (tag &optional (sc *store-controller*)) > (let ((entry (assoc tag (get-from-root *elephant-metadata*))) > (when entry > (cdr entry)))) > > (defmethod set-metadata (tag value &optional (sc *store-controller*)) > (unless (reserved-metadata-tags tag) > ...)) > > This way we make sure the user cannot shoot themselves in the foot by > overwriting reserved > metadata tags or the reserved root keyword. Migration should handle the > reserved keyword > specially by not migrating reserved tags but migrating any user-defined > tags. > > This might be overkill as users may only need access to > system-maintained metadata rather > than yet another namespace for storing data as the root should function > equally well... > > Thoughts? > Ian > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From read at robertlread.net Tue Mar 7 20:00:07 2006 From: read at robertlread.net (Robert L. Read) Date: Tue, 07 Mar 2006 14:00:07 -0600 Subject: [elephant-devel] Test In-Reply-To: <440DDB01.7070302@csail.mit.edu> References: <440DDB01.7070302@csail.mit.edu> Message-ID: <1141761608.4834.480.camel@localhost.localdomain> This proposed solution is just what I was imagining (only better described.) Certainly, whether we use it or not, 0.6.0 should start attaching a version to a store. On Tue, 2006-03-07 at 14:12 -0500, Ian Eslick wrote: > Probably we need to add a way to store simple database metadata (list > structure as a registry with strings, symbols and numbers) that contains > version and other information. Upon opening we can look up version > specific upgrade code if the elephant version is higher than the > database version and upgrade upon open, prompting the user for a target > spec if upgrade-by-copy or in-place. The easiest way to do this is to > reserve some keywords in the root by using a reserved value such as > 'elephant::*elephant-metadata*. > > In fact I recommend we agree on a standard for this and ensure that > 0.6.0 contains this metadata so we can at least recognize 0.6.0+ db's > when they are opened. I apologizing if I'm repeating earlier > discussions. I've been effectively offline for the past week and a half. > > Proposal: > > (add-to-root 'elephant::*elephant-metadata* > '((version 0 6 0) ;; format (version major minor sub1 ... subN) > (spec :bdb "foo") ;; format (spec ) > (lisp . :sbcl))) ;; format (lisp . ) > > In fact we might do the following as well: > 1) assert an error when add-to-root is used by the user to write > elephant::*elephant-metadata* > 2) create a simple interface around add-to-root which provides metadata > access > > (defmethod get-metadata (tag &optional (sc *store-controller*)) > (let ((entry (assoc tag (get-from-root *elephant-metadata*))) > (when entry > (cdr entry)))) > > (defmethod set-metadata (tag value &optional (sc *store-controller*)) > (unless (reserved-metadata-tags tag) > ...)) > > This way we make sure the user cannot shoot themselves in the foot by > overwriting reserved > metadata tags or the reserved root keyword. Migration should handle the > reserved keyword > specially by not migrating reserved tags but migrating any user-defined > tags. > > This might be overkill as users may only need access to > system-maintained metadata rather > than yet another namespace for storing data as the root should function > equally well... > > Thoughts? > Ian > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From ml13 at onlinehome.de Sun Mar 12 16:54:07 2006 From: ml13 at onlinehome.de (Kilian Sprotte) Date: Sun, 12 Mar 2006 17:54:07 +0100 Subject: [elephant-devel] small openmcl thing... Message-ID: Hi, first of all thank you very much for solving this terrifying openmcl mop problem!! (What ever it was...) I have a little fix for openmcl below on current CVS: src/elephant/controller.lisp (defclass store-controller () - ((spec :type (or pathname string) + ((spec :type (or pathname string cons) :accessor controller-spec :initarg :spec :documentation "Backend create functions should pass in :spec during make-instance") Openmcl is very picky about setf'ing a slot to a value that is not of declared type... It treats this declaration as an assertion, I would say. The above fix lets things work for me, but I guess it actually should be: + ((spec :type cons Will the spec passed to open-store always be a list? Cheers, Kilian Sprotte From eslick at csail.mit.edu Sun Mar 12 17:27:11 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Sun, 12 Mar 2006 12:27:11 -0500 Subject: [elephant-devel] small openmcl thing... In-Reply-To: References: Message-ID: <441459EF.6010301@csail.mit.edu> The new spec format will indeed always be a list so the old type specification can be replaced with cons or list. Ian Kilian Sprotte wrote: > Hi, > > first of all thank you very much for solving this terrifying openmcl > mop problem!! (What ever it was...) > > I have a little fix for openmcl below on current CVS: > > src/elephant/controller.lisp > > (defclass store-controller () > - ((spec :type (or pathname string) > + ((spec :type (or pathname string cons) > :accessor controller-spec > :initarg :spec > :documentation "Backend create functions should pass in :spec > during make-instance") > > Openmcl is very picky about setf'ing a slot to a value that is not of > declared type... It treats this declaration as an assertion, I would say. > > The above fix lets things work for me, but I guess it actually should be: > > + ((spec :type cons > > Will the spec passed to open-store always be a list? > > Cheers, > > Kilian Sprotte > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From ml13 at onlinehome.de Wed Mar 15 22:00:45 2006 From: ml13 at onlinehome.de (Kilian Sprotte) Date: Wed, 15 Mar 2006 23:00:45 +0100 Subject: [elephant-devel] confusion Message-ID: Hi, while actually trying to start using elephant with bdb, I am quite confused. For now, I am amusing myself filling the store with: (with-open-store (...) (dotimes (i 10000) (ele:add-to-root i (* i i)))) What I do is runing this snippet on the same db over and over again always starting lisp and everything from scratch. After some invocations I notice that the ondisk size of the db keeps growing... Starting from 11MB I quite quickly reach 30MB. What is happening? I thought to overwrite old values... I admit I dont know anything about Sleepycat, so enough reasons to get confused... -Kilian