From nowhere.man at levallois.eu.org Wed Nov 1 17:40:48 2006 From: nowhere.man at levallois.eu.org (Pierre THIERRY) Date: Wed, 1 Nov 2006 18:40:48 +0100 Subject: [elephant-devel] Concurrency problem? In-Reply-To: <1162306483.14560.285.camel@localhost.localdomain> References: <20061031134344.GU7585@bateleur.arcanes.fr.eu.org> <1162306483.14560.285.camel@localhost.localdomain> Message-ID: <20061101174047.GC13832@bateleur.arcanes.fr.eu.org> Scribit Robert L. Read dies 31/10/2006 hora 08:54: > 1) The serializer is definitely not thread-safe. [...] > > 2) CLSQL is not threadsafe unless you use a separate connection in > each thread. [...] Well, I was not aware of those problems. That is a quite huge step forward. > 1) If your access to Elephant is in some relatively narrow place, that > is, you can find a few pieces of code where all of the elephant > requests occur, you can put mutexes around those. I basically maintain all of my objects in Elephant with the help of persistent-metaclass... > 2) Depending on the demands of your application, you might be able to > put in a smaller number of higher-level mutexes I may try that first. > 3) I'd be happy to publish my current version of DCM to anyone who > wants it, but it is SBCL specific, and I personally don't know how to > do mutexes under any other LISP. I'll try to take a look at it, and see if I could convert it to bordeaux-threads. > I know that may not be terribly helpful At least now I have a clue on where the problem is. That *is* helpful... Exclusively, Nowhere man -- nowhere.man at levallois.eu.org OpenPGP 0xD9D50D8A -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From read at robertlread.net Wed Nov 1 17:58:36 2006 From: read at robertlread.net (Robert L. Read) Date: Wed, 01 Nov 2006 11:58:36 -0600 Subject: [elephant-devel] Concurrency problem? In-Reply-To: <20061101174047.GC13832@bateleur.arcanes.fr.eu.org> References: <20061031134344.GU7585@bateleur.arcanes.fr.eu.org> <1162306483.14560.285.camel@localhost.localdomain> <20061101174047.GC13832@bateleur.arcanes.fr.eu.org> Message-ID: <1162403916.14560.377.camel@localhost.localdomain> I have attached the gzipped tar of my current DCM directory. If you can't read it, I'll post it to the elephant project website. On Wed, 2006-11-01 at 18:40 +0100, Pierre THIERRY wrote: > Scribit Robert L. Read dies 31/10/2006 hora 08:54: > > 1) The serializer is definitely not thread-safe. [...] > > > > 2) CLSQL is not threadsafe unless you use a separate connection in > > each thread. [...] > > Well, I was not aware of those problems. That is a quite huge step > forward. > > > 1) If your access to Elephant is in some relatively narrow place, that > > is, you can find a few pieces of code where all of the elephant > > requests occur, you can put mutexes around those. > > I basically maintain all of my objects in Elephant with the help of > persistent-metaclass... > > > 2) Depending on the demands of your application, you might be able to > > put in a smaller number of higher-level mutexes > > I may try that first. > > > 3) I'd be happy to publish my current version of DCM to anyone who > > wants it, but it is SBCL specific, and I personally don't know how to > > do mutexes under any other LISP. > > I'll try to take a look at it, and see if I could convert it to > bordeaux-threads. > > > I know that may not be terribly helpful > > At least now I have a clue on where the problem is. That *is* helpful... > > Exclusively, > Nowhere man > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: dcm.tgz Type: application/x-compressed-tar Size: 72851 bytes Desc: not available URL: From gk1 at four-four.com Sat Nov 4 11:18:27 2006 From: gk1 at four-four.com (George Khouri) Date: Sat, 4 Nov 2006 03:18:27 -0800 Subject: [elephant-devel] Elephant 0.6.0 SQL fix on OpenMCL 1.0 Message-ID: Friends, Testing Elephant 0.6.0 against a postgres db, in OpenMCL 1.0, I found that the close-cursor method fails since it tries to set the value of the sql-cursor slot "curkey" to NIL, while its type spec in the class declaration is INTEGER. Perhaps other lisps allow this. If I change close-cursor to set curkey to -1 as below, the tests run fine. I don't know if there are other routines which check for a null value in that cursor slot, in which case my change wouldn't work. Perhaps the type spec should be removed from the slot declaration instead? The code is in elephant/src/db-clsql/sql-collections.lisp. (defclass sql-cursor (cursor) ((keys :accessor :sql-crsr-ks :initarg :sql-cursor-keys :initform '()) (curkey :accessor :sql-crsr-ck :initarg :sql-cursor-curkey :initform -1 :type integer)) (:documentation "A SQL cursor for traversing (primary) BTrees.")) ... (defmethod cursor-close ((cursor sql-cursor)) #-openmcl (setf (:sql-crsr-ck cursor) nil) #+openmcl (setf (:sql-crsr-ck cursor) -1) ; ** GK (setf (cursor-initialized-p cursor) nil)) Thanks, George -------- George Khouri gk1 at four-four.com From eslick at csail.mit.edu Sat Nov 4 15:19:46 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Sat, 04 Nov 2006 10:19:46 -0500 Subject: [elephant-devel] Concurrency problem? In-Reply-To: <1162306483.14560.285.camel@localhost.localdomain> References: <20061031134344.GU7585@bateleur.arcanes.fr.eu.org> <1162306483.14560.285.camel@localhost.localdomain> Message-ID: <454CAF92.9010406@csail.mit.edu> The current HEAD has a draft of a reentrant serializer, but only for allegro and I think SBCL. If you run db_deadlock on the open BDB database then all the transactions through BDB are thread safe. The next release will try to clean this up so it works for all the backends. I've been out of the country the past two weeks so hope to catch up on list mail in the next few days. Cheers, Ian Robert L. Read wrote: > I have had similar problems. My own solution has been to put mutual > exclusion locks into > DCM. Since my own app only accesses the database through DCM, this > works for me; > I think it would be inconvenient but not impossible if you are using > Elephant directly. > DCM is a "director-based" system that is in the contrib directory, > that aims more toward > a prevalence based system. > > There are/were at least three problems that have to be dealt with: > > 1) The serializer is definitely not thread-safe. In the next > release, we need to improve this, > but for the time being I have added mutexes (these are not checked in > yet.) > > 2) CLSQL is not threadsafe unless you use a separate connection in > each thread. > Elephant's basic use of CLSQL reuses the connections, and so you can have > big problems even at the CLSQL level in using Elephant in that way. > > So, based on what you have described, I can suggest a few solutions, > neither of > which are great: > > 1) If your access to Elephant is in some relatively narrow place, > that is, you can find > a few pieces of code where all of the elephant requests occur, you can > put mutexes around > those. (Code to define and use these mutexes is included at the end of > this message. It > defines a macro, "defemthodex", which is just like defmethod but takes > an extra argument > used to define the mutex. However, this is HIGHLY SBCL specific, and > would not > be trivial to adapt to anything other than DCM --- this is (in additio > to being busy) why > I have not put the thread-safe version of DCM in the contrib directory > yet. It is, however, > an example of how to use mutexes in SBCL, which you can easily copy.) > > 2) Depending on the demands of your application, you might be able to > put in a smaller > number of higher-level mutexes (that is, that operate around larger > bodies of code), and > still serve pages sufficiently rapidly. > > 3) If you use BDB, it will seem better, but the current version of > Elephant is still not > thread-safe, because of the serializer, although it will take you a > lot longer to hit a > problem than under CLSQL. > > I think really Ian and I, and whoever else is interested, need to plan > on solving this > problem in the next major release. To my knoledge, this would mean: > > 1) Writing a small connection-pool manager that is threadsafe in > Elephant, > 2) Making a reentrant serializer. (I know Ian is planning this > already - AMEN!) > 3) I'd be happy to publish my current version of DCM to anyone who > wants it, > but it is SBCL specific, and I personally don't know how to do mutexes > under > any other LISP. > > * * * > I know that may not be terribly helpful ---- but let's continue this > discussion until > we work out a solution. I am extraordinarily busy launching my > business, but am > committed to at least managing patches provided by others or getting > out of the way > if someone else (such as Ian) wants to be the official maintainer. > > > > (defvar *dcm-mutexes* (make-hash-table :test 'equal)) > > (defvar *a-mutex* (sb-thread::make-mutex :name "my lock")) > > (defun insure-mutex (name) > (let ((mtx (gethash name *dcm-mutexes*)) > ) > (or mtx (setf (gethash name *dcm-mutexes*) (sb-thread:make-mutex > :name name))) > ) > ) > > ;; This assumes that the the variable "dir" is being defined and that > we can can > ;; create > (defmacro defmethodex (mname dir args &body body) > `(defmethod ,mname ,(cons dir args) > ;; (format t "Thread ~A running ~%" sb-thread::*current-thread*) > (sb-thread:with-mutex ((insure-mutex (format nil "mutex-~A" ,(car dir)))) > ;; (format t "Thread ~A got the lock~%" > sb-thread::*current-thread*) > (let ((ret > , at body)) > ;; (format t "Thread ~A dropping lock~%" > sb-thread::*current-thread*) > ret > ) > ) > ) > ) > > > > On Tue, 2006-10-31 at 14:43 +0100, Pierre THIERRY wrote: >> I may have a problem with concurrency and elephant. I'm developing a web >> application using SBCL and Araneida (with it's threaded listener). >> Whereas araneida by irself doesn't seem to have problems dealing with >> many requests at a time (I tested it with apache's benckmark tool, with >> thousands of request, with from 1 to 16 concurrent requests to see how >> it reacts), my application just collapses when 3 requests come in a very >> short time. >> >> I had an error I did not copied, which was about a persistent slot of an >> object being unbound (but I know it should be bound when accessed at >> this time), and when trying to access one more time my app, I end up >> with this in the debugger: >> >> While accessing database # >> with expression "SELECT value FROM keyvalue WHERE ((clctn_id = 0) and (key = 'FSAAAAAyAAAANAAAACAAAABUAAAASQAAAFQAAABSAAAARQAAAA=='))": >> Error NIL / >> has occurred. >> [Condition of type CLSQL-SYS:SQL-DATABASE-DATA-ERROR] >> >> The database itself doesn't seem corrupted, as restarting the >> application from scratch, with the same store, works well. For the >> moment, I had to shield my app behind an HTTP cache so that requests are >> basicaly served to the user by the cache instead of my app... >> >> Concurrently, >> Nowhere man >> _______________________________________________ >> elephant-devel site list >> elephant-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/elephant-devel >> > ------------------------------------------------------------------------ > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From nowhere.man at levallois.eu.org Sun Nov 5 20:36:46 2006 From: nowhere.man at levallois.eu.org (Pierre THIERRY) Date: Sun, 5 Nov 2006 21:36:46 +0100 Subject: [elephant-devel] Concurrency problem? In-Reply-To: <454CAF92.9010406@csail.mit.edu> References: <20061031134344.GU7585@bateleur.arcanes.fr.eu.org> <1162306483.14560.285.camel@localhost.localdomain> <454CAF92.9010406@csail.mit.edu> Message-ID: <20061105203646.GB7160@bateleur.arcanes.fr.eu.org> Scribit Ian Eslick dies 04/11/2006 hora 10:19: > If you run db_deadlock on the open BDB database then all the > transactions through BDB are thread safe. I tried BDB, but without success. When doing (do-backend-tests) with the default BDB spec, it stops when compiling the wrap-errno macro. I tried with setting the *sleepycat-foreign-library-path* as - /usr/lib/libdb-4.2.so - /usr/lib/libdb-4.3.so - /usr/lib/libdb-4.4.so But there is no difference between them. I tried this with elephant HEAD, to see if it's globally thread safe. When using elephant 0.6.0, it stops just a form later, at the flags macro, and also exactly in the same way for the three versions of BDB. Thread unsafely, Nowhere man -- nowhere.man at levallois.eu.org OpenPGP 0xD9D50D8A -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From nowhere.man at levallois.eu.org Mon Nov 6 13:40:23 2006 From: nowhere.man at levallois.eu.org (Pierre THIERRY) Date: Mon, 6 Nov 2006 14:40:23 +0100 Subject: [elephant-devel] Concurrency problem? In-Reply-To: <20061101174047.GC13832@bateleur.arcanes.fr.eu.org> References: <20061031134344.GU7585@bateleur.arcanes.fr.eu.org> <1162306483.14560.285.camel@localhost.localdomain> <20061101174047.GC13832@bateleur.arcanes.fr.eu.org> Message-ID: <20061106134022.GD7160@bateleur.arcanes.fr.eu.org> > > 2) Depending on the demands of your application, you might be able > > to put in a smaller number of higher-level mutexes > I may try that first. Well. I just did the opposite. ;-) As all my use of Elephant was through classes with the persistent metaclass, I tried to protect all methods of the slot protocol in elephant/classes.lisp with a mutex. I used the bordeaux-threads library, for this modification to be usable on any multi-threading implementation instead of only one. FWIW, since this patch has been applied to elephant 0.6.0 in my application, there has not been any error. Of course, it slowed down the application significantly, but as it's a web application behind a caching proxy, that's no big deal in my case. Quickly, Nowhere man -- nowhere.man at levallois.eu.org OpenPGP 0xD9D50D8A -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From nowhere.man at levallois.eu.org Tue Nov 7 00:11:27 2006 From: nowhere.man at levallois.eu.org (Pierre THIERRY) Date: Tue, 7 Nov 2006 01:11:27 +0100 Subject: [elephant-devel] Concurrency problem? In-Reply-To: <20061106134022.GD7160@bateleur.arcanes.fr.eu.org> References: <20061031134344.GU7585@bateleur.arcanes.fr.eu.org> <1162306483.14560.285.camel@localhost.localdomain> <20061101174047.GC13832@bateleur.arcanes.fr.eu.org> <20061106134022.GD7160@bateleur.arcanes.fr.eu.org> Message-ID: <20061107001127.GA7406@bateleur.arcanes.fr.eu.org> Scribit Pierre THIERRY dies 06/11/2006 hora 14:40: > FWIW, since this patch [...] And since I'm not that much proud of the patch, I forgot to attach it! Quickly, Nowhere man -- nowhere.man at levallois.eu.org OpenPGP 0xD9D50D8A -------------- next part -------------- Index: elephant/elephant.asd =================================================================== --- elephant.orig/elephant.asd 2006-11-05 23:29:59.244904250 +0100 +++ elephant/elephant.asd 2006-11-05 23:32:24.469980250 +0100 @@ -112,7 +112,7 @@ (:file "backend")) :serial t :depends-on (memutil))))) - :depends-on (:uffi)) + :depends-on (:uffi :bordeaux-threads)) Index: elephant/src/elephant/classes.lisp =================================================================== --- elephant.orig/src/elephant/classes.lisp 2006-11-05 23:30:09.045516750 +0100 +++ elephant/src/elephant/classes.lisp 2006-11-06 00:14:03.970189250 +0100 @@ -233,44 +233,51 @@ ;; SLOT ACCESS PROTOCOLS ;; +(defvar *slot-lock* (make-lock "Slot access lock")) + (defmethod slot-value-using-class :around ((class persistent-metaclass) (instance persistent-object) (slot-def persistent-slot-definition)) "Get the slot value from the database." (declare (optimize (speed 3))) - (let ((name (slot-definition-name slot-def))) - (persistent-slot-reader (get-con instance) instance name))) + (with-lock-held (*slot-lock*) + (let ((name (slot-definition-name slot-def))) + (persistent-slot-reader (get-con instance) instance name)))) (defmethod (setf slot-value-using-class) :around (new-value (class persistent-metaclass) (instance persistent-object) (slot-def persistent-slot-definition)) "Set the slot value in the database." (declare (optimize (speed 3))) - (if (indexed class) - (indexed-slot-writer class instance slot-def new-value) - (let ((name (slot-definition-name slot-def))) - (persistent-slot-writer (get-con instance) new-value instance name)))) + (with-recursive-lock-held (*slot-lock*) + (if (indexed class) + (indexed-slot-writer class instance slot-def new-value) + (let ((name (slot-definition-name slot-def))) + (persistent-slot-writer (get-con instance) new-value instance name))))) (defmethod slot-boundp-using-class :around ((class persistent-metaclass) (instance persistent-object) (slot-def persistent-slot-definition)) "Checks if the slot exists in the database." (declare (optimize (speed 3))) - (let ((name (slot-definition-name slot-def))) - (persistent-slot-boundp (get-con instance) instance name))) + (with-recursive-lock-held (*slot-lock*) + (let ((name (slot-definition-name slot-def))) + (persistent-slot-boundp (get-con instance) instance name)))) (defmethod slot-boundp-using-class :around ((class persistent-metaclass) (instance persistent-object) (slot-name symbol)) "Checks if the slot exists in the database." (declare (optimize (speed 3))) - (loop for slot in (class-slots class) - for matches-p = (eq (slot-definition-name slot) slot-name) - until matches-p - finally (return (if (and matches-p - (subtypep (type-of slot) 'persistent-slot-definition)) - (persistent-slot-boundp (get-con instance) instance slot-name) - (call-next-method))))) + (with-recursive-lock-held (*slot-lock*) + (loop for slot in (class-slots class) + for matches-p = (eq (slot-definition-name slot) slot-name) + until matches-p + finally (return (if (and matches-p + (subtypep (type-of slot) 'persistent-slot-definition)) + (persistent-slot-boundp (get-con instance) instance slot-name) + (call-next-method)))))) (defmethod slot-makunbound-using-class :around ((class persistent-metaclass) (instance persistent-object) (slot-def persistent-slot-definition)) "Deletes the slot from the database." (declare (optimize (speed 3))) ;; NOTE: call remove-indexed-slot here instead? - (when (indexed slot-def) - (unregister-indexed-slot class (slot-definition-name slot-def))) - (persistent-slot-makunbound (get-con instance) instance (slot-definition-name slot-def))) + (with-recursive-lock-held (*slot-lock*) + (when (indexed slot-def) + (unregister-indexed-slot class (slot-definition-name slot-def))) + (persistent-slot-makunbound (get-con instance) instance (slot-definition-name slot-def)))) ;; ====================================================== ;; Handling metaclass overrides of normal slot operation Index: elephant/src/elephant/package.lisp =================================================================== --- elephant.orig/src/elephant/package.lisp 2006-11-05 23:29:59.400914000 +0100 +++ elephant/src/elephant/package.lisp 2006-11-05 23:30:24.186463000 +0100 @@ -20,7 +20,7 @@ (in-package :cl-user) (defpackage elephant - (:use common-lisp elephant-memutil) + (:use common-lisp elephant-memutil bordeaux-threads) (:nicknames ele :ele) (:documentation "Elephant: an object-oriented database for Common Lisp with -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From eslick at csail.mit.edu Tue Nov 7 00:41:53 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Mon, 06 Nov 2006 19:41:53 -0500 Subject: [elephant-devel] Concurrency problem? In-Reply-To: <20061105203646.GB7160@bateleur.arcanes.fr.eu.org> References: <20061031134344.GU7585@bateleur.arcanes.fr.eu.org> <1162306483.14560.285.camel@localhost.localdomain> <454CAF92.9010406@csail.mit.edu> <20061105203646.GB7160@bateleur.arcanes.fr.eu.org> Message-ID: <454FD651.7090008@csail.mit.edu> I just got back from a long trip and hope to look into this in a day or two - sorry for the delay. Ian Pierre THIERRY wrote: > Scribit Ian Eslick dies 04/11/2006 hora 10:19: > >> If you run db_deadlock on the open BDB database then all the >> transactions through BDB are thread safe. >> > > I tried BDB, but without success. When doing (do-backend-tests) with the > default BDB spec, it stops when compiling the wrap-errno macro. I tried > with setting the *sleepycat-foreign-library-path* as > > - /usr/lib/libdb-4.2.so > - /usr/lib/libdb-4.3.so > - /usr/lib/libdb-4.4.so > > But there is no difference between them. I tried this with elephant > HEAD, to see if it's globally thread safe. > > When using elephant 0.6.0, it stops just a form later, at the flags > macro, and also exactly in the same way for the three versions of BDB. > > Thread unsafely, > Nowhere man > > ------------------------------------------------------------------------ > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From nowhere.man at levallois.eu.org Tue Nov 7 16:44:32 2006 From: nowhere.man at levallois.eu.org (Pierre THIERRY) Date: Tue, 7 Nov 2006 17:44:32 +0100 Subject: [elephant-devel] Elephant and inheritance In-Reply-To: <452D0DF0.1070809@csail.mit.edu> References: <20061011142342.GI7422@bateleur.arcanes.fr.eu.org> <452D0DF0.1070809@csail.mit.edu> Message-ID: <20061107164432.GB7406@bateleur.arcanes.fr.eu.org> Scribit Ian Eslick dies 11/10/2006 hora 11:29: > If you have a non-persistent base class, should the semantics of the > slot storage change based on whether it's included in a persistent > subclass? When you inherit from a class, you expect the sub-class to have all the properties that the super-class has, plus it's specific ones. I think the current design of Elephant breaks that expectation. I really think that the most sensible way of dealing with this is to make all effective slots persistent, and provide options to alter that behaviour. This :persist option could accept either a keyword for a predefined persistence scheme or a list of slots. Let's take a base class: (defclass foo () ((bar))) We could have the following sub-classes: (defclass sub-foo-1 (foo) ((baz) (fubar)) (:metaclass persistent-metaclass)) ; persistent-slots: (bar baz fubar) ; this could be (:persist :effective-slots) (defclass sub-foo-2 (foo) ((baz) (fubar)) (:metaclass persistent-metaclass) (:persist :direct-slots)) ; persistent-slots: (baz fubar) (defclass sub-foo-3 (foo) ((baz) (fubar)) (:metaclass persistent-metaclass) (:persist :inherited-slots)) ; persistent-slots: (bar) (defclass sub-foo-4 (foo) ((baz) (fubar)) (:metaclass persistent-metaclass) (:persist (bar baz))) ; persistent-slots: (bar baz) There could also be a :transient option to achieve the opposite... As I never really used the MOP, I'm not sure how it is harder to achieve than the current behaviour, but I think the flexibility and expressiveness of this is worth the effort. > Now what happens if we inherit from sub-foo above? > > (defclass sub-sub-foo (sub-foo) > ((qux)) > (:metaclass ele:persistent-metaclass)) ;; protocol will assert an > error if subclass of persistent isn't persistent > > In this case the normal CLOS machinery will pick up the non-direct slots > from subclasses and the bar and myfoo > slots will be non-transient. There may be a way to override this to > search for subclass specifications of :persist-subclass-slots and then > go ahead and promote :transient effective slots to :persistent but all > the cases and interactions can start to become hairy. > > I think the currently policy is the simplest and least bug prone, even > though it forces the increased verboseness of the first example. > > Thoughts? I definitely agree on the hairiness of a more expressive alternative... I'd like to come up with a proposal for a sensible and consistent of dealing with these corner cases elegantly. I'll try to come up with a patch also, if I manage to grasp how MOP works. Hopefully, Nowhere man -- nowhere.man at levallois.eu.org OpenPGP 0xD9D50D8A -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From read at robertlread.net Tue Nov 7 17:46:59 2006 From: read at robertlread.net (Robert L. Read) Date: Tue, 07 Nov 2006 11:46:59 -0600 Subject: [elephant-devel] Elephant and inheritance In-Reply-To: <20061107164432.GB7406@bateleur.arcanes.fr.eu.org> References: <20061011142342.GI7422@bateleur.arcanes.fr.eu.org> <452D0DF0.1070809@csail.mit.edu> <20061107164432.GB7406@bateleur.arcanes.fr.eu.org> Message-ID: <1162921619.5145.8.camel@localhost.localdomain> To quote a folksy expression from Texas, "I ain't got a dog in this fight." I'm glad Pierre and Ian are working this out, however. On Tue, 2006-11-07 at 17:44 +0100, Pierre THIERRY wrote: > Scribit Ian Eslick dies 11/10/2006 hora 11:29: > > If you have a non-persistent base class, should the semantics of the > > slot storage change based on whether it's included in a persistent > > subclass? > > When you inherit from a class, you expect the sub-class to have all the > properties that the super-class has, plus it's specific ones. I think > the current design of Elephant breaks that expectation. > > I really think that the most sensible way of dealing with this is to > make all effective slots persistent, and provide options to alter that > behaviour. This :persist option could accept either a keyword for a > predefined persistence scheme or a list of slots. > > Let's take a base class: > > (defclass foo () > ((bar))) > > We could have the following sub-classes: > > (defclass sub-foo-1 (foo) > ((baz) > (fubar)) > (:metaclass persistent-metaclass)) > ; persistent-slots: (bar baz fubar) > ; this could be (:persist :effective-slots) > > (defclass sub-foo-2 (foo) > ((baz) > (fubar)) > (:metaclass persistent-metaclass) > (:persist :direct-slots)) > ; persistent-slots: (baz fubar) > > (defclass sub-foo-3 (foo) > ((baz) > (fubar)) > (:metaclass persistent-metaclass) > (:persist :inherited-slots)) > ; persistent-slots: (bar) > > (defclass sub-foo-4 (foo) > ((baz) > (fubar)) > (:metaclass persistent-metaclass) > (:persist (bar baz))) > ; persistent-slots: (bar baz) > > There could also be a :transient option to achieve the opposite... > > As I never really used the MOP, I'm not sure how it is harder to achieve > than the current behaviour, but I think the flexibility and > expressiveness of this is worth the effort. > > > Now what happens if we inherit from sub-foo above? > > > > (defclass sub-sub-foo (sub-foo) > > ((qux)) > > (:metaclass ele:persistent-metaclass)) ;; protocol will assert an > > error if subclass of persistent isn't persistent > > > > In this case the normal CLOS machinery will pick up the non-direct slots > > from subclasses and the bar and myfoo > > slots will be non-transient. There may be a way to override this to > > search for subclass specifications of :persist-subclass-slots and then > > go ahead and promote :transient effective slots to :persistent but all > > the cases and interactions can start to become hairy. > > > > I think the currently policy is the simplest and least bug prone, even > > though it forces the increased verboseness of the first example. > > > > Thoughts? > > I definitely agree on the hairiness of a more expressive alternative... > I'd like to come up with a proposal for a sensible and consistent of > dealing with these corner cases elegantly. I'll try to come up with a > patch also, if I manage to grasp how MOP works. > > Hopefully, > Nowhere man > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From blumberg at math.uchicago.edu Tue Nov 7 19:49:33 2006 From: blumberg at math.uchicago.edu (Andrew Blumberg) Date: Tue, 7 Nov 2006 13:49:33 -0600 (CST) Subject: [elephant-devel] Elephant and inheritance In-Reply-To: <20061107164432.GB7406@bateleur.arcanes.fr.eu.org> References: <20061011142342.GI7422@bateleur.arcanes.fr.eu.org> <452D0DF0.1070809@csail.mit.edu> <20061107164432.GB7406@bateleur.arcanes.fr.eu.org> Message-ID: as the designer of this part of elephant, maybe i'll add my opinions on the subject. the "usage profile" i envisioned when i wrote the MOP code was one (which reflects my coding habits) where there are a lot of little non-persistent mixin classes which affect dispatch and sometimes store intermediate information. very rarely should the slots associated to these mixins be made persistent when the mixin is added to a persistent class; you want it when you want it, but not by default. moreover, whether or not elephant breaks the expectation about sub-class behavior depends on what you think the property in of persistent classes is --- since i tend to think of that property as "might have some persistent slots", not "all slots are persistent", i regard the current behavior as consistent with my expectations. you might or might not think that's a reasonable way to view what persistence means, but it is coherent anyway. finally, i don't think the technical hair of adjusting the behavior of the mop here should be too bad (or rather, it won't be any hairier then the getting the current behavior to work correctly was), if that's what you want. i'll be happy to answer questions/ look at code if that will be helpful. - andrew On Tue, 7 Nov 2006, Pierre THIERRY wrote: > Scribit Ian Eslick dies 11/10/2006 hora 11:29: >> If you have a non-persistent base class, should the semantics of the >> slot storage change based on whether it's included in a persistent >> subclass? > > When you inherit from a class, you expect the sub-class to have all the > properties that the super-class has, plus it's specific ones. I think > the current design of Elephant breaks that expectation. > > I really think that the most sensible way of dealing with this is to > make all effective slots persistent, and provide options to alter that > behaviour. This :persist option could accept either a keyword for a > predefined persistence scheme or a list of slots. > > Let's take a base class: > > (defclass foo () > ((bar))) > > We could have the following sub-classes: > > (defclass sub-foo-1 (foo) > ((baz) > (fubar)) > (:metaclass persistent-metaclass)) > ; persistent-slots: (bar baz fubar) > ; this could be (:persist :effective-slots) > > (defclass sub-foo-2 (foo) > ((baz) > (fubar)) > (:metaclass persistent-metaclass) > (:persist :direct-slots)) > ; persistent-slots: (baz fubar) > > (defclass sub-foo-3 (foo) > ((baz) > (fubar)) > (:metaclass persistent-metaclass) > (:persist :inherited-slots)) > ; persistent-slots: (bar) > > (defclass sub-foo-4 (foo) > ((baz) > (fubar)) > (:metaclass persistent-metaclass) > (:persist (bar baz))) > ; persistent-slots: (bar baz) > > There could also be a :transient option to achieve the opposite... > > As I never really used the MOP, I'm not sure how it is harder to achieve > than the current behaviour, but I think the flexibility and > expressiveness of this is worth the effort. > >> Now what happens if we inherit from sub-foo above? >> >> (defclass sub-sub-foo (sub-foo) >> ((qux)) >> (:metaclass ele:persistent-metaclass)) ;; protocol will assert an >> error if subclass of persistent isn't persistent >> >> In this case the normal CLOS machinery will pick up the non-direct slots >> from subclasses and the bar and myfoo >> slots will be non-transient. There may be a way to override this to >> search for subclass specifications of :persist-subclass-slots and then >> go ahead and promote :transient effective slots to :persistent but all >> the cases and interactions can start to become hairy. >> >> I think the currently policy is the simplest and least bug prone, even >> though it forces the increased verboseness of the first example. >> >> Thoughts? > > I definitely agree on the hairiness of a more expressive alternative... > I'd like to come up with a proposal for a sensible and consistent of > dealing with these corner cases elegantly. I'll try to come up with a > patch also, if I manage to grasp how MOP works. > > Hopefully, > Nowhere man > From lists at infoway.net Thu Nov 9 03:38:37 2006 From: lists at infoway.net (Daniel Salama) Date: Wed, 8 Nov 2006 22:38:37 -0500 Subject: [elephant-devel] Install and Tutorial Message-ID: <91D3556E-333A-4DBF-BC80-C48C15E64CA0@infoway.net> I've been to the site lately and don't see much changes happening there. So I'm wondering if anyone can comment: 1) How to download the latest and greatest? Is the .gz file always the latest and greatest or should I get the CVS version? Is the CVS version stable? 2) Where can I see a tutorial using the latest changes made? The blog example (as documented) is VERY obsolete. Is there a newer tutorial somewhere else? Also, the HTML-ized tutorial claims to be the latest (as of May 27, 2006), but is there a tutorial residing somewhere else? 3) Other than the documentation in the file, is there any other documentation/tutorial on the proper use of DCM? Thanks, Daniel From read at robertlread.net Thu Nov 9 03:58:18 2006 From: read at robertlread.net (Robert L. Read) Date: Wed, 08 Nov 2006 21:58:18 -0600 Subject: [elephant-devel] Install and Tutorial In-Reply-To: <91D3556E-333A-4DBF-BC80-C48C15E64CA0@infoway.net> References: <91D3556E-333A-4DBF-BC80-C48C15E64CA0@infoway.net> Message-ID: <1163044699.5145.67.camel@localhost.localdomain> On Wed, 2006-11-08 at 22:38 -0500, Daniel Salama wrote: > I've been to the site lately and don't see much changes happening > there. So I'm wondering if anyone can comment: > > 1) How to download the latest and greatest? Is the .gz file always > the latest and greatest or should I get the CVS version? Is the CVS > version stable? The .gz is the latest and greatest which is stable and has been tested in a complete sense. The CVS version is of course newer; I estimate that it is *not* more stable. > > 2) Where can I see a tutorial using the latest changes made? The blog > example (as documented) is VERY obsolete. Is there a newer tutorial > somewhere else? Also, the HTML-ized tutorial claims to be the latest > (as of May 27, 2006), but is there a tutorial residing somewhere else? I think the HTML-ized tutorial is up-to-date with the last formal release; I put several hours of work into making sure that is true. It may be that the timestamp on it is wrong. > > 3) Other than the documentation in the file, is there any other > documentation/tutorial on the proper use of DCM? DCM is a contributed module; I'm happy to answer questions about it but have not written any documentation for it. The tests are pretty good examples, however. > > Thanks, > Daniel > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From eslick at csail.mit.edu Thu Nov 9 21:25:19 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Thu, 09 Nov 2006 16:25:19 -0500 Subject: [elephant-devel] Elephant 0.6.0 SQL fix on OpenMCL 1.0 In-Reply-To: References: Message-ID: <45539CBF.8070109@csail.mit.edu> I've had continuous problems with MCL's type specs. Do you have any idea how and why MCL treats them differently? I never use MCL so don't have a good sense of what standards to use to generate code that works under MCL. This specific problem might be fixed by setting the type to (or null integer). I'm back hacking on Elephant for a few days so this is a good time to chase any remaining issues to ground. Did you get the 0.6.0, BDB 4.3 and MCL working together? Ian George Khouri wrote: > Friends, > Testing Elephant 0.6.0 against a postgres db, in OpenMCL 1.0, I found that the close-cursor method fails since it tries to set the value of the sql-cursor slot "curkey" to NIL, while its type spec in the class declaration is INTEGER. Perhaps other lisps allow this. > If I change close-cursor to set curkey to -1 as below, the tests run fine. I don't know if there are other routines which check for a null value in that cursor slot, in which case my change wouldn't work. Perhaps the type spec should be removed from the slot declaration instead? > > The code is in elephant/src/db-clsql/sql-collections.lisp. > > (defclass sql-cursor (cursor) > ((keys :accessor :sql-crsr-ks :initarg :sql-cursor-keys :initform '()) > (curkey :accessor :sql-crsr-ck :initarg :sql-cursor-curkey :initform -1 :type integer)) > (:documentation "A SQL cursor for traversing (primary) BTrees.")) > > ... > > (defmethod cursor-close ((cursor sql-cursor)) > #-openmcl > (setf (:sql-crsr-ck cursor) nil) > #+openmcl > (setf (:sql-crsr-ck cursor) -1) ; ** GK > (setf (cursor-initialized-p cursor) nil)) > > Thanks, > George > -------- > George Khouri > gk1 at four-four.com > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel > From eslick at csail.mit.edu Thu Nov 9 21:30:24 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Thu, 09 Nov 2006 16:30:24 -0500 Subject: [elephant-devel] Concurrency problem? In-Reply-To: <20061105203646.GB7160@bateleur.arcanes.fr.eu.org> References: <20061031134344.GU7585@bateleur.arcanes.fr.eu.org> <1162306483.14560.285.camel@localhost.localdomain> <454CAF92.9010406@csail.mit.edu> <20061105203646.GB7160@bateleur.arcanes.fr.eu.org> Message-ID: <45539DF0.5090304@csail.mit.edu> Did you recompile libsleepycat.c between tests? It depends on the db.h that you've set in the Makefile. That has to be set to the BDB-4.3 directory for 0.6.0 Ian Pierre THIERRY wrote: > Scribit Ian Eslick dies 04/11/2006 hora 10:19: > >> If you run db_deadlock on the open BDB database then all the >> transactions through BDB are thread safe. >> > > I tried BDB, but without success. When doing (do-backend-tests) with the > default BDB spec, it stops when compiling the wrap-errno macro. I tried > with setting the *sleepycat-foreign-library-path* as > > - /usr/lib/libdb-4.2.so > - /usr/lib/libdb-4.3.so > - /usr/lib/libdb-4.4.so > > But there is no difference between them. I tried this with elephant > HEAD, to see if it's globally thread safe. > > When using elephant 0.6.0, it stops just a form later, at the flags > macro, and also exactly in the same way for the three versions of BDB. > > Thread unsafely, > Nowhere man > > ------------------------------------------------------------------------ > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From eslick at csail.mit.edu Thu Nov 9 22:26:19 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Thu, 09 Nov 2006 17:26:19 -0500 Subject: [elephant-devel] Elephant and inheritance In-Reply-To: <20061107164432.GB7406@bateleur.arcanes.fr.eu.org> References: <20061011142342.GI7422@bateleur.arcanes.fr.eu.org> <452D0DF0.1070809@csail.mit.edu> <20061107164432.GB7406@bateleur.arcanes.fr.eu.org> Message-ID: <4553AB0B.9000809@csail.mit.edu> > When you inherit from a class, you expect the sub-class to have all the > properties that the super-class has, plus it's specific ones. I think > the current design of Elephant breaks that expectation. > If the base class is persistent, then it's transient/persistent slot values will be propagated to the subclass. If it's a normal class, then all its slots become transient - their normal properties if in the baseclass. If you want all slots to be persistent then you should inherit from persistent base classes and build up a persistent hierarchy in the background. Of course this depends on how you read the :metaclass semantics. Am I making the entire hierarchy persistent or just the current class? If you have your head pretty deeply inside CLOS then I think the current behavior is more consistent. However, if you're used to more of a Java object model then reading :metaclass as specifying a property for all the effective slots of the current class is more appropriate. Why not override the base class slots you want to persist explicitly by redeclaring the slots to get the new semantics for storage while maintaining current functioning of base class methods? Class options are a little annoying to implement and when I first figured out how the MOP worked, and how it was implemented in Elephant I have to admit it took me quite awhile and I had used Lisp pretty heavily for a year and half. However, I'm happy to support a :persist class option if you can figure out how to implement. I was going to clean up the MOP implementation using :closer-mop, but I'm not sure it happens to have the fixes for the lisp-specific parts of the MOP that elephant uses so I'll back out those changes so the current tree doesn't get polluted while you're hacking on the MOP. Let me know what you're thinking so we can coordinate any changes (and I can provide some support and answer questions). Ian > I really think that the most sensible way of dealing with this is to > make all effective slots persistent, and provide options to alter that > behaviour. This :persist option could accept either a keyword for a > predefined persistence scheme or a list of slots. > > Let's take a base class: > > (defclass foo () > ((bar))) > > We could have the following sub-classes: > > (defclass sub-foo-1 (foo) > ((baz) > (fubar)) > (:metaclass persistent-metaclass)) > ; persistent-slots: (bar baz fubar) > ; this could be (:persist :effective-slots) > > (defclass sub-foo-2 (foo) > ((baz) > (fubar)) > (:metaclass persistent-metaclass) > (:persist :direct-slots)) > ; persistent-slots: (baz fubar) > > (defclass sub-foo-3 (foo) > ((baz) > (fubar)) > (:metaclass persistent-metaclass) > (:persist :inherited-slots)) > ; persistent-slots: (bar) > > (defclass sub-foo-4 (foo) > ((baz) > (fubar)) > (:metaclass persistent-metaclass) > (:persist (bar baz))) > ; persistent-slots: (bar baz) > > There could also be a :transient option to achieve the opposite... > > As I never really used the MOP, I'm not sure how it is harder to achieve > than the current behaviour, but I think the flexibility and > expressiveness of this is worth the effort. > > >> Now what happens if we inherit from sub-foo above? >> >> (defclass sub-sub-foo (sub-foo) >> ((qux)) >> (:metaclass ele:persistent-metaclass)) ;; protocol will assert an >> error if subclass of persistent isn't persistent >> >> In this case the normal CLOS machinery will pick up the non-direct slots >> from subclasses and the bar and myfoo >> slots will be non-transient. There may be a way to override this to >> search for subclass specifications of :persist-subclass-slots and then >> go ahead and promote :transient effective slots to :persistent but all >> the cases and interactions can start to become hairy. >> >> I think the currently policy is the simplest and least bug prone, even >> though it forces the increased verboseness of the first example. >> >> Thoughts? >> > > I definitely agree on the hairiness of a more expressive alternative... > I'd like to come up with a proposal for a sensible and consistent of > dealing with these corner cases elegantly. I'll try to come up with a > patch also, if I manage to grasp how MOP works. > > Hopefully, > Nowhere man > > ------------------------------------------------------------------------ > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From eslick at csail.mit.edu Thu Nov 9 22:27:54 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Thu, 09 Nov 2006 17:27:54 -0500 Subject: [elephant-devel] Elephant and inheritance In-Reply-To: <20061107164432.GB7406@bateleur.arcanes.fr.eu.org> References: <20061011142342.GI7422@bateleur.arcanes.fr.eu.org> <452D0DF0.1070809@csail.mit.edu> <20061107164432.GB7406@bateleur.arcanes.fr.eu.org> Message-ID: <4553AB6A.1010603@csail.mit.edu> PS - I'd vote to keep the current defaults but allow :persist :inherited-slots to override that. This avoids breaking backward compatibility for the existing user base. If enough people prefer that as the default we can fix it at some point in the future and force the (small) community to upgrade. Cheers, Ian Pierre THIERRY wrote: > >> If you have a non-persistent base class, should the semantics of the >> slot storage change based on whether it's included in a persistent >> subclass? >> > > When you inherit from a class, you expect the sub-class to have all the > properties that the super-class has, plus it's specific ones. I think > the current design of Elephant breaks that expectation. > > I really think that the most sensible way of dealing with this is to > make all effective slots persistent, and provide options to alter that > behaviour. This :persist option could accept either a keyword for a > predefined persistence scheme or a list of slots. > > Let's take a base class: > > (defclass foo () > ((bar))) > > We could have the following sub-classes: > > (defclass sub-foo-1 (foo) > ((baz) > (fubar)) > (:metaclass persistent-metaclass)) > ; persistent-slots: (bar baz fubar) > ; this could be (:persist :effective-slots) > > (defclass sub-foo-2 (foo) > ((baz) > (fubar)) > (:metaclass persistent-metaclass) > (:persist :direct-slots)) > ; persistent-slots: (baz fubar) > > (defclass sub-foo-3 (foo) > ((baz) > (fubar)) > (:metaclass persistent-metaclass) > (:persist :inherited-slots)) > ; persistent-slots: (bar) > > (defclass sub-foo-4 (foo) > ((baz) > (fubar)) > (:metaclass persistent-metaclass) > (:persist (bar baz))) > ; persistent-slots: (bar baz) > > There could also be a :transient option to achieve the opposite... > > As I never really used the MOP, I'm not sure how it is harder to achieve > than the current behaviour, but I think the flexibility and > expressiveness of this is worth the effort. > > >> Now what happens if we inherit from sub-foo above? >> >> (defclass sub-sub-foo (sub-foo) >> ((qux)) >> (:metaclass ele:persistent-metaclass)) ;; protocol will assert an >> error if subclass of persistent isn't persistent >> >> In this case the normal CLOS machinery will pick up the non-direct slots >> from subclasses and the bar and myfoo >> slots will be non-transient. There may be a way to override this to >> search for subclass specifications of :persist-subclass-slots and then >> go ahead and promote :transient effective slots to :persistent but all >> the cases and interactions can start to become hairy. >> >> I think the currently policy is the simplest and least bug prone, even >> though it forces the increased verboseness of the first example. >> >> Thoughts? >> > > I definitely agree on the hairiness of a more expressive alternative... > I'd like to come up with a proposal for a sensible and consistent of > dealing with these corner cases elegantly. I'll try to come up with a > patch also, if I manage to grasp how MOP works. > > Hopefully, > Nowhere man > > ------------------------------------------------------------------------ > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From eslick at csail.mit.edu Thu Nov 9 22:31:04 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Thu, 09 Nov 2006 17:31:04 -0500 Subject: [elephant-devel] trouble getting going (with open-store, ...) In-Reply-To: References: <1160036592.9944.18.camel@localhost> <45250215.7020809@csail.mit.edu> <1B109030-433B-4AAB-8783-B167C22F73BE@onlinehome.de> <452514F6.8000706@csail.mit.edu> Message-ID: <4553AC28.5010307@csail.mit.edu> Still interested in hacking on elephant + MCL? Kilian Sprotte wrote: > Am 05.10.2006 um 16:21 schrieb Ian Eslick: > >> Thanks for the help! The next two weeks are busy for me, and then I >> hope to get back to hacking properly on Elephant. > > Thats good, so I can prepare this slowly.... :) Basically, I have good > news: > With some minor inconveniences, I got sbcl running > and all the *testbdb-spec* tests passed in the first run. > Now (after restarting), I seem to be stuck on > prepares-sleepycat, test-seq1, test-seq2, cleansup-sleepycat. > But Ill repeat this being more careful. > > For openmcl, there will be a little more needed, but I think, > there will be quick solutions. > > Lets get together in 2 weeks, then! > > Cheers, > > Kilian > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From nowhere.man at levallois.eu.org Thu Nov 9 22:50:21 2006 From: nowhere.man at levallois.eu.org (Pierre THIERRY) Date: Thu, 9 Nov 2006 23:50:21 +0100 Subject: [elephant-devel] Elephant and inheritance In-Reply-To: <4553AB0B.9000809@csail.mit.edu> References: <20061011142342.GI7422@bateleur.arcanes.fr.eu.org> <452D0DF0.1070809@csail.mit.edu> <20061107164432.GB7406@bateleur.arcanes.fr.eu.org> <4553AB0B.9000809@csail.mit.edu> Message-ID: <20061109225021.GB12538@bateleur.arcanes.fr.eu.org> Scribit Ian Eslick dies 09/11/2006 hora 17:26: > If the base class is persistent, then it's transient/persistent slot > values will be propagated to the subclass. If it's a normal class, > then all its slots become transient - their normal properties if in > the baseclass. I understand this point of view. > If you have your head pretty deeply inside CLOS then I think the > current behavior is more consistent. I have a strong C++ background, and almost everything I know about OOP, I learned it with C++. It is a very different view of objects indeed. I must say that only the multipe dispatch and the concept of MOP were revolutions for me... (should I add I love them...) > Why not override the base class slots you want to persist explicitly > by redeclaring the slots to get the new semantics for storage while > maintaining current functioning of base class methods? Well, it breaks some use of the class hierarchy: you could alter all the hierarchy by changing it's base class. With the current scheme, if you add a slot, you have to track where it would be of any use and add it to the overridden slots. In this case, the class option is of great interest. > I was going to clean up the MOP implementation using :closer-mop, but > I'm not sure it happens to have the fixes for the lisp-specific parts > of the MOP that elephant uses so I'll back out those changes so the > current tree doesn't get polluted while you're hacking on the MOP. Don't ever stop your cleaning for me. I'm not sure when I will be able to give you a usable patch, and I've already begun to try and play with closer-mop, so in fact it could help me if you switch the MOP part to it. Quickly, Nowhere man -- nowhere.man at levallois.eu.org OpenPGP 0xD9D50D8A -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From nowhere.man at levallois.eu.org Thu Nov 9 22:52:31 2006 From: nowhere.man at levallois.eu.org (Pierre THIERRY) Date: Thu, 9 Nov 2006 23:52:31 +0100 Subject: [elephant-devel] Elephant and inheritance In-Reply-To: <4553AB6A.1010603@csail.mit.edu> References: <20061011142342.GI7422@bateleur.arcanes.fr.eu.org> <452D0DF0.1070809@csail.mit.edu> <20061107164432.GB7406@bateleur.arcanes.fr.eu.org> <4553AB6A.1010603@csail.mit.edu> Message-ID: <20061109225231.GC12538@bateleur.arcanes.fr.eu.org> Scribit Ian Eslick dies 09/11/2006 hora 17:27: > PS - I'd vote to keep the current defaults but allow :persist > :inherited-slots to override that. This avoids breaking backward > compatibility for the existing user base. That seems a sensible way of doing this, yes. Compatibly, Nowhere man -- nowhere.man at levallois.eu.org OpenPGP 0xD9D50D8A -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From nowhere.man at levallois.eu.org Thu Nov 9 23:19:29 2006 From: nowhere.man at levallois.eu.org (Pierre THIERRY) Date: Fri, 10 Nov 2006 00:19:29 +0100 Subject: [elephant-devel] Concurrency problem? In-Reply-To: <45539DF0.5090304@csail.mit.edu> References: <20061031134344.GU7585@bateleur.arcanes.fr.eu.org> <1162306483.14560.285.camel@localhost.localdomain> <454CAF92.9010406@csail.mit.edu> <20061105203646.GB7160@bateleur.arcanes.fr.eu.org> <45539DF0.5090304@csail.mit.edu> Message-ID: <20061109231929.GD12538@bateleur.arcanes.fr.eu.org> Scribit Ian Eslick dies 09/11/2006 hora 16:30: > Did you recompile libsleepycat.c between tests? It depends on the > db.h that you've set in the Makefile. That has to be set to the > BDB-4.3 directory for 0.6.0 I did not set the directory in the Makefile, but always did the test with a freshly copied source. So I restarted with a fresh source, with the following modifications: Index: elephant-0.6.0/Makefile =================================================================== --- elephant-0.6.0.orig/Makefile +++ elephant-0.6.0/Makefile @@ -32,7 +32,7 @@ # # But I will assume that Linux is more common? -DB43DIR=/usr/local/BerkeleyDB.4.3/ +DB43DIR=/usr # Dan Knapp contributed this line, which came form OS X? # DB43DIR=/sw # Other example paths Index: elephant-0.6.0/config.lisp =================================================================== --- elephant-0.6.0.orig/config.lisp +++ elephant-0.6.0/config.lisp @@ -28,7 +28,7 @@ ;; Sleepycat: this works on linux #+linux ;; "/db/ben/lisp/db43/lib/libdb.so" - "/usr/local/BerkeleyDB.4.3/lib/libdb-4.3.so" + "/usr/lib/libdb-4.3.so" ;; this works on FreeBSD #+(and (or bsd freebsd) (not (or darwin macosx))) "/usr/local/lib/db43/libdb.so" And then it didn't stop at WRAP-ERRNO but at the FLAGS macro... And I also discovered something, because I went to the *inferior-lisp* buffer of SLIME: fatal error encountered in SBCL pid 8244(tid 2789931936): GC invariant lost, file "thread.c", line 553 I had entered LDB. I'll try to see how other Lisp implementations behave. Unsuccessfully, Nowhere man -- nowhere.man at levallois.eu.org OpenPGP 0xD9D50D8A -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From eslick at csail.mit.edu Fri Nov 10 01:24:54 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Thu, 09 Nov 2006 20:24:54 -0500 Subject: [elephant-devel] Concurrency problem? In-Reply-To: <20061109231929.GD12538@bateleur.arcanes.fr.eu.org> References: <20061031134344.GU7585@bateleur.arcanes.fr.eu.org> <1162306483.14560.285.camel@localhost.localdomain> <454CAF92.9010406@csail.mit.edu> <20061105203646.GB7160@bateleur.arcanes.fr.eu.org> <45539DF0.5090304@csail.mit.edu> <20061109231929.GD12538@bateleur.arcanes.fr.eu.org> Message-ID: <4553D4E6.4000609@csail.mit.edu> This could also the pthreads problem discussed in an earlier mail which I admit I don't fully understand as I don't run on linux. However I'm almost certain we tested 0.6.0 and 4.3 and linux and SBCL as part of the 0.6.0 release so I'm not sure what the deal is. Could be the new SBCL not playing nice with some assumption left in 0.6.0. Also, if you are going to be editing the MOP - better you do it off of the CVS HEAD rather than 0.6.0. So you know, the current HEAD requires BDB 4.4. I probably won't do the closer port in the immediate future. We have a bunch of loose ends to tie off and I think we'll do a minor release once I've put all of them to bed and then get back to improvements like closure. Cheers, Ian Pierre THIERRY wrote: > Scribit Ian Eslick dies 09/11/2006 hora 16:30: > >> Did you recompile libsleepycat.c between tests? It depends on the >> db.h that you've set in the Makefile. That has to be set to the >> BDB-4.3 directory for 0.6.0 >> > > I did not set the directory in the Makefile, but always did the test > with a freshly copied source. So I restarted with a fresh source, with > the following modifications: > > Index: elephant-0.6.0/Makefile > =================================================================== > --- elephant-0.6.0.orig/Makefile > +++ elephant-0.6.0/Makefile > @@ -32,7 +32,7 @@ > # > > # But I will assume that Linux is more common? > -DB43DIR=/usr/local/BerkeleyDB.4.3/ > +DB43DIR=/usr > # Dan Knapp contributed this line, which came form OS X? > # DB43DIR=/sw > # Other example paths > Index: elephant-0.6.0/config.lisp > =================================================================== > --- elephant-0.6.0.orig/config.lisp > +++ elephant-0.6.0/config.lisp > @@ -28,7 +28,7 @@ > ;; Sleepycat: this works on linux > #+linux > ;; "/db/ben/lisp/db43/lib/libdb.so" > - "/usr/local/BerkeleyDB.4.3/lib/libdb-4.3.so" > + "/usr/lib/libdb-4.3.so" > ;; this works on FreeBSD > #+(and (or bsd freebsd) (not (or darwin macosx))) > "/usr/local/lib/db43/libdb.so" > > And then it didn't stop at WRAP-ERRNO but at the FLAGS macro... And I > also discovered something, because I went to the *inferior-lisp* buffer > of SLIME: > > fatal error encountered in SBCL pid 8244(tid 2789931936): > GC invariant lost, file "thread.c", line 553 > > I had entered LDB. I'll try to see how other Lisp implementations > behave. > > Unsuccessfully, > Nowhere man > > ------------------------------------------------------------------------ > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From nowhere.man at levallois.eu.org Fri Nov 10 02:00:41 2006 From: nowhere.man at levallois.eu.org (Pierre THIERRY) Date: Fri, 10 Nov 2006 03:00:41 +0100 Subject: [elephant-devel] Concurrency problem? In-Reply-To: <4553D4E6.4000609@csail.mit.edu> References: <20061031134344.GU7585@bateleur.arcanes.fr.eu.org> <1162306483.14560.285.camel@localhost.localdomain> <454CAF92.9010406@csail.mit.edu> <20061105203646.GB7160@bateleur.arcanes.fr.eu.org> <45539DF0.5090304@csail.mit.edu> <20061109231929.GD12538@bateleur.arcanes.fr.eu.org> <4553D4E6.4000609@csail.mit.edu> Message-ID: <20061110020041.GE12538@bateleur.arcanes.fr.eu.org> Scribit Ian Eslick dies 09/11/2006 hora 20:24: > Could be the new SBCL not playing nice with some assumption left in > 0.6.0. I'll install a sarge chroot and test this within it. > Also, if you are going to be editing the MOP - better you do it off of > the CVS HEAD rather than 0.6.0. So you know, the current HEAD > requires BDB 4.4. FWIW, I've got the same error when trying with CVS HEAD and BDB 4.4... Quickly, Nowhere man -- nowhere.man at levallois.eu.org OpenPGP 0xD9D50D8A -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From eslick at csail.mit.edu Sat Nov 11 04:15:08 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Fri, 10 Nov 2006 23:15:08 -0500 Subject: [elephant-devel] Calling windows users Message-ID: <45554E4C.6050506@csail.mit.edu> Do we have a windows user who would be willing to test and tweak a new feature in HEAD (originally contributed by pinterface at gmail.com) to build the libraries from asdf rather than by calling out to an external Makefile? Ian From nowhere.man at levallois.eu.org Sat Nov 11 10:17:27 2006 From: nowhere.man at levallois.eu.org (Pierre THIERRY) Date: Sat, 11 Nov 2006 11:17:27 +0100 Subject: [elephant-devel] Calling windows users In-Reply-To: <45554E4C.6050506@csail.mit.edu> References: <45554E4C.6050506@csail.mit.edu> Message-ID: <20061111101727.GK12538@bateleur.arcanes.fr.eu.org> Scribit Ian Eslick dies 10/11/2006 hora 23:15: > Do we have a windows user who would be willing to test and tweak a new > feature in HEAD (originally contributed by pinterface at gmail.com) to > build the libraries from asdf rather than by calling out to an > external Makefile? I have a Windows XP box on which I'm willing to run some of my Lisp code in the near future, and it will probably depends on Elephant, so I'll want to test this... Is there something to do to the HEAD code or should I just try and load a freshly checked out HEAD? Quickly, Nowhere man -- nowhere.man at levallois.eu.org OpenPGP 0xD9D50D8A -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From gk1 at four-four.com Sat Nov 11 11:13:47 2006 From: gk1 at four-four.com (George Khouri) Date: Sat, 11 Nov 2006 03:13:47 -0800 Subject: [elephant-devel] Elephant 0.6.0 SQL fix on OpenMCL 1.0 In-Reply-To: <45407334.3090202@csail.mit.edu> Message-ID: Ian, >I've had continuous problems with MCL's type specs. Do you have any >idea how and why MCL treats them differently? I never use MCL so don't >have a good sense of what standards to use to generate code that works >under MCL. This specific problem might be fixed by setting the type to >(or null integer). The short of it seems that openMCL chooses to be strict in assigning a slot value when that slot's type is specified in the class definition, where e.g., SBCL either doesn't check or doesn't care. OpenMCL constructs a type-predicate test when defining a class and it apples that test before assigning a slot value (c.f. source code for %set-std-slot-vector-value in l1-clos-boot.lisp). If the value doesn't meet the type spec, it errors. So in the sql-cursor case it's approximately: ... (if (typep new-val 'integer) (setf (%svref slot-vector loc) new-val) (error 'bad-slot-type ...)) ... SBCL lets you assign any type to a typed slot: This is SBCL 0.9.18, an implementation of ANSI Common Lisp. ... * (defclass foo () ((slot-one :accessor slot-one :initform -1 :type integer))) # * (defparameter my-class (make-instance 'foo)) MY-CLASS * (slot-one my-class) -1 * (setf (slot-one my-class) nil) NIL * (setf (slot-one my-class) "bar") "bar" * The Hyperspec Sec. 7.5.3: "The contents of a slot will always be of type (and T1 ... Tn) where T1 ...Tn are the values of the :type slot options contained in all of the slot specifiers ... The consequences of attempting to store in a slot a value that does not satisfy the type of the slot are undefined." >This specific problem might be fixed by setting the type to >(or null integer). That works in openMCL. > >I'm back hacking on Elephant for a few days so this is a good time to >chase any remaining issues to ground. Did you get the 0.6.0, BDB 4.3 >and MCL working together? Yes, in openMCL 1.0, the test suites all pass. OpenMCL is well on its way to the 1.1 release. I think the serializer will have to be modified to work with the new release, in which character and stream handling have been reworked to handle unicode. I can send along the release notes if you like. Here's the symptom in 1.1 pre-release 061024 with either elephant 6.0/db4.3 or elephant cvs/db4.4 (the ^@'s are unprintables): CL-USER> (ele:open-store '(:bdb "/Users/gkhouri/db/")) # CL-USER> (ELE:add-to-root "my key" "my Val") "my Val" CL-USER> (ele:get-from-root "my key") "^@^@^@m^@^@" T Thanks, George > >Ian > >George Khouri wrote: >> Friends, >> Testing Elephant 0.6.0 against a postgres db, in OpenMCL 1.0, I found that >the close-cursor method fails since it tries to set the value of the sql-cursor >slot "curkey" to NIL, while its type spec in the class declaration is INTEGER. >Perhaps other lisps allow this. >> If I change close-cursor to set curkey to -1 as below, the tests run fine. I >don't know if there are other routines which check for a null value in that >cursor slot, in which case my change wouldn't work. Perhaps the type spec should >be removed from the slot declaration instead? >> >> The code is in elephant/src/db-clsql/sql-collections.lisp. >> >> (defclass sql-cursor (cursor) >> ((keys :accessor :sql-crsr-ks :initarg :sql-cursor-keys :initform '()) >> (curkey :accessor :sql-crsr-ck :initarg :sql-cursor-curkey :initform -1 >:type integer)) >> (:documentation "A SQL cursor for traversing (primary) BTrees.")) >> >> ... >> >> (defmethod cursor-close ((cursor sql-cursor)) >> #-openmcl >> (setf (:sql-crsr-ck cursor) nil) >> #+openmcl >> (setf (:sql-crsr-ck cursor) -1) ; ** GK >> (setf (cursor-initialized-p cursor) nil)) >> >> Thanks, >> George >> -------- >> George Khouri >> gk1 at four-four.com >> _______________________________________________ >> elephant-devel site list >> elephant-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/elephant-devel >> From gk1 at four-four.com Sat Nov 11 11:16:32 2006 From: gk1 at four-four.com (George Khouri) Date: Sat, 11 Nov 2006 03:16:32 -0800 Subject: [elephant-devel] Elephant 0.6.0 SQL fix on OpenMCL 1.0 In-Reply-To: <4554960D.4090401@csail.mit.edu> Message-ID: Ian, OpenMCL 1.1-pre-061110 release excerpt: "About 15-20 character encoding schemes are defined (so far); these include UTF-8/16/32 and the big-endian/little-endian variants of the latter two and ISO-8859-* 8-bit encodings. " Unicode support is now "built-in." but only starting with the 1.1 pre-releases. Unfortunelately, there isn't a *feature* indicatiing unicode support. The only *features* changing from 1.0 to 1.1 are the "lack of" the :ccl-x" features (:CCL :CCL-2 :CCL-3 :CCL-4) and a few others, so you would have to do something like: #+(and openmcl (not ccl)) ...(ugh!) I'll be glad to do any testing on openMCL. George >Unicode support should be an easy fix; we've already patched it for SBCL >and ACL unicode. Is there a *feature* in MCL that indicates whether >string types are unicode or not? Also, what encoding does MCL use? >(UTF-8, UTF-16, etc) > >Ian > >George Khouri wrote: >> Ian, >> >> >>> I've had continuous problems with MCL's type specs. Do you have any >>> idea how and why MCL treats them differently? I never use MCL so don't >>> have a good sense of what standards to use to generate code that works >>> under MCL. This specific problem might be fixed by setting the type to >>> (or null integer). >>> >> >> The short of it seems that openMCL chooses to be strict in assigning a slot >value when that slot's type is specified in the class definition, where e.g., >SBCL either doesn't check or doesn't care. >> OpenMCL constructs a type-predicate test when defining a class and it apples >that test before assigning a slot value (c.f. source code for >%set-std-slot-vector-value in l1-clos-boot.lisp). If the value doesn't meet the >type spec, it errors. So in the sql-cursor case it's approximately: >> ... >> (if (typep new-val 'integer) >> (setf (%svref slot-vector loc) new-val) >> (error 'bad-slot-type >> ...)) ... >> >> SBCL lets you assign any type to a typed slot: >> >> This is SBCL 0.9.18, an implementation of ANSI Common Lisp. ... >> * (defclass foo () >> ((slot-one :accessor slot-one :initform -1 :type integer))) >> # >> * (defparameter my-class (make-instance 'foo)) >> MY-CLASS >> * (slot-one my-class) >> -1 >> * (setf (slot-one my-class) nil) >> NIL >> * (setf (slot-one my-class) "bar") >> "bar" >> * >> >> The Hyperspec Sec. 7.5.3: >> "The contents of a slot will always be of type (and T1 ... Tn) where T1 ...Tn >are the values of the :type slot options contained in all of the slot specifiers >... The consequences of attempting to store in a slot a value that does not >satisfy the type of the slot are undefined." >> >> >>> This specific problem might be fixed by setting the type to >>> (or null integer). >>> >> >> That works in openMCL. >> >> >>> I'm back hacking on Elephant for a few days so this is a good time to >>> chase any remaining issues to ground. Did you get the 0.6.0, BDB 4.3 >>> and MCL working together? >>> >> >> Yes, in openMCL 1.0, the test suites all pass. >> >> OpenMCL is well on its way to the 1.1 release. I think the serializer will >have to be modified to work with the new release, in which character and stream >handling have been reworked to handle unicode. I can send along the release >notes if you like. Here's the symptom in 1.1 pre-release 061024 with either >elephant 6.0/db4.3 or elephant cvs/db4.4 (the ^@'s are unprintables): >> CL-USER> (ele:open-store '(:bdb "/Users/gkhouri/db/")) >> # >> CL-USER> (ELE:add-to-root "my key" "my Val") >> "my Val" >> CL-USER> (ele:get-from-root "my key") >> "^@^@^@m^@^@" >> T >> >> Thanks, >> George >> >> >>> Ian >>> >>> >> >> >> >>> George Khouri wrote: >>> >>>> Friends, >>>> Testing Elephant 0.6.0 against a postgres db, in OpenMCL 1.0, I found that >>>> >>> the close-cursor method fails since it tries to set the value of the >sql-cursor >>> slot "curkey" to NIL, while its type spec in the class declaration is >INTEGER. >>> Perhaps other lisps allow this. >>> >>>> If I change close-cursor to set curkey to -1 as below, the tests run fine. >I >>>> >>> don't know if there are other routines which check for a null value in that >>> cursor slot, in which case my change wouldn't work. Perhaps the type spec >should >>> be removed from the slot declaration instead? >>> >>>> The code is in elephant/src/db-clsql/sql-collections.lisp. >>>> >>>> (defclass sql-cursor (cursor) >>>> ((keys :accessor :sql-crsr-ks :initarg :sql-cursor-keys :initform '()) >>>> (curkey :accessor :sql-crsr-ck :initarg :sql-cursor-curkey :initform -1 >>>> >>> :type integer)) >>> >>>> (:documentation "A SQL cursor for traversing (primary) BTrees.")) >>>> >>>> ... >>>> >>>> (defmethod cursor-close ((cursor sql-cursor)) >>>> #-openmcl >>>> (setf (:sql-crsr-ck cursor) nil) >>>> #+openmcl >>>> (setf (:sql-crsr-ck cursor) -1) ; ** GK >>>> (setf (cursor-initialized-p cursor) nil)) >>>> >>>> Thanks, >>>> George >>>> -------- >>>> George Khouri >>>> gk1 at four-four.com >>>> _______________________________________________ >>>> elephant-devel site list >>>> elephant-devel at common-lisp.net >>>> http://common-lisp.net/mailman/listinfo/elephant-devel >>>> >>>> From eslick at csail.mit.edu Sat Nov 11 15:16:09 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Sat, 11 Nov 2006 10:16:09 -0500 Subject: [elephant-devel] Calling windows users In-Reply-To: <20061111101727.GK12538@bateleur.arcanes.fr.eu.org> References: <45554E4C.6050506@csail.mit.edu> <20061111101727.GK12538@bateleur.arcanes.fr.eu.org> Message-ID: <4555E939.30008@csail.mit.edu> The problem is that Windows has no standard C compiler. I'm now thinking that rather than building the libraries; we should just manually release a 32-bit DLL for libmemutil.c and libsleepycat.c and skip the compilation step. I assume most Windows Elephant users will have one, but maybe it can be a configuration option. If the option is unset, the system defaults to loading the pre-packaged windows DLLs. Please try the latest HEAD on Win32 + BDB, it won't compile but if you follow the instructions in INSTALL you should be able to build one manually. I have a new Core 2 Duo PowerMac showing up on Monday and as it's 64-bit it will break my existing environment, so I'm going to focus on finishing the 64-bit port that Marco started a couple of weeks ago, but I can integrate any changes you suggest early next week. Thank you, Ian Pierre THIERRY wrote: > Scribit Ian Eslick dies 10/11/2006 hora 23:15: > >> Do we have a windows user who would be willing to test and tweak a new >> feature in HEAD (originally contributed by pinterface at gmail.com) to >> build the libraries from asdf rather than by calling out to an >> external Makefile? >> > > I have a Windows XP box on which I'm willing to run some of my Lisp code > in the near future, and it will probably depends on Elephant, so I'll > want to test this... > > Is there something to do to the HEAD code or should I just try and load > a freshly checked out HEAD? > > Quickly, > Nowhere man > > ------------------------------------------------------------------------ > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From lists at infoway.net Sun Nov 12 15:28:42 2006 From: lists at infoway.net (Daniel Salama) Date: Sun, 12 Nov 2006 10:28:42 -0500 Subject: [elephant-devel] Querying Advice Message-ID: <948E9C76-5F70-4245-9697-A59E50ABF9D6@infoway.net> Hi all, Coming from a Rails/MySQL world, we are having some difficulty trying to comprehend/model/implement querying the database using dynamically generated criteria. For example, in the life we are trying to leave, we present the user with advanced search screen, where they can select anywhere between 0 and 20 different search fields. When the user submits the screen, we dynamically construct the WHERE SQL statement based on whatever fields and information the user entered. Obviously, I could envision this being the easy part, however, any pointers would really help here. Then we rely on the SQL-engine to perform the proper computations in order to return to us the matching resultset. In addition, the user is then able to dynamically sort the results to their heart's desire. We simply dynamically generate the ORDER BY clause as well. As a learning task, we are trying to migrate some of this functionality to Elephant using BDB. After reading the HTML-ized tutorial and looking at some of the tests in the package, we still have a hard time understanding how to go about implementing something like this. From previous posts, we can use a cursor to jump into a subset of the data we need to get (e.g. based on some indexed or secondary indexed value in the search criteria). However, after that, I guess we would have to sequentially navigate thru the results in order to "manually" select each record based on all the other possible search arguments. I suppose, in a way, this can be done relatively painless by using macros (but we first would like to do the manual expansion to learn how it would look like). Any ideas or suggestions would certainly be appreciated here. Then we have the issue of the sorting. I suppose it falls into a similar situation: once we get the matching resultset from the previous step, we would have to perform some "efficient" sort algorithm on the data set dynamically based on the user's sorting desire. I also suppose we could create a macro for this as well. In a way, and after reading many posts out there, we are basically wanting to bypass all the RDBMS machinery and working directly with the data. After all, I guess many if not all of these RDBMS systems are simply a SQL-domain language interface performing all these computations and abstracting the direct access to the raw data, which is probably even using BDB in the backend. So, I suppose we have to learn to be able to roll-up our sleeves and deal with manipulating the data in a more direct way. Again, any ideas or suggestions here would be really appreciated. Thanks, Daniel From read at robertlread.net Sun Nov 12 20:48:46 2006 From: read at robertlread.net (Robert L. Read) Date: Sun, 12 Nov 2006 14:48:46 -0600 Subject: [elephant-devel] Querying Advice In-Reply-To: <948E9C76-5F70-4245-9697-A59E50ABF9D6@infoway.net> References: <948E9C76-5F70-4245-9697-A59E50ABF9D6@infoway.net> Message-ID: <1163364527.5145.244.camel@localhost.localdomain> This requires a philosophical response. In general, I think it will be way easier than you image, once you have been pointed in the right direction. Take my advice with a grain of salt. First of all, ask yourself, what is the size of your dataset? Can you fit it all into memory? If so, you have the full power of lisp at your command in dealing with the querying. You will not have to write any macros to do this. You might find the DCM package, in the "contrib" directory, a useful package, although it does not address querying; it is more of a cache handling issue. (DCM has only been tested under SBCL, as far as I know.) Under SBCL, when it comes to sorting you have "sort" and "stable-sort"; I think these are build in. I'll eat a candle if you don't find them to blazingly fast (although the predicates that you pass them might take some time.) I think really the only good way to answer this question in a deeper way is to provide some example code. I do exactly what you are talking about in my application (http://konsenti.com), (although I use DCM), so I ought to be able to produce an example program relatively quickly. You'll have to figure out how to map the GUI into those requests yourself, however. I'll try to post an example by Monday. On Sun, 2006-11-12 at 10:28 -0500, Daniel Salama wrote: > Hi all, > > Coming from a Rails/MySQL world, we are having some difficulty > trying > to comprehend/model/implement querying the database using > dynamically > generated criteria. > > For example, in the life we are trying to leave, we present the user > with advanced search screen, where they can select anywhere between > 0 > and 20 different search fields. When the user submits the screen, we > dynamically construct the WHERE SQL statement based on whatever > fields and information the user entered. > > Obviously, I could envision this being the easy part, however, any > pointers would really help here. Then we rely on the SQL-engine to > perform the proper computations in order to return to us the > matching > resultset. > > In addition, the user is then able to dynamically sort the results > to > their heart's desire. We simply dynamically generate the ORDER BY > clause as well. > > As a learning task, we are trying to migrate some of this > functionality to Elephant using BDB. After reading the HTML-ized > tutorial and looking at some of the tests in the package, we still > have a hard time understanding how to go about implementing > something > like this. From previous posts, we can use a cursor to jump into a > subset of the data we need to get (e.g. based on some indexed or > secondary indexed value in the search criteria). However, after > that, > I guess we would have to sequentially navigate thru the results in > order to "manually" select each record based on all the other > possible search arguments. I suppose, in a way, this can be done > relatively painless by using macros (but we first would like to do > the manual expansion to learn how it would look like). Any ideas or > suggestions would certainly be appreciated here. > > Then we have the issue of the sorting. I suppose it falls into a > similar situation: once we get the matching resultset from the > previous step, we would have to perform some "efficient" sort > algorithm on the data set dynamically based on the user's sorting > desire. I also suppose we could create a macro for this as well. > > In a way, and after reading many posts out there, we are basically > wanting to bypass all the RDBMS machinery and working directly with > the data. After all, I guess many if not all of these RDBMS systems > are simply a SQL-domain language interface performing all these > computations and abstracting the direct access to the raw data, > which > is probably even using BDB in the backend. So, I suppose we have to > learn to be able to roll-up our sleeves and deal with manipulating > the data in a more direct way. > > Again, any ideas or suggestions here would be really appreciated. > > Thanks, > Daniel -------------- next part -------------- An HTML attachment was scrubbed... URL: From eslick at csail.mit.edu Sun Nov 12 22:34:31 2006 From: eslick at csail.mit.edu (=?UTF-8?B?SWFuIEVzbGljaw==?=) Date: Sun, 12 Nov 2006 22:34:31 +0000 Subject: [elephant-devel] Querying Advice In-Reply-To: <1163364527.5145.244.camel@localhost.localdomain> References: <948E9C76-5F70-4245-9697-A59E50ABF9D6@infoway.net> <1163364527.5145.244.camel@localhost.localdomain> Message-ID: <192580707-1163370881-cardhu_blackberry.rim.net-1611534371-@bxe008-cell01.bisx.prod.on.blackberry> I have another take on this for queries where you are looking for high throughput but want to stick with the elephant persistence model (slot values on disk) or your data won't fit in memory. I'll send this out when I get my computer back to an Inet connection... Robert and I have discussed adding DCM features to the main elephant release which might open up more options for a query package for elephant. Ian Ian Sent via BlackBerry from T-Mobile -----Original Message----- From: "Robert L. Read" Date: Sun, 12 Nov 2006 14:48:46 To:Elephant bugs and development Subject: Re: [elephant-devel] Querying Advice This requires a philosophical response.? In general, I think it will be way easier than you image, once you have been pointed in the right direction.? Take my advice with a grain of salt. First of all, ask yourself, what is the size of your dataset?? Can you fit it all into memory? If so, you have the full power of lisp at your command in dealing with the querying.? You will not have to write any macros to do this.? You might find the DCM package, in the "contrib" directory, a useful package, although it does not address querying; it is more of a cache handling issue.? (DCM has only been tested under SBCL, as far as I know.) Under SBCL, when it comes to sorting you have "sort" and "stable-sort"; I think these are build in. I'll eat a candle if you don't find them to blazingly fast (although the predicates that you pass them might take some time.) I think really the only good way to answer this question in a deeper way is to provide some example code.? I do exactly what you are talking about in my application (http://konsenti.com: ), (although I use DCM), so I ought to be able to produce an example program relatively quickly. You'll have to figure out how to map the GUI into those requests yourself, however. I'll try to post an example by Monday. On Sun, 2006-11-12 at 10:28 -0500, Daniel Salama wrote: Hi all, Coming from a Rails/MySQL world, we are having some difficulty trying? to comprehend/model/implement querying the database using dynamically? generated criteria. For example, in the life we are trying to leave, we present the user? with advanced search screen, where they can select anywhere between 0? and 20 different search fields. When the user submits the screen, we? dynamically construct the WHERE SQL statement based on whatever? fields and information the user entered. Obviously, I could envision this being the easy part, however, any? pointers would really help here. Then we rely on the SQL-engine to? perform the proper computations in order to return to us the matching? resultset. In addition, the user is then able to dynamically sort the results to? their heart's desire. We simply dynamically generate the ORDER BY? clause as well. As a learning task, we are trying to migrate some of this? functionality to Elephant using BDB. After reading the HTML-ized? tutorial and looking at some of the tests in the package, we still? have a hard time understanding how to go about implementing something? like this. From previous posts, we can use a cursor to jump into a? subset of the data we need to get (e.g. based on some indexed or? secondary indexed value in the search criteria). However, after that,? I guess we would have to sequentially navigate thru the? results in? order to "manually" select each record based on all the other? possible search arguments. I suppose, in a way, this can be done? relatively painless by using macros (but we first would like to do? the manual expansion to learn how it would look like). Any ideas or? suggestions would certainly be appreciated here. Then we have the issue of the sorting. I suppose it falls into a? similar situation: once we get the matching resultset from the? previous step, we would have to perform some "efficient" sort? algorithm on the data set dynamically based on the user's sorting? desire. I also suppose we could create a macro for this as well. In a way, and after reading many posts out there, we are basically? wanting to bypass all the RDBMS machinery and working directly with? the data. After all, I guess many if not all of these RDBMS systems? are simply a SQL-domain language interface performing all these? computations and abstracting the direct access to the raw data, which? is probably even using BDB in the backend. So, I suppose we have to? learn to be able to roll-up our sleeves and deal with manipulating? the data in a more direct way. Again, any ideas or suggestions here would be really appreciated. Thanks, Daniel _______________________________________________ elephant-devel site list elephant-devel at common-lisp.net http://common-lisp.net/mailman/listinfo/elephant-devel From read at robertlread.net Sun Nov 12 23:45:20 2006 From: read at robertlread.net (Robert L. Read) Date: Sun, 12 Nov 2006 17:45:20 -0600 Subject: [elephant-devel] Querying Advice [w/code example] In-Reply-To: <1163364527.5145.244.camel@localhost.localdomain> References: <948E9C76-5F70-4245-9697-A59E50ABF9D6@infoway.net> <1163364527.5145.244.camel@localhost.localdomain> Message-ID: <1163375121.5145.289.camel@localhost.localdomain> Dear Daniel and Team, I think the code below, which I have tested on SBCL, illustrated a typical problem that Daniel Salama introduces. To paraphrase, you have a datatype (perhaps compound) which has a lot of slots; you have a GUI, perhaps web-based, that you use to both select or filter the large database, and to decide how to present sort the results. I've written the below example as if you operating directly on the slots. The fact that there are often intervening functions does not fundamentally change the problem. (An example of this is storing a timestamp as an integer, but presenting it in a human-readable format.) SQL supports a powerful querying ability based on both selection and sorting. One might think that this is an advantage of SQL; it is conventional reason that this is actually an advantage of using a relational database. However, since LISP treats functions as first- class citizens that can be constructed dynamically, you actually have a full Turing-complete capabilities in doing queries that SQL cannot match. This same ability applies to sorting; you can sort on any lexical order that you can program. In practice, however, one doesn't always need this power. More typically, a user will select fields that they want to use to filter the results (that is, construct a query from), and perhaps how they would like the results to be sorted. I assume that you know how to interpret an HTTP query or a McClim user interface or something to associate the GUI with underlying functions. (My personal framework has a way to do this, and UCW is probably the most common or famous way to do it now.) The code below generated 100 random "users". The bare act of defining this class defines accessor-functions that we can use in dynamically constructed lists as below: (list 'username-of 'balance-of). I have written very small functions that use such lists either to define define "lexicographic" sort orders based on the order of the functions within the list. That is, the primary sort criteria is the first function in the list, but of that function is equal for two values, the next is used and so on. If you load the below code and execute (show- off) several times I think you will see what I mean. You can then see how easily you can change the list of functions that are either in the selector or the sort criteria. If this is a web-based app, these list will be generated from the http-query, which is generated by the user's clicks. That is a "columnar" based approach; but it one can do something similar but more powerful based on computed functions that aren't based on individual columns, but on the entire data element. For example "find users whose username is equal to their password" cannot be done in this way --- but can be done by just using a function #(lambda (x) (equal (username-of x) (password-of x)). SQL can do this --- but LISP can could use any function there, such as "find users who have both short usernames and passwords that can be cracked by routine-x". Instead of adding things to the root or the store-controller directly, one would generally prefer to use consistent classes: http://common-lisp.net/project/elephant/doc/Persistent- Classes.html#Persistent-Classes This doesn't change the nature of the problem. If you like, you can create an index on any slot in a very convenient way: http://common- lisp.net/project/elephant/doc/Class-Indices.html#Class-Indices This holds out the possibility of NOT having to iterate over the entire data-set, but rather honing in directly upon matching values. (You can in fact create functional indexes based on any function at all in Elephant, which is something that SQL can't do conveniently, but the times you will need to do this are rare.) It would take maybe 20 minutes more coding to uses Ian's "get-instances- by-range" directly, and in a very efficient manner for performing the query (if the GUI elements correspond to the class's slots.) This would be very efficient; but of course you should not do this until you know that this is really the bottleneck in your system. By using cursors, you can avoid reading the entire data into memory, and thus process huge datasets. However, one should note that Ian's code makes creating indices on slots zero-effort; but indexes always have overhead. The real question is: when will your queries actually utilize the index? (That is, if you always select on one column/slot, then that one should be indexed....but if your query pattern is more complicated, it becomes fuzzy.) Let me know if you find this useful; after I get feedback from you and Ian has made his post, perhaps we will put this in the documentation. (asdf:operate 'asdf:load-op :elephant-tests) (in-package "ELEPHANT-TESTS") (setf *default-spec* *testbdb-spec*) (defclass User () ((username :type 'string :initform "" :initarg :uname :accessor username-of) (password :type 'string :initform "" :initarg :pword :accessor password-of) (email :type 'string :initform "" :initarg :email :accessor email- of) (fullname :type 'string :initform "" :initarg :fullname :accessor fullname-of) (balance :type 'integer :initform 0 :initarg :balance :accessor balance-of) )) (defun random-letter () (if (= 0 (random 2)) (code-char (+ 65 (random 26))) (code-char (+ 97 (random 26))))) (defun random-password () (let ((pw "")) (dotimes (i 8) (setf pw (format nil "~A~A" pw (random-letter)))) pw)) (defun random-users (n) (dotimes (x n) (let ((u (make-instance 'User :uname (format nil "user~A" x) :pword (random-password) :email (format nil "user~A at .nowheresville.org" x) :fullname (format nil "~A~A ~A~A" (random-password) x (random- password) x) :balance (random 100)))) (add-to-root x u) ) )) (defun my-lexco (a b) (cond ((typep a 'string) (string< a b)) ((typep a 'number) (< a b)) (t (string< (format nil "~A" a) (format nil "~A" b))) )) (defun sort-by-columns (objs lst) (sort objs #'(lambda (a b) (let ((res t)) (dolist (fun lst) (progn (format t "fun = ~A~%" fun) (let ((av (funcall fun a)) (bv (funcall fun b ))) (if (equal av bv) nil (return (my-lexco av bv)))) res)))) )) (defun select-by-many-selectors (objs lst) (let ((ret nil)) (dolist (o objs) (let ((selected t)) (dolist (fun lst) (setf selected (and selected (funcall fun o))) ) (if selected (push o ret)) )) ret)) (defun get-all-objects (n) (let ((res nil)) (dotimes (x n) (push (get-from-root x) res)) res)) (defmethod print-object ((obj User) stream) (format stream "(username, password, email, fullname, balance) = (~A, ~A, ~A, ~A, ~A)" (username-of obj) (password-of obj) (email-of obj) (fullname-of obj) (balance-of obj) )) (defun show-off () (let ((n 100)) (open-store *default-spec*) ;; Let's create 100 users.... (random-users n) ;; Now we'll sort based on a list of functions.. (sort-by-columns ;; But first we'll filter by a list of functions that must be true of each object... (select-by-many-selectors (get-all-objects n) ;; This list of functions (which has only (list #'(lambda (x) (< (balance-of x) 50)) #'(lambda (x) (equal #\3 (aref (username-of x) (- (length (username-of x)) 1)))))) ;; This is the list of functions that we will (list #'(lambda (x) (floor (/ (balance-of x) 10))) #'username- of)))) -------------- next part -------------- An HTML attachment was scrubbed... URL: From lists at infoway.net Mon Nov 13 04:17:36 2006 From: lists at infoway.net (Daniel Salama) Date: Sun, 12 Nov 2006 23:17:36 -0500 Subject: [elephant-devel] Querying Advice In-Reply-To: <1163364527.5145.244.camel@localhost.localdomain> References: <948E9C76-5F70-4245-9697-A59E50ABF9D6@infoway.net> <1163364527.5145.244.camel@localhost.localdomain> Message-ID: <3D430634-748D-44BE-9B44-A8EB5AB5EBAB@infoway.net> On Nov 12, 2006, at 3:48 PM, Robert L. Read wrote: > This requires a philosophical response. In general, I think it > will be way easier than > you image, once you have been pointed in the right direction. Take > my advice with > a grain of salt. I certainly hope so. As I have been learning lisp, I have full confidence that with the proper knowledge and the right guidance, this is definitely a manageable task. > > First of all, ask yourself, what is the size of your dataset? Can > you fit it all into memory? > If so, you have the full power of lisp at your command in dealing > with the querying. You > will not have to write any macros to do this. You might find the > DCM package, in the "contrib" > directory, a useful package, although it does not address querying; > it is more of a cache handling > issue. (DCM has only been tested under SBCL, as far as I know.) In general, I do think that the dataset fits in memory. However, we have not fully loaded all of our data into Elephant. Simply looking at the MySQL file storage for the database, it occupies 900MB of disk space, including indices. However, when we did loads of some of the tables into Elephant, the size of the Elephant data files were, at least, 5 times the size. I don't know the reason why. I don't know if that's the nature of BDB when it stores "arbitrarily" any type of object in the k,v pair. I don't know if we had some circular references when we loaded our model and that just simply increased the data file by that much (although I wouldn't think so, since if that was the case, I would expect that it only stored references to objects and not duplicating the objects). Regardless, our dev server currently has 4GB of RAM. I think that once properly loaded, all the data should be able to fit in memory. Now, for the target application, I prefer not to rely in the data fitting in memory. Reason being is that the nature of the application requires the data to be available for several years. This 900MB is the results of only 1 year for one company. As we get more companies to use the application and keep the data online for several years, the assumption of the data fitting into memory will no longer be applicable. I have been looking into the DCM package and I think that it certainly looks promising. We haven't used it yet, but certainly hope that sooner, rather than later, will be made part of Elephant permanently. I also hope that it's not targeted mainly at the in- memory database type of application, but rather, as an efficient caching mechanism for persistent data (regardless of where it's being permanently stored). With regards to: "...If so, you have the full power of lisp at your command in dealing with the querying...", I agree with you. However, where I'm trying to get at is how "easy" would it be to generate these type of dynamic queries in a generic way. Of course, we could always hard code all the cases for each of our different searchable screens, but the thought of that simply just makes me vomit :) > > Under SBCL, when it comes to sorting you have "sort" and "stable- > sort"; I think these are build in. > I'll eat a candle if you don't find them to blazingly fast > (although the predicates that you pass them > might take some time.) I thought the answer to my sorting question is exactly addressed by your comment. I suppose that once I have the resulting dataset, I could run it by "sort". They key would be how to make it arbitrarily sortable (in a similar way as the dynamic query) > > I think really the only good way to answer this question in a > deeper way is to provide some > example code. I do exactly what you are talking about in my > application (http://konsenti.com), > (although I use DCM), so I ought to be able to produce an example > program relatively quickly. > You'll have to figure out how to map the GUI into those requests > yourself, however. > I believe (and hope) that we should have no problem mapping the GUI to the requests. Just out of curiosity (and I don't mean to divert from the topic of this thread): if you're using DCM for your konsenti (BTW, nice concept) site, how do you protect your in-memory data? Do you just write an image to disk every once in a while for back ups? How resilient is this to hardware failure and you loosing data since the last image (if that's your approach)? > I'll try to post an example by Monday. > Thanks, Daniel -------------- next part -------------- An HTML attachment was scrubbed... URL: From lists at infoway.net Mon Nov 13 04:23:15 2006 From: lists at infoway.net (Daniel Salama) Date: Sun, 12 Nov 2006 23:23:15 -0500 Subject: [elephant-devel] Querying Advice In-Reply-To: <192580707-1163370881-cardhu_blackberry.rim.net-1611534371-@bxe008-cell01.bisx.prod.on.blackberry> References: <948E9C76-5F70-4245-9697-A59E50ABF9D6@infoway.net> <1163364527.5145.244.camel@localhost.localdomain> <192580707-1163370881-cardhu_blackberry.rim.net-1611534371-@bxe008-cell01.bisx.prod.on.blackberry> Message-ID: <48F699F8-4053-42CF-9815-32C5F8F4982D@infoway.net> I'd love to hear/read about your take on the subject. It would be really great to be able to have some querying or even better a reporting tool that worked with Elephant. May be something a la Crystal Reports, but that may be shooting too high :). How about something like http://www.agata.org.br/? Anyway, once we get a better understanding of what we are doing, how things work in Elephant, and how we are doing things, we'll give it a shot at trying to contribute some sort of querying or reporting tool. We need it anyway, so if we can do it, we'll contribute it. I was a bit surprised that we couldn't find any CL-based reporting package out there :( Thanks, Daniel On Nov 12, 2006, at 5:34 PM, Ian Eslick wrote: > I have another take on this for queries where you are looking for > high throughput but want to stick with the elephant persistence > model (slot values on disk) or your data won't fit in memory. I'll > send this out when I get my computer back to an Inet connection... > > Robert and I have discussed adding DCM features to the main > elephant release which might open up more options for a query > package for elephant. > > Ian > > Ian > Sent via BlackBerry from T-Mobile -------------- next part -------------- An HTML attachment was scrubbed... URL: From eslick at csail.mit.edu Sun Nov 12 18:11:40 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Sun, 12 Nov 2006 13:11:40 -0500 Subject: [elephant-devel] Complex queries Message-ID: <455763DC.3040708@csail.mit.edu> I'm interested in this problem myself, but haven't had the time to do the appropriate research. Many SQL query engines are built on top of a BTree infrastructure so building and performing optimized queries should be quite doable in Elephant. However, I do know that efficient operations are fairly complicated. I'd be happy to support your work by including a search interface + simple compiler into the main Elephant release. I'm assuming you want to search for persistent objects and not standard objects stored in BTrees. There are two ways to extract a set of objects associated with a particular slot value: 1) Get all objects of the desired type and do a linear search 2) Create a slot-index for searchable slots. You can use the accessor functions in get-instances-by-value and get-instances-by-range. This returns the persistent object. Current cursor accesses require full deserialization of the target object but fortunately this doesn't include the slot, just the oid+classname, etc. If you extract a range of objects using a slot index, they'll already be ordered according to the sort order of the index values. You can do joins by filtering each list by whether the object exists in the next list or not. i.e. age = 18 (oid1, oid2, oid3, oid4, oid5, oid6, oid7 ... ) name = "Cindy" (oid4, oid7, oid3 ...) state = MA (oid10, oid4, oid7 ...) Filtered objects = oid4, oid7 ... Optimization issues: 1) Returned objects are in value order, not OID or object order; so each merge of lists is naively C*N^2 where C is the number of constraints (3 above) and N is the average size of a returned list. This can be avoided by sorting each list (N log N) prior to merging which is effectively M log M (where M = the sum of all returned objects for all the constraints). In fact if you append the lists (N) and then sort removing duplicates M log M you get the same result. You then need another N log N walk of the list to order the elements according to the original list of constraints. I'm sure there's a better way, but this is a decent first order take. 2) Now this works fine when the # of objects in a given query are small. But let's say you want to pick all adults with the name "Jerry" from a 1M entry database of people. This is a range query (get-instances-by-range 'person 'age 18 120) and a value query (get-instances-by-value 'person 'first-name "Jerry"). However, the # of Jerry instances and in particular the # of age > 18 instances will be enormous (at least 500k). You might not even be able to fit all of these in memory! In this case we need to know how large the sets of objects are under given constraints. (I'm not sure if BDB provides a way to get access to the size of the index between two values - the total size of the tree + the height of the nodes should be telling as the BTrees are balanced...). In this case we can try to use cursors or slot accessors to do the filtering based on the total size and accessibility of the queries. At this point it gets somewhat complicated and I think it's worthwhile to appeal to the SQL implementation literature. Also, what happens if a slot is not indexed? How do we search multiple object classes at the same time? For example, how would we specify a search over instances and it's super classes when the superclass also contain the slot constraints? (Currently class indexes are specific to a class name) Is this helpful? A first cut approach as described in #1 above will probably flesh out a large number of issues and shouldn't take that much time. If you are motivated, or have large queries, then it's appropriate to spend some time talking about how to handle the issues of large sets in #2. Query language: Does it make sense to have a full query language, or a simpler macros to make this simple? The easy macro approach probably doesn't need anything sophisticated: (retrieve-instances-by-constraints ((person :with-superclasses t) employee) ((age 18 120) (name "George") (state :MA)) (:symbols-as-strings t :order t)) Here we search for person objects, parents of person objects and employee objects for the three constraints. The search option :symbols-as-strings means to treat :MA as matching :MA or "MA" - that means two slot queries. Not sure that's a good idea, but it's an example of modifying how the query operates. :order means to order the resulting objects by their age, in this case. The order of ranges in the constraint list would determine the total order. What if we want to do join queries? How about OR constraints? (state (or :MA :NY)) Ian From eslick at csail.mit.edu Mon Nov 13 05:18:52 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Mon, 13 Nov 2006 00:18:52 -0500 Subject: [elephant-devel] Querying Advice In-Reply-To: <3D430634-748D-44BE-9B44-A8EB5AB5EBAB@infoway.net> References: <948E9C76-5F70-4245-9697-A59E50ABF9D6@infoway.net> <1163364527.5145.244.camel@localhost.localdomain> <3D430634-748D-44BE-9B44-A8EB5AB5EBAB@infoway.net> Message-ID: <4558003C.2070104@csail.mit.edu> >> >> First of all, ask yourself, what is the size of your dataset? Can >> you fit it all into memory? >> If so, you have the full power of lisp at your command in dealing >> with the querying. You >> will not have to write any macros to do this. You might find the DCM >> package, in the "contrib" >> directory, a useful package, although it does not address querying; >> it is more of a cache handling >> issue. (DCM has only been tested under SBCL, as far as I know.) > > In general, I do think that the dataset fits in memory. However, we > have not fully loaded all of our data into Elephant. Simply looking at > the MySQL file storage for the database, it occupies 900MB of disk > space, including indices. However, when we did loads of some of the > tables into Elephant, the size of the Elephant data files were, at > least, 5 times the size. I don't know the reason why. I don't know if > that's the nature of BDB when it stores "arbitrarily" any type of > object in the k,v pair. I don't know if we had some circular > references when we loaded our model and that just simply increased the > data file by that much (although I wouldn't think so, since if that > was the case, I would expect that it only stored references to objects > and not duplicating the objects). Regardless, our dev server currently > has 4GB of RAM. I think that once properly loaded, all the data should > be able to fit in memory. Were you storing persistent-metaclass objects or simply normal objects? Normal objects are huge relative to persistent objects as currently all the slot names are also serialized (potentially in 32-bit unicode if on SBCL). I have some improvements planned to reduce the storage and processing overhead for strings, but it will take some evaluation to make sure performance doesn't suffer over much. Some of this will make it into the forthcoming 0.6.1 release I'm trying to finish a bunch of work for prior to my twin daughters are born. Also SBCL strings in the 0.6.0 release are stored in their internal 32-bit form so we can memcpy them back and forth between elephant and the internal format. The new serializer will store them in the minimal format and then convert back to the 32-bit internal form for unicode enabled SBCL. > Now, for the target application, I prefer not to rely in the data > fitting in memory. Reason being is that the nature of the application > requires the data to be available for several years. This 900MB is the > results of only 1 year for one company. As we get more companies to > use the application and keep the data online for several years, the > assumption of the data fitting into memory will no longer be applicable. See my delayed e-mail on a hybrid query approach... > I have been looking into the DCM package and I think that it certainly > looks promising. We haven't used it yet, but certainly hope that > sooner, rather than later, will be made part of Elephant permanently. > I also hope that it's not targeted mainly at the in-memory database > type of application, but rather, as an efficient caching mechanism for > persistent data (regardless of where it's being permanently stored). DCM is, as I understand it, a form of prevalence with explicit checkpointing using Elephant as the backend. > With regards to: "...If so, you have the full power of lisp at your > command in dealing with the querying...", I agree with you. However, > where I'm trying to get at is how "easy" would it be to generate these > type of dynamic queries in a generic way. Of course, we could always > hard code all the cases for each of our different searchable screens, > but the thought of that simply just makes me vomit :) Associate a constraint on the display with the user action for selections and an ordering function for specific columns. If you want to sort by multiple columns, then compose your functions: Selection constraint fn: #'(lambda (obj) (= (slot-accessor obj) value) Sorting fn: #'(lambda (obj1 obj2) (string< (name obj1) (name obj2))) #'(lambda (obj1 obj2) (< (age obj1) (age obj2))) Composed sorting function: (defun compose-sorts (sort1 sort2) (lambda (o1 o2) (or (funcall sort1 o1 o2) (funcall sort2 o1 o2)))) Same thing with constraints: (defun compose-constraint (c1 c2) (lambda (obj) (and (funcall c1 obj) (funcall c2 obj)))) Or to be a little more fancy: (defun compose-constraints (clist) (lambda (obj) (every #'(lambda (c) (funcall c obj)) clist))) (defun compose-sorts (slist) (lambda (o1 o2) (some #'(lambda (sort) (funcall sort o1 o2))))) This all comes down to: (defun select-results (objects) (select-if (compose-constraints (make-name-constraints "Bob") (make-age-constraint 18 120)) objects)) (defun order-results (objects) (sort (copy-list objects) (compose-sorts #'by-occupation #'by-name #'by-date-of-birth))) This way you can have a set of widgets with static constraints and then you can compose functions over the constraints based on the dynamic query. There may be more elegant ways to do this but I think this is what Robert was driving at. To do this on elephant simply requires that you (get-instances-by-class 'person) to get all instances of person and then run the lisp query over it. If you access person alot, these persistent references will be cached in memory. Or they're all in the cache including slot values in DCM - in which case these queries will run really fast! >> >> Under SBCL, when it comes to sorting you have "sort" and >> "stable-sort"; I think these are build in. >> I'll eat a candle if you don't find them to blazingly fast (although >> the predicates that you pass them >> might take some time.) > > I thought the answer to my sorting question is exactly addressed by > your comment. I suppose that once I have the resulting dataset, I > could run it by "sort". They key would be how to make it arbitrarily > sortable (in a similar way as the dynamic query) Sorting should be easy to do in this manner once you've found your target group of objects and when you've cached all the slot values in-memory DCM style. My proposal includes some sorting in the retrieval to avoid accessing and serializing the slot values. Every step of a cursor over a secondary index deserializes the key, the primary BTree key and the value (the persistent object record) from that primary BTree. This does not deserialize the slot values so is reasonably fast. >> >> I think really the only good way to answer this question in a deeper >> way is to provide some >> example code. I do exactly what you are talking about in my >> application (http://konsenti.com), >> (although I use DCM), so I ought to be able to produce an example >> program relatively quickly. >> You'll have to figure out how to map the GUI into those requests >> yourself, however. >> > > I believe (and hope) that we should have no problem mapping the GUI to > the requests. Just out of curiosity (and I don't mean to divert from > the topic of this thread): if you're using DCM for your konsenti (BTW, > nice concept) site, how do you protect your in-memory data? Do you > just write an image to disk every once in a while for back ups? How > resilient is this to hardware failure and you loosing data since the > last image (if that's your approach)? > Every time you finish or 'commit' a user transaction (form submit), just write the object data back to the Elephant persistent store before acking to the user. Ian From lists at infoway.net Mon Nov 13 05:19:40 2006 From: lists at infoway.net (Daniel Salama) Date: Mon, 13 Nov 2006 00:19:40 -0500 Subject: [elephant-devel] Querying Advice [w/code example] In-Reply-To: <1163375121.5145.289.camel@localhost.localdomain> References: <948E9C76-5F70-4245-9697-A59E50ABF9D6@infoway.net> <1163364527.5145.244.camel@localhost.localdomain> <1163375121.5145.289.camel@localhost.localdomain> Message-ID: Wow! I wasn't expecting to hear back from you until Monday :) I haven't been able to run your code yet. I downloaded SBCL for PPC but have not been able to successfully compile Elephant with it. I will continue trying. I used to have it working before, but stopped using SBCL a while ago for OpenMCL. Anyway, some comments before I can run your code and provide more feedback: 100% agree with all this. However, one thing that might have been oversimplified is when it comes down to complex relational queries. There are two main issues to address here: 1) The nature of dynamic queries as presented in my original email: your explanation and code seems to address this in a subtle way and, although I believe you when you say it works, I'd say that it could be enhanced in a way to make it generic enough. I remember the how Peter Siebel presents such a nice and simple interface to querying a CD/MP3 database in Chapter 3 of his Practical Common Lisp book. In a way, he kind of developed a small domain language for querying and I think something like that can be implemented with the sample code you provided below. 2) What happens when you have multiple relational tables... or more appropriate in our case, when you want to query an object model based on different object and object relational hierarchies. Say, for example, you have class person which has references to an object called demographics and an object postal-code, which, in turns, has a reference to an object state. How would you apply your proposal to a query such as "give me all 'males' under the age of 30 who live in 'NY' sorted by age in descending order and then by last, first names". It obviously is very simple to do this in SQL (if the proper relations exist between the tables), but may not be (or at least doesn't seem to be initially) as trivial in Elephant or applying your code below. Your dynamically constructed list of query-able columns approach may be a bit limited. Also, class indices may not necessarily be appropriate either in these situation. Maybe I haven't delved deeply enough into your proposal and/or haven't given it sufficient thought (I apologize for that if I belittled your proposal). I definitely understand the potential overhead of using secondary indices and agree that their use should be determined by how often you really need to search by those slots. I will continue trying to make SBCL work again in my PPC and then run your code. Maybe play with it and try to simulate different scenarios such as I mentioned above. Will then send more comments. Thanks again, Daniel P.S. I don't mean to distract anyone from the main thread of this email. But I noticed how you make use of format to append values to strings (e.g. random-password). Is that like the only way to do so, or maybe the more efficient way of doing so? Thanks again On Nov 12, 2006, at 6:45 PM, Robert L. Read wrote: > Dear Daniel and Team, > > I think the code below, which I have tested on SBCL, > illustrated a typical problem that Daniel Salama introduces. To > paraphrase, you have a datatype (perhaps compound) which has a lot > of slots; you have a GUI, perhaps web-based, that you use to both > select or filter the large database, and to decide how to present > sort the results. I've written the below example as if you > operating directly on the slots. The fact that there are often > intervening functions does not fundamentally change the problem. > (An example of this is storing a timestamp as an integer, but > presenting it in a human-readable format.) > SQL supports a powerful querying ability based on both > selection and sorting. One might think that this is an advantage > of SQL; it is conventional reason that this is actually an > advantage of using a relational database. However, since LISP > treats functions as first-class citizens that can be constructed > dynamically, you actually have a full Turing-complete capabilities > in doing queries that SQL cannot match. This same ability applies > to sorting; you can sort on any lexical order that you can program. > In practice, however, one doesn't always need this power. More > typically, a user will select fields that they want to use to > filter the results (that is, construct a query from), and perhaps > how they would like the results to be sorted. I assume that you > know how to interpret an HTTP query or a McClim user interface or > something to associate the GUI with underlying functions. (My > personal framework has a way to do this, and UCW is probably the > most common or famous way to do it now.) > The code below generated 100 random "users". The bare act of > defining this class defines accessor-functions that we can use in > dynamically constructed lists as below: (list 'username-of 'balance- > of). I have written very small functions that use such lists > either to define define "lexicographic" sort orders based on the > order of the functions within the list. That is, the primary sort > criteria is the first function in the list, but of that function is > equal for two values, the next is used and so on. If you load the > below code and execute (show-off) several times I think you will > see what I mean. You can then see how easily you can change the > list of functions that are either in the selector or the sort > criteria. If this is a web-based app, these list will be generated > from the http-query, which is generated by the user's clicks. > That is a "columnar" based approach; but it one can do > something similar but more powerful based on computed functions > that aren't based on individual columns, but on the entire data > element. For example "find users whose username is equal to their > password" cannot be done in this way --- but can be done by just > using a function #(lambda (x) (equal (username-of x) (password-of > x)). SQL can do this --- but LISP can could use any function > there, such as "find users who have both short usernames and > passwords that can be cracked by routine-x". > Instead of adding things to the root or the store-controller > directly, one would generally prefer > to use consistent classes: > http://common-lisp.net/project/elephant/doc/Persistent- > Classes.html#Persistent-Classes > > This doesn't change the nature of the problem. If you like, you > can create an index on any slot in a very convenient way: http:// > common-lisp.net/project/elephant/doc/Class-Indices.html#Class-Indices > This holds out the possibility of NOT having to iterate over the > entire data-set, but rather honing in directly upon > matching values. (You can in fact create functional indexes based > on any function at all in Elephant, which is something that SQL > can't do conveniently, but the times you will need to do this are > rare.) > > It would take maybe 20 minutes more coding to uses Ian's "get- > instances-by-range" directly, and in a very efficient manner for > performing the query (if the GUI elements correspond to the class's > slots.) This would be very efficient; but of course you should > not do this until you know that this is really the bottleneck in > your system. By using cursors, you can avoid reading the entire > data into memory, and thus process huge datasets. > However, one should note that Ian's code makes creating indices > on slots zero-effort; but indexes always have overhead. The real > question is: when will your queries actually utilize the index? > (That is, if you always select on one column/slot, then that one > should be indexed....but if your query pattern is more complicated, > it becomes fuzzy.) > Let me know if you find this useful; after I get feedback from > you and Ian has made his post, perhaps we will put > this in the documentation. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lists at infoway.net Mon Nov 13 05:42:02 2006 From: lists at infoway.net (Daniel Salama) Date: Mon, 13 Nov 2006 00:42:02 -0500 Subject: [elephant-devel] Complex queries In-Reply-To: <455763DC.3040708@csail.mit.edu> References: <455763DC.3040708@csail.mit.edu> Message-ID: <44E13D01-2F38-44E8-9A65-B04D59575920@infoway.net> Thanks. I'll be great to contribute anything to the project and a pleasure working with you guys. On Nov 12, 2006, at 1:11 PM, Ian Eslick wrote: > I'm interested in this problem myself, but haven't had the time to do > the appropriate research. Many SQL query engines are built on top > of a > BTree infrastructure so building and performing optimized queries > should > be quite doable in Elephant. However, I do know that efficient > operations are fairly complicated. > > I'd be happy to support your work by including a search interface + > simple compiler into the main Elephant release. > > I'm assuming you want to search for persistent objects and not > standard > objects stored in BTrees. Correct. Just persistent objects > > There are two ways to extract a set of objects associated with a > particular slot value: > 1) Get all objects of the desired type and do a linear search > 2) Create a slot-index for searchable slots. You can use the accessor > functions in get-instances-by-value and get-instances-by-range. This > returns the persistent object. Current cursor accesses require full > deserialization of the target object but fortunately this doesn't > include the slot, just the oid+classname, etc. Pardon my lisp ignorance, but is there a way to introspect an object/ class in lisp and find out whether or not a slot is "indexable"? That way, maybe whatever code is querying for data can take advantage of any slot-index before defaulting to linear searching. > > If you extract a range of objects using a slot index, they'll > already be > ordered according to the sort order of the index values. You can do > joins by filtering each list by whether the object exists in the next > list or not. > > i.e. > > age = 18 (oid1, oid2, oid3, oid4, oid5, oid6, oid7 ... ) > name = "Cindy" (oid4, oid7, oid3 ...) > state = MA (oid10, oid4, oid7 ...) > > Filtered objects = oid4, oid7 ... > I would assume that you might have meant intersection instead of join? Right? > Optimization issues: > > 1) Returned objects are in value order, not OID or object order; so > each > merge of lists is naively C*N^2 where C is the number of > constraints (3 > above) and N is the average size of a returned list. This can be > avoided by sorting each list (N log N) prior to merging which is > effectively M log M (where M = the sum of all returned objects for all > the constraints). In fact if you append the lists (N) and then sort > removing duplicates M log M you get the same result. You then need > another N log N walk of the list to order the elements according to > the > original list of constraints. > > I'm sure there's a better way, but this is a decent first order take. > It's too late at night for me to compute/validate this Big O notation this but will confirm/comment tomorrow :) > 2) Now this works fine when the # of objects in a given query are > small. But let's say you want to pick all adults with the name > "Jerry" > from a 1M entry database of people. This is a range query > (get-instances-by-range 'person 'age 18 120) and a value query > (get-instances-by-value 'person 'first-name "Jerry"). However, the > # of > Jerry instances and in particular the # of age > 18 instances will be > enormous (at least 500k). You might not even be able to fit all of > these in memory! > This is exactly my main concern. Specially when, for example, the age of a person is a slot of a class called demographic to which the class person has a relationship. > In this case we need to know how large the sets of objects are under > given constraints. (I'm not sure if BDB provides a way to get > access to > the size of the index between two values - the total size of the > tree + > the height of the nodes should be telling as the BTrees are > balanced...). In this case we can try to use cursors or slot > accessors > to do the filtering based on the total size and accessibility of the > queries. At this point it gets somewhat complicated and I think it's > worthwhile to appeal to the SQL implementation literature. Agree. I'll see if I can research any SQL implementation on how they compute this. After all, SQL is all about boolean math, so they have to have a way to easily compute this. > Also, what happens if a slot is not indexed? How do we search > multiple > object classes at the same time? For example, how would we specify a > search over instances and it's super classes when the superclass also > contain the slot constraints? (Currently class indexes are > specific to > a class name) I suppose we'll find the answer of this somewhere in a SQL implementation, unless anyone already knows and can contribute that knowledge (I guess those are the pros and cons of going to an object database rather than a relational database). > > > Is this helpful? A first cut approach as described in #1 above will > probably flesh out a large number of issues and shouldn't take that > much > time. If you are motivated, or have large queries, then it's > appropriate to spend some time talking about how to handle the > issues of > large sets in #2. > I think #2 is where we (and I don't mean just my team) want to go since it will provide the most flexibility. > Query language: > > Does it make sense to have a full query language, or a simpler > macros to > make this simple? The easy macro approach probably doesn't need > anything sophisticated: > > (retrieve-instances-by-constraints ((person :with-superclasses t) > employee) > ((age 18 120) > (name "George") > (state :MA)) > (:symbols-as-strings t :order t)) > > Here we search for person objects, parents of person objects and > employee objects for the three constraints. The search option > :symbols-as-strings means to treat :MA as matching :MA or "MA" - that > means two slot queries. Not sure that's a good idea, but it's an > example of modifying how the query operates. :order means to order > the > resulting objects by their age, in this case. The order of ranges in > the constraint list would determine the total order. > IMHO, macros are just fine. You can always repackage them in a separate domain language. > What if we want to do join queries? > How about OR constraints? (state (or :MA :NY)) > I have the same questions. > > Ian > Thanks, Daniel From lists at infoway.net Mon Nov 13 05:46:30 2006 From: lists at infoway.net (Daniel Salama) Date: Mon, 13 Nov 2006 00:46:30 -0500 Subject: [elephant-devel] Querying Advice In-Reply-To: <4558003C.2070104@csail.mit.edu> References: <948E9C76-5F70-4245-9697-A59E50ABF9D6@infoway.net> <1163364527.5145.244.camel@localhost.localdomain> <3D430634-748D-44BE-9B44-A8EB5AB5EBAB@infoway.net> <4558003C.2070104@csail.mit.edu> Message-ID: <88B23BCB-2A6C-4943-91E4-2F9CDC48C4C0@infoway.net> > Were you storing persistent-metaclass objects or simply normal > objects? > Normal objects are huge relative to persistent objects as currently > all > the slot names are also serialized (potentially in 32-bit unicode > if on > SBCL). I have some improvements planned to reduce the storage and > processing overhead for strings, but it will take some evaluation to > make sure performance doesn't suffer over much. Some of this will > make > it into the forthcoming 0.6.1 release I'm trying to finish a bunch of > work for prior to my twin daughters are born. > > Also SBCL strings in the 0.6.0 release are stored in their internal > 32-bit form so we can memcpy them back and forth between elephant and > the internal format. The new serializer will store them in the > minimal > format and then convert back to the 32-bit internal form for unicode > enabled SBCL. We were storing persistent objects. However, instead of storing them all into the root, we were storing them into a normal collection class (I think it was even a BTree), such that we would have a collection of "people". Maybe this was the wrong approach. We were coming from the RDBMS world so what we ended up doing is storing the root, different collections of objects, where each collection would "kind-of" resemble the different tables we have in MySQL. > DCM is, as I understand it, a form of prevalence with explicit > checkpointing using Elephant as the backend. Cool. Will look further into it. >> With regards to: "...If so, you have the full power of lisp at your >> command in dealing with the querying...", I agree with you. However, >> where I'm trying to get at is how "easy" would it be to generate >> these >> type of dynamic queries in a generic way. Of course, we could always >> hard code all the cases for each of our different searchable screens, >> but the thought of that simply just makes me vomit :) > Associate a constraint on the display with the user action for > selections and an ordering function for specific columns. If you want > to sort by multiple columns, then compose your functions: > > Selection constraint fn: > #'(lambda (obj) (= (slot-accessor obj) value) > > Sorting fn: > #'(lambda (obj1 obj2) (string< (name obj1) (name obj2))) > #'(lambda (obj1 obj2) (< (age obj1) (age obj2))) > > Composed sorting function: > (defun compose-sorts (sort1 sort2) > (lambda (o1 o2) > (or (funcall sort1 o1 o2) > (funcall sort2 o1 o2)))) > > Same thing with constraints: > (defun compose-constraint (c1 c2) > (lambda (obj) > (and (funcall c1 obj) > (funcall c2 obj)))) > > Or to be a little more fancy: > (defun compose-constraints (clist) > (lambda (obj) > (every #'(lambda (c) (funcall c obj)) clist))) > > (defun compose-sorts (slist) > (lambda (o1 o2) > (some #'(lambda (sort) (funcall sort o1 o2))))) > > This all comes down to: > > (defun select-results (objects) > (select-if (compose-constraints (make-name-constraints "Bob") > (make-age-constraint 18 120)) > objects)) > > (defun order-results (objects) > (sort (copy-list objects) > (compose-sorts #'by-occupation #'by-name #'by-date-of- > birth))) > > This way you can have a set of widgets with static constraints and > then > you can compose functions over the constraints based on the dynamic > query. There may be more elegant ways to do this but I think this is > what Robert was driving at. > > To do this on elephant simply requires that you (get-instances-by- > class > 'person) to get all instances of person and then run the lisp query > over > it. If you access person alot, these persistent references will be > cached in memory. Or they're all in the cache including slot > values in > DCM - in which case these queries will run really fast! I think I understand this. Will try to play with these ideas further and see where we go. > Every time you finish or 'commit' a user transaction (form submit), > just > write the object data back to the Elephant persistent store before > acking to the user. Is this done automatic by the way DCM is configured or is it a manual commit you do in DCM/Elephant to persistent store? > > > Ian Thanks, Daniel From nowhere.man at levallois.eu.org Mon Nov 13 11:52:03 2006 From: nowhere.man at levallois.eu.org (Pierre THIERRY) Date: Mon, 13 Nov 2006 12:52:03 +0100 Subject: [elephant-devel] Querying Advice In-Reply-To: <948E9C76-5F70-4245-9697-A59E50ABF9D6@infoway.net> References: <948E9C76-5F70-4245-9697-A59E50ABF9D6@infoway.net> Message-ID: <20061113115203.GN12538@bateleur.arcanes.fr.eu.org> Scribit Daniel Salama dies 12/11/2006 hora 10:28: > I guess we would have to sequentially navigate thru the results in > order to "manually" select each record based on all the other > possible search arguments. I suppose, in a way, this can be done > relatively painless by using macros I don't think custom macros are needed here. The search could be made with the various searching functions of Lisp and a predicate built from the constraints. Here is an example of a predicate building function for an application about real estate: (defun search-predicate (&key min-price max-price min-surface max-surface) (lambda (x) (and (if min-price (>= (price x) min-price) t) (if max-price (<= (price x) max-price) t) (if min-surface (>= (surface x) min-surface) t) (if max-surface (<= (surface x) max-surface) t)))) Macros could be used to return an optimized closure where some useless branching is avoided, but some implementation may already do that optimization by themselves (I think at least SBCL removes unreachable code when compiling). It should be pretty easy to write some macros to avoid writing the predicated building function by hand, and we may provide some more as some sort of query language for the most common cases (min and max for the result of a function, and so on). I think we should keep the ability for the user to provide it's own predicate code (it would be so unlispy not to...). > Then we have the issue of the sorting. I suppose it falls into a > similar situation: once we get the matching resultset from the > previous step, we would have to perform some "efficient" sort > algorithm on the data set dynamically based on the user's sorting > desire. I also suppose we could create a macro for this as well. I'm sure sorting is an absolutely pure functional thing. It's "just" a matter of finding an efficient algorithm here. Beware of premature optimization though. I think we should build a correct solution first, and check the state of the art of database implementations if profiling show us a bad performance. Functionally, Nowhere man -- nowhere.man at levallois.eu.org OpenPGP 0xD9D50D8A -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From eslick at csail.mit.edu Mon Nov 13 12:37:33 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Mon, 13 Nov 2006 07:37:33 -0500 Subject: [elephant-devel] Querying Advice In-Reply-To: <88B23BCB-2A6C-4943-91E4-2F9CDC48C4C0@infoway.net> References: <948E9C76-5F70-4245-9697-A59E50ABF9D6@infoway.net> <1163364527.5145.244.camel@localhost.localdomain> <3D430634-748D-44BE-9B44-A8EB5AB5EBAB@infoway.net> <4558003C.2070104@csail.mit.edu> <88B23BCB-2A6C-4943-91E4-2F9CDC48C4C0@infoway.net> Message-ID: <4558670D.5090701@csail.mit.edu> Daniel Salama wrote: >> Were you storing persistent-metaclass objects or simply normal objects? >> Normal objects are huge relative to persistent objects as currently all >> the slot names are also serialized (potentially in 32-bit unicode if on >> SBCL). I have some improvements planned to reduce the storage and >> processing overhead for strings, but it will take some evaluation to >> make sure performance doesn't suffer over much. Some of this will make >> it into the forthcoming 0.6.1 release I'm trying to finish a bunch of >> work for prior to my twin daughters are born. >> >> Also SBCL strings in the 0.6.0 release are stored in their internal >> 32-bit form so we can memcpy them back and forth between elephant and >> the internal format. The new serializer will store them in the minimal >> format and then convert back to the 32-bit internal form for unicode >> enabled SBCL. > > We were storing persistent objects. However, instead of storing them > all into the root, we were storing them into a normal collection class > (I think it was even a BTree), such that we would have a collection of > "people". Maybe this was the wrong approach. We were coming from the > RDBMS world so what we ended up doing is storing the root, different > collections of objects, where each collection would "kind-of" resemble > the different tables we have in MySQL. > Class indexing accomplishes the same thing - it creates a BTree for each indexed class type that contains all instances of the class. Slot indexes can be added/deleted and are secondary BTrees with the class BTree as primary. >> Every time you finish or 'commit' a user transaction (form submit), just >> write the object data back to the Elephant persistent store before >> acking to the user. > > Is this done automatic by the way DCM is configured or is it a manual > commit you do in DCM/Elephant to persistent store? > I think this is manual, but you'll have to ask Robert as I haven't had time to dig into his DCM code yet. Ian From eslick at csail.mit.edu Mon Nov 13 12:57:16 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Mon, 13 Nov 2006 07:57:16 -0500 Subject: [elephant-devel] Querying Advice In-Reply-To: <20061113115203.GN12538@bateleur.arcanes.fr.eu.org> References: <948E9C76-5F70-4245-9697-A59E50ABF9D6@infoway.net> <20061113115203.GN12538@bateleur.arcanes.fr.eu.org> Message-ID: <45586BAC.9030802@csail.mit.edu> Performance and functionality of search/sort will depend a great deal on the model we have of how data is stored, how it is cached, how it is indexed and what kind of queries we want to support. Robert's approach is more than adequate if everything is maintained in lists in memory; it is more expensive if we're searching by de-serializing slot values from disk one at a time. An index can be used to clean that up, but sequencing the indexes can have an enormous impact on performance and can perform queries that a naive approach would fail on because of exhausting memory. I think a macro system makes sense when we find we want to have a query syntax (a simpler and more elegant version of what SQL tries to do) that concisely expresses the class of operations we want to perform and behind which we need to put a query sequencer or compiler to optimize retrieval. I do think that Pierre is right that we need to list a set of queries we want to perform and not worry about performance up front. We can worry about sequencing, etc when we have examples of slow queries that start to rise to the level of user perception. Let's build up this infrastructure incrementally so we can figure out what the problems are at low cost. Also, I'm going to promote a faster serializer shortly that should speed up both SQL and BDB backends which should take some modest constant factors off of query runtime from disk (vs. DCM). Cheers, Ian Pierre THIERRY wrote: > Scribit Daniel Salama dies 12/11/2006 hora 10:28: > >> I guess we would have to sequentially navigate thru the results in >> order to "manually" select each record based on all the other >> possible search arguments. I suppose, in a way, this can be done >> relatively painless by using macros >> > > I don't think custom macros are needed here. The search could be made > with the various searching functions of Lisp and a predicate built from > the constraints. Here is an example of a predicate building function for > an application about real estate: > > (defun search-predicate (&key min-price max-price min-surface max-surface) > (lambda (x) > (and > (if min-price (>= (price x) min-price) t) > (if max-price (<= (price x) max-price) t) > (if min-surface (>= (surface x) min-surface) t) > (if max-surface (<= (surface x) max-surface) t)))) > > Macros could be used to return an optimized closure where some useless > branching is avoided, but some implementation may already do that > optimization by themselves (I think at least SBCL removes unreachable > code when compiling). > > It should be pretty easy to write some macros to avoid writing the > predicated building function by hand, and we may provide some more as > some sort of query language for the most common cases (min and max for > the result of a function, and so on). I think we should keep the ability > for the user to provide it's own predicate code (it would be so > unlispy not to...). > > >> Then we have the issue of the sorting. I suppose it falls into a >> similar situation: once we get the matching resultset from the >> previous step, we would have to perform some "efficient" sort >> algorithm on the data set dynamically based on the user's sorting >> desire. I also suppose we could create a macro for this as well. >> > > I'm sure sorting is an absolutely pure functional thing. It's "just" a > matter of finding an efficient algorithm here. Beware of premature > optimization though. I think we should build a correct solution first, > and check the state of the art of database implementations if profiling > show us a bad performance. > > Functionally, > Nowhere man > > ------------------------------------------------------------------------ > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From venkat at rayservers.com Mon Nov 13 14:16:50 2006 From: venkat at rayservers.com (Venkat Manakkal) Date: Mon, 13 Nov 2006 09:16:50 -0500 Subject: [elephant-devel] Querying Advice [w/code example] In-Reply-To: References: <948E9C76-5F70-4245-9697-A59E50ABF9D6@infoway.net> <1163364527.5145.244.camel@localhost.localdomain> <1163375121.5145.289.camel@localhost.localdomain> Message-ID: <45587E52.2000800@rayservers.com> Daniel Salama wrote: > 1) The nature of dynamic queries as presented in my original email: your Just briefly coming out of lurk mode, when designing your application you might want to use something like GBBopen from gbbopen.org that implements a main memory blackboard for dynamic queries to in memory objects. It also provides for dynamic re-ordering of execution of what are called knowledge-sources, as well as events and event triggers that can be used along with a control shell mechanism. If you write your application in such a way that persistence is handled by Berkley db and dynamic queries are in memory, you can develop an application specific language that will rock. Best regards, ---Venkat. From read at robertlread.net Mon Nov 13 14:46:04 2006 From: read at robertlread.net (Robert L. Read) Date: Mon, 13 Nov 2006 08:46:04 -0600 Subject: [elephant-devel] Querying Advice In-Reply-To: <3D430634-748D-44BE-9B44-A8EB5AB5EBAB@infoway.net> References: <948E9C76-5F70-4245-9697-A59E50ABF9D6@infoway.net> <1163364527.5145.244.camel@localhost.localdomain> <3D430634-748D-44BE-9B44-A8EB5AB5EBAB@infoway.net> Message-ID: <1163429164.5145.315.camel@localhost.localdomain> On Sun, 2006-11-12 at 23:17 -0500, Daniel Salama wrote: > I believe (and hope) that we should have no problem mapping the GUI to > the requests. Just out of curiosity (and I don't mean to divert from > the topic of this thread): if you're using DCM for your konsenti (BTW, > nice concept) site, how do you protect your in-memory data? Do you > just write an image to disk every once in a while for back ups? How > resilient is this to hardware failure and you loosing data since the > last image (if that's your approach)? DCM does write-through; any change in a data object is explicitly pushed to the datastore via "register-obj". If you can't fit your dataset into memory, then you will need to you the "gdcm" file, which implements a "generational" notion. Every object has a generation, and you can choose a directory strategy for each generation. The obvious thing to do is to keep the "zero" generation in memory, and everything in the datastore. Thus access to the first generation is very very fast, but when you reach into other geneations you pay the normal database retrieval cost. The BDB files are probably larger because our serializer is not as efficient as it could be. -------------- next part -------------- An HTML attachment was scrubbed... URL: From lists at infoway.net Mon Nov 13 14:45:26 2006 From: lists at infoway.net (Daniel Salama) Date: Mon, 13 Nov 2006 09:45:26 -0500 Subject: [elephant-devel] Querying Advice In-Reply-To: <20061113115203.GN12538@bateleur.arcanes.fr.eu.org> References: <948E9C76-5F70-4245-9697-A59E50ABF9D6@infoway.net> <20061113115203.GN12538@bateleur.arcanes.fr.eu.org> Message-ID: <66404002-3612-4DAC-B85D-E8BEFCDD9827@infoway.net> I agree that macros are not a necessity. I was thinking in abstracting the predicate building functions so that you wouldn't have to "hard-code" some of the functionality. In essence, kind of like building a querying language. Granted, I agree that it should be flexible enough to allow the user to provide custom predicate functions. Thanks, Daniel On Nov 13, 2006, at 6:52 AM, Pierre THIERRY wrote: > Scribit Daniel Salama dies 12/11/2006 hora 10:28: >> I guess we would have to sequentially navigate thru the results in >> order to "manually" select each record based on all the other >> possible search arguments. I suppose, in a way, this can be done >> relatively painless by using macros > > I don't think custom macros are needed here. The search could be made > with the various searching functions of Lisp and a predicate built > from > the constraints. Here is an example of a predicate building > function for > an application about real estate: > > (defun search-predicate (&key min-price max-price min-surface max- > surface) > (lambda (x) > (and > (if min-price (>= (price x) min-price) t) > (if max-price (<= (price x) max-price) t) > (if min-surface (>= (surface x) min-surface) t) > (if max-surface (<= (surface x) max-surface) t)))) > > Macros could be used to return an optimized closure where some useless > branching is avoided, but some implementation may already do that > optimization by themselves (I think at least SBCL removes unreachable > code when compiling). > > It should be pretty easy to write some macros to avoid writing the > predicated building function by hand, and we may provide some more as > some sort of query language for the most common cases (min and max for > the result of a function, and so on). I think we should keep the > ability > for the user to provide it's own predicate code (it would be so > unlispy not to...). > >> Then we have the issue of the sorting. I suppose it falls into a >> similar situation: once we get the matching resultset from the >> previous step, we would have to perform some "efficient" sort >> algorithm on the data set dynamically based on the user's sorting >> desire. I also suppose we could create a macro for this as well. > > I'm sure sorting is an absolutely pure functional thing. It's "just" a > matter of finding an efficient algorithm here. Beware of premature > optimization though. I think we should build a correct solution first, > and check the state of the art of database implementations if > profiling > show us a bad performance. > > Functionally, > Nowhere man > -- > nowhere.man at levallois.eu.org > OpenPGP 0xD9D50D8A > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From lists at infoway.net Mon Nov 13 14:47:15 2006 From: lists at infoway.net (Daniel Salama) Date: Mon, 13 Nov 2006 09:47:15 -0500 Subject: [elephant-devel] Querying Advice In-Reply-To: <4558670D.5090701@csail.mit.edu> References: <948E9C76-5F70-4245-9697-A59E50ABF9D6@infoway.net> <1163364527.5145.244.camel@localhost.localdomain> <3D430634-748D-44BE-9B44-A8EB5AB5EBAB@infoway.net> <4558003C.2070104@csail.mit.edu> <88B23BCB-2A6C-4943-91E4-2F9CDC48C4C0@infoway.net> <4558670D.5090701@csail.mit.edu> Message-ID: <38E9823B-E4BD-4380-987B-8C33FFE252DE@infoway.net> On Nov 13, 2006, at 7:37 AM, Ian Eslick wrote: > > > Class indexing accomplishes the same thing - it creates a BTree for > each > indexed class type that contains all instances of the class. Slot > indexes can be added/deleted and are secondary BTrees with the class > BTree as primary. So, would you say that we based on our approach, we could just store a bunch of random objects (whether person, state, zip, order, etc) in the root and use class indices? Sounds interesting. > > Ian Thanks, Daniel From lists at infoway.net Mon Nov 13 14:49:29 2006 From: lists at infoway.net (Daniel Salama) Date: Mon, 13 Nov 2006 09:49:29 -0500 Subject: [elephant-devel] Querying Advice In-Reply-To: <45586BAC.9030802@csail.mit.edu> References: <948E9C76-5F70-4245-9697-A59E50ABF9D6@infoway.net> <20061113115203.GN12538@bateleur.arcanes.fr.eu.org> <45586BAC.9030802@csail.mit.edu> Message-ID: <59F2EE95-4157-474F-9F20-9DB7506E0740@infoway.net> On Nov 13, 2006, at 7:57 AM, Ian Eslick wrote: > I think a macro system makes sense when we find we want to have a > query > syntax (a simpler and more elegant version of what SQL tries to do) > that > concisely expresses the class of operations we want to perform and > behind which we need to put a query sequencer or compiler to optimize > retrieval. That's where I'm aiming at > > I do think that Pierre is right that we need to list a set of > queries we > want to perform and not worry about performance up front. We can > worry > about sequencing, etc when we have examples of slow queries that start > to rise to the level of user perception. Let's build up this > infrastructure incrementally so we can figure out what the problems > are > at low cost. Agree 100% > > Also, I'm going to promote a faster serializer shortly that should > speed > up both SQL and BDB backends which should take some modest constant > factors off of query runtime from disk (vs. DCM). > > Cheers, > Ian That would be splendid Thanks, Daniel From lists at infoway.net Mon Nov 13 14:50:43 2006 From: lists at infoway.net (Daniel Salama) Date: Mon, 13 Nov 2006 09:50:43 -0500 Subject: [elephant-devel] Querying Advice [w/code example] In-Reply-To: <45587E52.2000800@rayservers.com> References: <948E9C76-5F70-4245-9697-A59E50ABF9D6@infoway.net> <1163364527.5145.244.camel@localhost.localdomain> <1163375121.5145.289.camel@localhost.localdomain> <45587E52.2000800@rayservers.com> Message-ID: <4AD524C6-40FF-41B5-937A-B12CB96D9A7C@infoway.net> I'm not familiar with GBBopen and will read up on it. Would anyone care to comment on it? For the purpose of trying to "develop" a querying facility for Elephant, will it be useful/needed? Thanks, Daniel On Nov 13, 2006, at 9:16 AM, Venkat Manakkal wrote: > Daniel Salama wrote: >> 1) The nature of dynamic queries as presented in my original >> email: your > > Just briefly coming out of lurk mode, when designing your application > you might want to use something like GBBopen from gbbopen.org that > implements a main memory blackboard for dynamic queries to in memory > objects. > > It also provides for dynamic re-ordering of execution of what are > called > knowledge-sources, as well as events and event triggers that can be > used > along with a control shell mechanism. > > If you write your application in such a way that persistence is > handled > by Berkley db and dynamic queries are in memory, you can develop an > application specific language that will rock. > > Best regards, > > ---Venkat. From eslick at csail.mit.edu Mon Nov 13 15:00:40 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Mon, 13 Nov 2006 10:00:40 -0500 Subject: [elephant-devel] Querying Advice [w/code example] In-Reply-To: <4AD524C6-40FF-41B5-937A-B12CB96D9A7C@infoway.net> References: <948E9C76-5F70-4245-9697-A59E50ABF9D6@infoway.net> <1163364527.5145.244.camel@localhost.localdomain> <1163375121.5145.289.camel@localhost.localdomain> <45587E52.2000800@rayservers.com> <4AD524C6-40FF-41B5-937A-B12CB96D9A7C@infoway.net> Message-ID: <45588898.7040005@csail.mit.edu> Daniel, I think GBBopen would be an application-oriented approach to indexing rather than an infrastructure that one might use to build a query engine for elephant; from what I can tell you have to buy into their model of objects, spaces and dimensions, etc. You could mix it with DCM, perhaps - but I think that's alot of hair to take on to do what I think you're talking about. A facility to support dynamic indexing and sorting should be fairly cheap (a few files of a few 100 lines each max). Here's an example of class indexing: (defpclass person () ((name :accessor name :initarg :name :index t) (age :accessor age :initarg :age :index t) (occupation :accessor occupation :initarg :occupation :index t))) ;; defpclass is a macro that adds :metaclass persistent-metaclass to the class options (make-instance 'person :name "Ian" :age 33 :occupation "student") (make-instance 'person :name "Alyssa" :age 40 :occupation "hacker") (defmethod print-object ((obj person) stream) (format stream "" (name obj))) (get-instances-by-class 'person) => ( ) (get-instances-by-value 'person 'name "Ian") => () (get-instances-by-range 'person 'age 30 45) => ( ) Ian Daniel Salama wrote: > I'm not familiar with GBBopen and will read up on it. Would anyone > care to comment on it? For the purpose of trying to "develop" a > querying facility for Elephant, will it be useful/needed? > > Thanks, > Daniel > > On Nov 13, 2006, at 9:16 AM, Venkat Manakkal wrote: > >> Daniel Salama wrote: >>> 1) The nature of dynamic queries as presented in my original email: >>> your >> >> Just briefly coming out of lurk mode, when designing your application >> you might want to use something like GBBopen from gbbopen.org that >> implements a main memory blackboard for dynamic queries to in memory >> objects. >> >> It also provides for dynamic re-ordering of execution of what are called >> knowledge-sources, as well as events and event triggers that can be used >> along with a control shell mechanism. >> >> If you write your application in such a way that persistence is handled >> by Berkley db and dynamic queries are in memory, you can develop an >> application specific language that will rock. >> >> Best regards, >> >> ---Venkat. > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From read at robertlread.net Mon Nov 13 15:18:58 2006 From: read at robertlread.net (Robert L. Read) Date: Mon, 13 Nov 2006 09:18:58 -0600 Subject: [elephant-devel] Querying Advice [w/code example] In-Reply-To: References: <948E9C76-5F70-4245-9697-A59E50ABF9D6@infoway.net> <1163364527.5145.244.camel@localhost.localdomain> <1163375121.5145.289.camel@localhost.localdomain> Message-ID: <1163431138.5145.339.camel@localhost.localdomain> Some notes; forgive me if these are out of order: *) (format t "blah~A" x) is not as efficient as (concatenate 'string "blah" x), but I typically use it for debugging and its fine unless the strings are long. *) If you have many tables/persistent classes, then you are quite correct, the code which I posted won't handle this in a direct way. However, there is no reason that one of the small functions in question can't access the data store, via "get-from-root", a btree-access, or a persistent class access. A SQL advocate would say that once these queries become complicate enough, a query optimizer can do a better job organizing the access. There is some truth in this; I get around this by using a prevalence like-model in anyway. To make SQL work really well in this way, you have to design your indicespretty carefully; the same is basically true of using Elephant (except that its easier, due to Ian's coding) and that if you put everything in memory it almost doesn't matter. (Of course, you may end up having to build indices in memory as well, but I wouldn't do that until careful profiling showed it to be needed.) So in my code example, a reasonable strategy is to "loop over" the basic data set, and go out to other data sets only when they are needed. In a complex data situation, it can be hard to distinguish which class should be considered basic; frankly in my experience things that complicated rarely arise. The ideal situation is to run the loop over the dataset that contributes the most objects, in terms of cardinality, to the result, and look for chances not to reach into other data sets. Of course, the more smarts you put into working on this, the closer you come to duplicated the work that a standard "query optimizer" in a RDBMS system does. There might be times when an RDBMS is actually better; but for my money the prevalence model matches normal applications more often. The more dynamic your set of queries is, the less well a query optimizer will perform, whether inside an RDBMS or an expression of your own code; so use the K.I.S.S. principle if you want maximum query flexibility. *) Pierre pointed out a good use of macros ---- You could used them to make a nice little query syntax. Note that CL-SQL does this quite nicely, and in the SQL-backend of elephant we use this a little bit. The issue of whether you need a macro system for this or not somewhat depends on how the GUI actions will be translated into the LISP functions. For example, if you wanted to let someone actually type in: "field1>4 && field3 ~= "\d+*fizmo"", then a macro language is a good way to do that. I uses a macro package called LML for producing (X)HTML, and it is a compelling reason to use lisp. (BTW, that query is a bit of a joke---I doubt anyone would want to expose a Perl-like query-syntax to an end-user; but my point is, you could---and with the CL-PCCRE package (which rocks) and a prevalence like model, it would be fast.) [snipey personality on: Try doing that in Java.] BTW, with just a little thought, you should be able to adopt the code that I posted to OpenMCL, I think. That might be faster than installing SBCL. On Mon, 2006-11-13 at 00:19 -0500, Daniel Salama wrote: > Wow! I wasn't expecting to hear back from you until Monday :) > > > I haven't been able to run your code yet. I downloaded SBCL for PPC > but have not been able to successfully compile Elephant with it. I > will continue trying. I used to have it working before, but stopped > using SBCL a while ago for OpenMCL. > > > Anyway, some comments before I can run your code and provide more > feedback: > > > 100% agree with all this. However, one thing that might have been > oversimplified is when it comes down to complex relational queries. > There are two main issues to address here: > > > 1) The nature of dynamic queries as presented in my original email: > your explanation and code seems to address this in a subtle way and, > although I believe you when you say it works, I'd say that it could be > enhanced in a way to make it generic enough. I remember the how Peter > Siebel presents such a nice and simple interface to querying a CD/MP3 > database in Chapter 3 of his Practical Common Lisp book. In a way, he > kind of developed a small domain language for querying and I think > something like that can be implemented with the sample code you > provided below. > > > 2) What happens when you have multiple relational tables... or more > appropriate in our case, when you want to query an object model based > on different object and object relational hierarchies. Say, for > example, you have class person which has references to an object > called demographics and an object postal-code, which, in turns, has a > reference to an object state. How would you apply your proposal to a > query such as "give me all 'males' under the age of 30 who live in > 'NY' sorted by age in descending order and then by last, first names". > It obviously is very simple to do this in SQL (if the proper relations > exist between the tables), but may not be (or at least doesn't seem to > be initially) as trivial in Elephant or applying your code below. Your > dynamically constructed list of query-able columns approach may be a > bit limited. Also, class indices may not necessarily be appropriate > either in these situation. > > > Maybe I haven't delved deeply enough into your proposal and/or haven't > given it sufficient thought (I apologize for that if I belittled your > proposal). > > > I definitely understand the potential overhead of using secondary > indices and agree that their use should be determined by how often you > really need to search by those slots. > > > I will continue trying to make SBCL work again in my PPC and then run > your code. Maybe play with it and try to simulate different scenarios > such as I mentioned above. Will then send more comments. > > > Thanks again, > Daniel > > > P.S. I don't mean to distract anyone from the main thread of this > email. But I noticed how you make use of format to append values to > strings (e.g. random-password). Is that like the only way to do so, or > maybe the more efficient way of doing so? Thanks again -------------- next part -------------- An HTML attachment was scrubbed... URL: From read at robertlread.net Mon Nov 13 15:22:27 2006 From: read at robertlread.net (Robert L. Read) Date: Mon, 13 Nov 2006 09:22:27 -0600 Subject: [elephant-devel] Querying Advice In-Reply-To: <38E9823B-E4BD-4380-987B-8C33FFE252DE@infoway.net> References: <948E9C76-5F70-4245-9697-A59E50ABF9D6@infoway.net> <1163364527.5145.244.camel@localhost.localdomain> <3D430634-748D-44BE-9B44-A8EB5AB5EBAB@infoway.net> <4558003C.2070104@csail.mit.edu> <88B23BCB-2A6C-4943-91E4-2F9CDC48C4C0@infoway.net> <4558670D.5090701@csail.mit.edu> <38E9823B-E4BD-4380-987B-8C33FFE252DE@infoway.net> Message-ID: <1163431348.5145.342.camel@localhost.localdomain> On Mon, 2006-11-13 at 09:47 -0500, Daniel Salama wrote: > So, would you say that we based on our approach, we could just store > a bunch of random objects (whether person, state, zip, order, etc) > in > the root and use class indices? Sounds interesting. > I have to go to the dentist now, but this sounds a bit confused ---- you would EITHER put things in the root or use persistent classes; doing both doesn't make sense to me. -------------- next part -------------- An HTML attachment was scrubbed... URL: From lists at infoway.net Mon Nov 13 18:43:03 2006 From: lists at infoway.net (Daniel Salama) Date: Mon, 13 Nov 2006 13:43:03 -0500 Subject: [elephant-devel] Querying Advice [w/code example] In-Reply-To: <1163375121.5145.289.camel@localhost.localdomain> References: <948E9C76-5F70-4245-9697-A59E50ABF9D6@infoway.net> <1163364527.5145.244.camel@localhost.localdomain> <1163375121.5145.289.camel@localhost.localdomain> Message-ID: <1461D8DA-03F1-4DE8-B97C-1C4ABFA3BC81@infoway.net> Ok. I got Elephant to work again with SBCL on PPC. I guess I was still using BDB 4.3 when 4.4 seems to be required. I couldn't find that anywhere in the docs. I see what you're saying and can start to envision where this can go. I will keep playing and provide more feedback later today. FYI, for curiosity purposes, I just ran (random-users 5000) on an empty database and it took 5 min 56 secs to complete. At the end, the %ELEPHANT file was 9MB in size and both log files where 10MB. That's kind of big considering what it does. I then made the random-users2 function to look like: (defun random-users2 (n) (setq *auto-commit* nil) (with-transaction () (dotimes (x n) (let ((u (make-instance 'User :uname (format nil "user~A" x) :pword (random-password) :email (format nil "user~A at .nowheresville.org" x) :fullname (format nil "~A~A ~A~A" (random-password) x (random-password) x) :balance (random 100)))) (add-to-root x u)))) (setq *auto-commit* t)) When I ran (random-users2 5000) I got a Berkeley DB Error: Cannot allocate memory, so I changed random-users2 to: (defun random-users2 (n) (setq *auto-commit* nil) (start-ele-transaction) (dotimes (x n) (if (eq (mod x 1000) 0) (progn (commit-transaction) (start-ele-transaction))) (let ((u (make-instance 'User :uname (format nil "user~A" x) :pword (random-password) :email (format nil "user~A at .nowheresville.org" x) :fullname (format nil "~A~A ~A~A" (random-password) x (random-password) x) :balance (random 100)))) (add-to-root x u))) (commit-transaction) (setq *auto-commit* t)) When I ran (random-users2 5000) on an empty database, this time it only 25 secs to complete. The %ELEPHANT file was still 9MB in size and both log files where also 10MB. I suppose your code was not designed for performance and only for illustration purposes. Needless to say, knowing how to use transactions certainly helps and can dramatically affect application performance. My only concern at this moment (which I also mentioned in another email) is the size of the data files. Whether or not that only reflects the persistent storage and not necessarily the memory footprint, I don't know. Therefore, if I loaded my database with 650,000 customer records, the data files will easily exceed 1GB of storage, and that's just one "table". Thanks, Daniel On Nov 12, 2006, at 6:45 PM, Robert L. Read wrote: > Dear Daniel and Team, > > I think the code below, which I have tested on SBCL, > illustrated a typical problem that Daniel Salama introduces. To > paraphrase, you have a datatype (perhaps compound) which has a lot > of slots; you have a GUI, perhaps web-based, that you use to both > select or filter the large database, and to decide how to present > sort the results. I've written the below example as if you > operating directly on the slots. The fact that there are often > intervening functions does not fundamentally change the problem. > (An example of this is storing a timestamp as an integer, but > presenting it in a human-readable format.) > SQL supports a powerful querying ability based on both > selection and sorting. One might think that this is an advantage > of SQL; it is conventional reason that this is actually an > advantage of using a relational database. However, since LISP > treats functions as first-class citizens that can be constructed > dynamically, you actually have a full Turing-complete capabilities > in doing queries that SQL cannot match. This same ability applies > to sorting; you can sort on any lexical order that you can program. > In practice, however, one doesn't always need this power. More > typically, a user will select fields that they want to use to > filter the results (that is, construct a query from), and perhaps > how they would like the results to be sorted. I assume that you > know how to interpret an HTTP query or a McClim user interface or > something to associate the GUI with underlying functions. (My > personal framework has a way to do this, and UCW is probably the > most common or famous way to do it now.) > The code below generated 100 random "users". The bare act of > defining this class defines accessor-functions that we can use in > dynamically constructed lists as below: (list 'username-of 'balance- > of). I have written very small functions that use such lists > either to define define "lexicographic" sort orders based on the > order of the functions within the list. That is, the primary sort > criteria is the first function in the list, but of that function is > equal for two values, the next is used and so on. If you load the > below code and execute (show-off) several times I think you will > see what I mean. You can then see how easily you can change the > list of functions that are either in the selector or the sort > criteria. If this is a web-based app, these list will be generated > from the http-query, which is generated by the user's clicks. > That is a "columnar" based approach; but it one can do > something similar but more powerful based on computed functions > that aren't based on individual columns, but on the entire data > element. For example "find users whose username is equal to their > password" cannot be done in this way --- but can be done by just > using a function #(lambda (x) (equal (username-of x) (password-of > x)). SQL can do this --- but LISP can could use any function > there, such as "find users who have both short usernames and > passwords that can be cracked by routine-x". > Instead of adding things to the root or the store-controller > directly, one would generally prefer > to use consistent classes: > http://common-lisp.net/project/elephant/doc/Persistent- > Classes.html#Persistent-Classes > > This doesn't change the nature of the problem. If you like, you > can create an index on any slot in a very convenient way: http:// > common-lisp.net/project/elephant/doc/Class-Indices.html#Class-Indices > This holds out the possibility of NOT having to iterate over the > entire data-set, but rather honing in directly upon > matching values. (You can in fact create functional indexes based > on any function at all in Elephant, which is something that SQL > can't do conveniently, but the times you will need to do this are > rare.) > > It would take maybe 20 minutes more coding to uses Ian's "get- > instances-by-range" directly, and in a very efficient manner for > performing the query (if the GUI elements correspond to the class's > slots.) This would be very efficient; but of course you should > not do this until you know that this is really the bottleneck in > your system. By using cursors, you can avoid reading the entire > data into memory, and thus process huge datasets. > However, one should note that Ian's code makes creating indices > on slots zero-effort; but indexes always have overhead. The real > question is: when will your queries actually utilize the index? > (That is, if you always select on one column/slot, then that one > should be indexed....but if your query pattern is more complicated, > it becomes fuzzy.) > Let me know if you find this useful; after I get feedback from > you and Ian has made his post, perhaps we will put > this in the documentation. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eslick at csail.mit.edu Mon Nov 13 18:48:32 2006 From: eslick at csail.mit.edu (=?UTF-8?B?SWFuIEVzbGljaw==?=) Date: Mon, 13 Nov 2006 18:48:32 +0000 Subject: [elephant-devel] Querying Advice [w/code example] In-Reply-To: <1461D8DA-03F1-4DE8-B97C-1C4ABFA3BC81@infoway.net> References: <948E9C76-5F70-4245-9697-A59E50ABF9D6@infoway.net><1163364527.5145.244.camel@localhost.localdomain><1163375121.5145.289.camel@localhost.localdomain> <1461D8DA-03F1-4DE8-B97C-1C4ABFA3BC81@infoway.net> Message-ID: <1234847546-1163443721-cardhu_blackberry.rim.net-1352196992-@bxe050-cell01.bisx.prod.on.blackberry> Are running from HEAD or 0.6.0? I'll answer the size question tonight (on travel today). -Ian Sent via BlackBerry from T-Mobile -----Original Message----- From: Daniel Salama Date: Mon, 13 Nov 2006 13:43:03 To:Elephant bugs and development Subject: Re: [elephant-devel] Querying Advice [w/code example] Ok. I got Elephant to work again with SBCL on PPC. I guess I was still using BDB 4.3 when 4.4 seems to be required. I couldn't find that anywhere in the docs. I see what you're saying and can start to envision where this can go. I will keep playing and provide more feedback later today. FYI, for curiosity purposes, I just ran (random-users 5000) on an empty database and it took 5 min 56 secs to complete. At the end, the %ELEPHANT file was 9MB in size and both log files where 10MB. That's kind of big considering what it does. I then made the random-users2 function to look like: (defun random-users2 (n) ? (setq *auto-commit* nil) ? (with-transaction () ? ? (dotimes (x n) ? ? ? (let ((u (make-instance ? ? ? ? ? ? ? ? 'User ? ? ? ? ? ? ? ? :uname (format nil "user~A" x) ? ? ? ? ? ? ? ? :pword (random-password) ? ? ? ? ? ? ? ? :email (format nil "user~A at .nowheresville.org" x) ? ? ? ? ? ? ? ? :fullname (format nil "~A~A ~A~A" (random-password) x (random-password) x) ? ? ? ? ? ? ? ? :balance (random 100)))) ? ? ? ? (add-to-root x u)))) ? (setq *auto-commit* t)) When I ran (random-users2 5000) I got a Berkeley DB Error: Cannot allocate memory, so I changed random-users2 to: (defun random-users2 (n) ? (setq *auto-commit* nil) ? (start-ele-transaction) ? (dotimes (x n) ? ? (if (eq (mod x 1000) ? ? ? ? ? ? 0) ? ? ? ? (progn ? ? ? ? ? (commit-transaction) ? ? ? ? ? (start-ele-transaction))) ? ? (let ((u (make-instance ? ? ? ? ? ? ? 'User ? ? ? ? ? ? ? :uname (format nil "user~A" x) ? ? ? ? ? ? ? :pword (random-password) ? ? ? ? ? ? ? :email (format nil "user~A at .nowheresville.org" x) ? ? ? ? ? ? ? :fullname (format nil "~A~A ~A~A" (random-password) x (random-password) x) ? ? ? ? ? ? ? :balance (random 100)))) ? ? ? (add-to-root x u))) ? (commit-transaction) ? (setq *auto-commit* t)) When I ran (random-users2 5000) on an empty database, this time it only 25 secs to complete. The %ELEPHANT file was still 9MB in size and both log files where also 10MB. I suppose your code was not designed for performance and only for illustration purposes. Needless to say, knowing how to use transactions certainly helps and can dramatically affect application performance. My only concern at this moment (which I also mentioned in another email) is the size of the data files. Whether or not that only reflects the persistent storage and not necessarily the memory footprint, I don't know. Therefore, if I loaded my database with 650,000 customer records, the data files will easily exceed 1GB of storage, and that's just one "table". Thanks, Daniel On Nov 12, 2006, at 6:45 PM, Robert L. Read wrote: Dear Daniel and Team, ??? I think the code below, which I have tested on SBCL, illustrated a typical problem that Daniel Salama introduces.? To paraphrase, you have a datatype (perhaps compound) which has a lot of slots; you have a GUI, perhaps web-based, that you use to both select or filter the large database, and to decide how to present sort the results.? I've written the below example as if you operating directly on the slots.? The fact that there are often intervening functions does not fundamentally change the problem.? (An example of this is storing a timestamp as an integer, but presenting it in a human-readable format.) ??? SQL supports a powerful querying ability based on both selection and sorting.? One might think that this is an advantage of SQL; it is conventional reason that this is actually an advantage of using a relational database.? However, since LISP treats functions as first-class citizens that can be constructed dynamically, you actually have a full Turing-complete capabilities in doing queries that SQL cannot match.? This same ability applies to sorting; you can sort on any lexical order that you can program. ??? In practice, however, one doesn't always need this power.? More typically, a user will select fields that they want to use to filter the results (that is, construct a query from), and perhaps how they would like the results to be sorted.? I assume that you know how to interpret an HTTP query or a McClim user interface or something to associate the GUI with underlying functions.? (My personal framework has a way to do this, and UCW is probably the most common or famous way to do it now.) ??? The code below generated 100 random "users".? The bare act of defining this class defines accessor-functions that we can use in dynamically constructed lists as below: (list 'username-of 'balance-of).? I have written very small functions that use such lists either to define define "lexicographic" sort orders based on the order of the functions within the list.? That is, the primary sort criteria is the first function in the list, but of that function is equal for two values, the next is used and so on.? If you load the below code and execute (show-off) several times I think you will see what I mean.? You can then see how easily you can change the list of functions that are either in the selector or the sort criteria. If this is a web-based app, these list will be generated from the http-query, which is generated by the user's clicks. ??? That is a "columnar" based approach; but it one can do something similar but more powerful based on computed functions that aren't based on individual columns, but on the entire data element.? For example "find users whose username is equal to their password" cannot be done in this way --- but can be done by just using a function #(lambda (x) (equal (username-of x) (password-of x)).? SQL can do this --- but LISP can could use any function there, such as "find users who have both short usernames and passwords that can be cracked by routine-x". ??? Instead of adding things to the root or the store-controller directly, one would generally prefer to use consistent classes: http://common-lisp.net/project/elephant/doc/Persistent-Classes.html#Persistent-Classes: This doesn't change the nature of the problem.? If you like, you can create an index on any slot in a very convenient way: http://common-lisp.net/project/elephant/doc/Class-Indices.html#Class-Indices: This holds out the possibility of NOT having to iterate over the entire data-set, but rather honing in directly upon matching values. (You can in fact create functional indexes based on any function at all in Elephant, which is something that SQL can't do conveniently, but the times you will need to do this are rare.) It would take maybe 20 minutes more coding to uses Ian's "get-instances-by-range" directly, and in a very efficient manner for performing the query (if the GUI elements correspond to the class's slots.)?? This would be very efficient; but of course you should not do this until you know that this is really the bottleneck in your system.? By using cursors, you can avoid reading the entire data into memory, and thus process huge datasets. ??? However, one should note that Ian's code makes creating indices on slots zero-effort; but indexes always have overhead.? The real question is: when will your queries actually utilize the index?? (That is, if you always select on one column/slot, then that one should be indexed....but if your query pattern is more complicated, it becomes fuzzy.) ??? Let me know if you find this useful;? after I get feedback from you and Ian has made his post, perhaps we will put this in the documentation. _______________________________________________ elephant-devel site list elephant-devel at common-lisp.net http://common-lisp.net/mailman/listinfo/elephant-devel From lists at infoway.net Mon Nov 13 18:50:35 2006 From: lists at infoway.net (Daniel Salama) Date: Mon, 13 Nov 2006 13:50:35 -0500 Subject: [elephant-devel] Querying Advice [w/code example] In-Reply-To: <1234847546-1163443721-cardhu_blackberry.rim.net-1352196992-@bxe050-cell01.bisx.prod.on.blackberry> References: <948E9C76-5F70-4245-9697-A59E50ABF9D6@infoway.net><1163364527.5145.244.camel@localhost.localdomain><1163375121.5145.289.camel@localhost.localdomain> <1461D8DA-03F1-4DE8-B97C-1C4ABFA3BC81@infoway.net> <1234847546-1163443721-cardhu_blackberry.rim.net-1352196992-@bxe050-cell01.bisx.prod.on.blackberry> Message-ID: I'm running 0.6.0 downloaded from the web site. On Nov 13, 2006, at 1:48 PM, Ian Eslick wrote: > Are running from HEAD or 0.6.0? I'll answer the size question > tonight (on travel today). -Ian > Sent via BlackBerry from T-Mobile From read at robertlread.net Mon Nov 13 19:17:17 2006 From: read at robertlread.net (Robert L. Read) Date: Mon, 13 Nov 2006 13:17:17 -0600 Subject: [elephant-devel] Querying Advice [w/code example] In-Reply-To: <1461D8DA-03F1-4DE8-B97C-1C4ABFA3BC81@infoway.net> References: <948E9C76-5F70-4245-9697-A59E50ABF9D6@infoway.net> <1163364527.5145.244.camel@localhost.localdomain> <1163375121.5145.289.camel@localhost.localdomain> <1461D8DA-03F1-4DE8-B97C-1C4ABFA3BC81@infoway.net> Message-ID: <1163445437.5145.354.camel@localhost.localdomain> I wrote that code in about an hour and a half; it is a throw-away spike solution. Furthermore I developed it using the SQL backend rather than the BDB. I would never claim it is an ideal solution. Morever, coding style isn't really my strong suit. As Ian mentioned, our serialization could be improved. However, (/ 9M 5000) = 180 bytes/User. 180 bytes of storage total for a data object as below that has to contain 45 or so separate characters is not crazy; there are lot's of SQL systems that would be worse under normal circumstances (at least that was true 8 years ago...) The size of the log files are probably irrelevant. On Mon, 2006-11-13 at 13:43 -0500, Daniel Salama wrote: > Ok. > > > I got Elephant to work again with SBCL on PPC. I guess I was still > using BDB 4.3 when 4.4 seems to be required. I couldn't find that > anywhere in the docs. > > > I see what you're saying and can start to envision where this can go. > I will keep playing and provide more feedback later today. > > > FYI, for curiosity purposes, I just ran (random-users 5000) on an > empty database and it took 5 min 56 secs to complete. At the end, the > %ELEPHANT file was 9MB in size and both log files where 10MB. That's > kind of big considering what it does. > > > I then made the random-users2 function to look like: > > > (defun random-users2 (n) > (setq *auto-commit* nil) > (with-transaction () > (dotimes (x n) > (let ((u (make-instance > 'User > :uname (format nil "user~A" x) > :pword (random-password) > :email (format nil "user~A at .nowheresville.org" x) > :fullname (format nil "~A~A ~A~A" (random-password) x > (random-password) x) > :balance (random 100)))) > (add-to-root x u)))) > (setq *auto-commit* t)) > > > When I ran (random-users2 5000) I got a Berkeley DB Error: Cannot > allocate memory, so I changed random-users2 to: > > > (defun random-users2 (n) > (setq *auto-commit* nil) > (start-ele-transaction) > (dotimes (x n) > (if (eq (mod x 1000) > 0) > (progn > (commit-transaction) > (start-ele-transaction))) > (let ((u (make-instance > 'User > :uname (format nil "user~A" x) > :pword (random-password) > :email (format nil "user~A at .nowheresville.org" x) > :fullname (format nil "~A~A ~A~A" (random-password) x > (random-password) x) > :balance (random 100)))) > (add-to-root x u))) > (commit-transaction) > (setq *auto-commit* t)) > > > When I ran (random-users2 5000) on an empty database, this time it > only 25 secs to complete. The %ELEPHANT file was still 9MB in size and > both log files where also 10MB. > > > I suppose your code was not designed for performance and only for > illustration purposes. Needless to say, knowing how to use > transactions certainly helps and can dramatically affect application > performance. > > > My only concern at this moment (which I also mentioned in another > email) is the size of the data files. Whether or not that only > reflects the persistent storage and not necessarily the memory > footprint, I don't know. Therefore, if I loaded my database with > 650,000 customer records, the data files will easily exceed 1GB of > storage, and that's just one "table". > > > Thanks, > Daniel > > On Nov 12, 2006, at 6:45 PM, Robert L. Read wrote: > > > > Dear Daniel and Team, > > > > I think the code below, which I have tested on SBCL, illustrated > > a typical problem that Daniel Salama introduces. To paraphrase, you > > have a datatype (perhaps compound) which has a lot of slots; you > > have a GUI, perhaps web-based, that you use to both select or filter > > the large database, and to decide how to present sort the results. > > I've written the below example as if you operating directly on the > > slots. The fact that there are often intervening functions does not > > fundamentally change the problem. (An example of this is storing a > > timestamp as an integer, but presenting it in a human-readable > > format.) > > SQL supports a powerful querying ability based on both selection > > and sorting. One might think that this is an advantage of SQL; it > > is conventional reason that this is actually an advantage of using a > > relational database. However, since LISP treats functions as first- > > class citizens that can be constructed dynamically, you actually > > have a full Turing-complete capabilities in doing queries that SQL > > cannot match. This same ability applies to sorting; you can sort on > > any lexical order that you can program. > > In practice, however, one doesn't always need this power. More > > typically, a user will select fields that they want to use to filter > > the results (that is, construct a query from), and perhaps how they > > would like the results to be sorted. I assume that you know how to > > interpret an HTTP query or a McClim user interface or something to > > associate the GUI with underlying functions. (My personal framework > > has a way to do this, and UCW is probably the most common or famous > > way to do it now.) > > The code below generated 100 random "users". The bare act of > > defining this class defines accessor-functions that we can use in > > dynamically constructed lists as below: (list 'username-of 'balance- > > of). I have written very small functions that use such lists either > > to define define "lexicographic" sort orders based on the order of > > the functions within the list. That is, the primary sort criteria > > is the first function in the list, but of that function is equal for > > two values, the next is used and so on. If you load the below code > > and execute (show-off) several times I think you will see what I > > mean. You can then see how easily you can change the list of > > functions that are either in the selector or the sort criteria. If > > this is a web-based app, these list will be generated from the http- > > query, which is generated by the user's clicks. > > That is a "columnar" based approach; but it one can do something > > similar but more powerful based on computed functions that aren't > > based on individual columns, but on the entire data element. For > > example "find users whose username is equal to their password" > > cannot be done in this way --- but can be done by just using a > > function #(lambda (x) (equal (username-of x) (password-of x)). SQL > > can do this --- but LISP can could use any function there, such as > > "find users who have both short usernames and passwords that can be > > cracked by routine-x". > > Instead of adding things to the root or the store-controller > > directly, one would generally prefer > > to use consistent classes: > > http://common-lisp.net/project/elephant/doc/Persistent- > > Classes.html#Persistent-Classes > > > > This doesn't change the nature of the problem. If you like, you can > > create an index on any slot in a very convenient way: http://common- > > lisp.net/project/elephant/doc/Class-Indices.html#Class-Indices > > This holds out the possibility of NOT having to iterate over the > > entire data-set, but rather honing in directly upon > > matching values. (You can in fact create functional indexes based on > > any function at all in Elephant, which is something that SQL can't > > do conveniently, but the times you will need to do this are rare.) > > > > It would take maybe 20 minutes more coding to uses Ian's "get- > > instances-by-range" directly, and in a very efficient manner for > > performing the query (if the GUI elements correspond to the class's > > slots.) This would be very efficient; but of course you should not > > do this until you know that this is really the bottleneck in your > > system. By using cursors, you can avoid reading the entire data > > into memory, and thus process huge datasets. > > However, one should note that Ian's code makes creating indices > > on slots zero-effort; but indexes always have overhead. The real > > question is: when will your queries actually utilize the index? > > (That is, if you always select on one column/slot, then that one > > should be indexed....but if your query pattern is more complicated, > > it becomes fuzzy.) > > Let me know if you find this useful; after I get feedback from > > you and Ian has made his post, perhaps we will put > > this in the documentation. > > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From venkat at rayservers.com Mon Nov 13 23:22:04 2006 From: venkat at rayservers.com (Venkat Manakkal) Date: Mon, 13 Nov 2006 18:22:04 -0500 Subject: [elephant-devel] Querying Advice [w/code example] In-Reply-To: <4AD524C6-40FF-41B5-937A-B12CB96D9A7C@infoway.net> References: <948E9C76-5F70-4245-9697-A59E50ABF9D6@infoway.net> <1163364527.5145.244.camel@localhost.localdomain> <1163375121.5145.289.camel@localhost.localdomain> <45587E52.2000800@rayservers.com> <4AD524C6-40FF-41B5-937A-B12CB96D9A7C@infoway.net> Message-ID: <4558FE1C.3080006@rayservers.com> Daniel Salama wrote: > I'm not familiar with GBBopen and will read up on it. Would anyone care > to comment on it? For the purpose of trying to "develop" a querying > facility for Elephant, will it be useful/needed? Not at all, its another framework for application development I have found very useful when complex decisions have to be made. Best regards, ---Venkat. From lists at infoway.net Tue Nov 14 02:00:28 2006 From: lists at infoway.net (Daniel Salama) Date: Mon, 13 Nov 2006 21:00:28 -0500 Subject: [elephant-devel] Querying Advice In-Reply-To: <1163431348.5145.342.camel@localhost.localdomain> References: <948E9C76-5F70-4245-9697-A59E50ABF9D6@infoway.net> <1163364527.5145.244.camel@localhost.localdomain> <3D430634-748D-44BE-9B44-A8EB5AB5EBAB@infoway.net> <4558003C.2070104@csail.mit.edu> <88B23BCB-2A6C-4943-91E4-2F9CDC48C4C0@infoway.net> <4558670D.5090701@csail.mit.edu> <38E9823B-E4BD-4380-987B-8C33FFE252DE@infoway.net> <1163431348.5145.342.camel@localhost.localdomain> Message-ID: I think I just understood what you've all been saying. Sorry about this. So, if I make my class persistent, it's persistent... period! I don't need to add it to a collection or to the root. I guess that's also what Ian was saying about using class indices as well. Thanks, Daniel On Nov 13, 2006, at 10:22 AM, Robert L. Read wrote: > On Mon, 2006-11-13 at 09:47 -0500, Daniel Salama wrote: >> So, would you say that we based on our approach, we could just store >> a bunch of random objects (whether person, state, zip, order, etc) in >> the root and use class indices? Sounds interesting. >> > > I have to go to the dentist now, but this sounds a bit confused > ---- you would EITHER put > things in the root or use persistent classes; doing both doesn't > make sense to me. > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From read at robertlread.net Tue Nov 14 02:27:33 2006 From: read at robertlread.net (Robert L. Read) Date: Mon, 13 Nov 2006 20:27:33 -0600 Subject: [elephant-devel] Querying Advice In-Reply-To: References: <948E9C76-5F70-4245-9697-A59E50ABF9D6@infoway.net> <1163364527.5145.244.camel@localhost.localdomain> <3D430634-748D-44BE-9B44-A8EB5AB5EBAB@infoway.net> <4558003C.2070104@csail.mit.edu> <88B23BCB-2A6C-4943-91E4-2F9CDC48C4C0@infoway.net> <4558670D.5090701@csail.mit.edu> <38E9823B-E4BD-4380-987B-8C33FFE252DE@infoway.net> <1163431348.5145.342.camel@localhost.localdomain> Message-ID: <1163471254.5145.362.camel@localhost.localdomain> On Mon, 2006-11-13 at 21:00 -0500, Daniel Salama wrote: > I think I just understood what you've all been saying. Sorry about > this. > > > So, if I make my class persistent, it's persistent... period! I don't > need to add it to a collection or to the root. I guess that's also > what Ian was saying about using class indices as well. That's correct. > > > Thanks, > Daniel > > On Nov 13, 2006, at 10:22 AM, Robert L. Read wrote: > > > > On Mon, 2006-11-13 at 09:47 -0500, Daniel Salama wrote: > > > > > So, would you say that we based on our approach, we could just > > > store > > > a bunch of random objects (whether person, state, zip, order, etc) > > > in > > > the root and use class indices? Sounds interesting. > > > > > > > > > I have to go to the dentist now, but this sounds a bit confused ---- > > you would EITHER put > > things in the root or use persistent classes; doing both doesn't > > make sense to me. > > > > > > _______________________________________________ > > elephant-devel site list > > elephant-devel at common-lisp.net > > http://common-lisp.net/mailman/listinfo/elephant-devel > > > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From nowhere.man at levallois.eu.org Tue Nov 14 13:08:03 2006 From: nowhere.man at levallois.eu.org (Pierre THIERRY) Date: Tue, 14 Nov 2006 14:08:03 +0100 Subject: [elephant-devel] Querying Advice In-Reply-To: References: <948E9C76-5F70-4245-9697-A59E50ABF9D6@infoway.net> <1163364527.5145.244.camel@localhost.localdomain> <3D430634-748D-44BE-9B44-A8EB5AB5EBAB@infoway.net> <4558003C.2070104@csail.mit.edu> <88B23BCB-2A6C-4943-91E4-2F9CDC48C4C0@infoway.net> <4558670D.5090701@csail.mit.edu> <38E9823B-E4BD-4380-987B-8C33FFE252DE@infoway.net> <1163431348.5145.342.camel@localhost.localdomain> Message-ID: <20061114130803.GR12538@bateleur.arcanes.fr.eu.org> Scribit Daniel Salama dies 13/11/2006 hora 21:00: > So, if I make my class persistent, it's persistent... period! I don't > need to add it to a collection or to the root. Well, I did not know that before that discussion. I think it's not clearly indicated in the docs... And then I wonder: if I use persistent classes, how do I remove an object from the store? Curiously, Nowhere man -- nowhere.man at levallois.eu.org OpenPGP 0xD9D50D8A -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From eslick at csail.mit.edu Tue Nov 14 14:04:24 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Tue, 14 Nov 2006 09:04:24 -0500 Subject: [elephant-devel] Querying Advice In-Reply-To: <20061114130803.GR12538@bateleur.arcanes.fr.eu.org> References: <948E9C76-5F70-4245-9697-A59E50ABF9D6@infoway.net> <1163364527.5145.244.camel@localhost.localdomain> <3D430634-748D-44BE-9B44-A8EB5AB5EBAB@infoway.net> <4558003C.2070104@csail.mit.edu> <88B23BCB-2A6C-4943-91E4-2F9CDC48C4C0@infoway.net> <4558670D.5090701@csail.mit.edu> <38E9823B-E4BD-4380-987B-8C33FFE252DE@infoway.net> <1163431348.5145.342.camel@localhost.localdomain> <20061114130803.GR12538@bateleur.arcanes.fr.eu.org> Message-ID: <4559CCE8.4050408@csail.mit.edu> Persistent classes in the current model aren't easy to fully delete - you can only make the unreachable by having no reference to them reachable from the controller or class roots. You can drop an object from the index and there is a way to delete the object from the main system BTree - but until 4.4 you couldn't reclaim that space (unless it happened to be reused opportunistically). In 4.4 you can compact the DB, but elephant does not take advantage of this yet. Ideally there would be an automated way to 'migrate' which is a stop-and-copy model of GC - everything reachable from the controller and class roots gets copied, anything that isn't reachable stays in the old DB. This compacts and cleans the DB. I hope that we'll clean up the model of space reclamation and provide something akin to GC in the not too distant future. It's not a high priority in large part because on-disk space has little implication on in-memory space and disk is ~ free. (Although I do suffer from too many backup copies of my 6GB DB - fortunately I just upgraded my computer and doubled my hard drive space so I'm OK for a awhile longer!) Ian Pierre THIERRY wrote: > Scribit Daniel Salama dies 13/11/2006 hora 21:00: > >> So, if I make my class persistent, it's persistent... period! I don't >> need to add it to a collection or to the root. >> > > Well, I did not know that before that discussion. I think it's not > clearly indicated in the docs... > > And then I wonder: if I use persistent classes, how do I remove an > object from the store? > > Curiously, > Nowhere man > > ------------------------------------------------------------------------ > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From lists at infoway.net Tue Nov 14 19:50:19 2006 From: lists at infoway.net (Daniel Salama) Date: Tue, 14 Nov 2006 14:50:19 -0500 Subject: [elephant-devel] Querying Advice In-Reply-To: <4559CCE8.4050408@csail.mit.edu> References: <948E9C76-5F70-4245-9697-A59E50ABF9D6@infoway.net> <1163364527.5145.244.camel@localhost.localdomain> <3D430634-748D-44BE-9B44-A8EB5AB5EBAB@infoway.net> <4558003C.2070104@csail.mit.edu> <88B23BCB-2A6C-4943-91E4-2F9CDC48C4C0@infoway.net> <4558670D.5090701@csail.mit.edu> <38E9823B-E4BD-4380-987B-8C33FFE252DE@infoway.net> <1163431348.5145.342.camel@localhost.localdomain> <20061114130803.GR12538@bateleur.arcanes.fr.eu.org> <4559CCE8.4050408@csail.mit.edu> Message-ID: <0A06D50B-317F-4682-8117-4482A46026B2@infoway.net> Ian et al, Based on my comment to Robert and that of Pierre, could you, or anyone, please clarify this for me (and maybe others): If making a class persistent means that there is no need to add it to root or to any persistent collection, when I look at Robert's sample code, I see (excerpts): (defclass User () ((username :type 'string :initform "" :initarg :uname :accessor username-of) (password :type 'string :initform "" :initarg :pword :accessor password-of) (email :type 'string :initform "" :initarg :email :accessor email- of) (fullname :type 'string :initform "" :initarg :fullname :accessor fullname-of) (balance :type 'integer :initform 0 :initarg :balance :accessor balance-of))) (defun random-users (n) (dotimes (x n) (let ((u (make-instance 'User :uname (format nil "user~A" x) :pword (random-password) :email (format nil "user~A at .nowheresville.org" x) :fullname (format nil "~A~A ~A~A" (random-password) x (random-password) x) :balance (random 100)))) (add-to-root x u)))) There is an explicit add-to-root in random-users. I suppose the reason for this is because User class does not inherit from persistent-metaclass and in order to be able to "search for" or retrieve that object (could this also be the reason for the additional storage overhead, as you pointed out yesterday?). Right? So, if my understanding is correct, defining User with defpclass instead would mean that you don't have to add-to-root because it will be automatically persisted. However, after the function exits, there will be no reference to that persistent object and will therefore be eventually garbage collected (whether or not the persistent space will be reclaimed is a different story, as you mentioned in your email). Is that right? If so, how could I avoid for the User objects to be garbage collected in this case, since there really is no other reference to these objects after creating them? Or, if the objects are NOT garbage collected, how could I manually "delete" any of them? Now, if I (or Robert) had defined the User class as: (defpclass User () ((username :type 'string :initform "" :initarg :uname :accessor username-of :index t) (password :type 'string :initform "" :initarg :pword :accessor password-of) (email :type 'string :initform "" :initarg :email :accessor email- of) (fullname :type 'string :initform "" :initarg :fullname :accessor fullname-of) (balance :type 'integer :initform 0 :initarg :balance :accessor balance-of))) where (notice how it's defined with defpclass) the username slot is indexed, the system would automatically store a reference to the object in the slot index, and there would be no need to use the add- to-root in random-users. Correct? If that's the case, how would I then go about removing this user object from persistence? Would it be by setting the indexed slot value to NIL? On another note, if I want to create a collection of users, I don't have to store these users in a collection. Simply making them inherit from persistent-metaclass and indexing them would do so automatically (just like the example above). Right? How about this, then: (asdf:operate 'asdf:load-op :elephant-tests) (in-package :elephant-tests) (setf *default-spec* *testbdb-spec*) (open-store *default-spec*) (defpclass state () ((abbr :type 'string :initform "" :initarg :abbr :accessor abbr- of :index t) (name :type 'string :initform "" :initarg :name :accessor name-of))) (defpclass zip-code () ((zip :type 'string :initform "" :initarg :zip :accessor zip- of :index t) (city :type 'string :initform "" :initarg :city :accessor city-of) (county :type 'string :initform "" :initarg :county :accessor county-of) (state :initform nil :initarg :state :accessor state-of))) (defmethod print-object ((obj state) stream) (format stream "State (abbr, name) = (~A, ~A)" (abbr-of obj) (name- of obj))) (defmethod print-object ((obj zip-code) stream) (format stream "Zip (zip, city, county, state) = (~A, ~A, ~A, ~A)" (zip-of obj) (city-of obj) (county-of obj) (state-of obj))) (let* ((s1 (make-instance 'state :abbr "FL" :name "Florida")) (s2 (make-instance 'state :abbr "NY" :name "New York")) (z1 (make-instance 'zip-code :zip "33015" :city "Miami" :county "Dade" :state s1)) (z2 (make-instance 'zip-code :zip "13605" :city "Adams" :county "Jefferson" :state s2)) (z2 (make-instance 'zip-code :zip "33160" :city "Sunny Isles Beach" :county "Dade" :state s1))) (print s1) (print s2) (print z1) (print z2) (print z3)) Here, I'm creating a couple of state objects and a couple of zip-code objects. Since a zip-code can only belong to one state, they have a reference back to the state in order to "quickly" determine the state they belong to. A couple of questions/comments here: 1) If I wanted to ask a state to give me all the zip-code(s) within it, would I create a slot in the state class to hold a collection of zip-code references? Or would I simply create a state-class method like: (defmethod get-zip-codes ((obj state)) (get-instances-by-value 'zip-code 'state obj)) This does not work because the state slot in zip-code is not indexed. Also, from the code above, I don't know how to get a reference to the btree in order to create a cursor so that I can linearly traverse the zip-code(s) to return only those zip-code(s) which belong to that state. Of course, I would probably want to index the state slot of zip-code because there are tens of thousands of zip codes and I wouldn't want to linearly traverse them, but I just wanted to illustrate the problem to get some additional feedback (the overhead of maintaining a secondary index on state wouldn't matter too much to me because changes to this class/objects are very rare but access is much more frequent) 2) Maybe this is a totally different issue and mainly caused by my lisp ignorance: If I have: (defparameter *s1* (get-instances-by-value 'state 'abbr "FL")) (defparameter *s2* (get-instances-by-value 'state 'abbr "NY")) (defparameter *z1* (get-instances-by-value 'zip-code 'zip "33015")) (defparameter *z2* (get-instances-by-value 'zip-code 'zip "13605")) (defparameter *z3* (get-instances-by-value 'zip-code 'zip "33160")) and I want to remove the association of zip code 33160 from the "FL" state object, I would think all I have to do is: (setf (state-of *z3*) nil) However, when I do so, I get this error message: There is no applicable method for the generic function # when called with arguments (NIL (Zip (zip, city, county, state) = (33160, Sunny Isles Beach, Dade, State (abbr, name) = (FL, Florida)))). [Condition of type SIMPLE-ERROR] Restarts: 0: [ABORT-REQUEST] Abort handling SLIME request. 1: [ABORT] Exit debugger, returning to top level. Backtrace: 0: ((SB-PCL::FAST-METHOD NO-APPLICABLE-METHOD (T)) # # # (NIL (Zip (zip, city, county, state) = (33160, Sunny Isles Beach, Dade, State (abbr, name) = (FL, Florida))))) 1: ((SB-PCL::FAST-METHOD NO-APPLICABLE-METHOD (T)) # # # (NIL (Zip (zip, city, county, state) = (33160, Sunny Isles Beach, Dade, State (abbr, name) = (FL, Florida))))) 2: (SB-INT:EVAL-IN-LEXENV (SETF (STATE-OF *Z3*) NIL) #) 3: (SWANK::EVAL-REGION "(setf (state-of *z3*) nil)" T) 3) So, going back to the previous questions in the email, if I wanted to permanently remove zip code "33015", how would I go about it? Thanks again and thanks for the patience. - Daniel On Nov 14, 2006, at 9:04 AM, Ian Eslick wrote: > Persistent classes in the current model aren't easy to fully delete - > you can only make the unreachable by having no reference to them > reachable from the controller or class roots. You can drop an object > from the index and there is a way to delete the object from the main > system BTree - but until 4.4 you couldn't reclaim that space > (unless it > happened to be reused opportunistically). In 4.4 you can compact the > DB, but elephant does not take advantage of this yet. > > Ideally there would be an automated way to 'migrate' which is a > stop-and-copy model of GC - everything reachable from the > controller and > class roots gets copied, anything that isn't reachable stays in the > old > DB. This compacts and cleans the DB. I hope that we'll clean up the > model of space reclamation and provide something akin to GC in the not > too distant future. It's not a high priority in large part because > on-disk space has little implication on in-memory space and disk is ~ > free. (Although I do suffer from too many backup copies of my 6GB > DB - > fortunately I just upgraded my computer and doubled my hard drive > space > so I'm OK for a awhile longer!) > > Ian > > Pierre THIERRY wrote: >> Scribit Daniel Salama dies 13/11/2006 hora 21:00: >> >>> So, if I make my class persistent, it's persistent... period! I >>> don't >>> need to add it to a collection or to the root. >>> >> >> Well, I did not know that before that discussion. I think it's not >> clearly indicated in the docs... >> >> And then I wonder: if I use persistent classes, how do I remove an >> object from the store? >> >> Curiously, >> Nowhere man >> >> --------------------------------------------------------------------- >> --- >> >> _______________________________________________ >> elephant-devel site list >> elephant-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/elephant-devel > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From read at robertlread.net Tue Nov 14 20:20:28 2006 From: read at robertlread.net (Robert L. Read) Date: Tue, 14 Nov 2006 14:20:28 -0600 Subject: [elephant-devel] Querying Advice In-Reply-To: <0A06D50B-317F-4682-8117-4482A46026B2@infoway.net> References: <948E9C76-5F70-4245-9697-A59E50ABF9D6@infoway.net> <1163364527.5145.244.camel@localhost.localdomain> <3D430634-748D-44BE-9B44-A8EB5AB5EBAB@infoway.net> <4558003C.2070104@csail.mit.edu> <88B23BCB-2A6C-4943-91E4-2F9CDC48C4C0@infoway.net> <4558670D.5090701@csail.mit.edu> <38E9823B-E4BD-4380-987B-8C33FFE252DE@infoway.net> <1163431348.5145.342.camel@localhost.localdomain> <20061114130803.GR12538@bateleur.arcanes.fr.eu.org> <4559CCE8.4050408@csail.mit.edu> <0A06D50B-317F-4682-8117-4482A46026B2@infoway.net> Message-ID: <1163535629.5145.427.camel@localhost.localdomain> On Tue, 2006-11-14 at 14:50 -0500, Daniel Salama wrote: > Ian et al, > > Based on my comment to Robert and that of Pierre, could you, or > anyone, please clarify this for me (and maybe others): > > If making a class persistent means that there is no need to add it to > root or to any persistent collection, when I look at Robert's sample > code, I see (excerpts): > > (defclass User () > ((username :type 'string :initform "" :initarg :uname :accessor > username-of) > (password :type 'string :initform "" :initarg :pword :accessor > password-of) > (email :type 'string :initform "" :initarg :email :accessor email- > of) > (fullname :type 'string :initform "" :initarg :fullname :accessor > fullname-of) > (balance :type 'integer :initform 0 :initarg :balance :accessor > balance-of))) > > (defun random-users (n) > (dotimes (x n) > (let ((u (make-instance > 'User > :uname (format nil "user~A" x) > :pword (random-password) > :email (format nil "user~A at .nowheresville.org" x) > :fullname (format nil "~A~A ~A~A" (random-password) x > (random-password) x) > :balance (random 100)))) > (add-to-root x u)))) > > There is an explicit add-to-root in random-users. I suppose the > reason for this is because User class does not inherit from > persistent-metaclass and in order to be able to "search for" or > retrieve that object (could this also be the reason for the > additional storage overhead, as you pointed out yesterday?). Right? You must either do something like add-to-root, or inherit from persistent-class. In that, you are correct. Once an object is persisted by either mechanism, it won't be "garbage collected" from the store; you can kill you lisp image and recreated it and then retrieve the object. Garbage collection within memory is a separate issue, and less important---you can always get back to the object. I feel that I just muddied things by using "add-to-root" here --- creating a persistent class is almost always better for a serious application. I think Ian should answer the rest of this email. > So, if my understanding is correct, defining User with defpclass > instead would mean that you don't have to add-to-root because it will > be automatically persisted. However, after the function exits, there > will be no reference to that persistent object and will therefore be > eventually garbage collected (whether or not the persistent space > will be reclaimed is a different story, as you mentioned in your > email). Is that right? If so, how could I avoid for the User objects > to be garbage collected in this case, since there really is no other > reference to these objects after creating them? Or, if the objects > are NOT garbage collected, how could I manually "delete" any of them? > > Now, if I (or Robert) had defined the User class as: > > (defpclass User () > ((username :type 'string :initform "" :initarg :uname :accessor > username-of :index t) > (password :type 'string :initform "" :initarg :pword :accessor > password-of) > (email :type 'string :initform "" :initarg :email :accessor email- > of) > (fullname :type 'string :initform "" :initarg :fullname :accessor > fullname-of) > (balance :type 'integer :initform 0 :initarg :balance :accessor > balance-of))) > > where (notice how it's defined with defpclass) the username slot is > indexed, the system would automatically store a reference to the > object in the slot index, and there would be no need to use the add- > to-root in random-users. Correct? If that's the case, how would I > then go about removing this user object from persistence? Would it be > by setting the indexed slot value to NIL? > > On another note, if I want to create a collection of users, I don't > have to store these users in a collection. Simply making them inherit > from persistent-metaclass and indexing them would do so automatically > (just like the example above). Right? How about this, then: > > (asdf:operate 'asdf:load-op :elephant-tests) > > (in-package :elephant-tests) > > (setf *default-spec* *testbdb-spec*) > > (open-store *default-spec*) > > (defpclass state () > ((abbr :type 'string :initform "" :initarg :abbr :accessor abbr- > of :index t) > (name :type 'string :initform "" :initarg :name :accessor name-of))) > > (defpclass zip-code () > ((zip :type 'string :initform "" :initarg :zip :accessor zip- > of :index t) > (city :type 'string :initform "" :initarg :city :accessor city-of) > (county :type 'string :initform "" :initarg :county :accessor > county-of) > (state :initform nil :initarg :state :accessor state-of))) > > (defmethod print-object ((obj state) stream) > (format stream "State (abbr, name) = (~A, ~A)" (abbr-of obj) (name- > of obj))) > > (defmethod print-object ((obj zip-code) stream) > (format stream "Zip (zip, city, county, state) = (~A, ~A, ~A, ~A)" > (zip-of obj) (city-of obj) (county-of obj) (state-of obj))) > > (let* ((s1 (make-instance 'state :abbr "FL" :name "Florida")) > (s2 (make-instance 'state :abbr "NY" :name "New York")) > (z1 (make-instance 'zip-code :zip "33015" :city > "Miami" :county "Dade" :state s1)) > (z2 (make-instance 'zip-code :zip "13605" :city > "Adams" :county "Jefferson" :state s2)) > (z2 (make-instance 'zip-code :zip "33160" :city "Sunny Isles > Beach" :county "Dade" :state s1))) > (print s1) > (print s2) > (print z1) > (print z2) > (print z3)) > > Here, I'm creating a couple of state objects and a couple of zip-code > objects. Since a zip-code can only belong to one state, they have a > reference back to the state in order to "quickly" determine the state > they belong to. > > A couple of questions/comments here: > > 1) If I wanted to ask a state to give me all the zip-code(s) within > it, would I create a slot in the state class to hold a collection of > zip-code references? Or would I simply create a state-class method like: > > (defmethod get-zip-codes ((obj state)) > (get-instances-by-value 'zip-code 'state obj)) > > This does not work because the state slot in zip-code is not indexed. > Also, from the code above, I don't know how to get a reference to the > btree in order to create a cursor so that I can linearly traverse the > zip-code(s) to return only those zip-code(s) which belong to that state. > > Of course, I would probably want to index the state slot of zip-code > because there are tens of thousands of zip codes and I wouldn't want > to linearly traverse them, but I just wanted to illustrate the > problem to get some additional feedback (the overhead of maintaining > a secondary index on state wouldn't matter too much to me because > changes to this class/objects are very rare but access is much more > frequent) > > 2) Maybe this is a totally different issue and mainly caused by my > lisp ignorance: > > If I have: > > (defparameter *s1* (get-instances-by-value 'state 'abbr "FL")) > (defparameter *s2* (get-instances-by-value 'state 'abbr "NY")) > (defparameter *z1* (get-instances-by-value 'zip-code 'zip "33015")) > (defparameter *z2* (get-instances-by-value 'zip-code 'zip "13605")) > (defparameter *z3* (get-instances-by-value 'zip-code 'zip "33160")) > > and I want to remove the association of zip code 33160 from the "FL" > state object, I would think all I have to do is: > > (setf (state-of *z3*) nil) > > However, when I do so, I get this error message: > > There is no applicable method for the generic function > # > when called with arguments > (NIL > (Zip (zip, city, county, state) = (33160, Sunny Isles Beach, > Dade, State (abbr, name) = (FL, Florida)))). > [Condition of type SIMPLE-ERROR] > > Restarts: > 0: [ABORT-REQUEST] Abort handling SLIME request. > 1: [ABORT] Exit debugger, returning to top level. > > Backtrace: > 0: ((SB-PCL::FAST-METHOD NO-APPLICABLE-METHOD (T)) # argument> # # STATE-OF) (1)> (NIL (Zip (zip, city, county, state) = (33160, Sunny > Isles Beach, Dade, State (abbr, name) = (FL, Florida))))) > 1: ((SB-PCL::FAST-METHOD NO-APPLICABLE-METHOD (T)) # argument> # # STATE-OF) (1)> (NIL (Zip (zip, city, county, state) = (33160, Sunny > Isles Beach, Dade, State (abbr, name) = (FL, Florida))))) > 2: (SB-INT:EVAL-IN-LEXENV (SETF (STATE-OF *Z3*) NIL) #) > 3: (SWANK::EVAL-REGION "(setf (state-of *z3*) nil)" T) > > 3) So, going back to the previous questions in the email, if I wanted > to permanently remove zip code "33015", how would I go about it? > > Thanks again and thanks for the patience. > > - Daniel > > On Nov 14, 2006, at 9:04 AM, Ian Eslick wrote: > > > Persistent classes in the current model aren't easy to fully delete - > > you can only make the unreachable by having no reference to them > > reachable from the controller or class roots. You can drop an object > > from the index and there is a way to delete the object from the main > > system BTree - but until 4.4 you couldn't reclaim that space > > (unless it > > happened to be reused opportunistically). In 4.4 you can compact the > > DB, but elephant does not take advantage of this yet. > > > > Ideally there would be an automated way to 'migrate' which is a > > stop-and-copy model of GC - everything reachable from the > > controller and > > class roots gets copied, anything that isn't reachable stays in the > > old > > DB. This compacts and cleans the DB. I hope that we'll clean up the > > model of space reclamation and provide something akin to GC in the not > > too distant future. It's not a high priority in large part because > > on-disk space has little implication on in-memory space and disk is ~ > > free. (Although I do suffer from too many backup copies of my 6GB > > DB - > > fortunately I just upgraded my computer and doubled my hard drive > > space > > so I'm OK for a awhile longer!) > > > > Ian > > > > Pierre THIERRY wrote: > >> Scribit Daniel Salama dies 13/11/2006 hora 21:00: > >> > >>> So, if I make my class persistent, it's persistent... period! I > >>> don't > >>> need to add it to a collection or to the root. > >>> > >> > >> Well, I did not know that before that discussion. I think it's not > >> clearly indicated in the docs... > >> > >> And then I wonder: if I use persistent classes, how do I remove an > >> object from the store? > >> > >> Curiously, > >> Nowhere man > >> > >> --------------------------------------------------------------------- > >> --- > >> > >> _______________________________________________ > >> elephant-devel site list > >> elephant-devel at common-lisp.net > >> http://common-lisp.net/mailman/listinfo/elephant-devel > > _______________________________________________ > > elephant-devel site list > > elephant-devel at common-lisp.net > > http://common-lisp.net/mailman/listinfo/elephant-devel > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From lists at infoway.net Tue Nov 14 20:34:01 2006 From: lists at infoway.net (Daniel Salama) Date: Tue, 14 Nov 2006 15:34:01 -0500 Subject: [elephant-devel] Querying Advice In-Reply-To: <1163535629.5145.427.camel@localhost.localdomain> References: <948E9C76-5F70-4245-9697-A59E50ABF9D6@infoway.net> <1163364527.5145.244.camel@localhost.localdomain> <3D430634-748D-44BE-9B44-A8EB5AB5EBAB@infoway.net> <4558003C.2070104@csail.mit.edu> <88B23BCB-2A6C-4943-91E4-2F9CDC48C4C0@infoway.net> <4558670D.5090701@csail.mit.edu> <38E9823B-E4BD-4380-987B-8C33FFE252DE@infoway.net> <1163431348.5145.342.camel@localhost.localdomain> <20061114130803.GR12538@bateleur.arcanes.fr.eu.org> <4559CCE8.4050408@csail.mit.edu> <0A06D50B-317F-4682-8117-4482A46026B2@infoway.net> <1163535629.5145.427.camel@localhost.localdomain> Message-ID: <50B1E3E9-F389-4772-AF7C-FC9EEFDF559B@infoway.net> Robert, Thanks for the clarification. I think I got it and agree. BTW, after running your code using persistent objects and adding 5000 users, the total on-disk storage is now 5MB (as compared to 9MB by using non-persistent objects), which concurs with what Ian mentioned yesterday. Anyway, just wanted to point this out for the rest of us. Thanks again, Daniel On Nov 14, 2006, at 3:20 PM, Robert L. Read wrote: > You must either do something like add-to-root, or inherit from > persistent-class. In that, > you are correct. Once an object is persisted by either mechanism, > it won't be "garbage collected" > from the store; you can kill you lisp image and recreated it and > then retrieve the object. > Garbage collection within memory is a separate issue, and less > important---you can always > get back to the object. > > I feel that I just muddied things by using "add-to-root" here --- > creating a persistent > class is almost always better for a serious application. -------------- next part -------------- An HTML attachment was scrubbed... URL: From eslick at csail.mit.edu Wed Nov 15 11:58:50 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Wed, 15 Nov 2006 06:58:50 -0500 Subject: [elephant-devel] Querying Advice In-Reply-To: <0A06D50B-317F-4682-8117-4482A46026B2@infoway.net> References: <948E9C76-5F70-4245-9697-A59E50ABF9D6@infoway.net> <1163364527.5145.244.camel@localhost.localdomain> <3D430634-748D-44BE-9B44-A8EB5AB5EBAB@infoway.net> <4558003C.2070104@csail.mit.edu> <88B23BCB-2A6C-4943-91E4-2F9CDC48C4C0@infoway.net> <4558670D.5090701@csail.mit.edu> <38E9823B-E4BD-4380-987B-8C33FFE252DE@infoway.net> <1163431348.5145.342.camel@localhost.localdomain> <20061114130803.GR12538@bateleur.arcanes.fr.eu.org> <4559CCE8.4050408@csail.mit.edu> <0A06D50B-317F-4682-8117-4482A46026B2@infoway.net> Message-ID: <455B00FA.6090206@csail.mit.edu> Daniel, There are a couple of things about the Elephant model that should be made clear. Elephant is not a very high level DB system. It's focus is persistence of data. In effect, it is a collection of indexing mechanisms and a metaobject protocol that helps us to store/retrieve data. There are three ways to persist data. 1) Put it in the root, or another BTree that you have manually created and put in the rot 2) Create a persistent object. A persistent object has two manifestations: a) A stub containing an OID and a reference to the controller the object is stored in b) A set of slot values stored in a master slot-value BTree When you do a slot access, the OID and slot name are used to go into this (hidden) master slot-value BTree to find the actual slot value. This is done as a tradeoff so if you are writing an integer slot in an object with a large string, you don't have to load that string into memory to get ACID properties on that integer slot. When you load a persistent object from the root or another index, all that is loaded into memory is the stub. All slot accesses go independently back to the on-disk store to get their values. This means that the on-disk value and locking together guarantee that any number of processes or threads within a process can use the same Elephant store and guarantee ACID semantics (BDB & SQL handle locking for us). For most lisp applications with a single image, bookkeeping like Robert's system make more sense, but you have to worry about locking yourself. DCM pre-loads all the slot values but updates the on-disk version on writes for CID properties - atomicity has to be explicitly managed. We're hoping to make slot caching a natural feature of elephant eventually. 3) Use slot and class indices. Underneath, this is just #1 with functions to make the BTrees simple to use. When you created an indexed object, the object is added to a BTree and a secondary index is created for slot values on top of the master slot-value index. It's the class index that guarantees you can reach the object. Now if you are hacking on elephant, this is the mental model you need. Eventually we're hoping that it becomes sophisticated enough you rarely need this level of detail (query language, flexible persistence semantics, etc). However the persistence of lisp values and the persistent vs. normal class objects means it will never be perfect. Manual deletions are actually quite hard to do in such a system, because if you delete an object and forget to delete the references then you end up with a corrupted data representation that will fail silently (return garbage or null slot values) if you reload a reference. Also deleting an object means removing it from all referring data structures and also deleting (separately) each slot value. Thus the only safe method of deletion is GC which is not yet implemented except by manually migrating your DB which copies everything reachable from root & class-root. I'm a little worried for large DBs that GC can't happen when the DB itself is some factor larger than the available working memory. I know how to fix that, but it's a little involved. Some more comments below. PS - All BDB log files start out at 10MB. They store bookkeeping information so take up more room than you might expect from the stored data itself. Daniel Salama wrote: > Ian et al, > > Based on my comment to Robert and that of Pierre, could you, or > anyone, please clarify this for me (and maybe others): > > If making a class persistent means that there is no need to add it to > root or to any persistent collection, when I look at Robert's sample > code, I see (excerpts): > > (defclass User () > ((username :type 'string :initform "" :initarg :uname :accessor > username-of) > (password :type 'string :initform "" :initarg :pword :accessor > password-of) > (email :type 'string :initform "" :initarg :email :accessor email-of) > (fullname :type 'string :initform "" :initarg :fullname :accessor > fullname-of) > (balance :type 'integer :initform 0 :initarg :balance :accessor > balance-of))) > > (defun random-users (n) > (dotimes (x n) > (let ((u (make-instance > 'User > :uname (format nil "user~A" x) > :pword (random-password) > :email (format nil "user~A at .nowheresville.org" x) > :fullname (format nil "~A~A ~A~A" (random-password) x > (random-password) x) > :balance (random 100)))) > (add-to-root x u)))) > > There is an explicit add-to-root in random-users. I suppose the reason > for this is because User class does not inherit from > persistent-metaclass and in order to be able to "search for" or > retrieve that object (could this also be the reason for the additional > storage overhead, as you pointed out yesterday?). Right? So, if my > understanding is correct, defining User with defpclass instead would > mean that you don't have to add-to-root because it will be > automatically persisted. However, after the function exits, there will > be no reference to that persistent object and will therefore be > eventually garbage collected (whether or not the persistent space will > be reclaimed is a different story, as you mentioned in your email). Is > that right? If so, how could I avoid for the User objects to be > garbage collected in this case, since there really is no other > reference to these objects after creating them? Or, if the objects are > NOT garbage collected, how could I manually "delete" any of them? > > Now, if I (or Robert) had defined the User class as: > > (defpclass User () > ((username :type 'string :initform "" :initarg :uname :accessor > username-of :index t) > (password :type 'string :initform "" :initarg :pword :accessor > password-of) > (email :type 'string :initform "" :initarg :email :accessor email-of) > (fullname :type 'string :initform "" :initarg :fullname :accessor > fullname-of) > (balance :type 'integer :initform 0 :initarg :balance :accessor > balance-of))) > > where (notice how it's defined with defpclass) the username slot is > indexed, the system would automatically store a reference to the > object in the slot index, and there would be no need to use the > add-to-root in random-users. Correct? If that's the case, how would I > then go about removing this user object from persistence? Would it be > by setting the indexed slot value to NIL? See above. > On another note, if I want to create a collection of users, I don't > have to store these users in a collection. Simply making them inherit > from persistent-metaclass and indexing them would do so automatically > (just like the example above). Right? How about this, then: > > (asdf:operate 'asdf:load-op :elephant-tests) > > (in-package :elephant-tests) > > (setf *default-spec* *testbdb-spec*) > > (open-store *default-spec*) > > (defpclass state () > ((abbr :type 'string :initform "" :initarg :abbr :accessor abbr-of > :index t) > (name :type 'string :initform "" :initarg :name :accessor name-of))) > > (defpclass zip-code () > ((zip :type 'string :initform "" :initarg :zip :accessor zip-of > :index t) > (city :type 'string :initform "" :initarg :city :accessor city-of) > (county :type 'string :initform "" :initarg :county :accessor > county-of) > (state :initform nil :initarg :state :accessor state-of))) > > (defmethod print-object ((obj state) stream) > (format stream "State (abbr, name) = (~A, ~A)" (abbr-of obj) > (name-of obj))) > > (defmethod print-object ((obj zip-code) stream) > (format stream "Zip (zip, city, county, state) = (~A, ~A, ~A, ~A)" > (zip-of obj) (city-of obj) (county-of obj) (state-of obj))) > > (let* ((s1 (make-instance 'state :abbr "FL" :name "Florida")) > (s2 (make-instance 'state :abbr "NY" :name "New York")) > (z1 (make-instance 'zip-code :zip "33015" :city "Miami" :county > "Dade" :state s1)) > (z2 (make-instance 'zip-code :zip "13605" :city "Adams" :county > "Jefferson" :state s2)) > (z2 (make-instance 'zip-code :zip "33160" :city "Sunny Isles > Beach" :county "Dade" :state s1))) > (print s1) > (print s2) > (print z1) > (print z2) > (print z3)) > > Here, I'm creating a couple of state objects and a couple of zip-code > objects. Since a zip-code can only belong to one state, they have a > reference back to the state in order to "quickly" determine the state > they belong to. > > A couple of questions/comments here: > > 1) If I wanted to ask a state to give me all the zip-code(s) within > it, would I create a slot in the state class to hold a collection of > zip-code references? Or would I simply create a state-class method like: > > (defmethod get-zip-codes ((obj state)) > (get-instances-by-value 'zip-code 'state obj)) > > This does not work because the state slot in zip-code is not indexed. > Also, from the code above, I don't know how to get a reference to the > btree in order to create a cursor so that I can linearly traverse the > zip-code(s) to return only those zip-code(s) which belong to that state. Yes, you have to decide what queries you want to do and make trade-offs between space and time. Of course it's easy to make the decision "lots of zip codes, small ranges in queries, large number of zip codes per state = index OR moderate # zip codes, large ranges per state, small number of states = add a slot and use a list or array" Then you add the :index marker to the zip code or add a new slot to state to keep track of the association. There is no general notion of association as you'll see below. However that might be a nice thing to add - you start to get relational notion into the simple object persistence model. I guess that's the confusion. Elephant is NOT an object database - it is a persistent object and collection facility. The DB functionality you (today) have to write yourself. It might be nice to add a layer or two that makes it more ODB like. But as Robert has done with DCM - you can craft an object/storage model that works for your application. Keeping references to objects in slots is an implicit DB but it does require that you think in a way that RDB's do not. > Of course, I would probably want to index the state slot of zip-code > because there are tens of thousands of zip codes and I wouldn't want > to linearly traverse them, but I just wanted to illustrate the problem > to get some additional feedback (the overhead of maintaining a > secondary index on state wouldn't matter too much to me because > changes to this class/objects are very rare but access is much more > frequent) > > 2) Maybe this is a totally different issue and mainly caused by my > lisp ignorance: > > If I have: > > (defparameter *s1* (get-instances-by-value 'state 'abbr "FL")) > (defparameter *s2* (get-instances-by-value 'state 'abbr "NY")) > (defparameter *z1* (get-instances-by-value 'zip-code 'zip "33015")) > (defparameter *z2* (get-instances-by-value 'zip-code 'zip "13605")) > (defparameter *z3* (get-instances-by-value 'zip-code 'zip "33160")) > > and I want to remove the association of zip code 33160 from the "FL" > state object, I would think all I have to do is: > > (setf (state-of *z3*) nil) First problem here is that get-instances (note the plural) returns a list. However that will work. You orphan the zip code by removing it's reference to a state. However there is a singular function that returns the first instance or nil. (defparameter *z3* (get-instance-by-value 'zip-code 'zip "13605))) (setf (state-of *z3*) nil) So you had the right idea but just missed that detail. > > However, when I do so, I get this error message: > > There is no applicable method for the generic function > # > when called with arguments > (NIL > (Zip (zip, city, county, state) = (33160, Sunny Isles Beach, Dade, > State (abbr, name) = (FL, Florida)))). > [Condition of type SIMPLE-ERROR] > > Restarts: > 0: [ABORT-REQUEST] Abort handling SLIME request. > 1: [ABORT] Exit debugger, returning to top level. > > Backtrace: > 0: ((SB-PCL::FAST-METHOD NO-APPLICABLE-METHOD (T)) # argument> # # STATE-OF) (1)> (NIL (Zip (zip, city, county, state) = (33160, Sunny > Isles Beach, Dade, State (abbr, name) = (FL, Florida))))) > 1: ((SB-PCL::FAST-METHOD NO-APPLICABLE-METHOD (T)) # argument> # # STATE-OF) (1)> (NIL (Zip (zip, city, county, state) = (33160, Sunny > Isles Beach, Dade, State (abbr, name) = (FL, Florida))))) > 2: (SB-INT:EVAL-IN-LEXENV (SETF (STATE-OF *Z3*) NIL) #) > 3: (SWANK::EVAL-REGION "(setf (state-of *z3*) nil)" T) > > 3) So, going back to the previous questions in the email, if I wanted > to permanently remove zip code "33015", how would I go about it? > > Thanks again and thanks for the patience. > > - Daniel > > On Nov 14, 2006, at 9:04 AM, Ian Eslick wrote: > >> Persistent classes in the current model aren't easy to fully delete - >> you can only make the unreachable by having no reference to them >> reachable from the controller or class roots. You can drop an object >> from the index and there is a way to delete the object from the main >> system BTree - but until 4.4 you couldn't reclaim that space (unless it >> happened to be reused opportunistically). In 4.4 you can compact the >> DB, but elephant does not take advantage of this yet. >> >> Ideally there would be an automated way to 'migrate' which is a >> stop-and-copy model of GC - everything reachable from the controller and >> class roots gets copied, anything that isn't reachable stays in the old >> DB. This compacts and cleans the DB. I hope that we'll clean up the >> model of space reclamation and provide something akin to GC in the not >> too distant future. It's not a high priority in large part because >> on-disk space has little implication on in-memory space and disk is ~ >> free. (Although I do suffer from too many backup copies of my 6GB DB - >> fortunately I just upgraded my computer and doubled my hard drive space >> so I'm OK for a awhile longer!) >> >> Ian >> >> Pierre THIERRY wrote: >>> Scribit Daniel Salama dies 13/11/2006 hora 21:00: >>> >>>> So, if I make my class persistent, it's persistent... period! I don't >>>> need to add it to a collection or to the root. >>>> >>> >>> Well, I did not know that before that discussion. I think it's not >>> clearly indicated in the docs... >>> >>> And then I wonder: if I use persistent classes, how do I remove an >>> object from the store? >>> >>> Curiously, >>> Nowhere man >>> >>> ------------------------------------------------------------------------ >>> >>> >>> _______________________________________________ >>> elephant-devel site list >>> elephant-devel at common-lisp.net >>> http://common-lisp.net/mailman/listinfo/elephant-devel >> _______________________________________________ >> elephant-devel site list >> elephant-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/elephant-devel > From eslick at csail.mit.edu Wed Nov 15 17:44:35 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Wed, 15 Nov 2006 12:44:35 -0500 Subject: [elephant-devel] Querying Advice In-Reply-To: <8645F361-ABE9-4811-B550-22474804C31F@infoway.net> References: <948E9C76-5F70-4245-9697-A59E50ABF9D6@infoway.net> <1163364527.5145.244.camel@localhost.localdomain> <3D430634-748D-44BE-9B44-A8EB5AB5EBAB@infoway.net> <4558003C.2070104@csail.mit.edu> <88B23BCB-2A6C-4943-91E4-2F9CDC48C4C0@infoway.net> <4558670D.5090701@csail.mit.edu> <38E9823B-E4BD-4380-987B-8C33FFE252DE@infoway.net> <1163431348.5145.342.camel@localhost.localdomain> <20061114130803.GR12538@bateleur.arcanes.fr.eu.org> <4559CCE8.4050408@csail.mit.edu> <0A06D50B-317F-4682-8117-4482A46026B2@infoway.net> <455B00FA.6090206@csail.mit.edu> <8645F361-ABE9-4811-B550-22474804C31F@infoway.net> Message-ID: <455B5203.5090803@csail.mit.edu> It's not entirely misleading. Berkeley DB is a database in the sense that it provides efficient indexing and retrieval. However it's much lower level than SQL (and is often used to implement relational databases). Elephant is an ODB in the sense that BDB - it stores objects, values and can efficiently reference objects. It does not however provide the rich infrastructure of relational databases, but it is also less restrictive. Right now you have to design your own layer over this basic set of facilities. You have to figure out what slots to create to connect one object to another; what indices to create, how to manage your memory usage and performance, etc. I'm definitely interested in moving Elephant more towards the OODB model. I can perhaps see a couple of steps: 1) Continue to solidify and clarify the basic functionality 2) Support two or three major usage models or architectural concepts a) Simple object storage and retrieval b) DCM and a heavily-structured object usage model c) Query subsystem that auto-creates indices and associations based on what queries are defined d) Major application types: web application, e-commerce, data analysis, bulk store with efficient searching, etc. e) Copy more of the AllegroStore feature set? f) Provide a relational model on top of the basic object persistence model Ian Daniel Salama wrote: > Ian, > > Thank you for the detailed explanation. > > I think, at least, I was under the impression that Elephant was an > ODB. At least, that's how the first sentence of the web site presents > it: "Elephant is an object database for Common Lisp" :) Maybe this > could confuse other newcomers so you may want to revisit the web site. > > Now, under the understanding that it is a framework to "transparently" > persist data, it makes it more apparent that any object model needs to > be done in Lisp and in memory and Elephant will simply aid in persist > the data in Lisp's memory (at least that which you want to persist). > > Going back to all this email thread, makes me wonder, which direction > do you/we all want Elephant to go to. In your email you mention that > it would be nice to incorporation some ODB-like features into > Elephant. Well, I think the approach to query data may be affected as > to the direction we want to take. If we just have a framework to query > data, it will serve one purpose. If we incorporate a "layer or two" to > make it more ODB-like, that could change the panorama of where > Elephant goes, potentially involving changes to DCM and also whatever > querying framework is "developed". After all, it could end up looking > something like CL-SQL with the ability to _also_ use BDB as the backend :) > > As for the indexing and deletion, it makes more sense. I still need to > understand better how class and slot indices work. From your > explanation, it seems as if the Elephant machinery is automatically > creating these BTree indices on the class/slots. I will look into the > code for the get-instance* methods to learn more how it works behind > the scenes (I suppose that's more like the Lispy mentality of learning > by looking at the library's source code - something which I still need > to get used to) > > Will probably bother more later as questions/concerns come up. > However, as I mentioned before, I'd like to contribute to the project, > specially in the querying layer/framework and possibly, if that's > where we go, the ODB layer(s). I just need to know what will be the > direction we want to go. > > Thanks again, > Daniel > > On Nov 15, 2006, at 6:58 AM, Ian Eslick wrote: > >> Daniel, >> >> There are a couple of things about the Elephant model that should be >> made clear. Elephant is not a very high level DB system. It's focus is >> persistence of data. In effect, it is a collection of indexing >> mechanisms and a metaobject protocol that helps us to store/retrieve >> data. There are three ways to persist data. >> >> 1) Put it in the root, or another BTree that you have manually created >> and put in the rot >> 2) Create a persistent object. A persistent object has two >> manifestations: >> a) A stub containing an OID and a reference to the controller the >> object is stored in >> b) A set of slot values stored in a master slot-value BTree >> >> When you do a slot access, the OID and slot name are used to go into >> this (hidden) master slot-value BTree to find the actual slot value. >> This is done as a tradeoff so if you are writing an integer slot in an >> object with a large string, you don't have to load that string into >> memory to get ACID properties on that integer slot. >> >> When you load a persistent object from the root or another index, all >> that is loaded into memory is the stub. All slot accesses go >> independently back to the on-disk store to get their values. This means >> that the on-disk value and locking together guarantee that any number of >> processes or threads within a process can use the same Elephant store >> and guarantee ACID semantics (BDB & SQL handle locking for us). For >> most lisp applications with a single image, bookkeeping like Robert's >> system make more sense, but you have to worry about locking yourself. >> DCM pre-loads all the slot values but updates the on-disk version on >> writes for CID properties - atomicity has to be explicitly managed. >> We're hoping to make slot caching a natural feature of elephant >> eventually. >> >> 3) Use slot and class indices. Underneath, this is just #1 with >> functions to make the BTrees simple to use. When you created an indexed >> object, the object is added to a BTree and a secondary index is created >> for slot values on top of the master slot-value index. It's the class >> index that guarantees you can reach the object. >> >> Now if you are hacking on elephant, this is the mental model you need. >> Eventually we're hoping that it becomes sophisticated enough you rarely >> need this level of detail (query language, flexible persistence >> semantics, etc). However the persistence of lisp values and the >> persistent vs. normal class objects means it will never be perfect. >> >> Manual deletions are actually quite hard to do in such a system, because >> if you delete an object and forget to delete the references then you end >> up with a corrupted data representation that will fail silently (return >> garbage or null slot values) if you reload a reference. Also deleting >> an object means removing it from all referring data structures and also >> deleting (separately) each slot value. Thus the only safe method of >> deletion is GC which is not yet implemented except by manually migrating >> your DB which copies everything reachable from root & class-root. I'm a >> little worried for large DBs that GC can't happen when the DB itself is >> some factor larger than the available working memory. I know how to fix >> that, but it's a little involved. >> >> Some more comments below. >> >> PS - All BDB log files start out at 10MB. They store bookkeeping >> information so take up more room than you might expect from the stored >> data itself. >> >> Daniel Salama wrote: >>> Ian et al, >>> >>> Based on my comment to Robert and that of Pierre, could you, or >>> anyone, please clarify this for me (and maybe others): >>> >>> If making a class persistent means that there is no need to add it to >>> root or to any persistent collection, when I look at Robert's sample >>> code, I see (excerpts): >>> >>> (defclass User () >>> ((username :type 'string :initform "" :initarg :uname :accessor >>> username-of) >>> (password :type 'string :initform "" :initarg :pword :accessor >>> password-of) >>> (email :type 'string :initform "" :initarg :email :accessor email-of) >>> (fullname :type 'string :initform "" :initarg :fullname :accessor >>> fullname-of) >>> (balance :type 'integer :initform 0 :initarg :balance :accessor >>> balance-of))) >>> >>> (defun random-users (n) >>> (dotimes (x n) >>> (let ((u (make-instance >>> 'User >>> :uname (format nil "user~A" x) >>> :pword (random-password) >>> :email (format nil "user~A at .nowheresville.org" x) >>> :fullname (format nil "~A~A ~A~A" (random-password) x >>> (random-password) x) >>> :balance (random 100)))) >>> (add-to-root x u)))) >>> >>> There is an explicit add-to-root in random-users. I suppose the reason >>> for this is because User class does not inherit from >>> persistent-metaclass and in order to be able to "search for" or >>> retrieve that object (could this also be the reason for the additional >>> storage overhead, as you pointed out yesterday?). Right? So, if my >>> understanding is correct, defining User with defpclass instead would >>> mean that you don't have to add-to-root because it will be >>> automatically persisted. However, after the function exits, there will >>> be no reference to that persistent object and will therefore be >>> eventually garbage collected (whether or not the persistent space will >>> be reclaimed is a different story, as you mentioned in your email). Is >>> that right? If so, how could I avoid for the User objects to be >>> garbage collected in this case, since there really is no other >>> reference to these objects after creating them? Or, if the objects are >>> NOT garbage collected, how could I manually "delete" any of them? >>> >>> Now, if I (or Robert) had defined the User class as: >>> >>> (defpclass User () >>> ((username :type 'string :initform "" :initarg :uname :accessor >>> username-of :index t) >>> (password :type 'string :initform "" :initarg :pword :accessor >>> password-of) >>> (email :type 'string :initform "" :initarg :email :accessor email-of) >>> (fullname :type 'string :initform "" :initarg :fullname :accessor >>> fullname-of) >>> (balance :type 'integer :initform 0 :initarg :balance :accessor >>> balance-of))) >>> >>> where (notice how it's defined with defpclass) the username slot is >>> indexed, the system would automatically store a reference to the >>> object in the slot index, and there would be no need to use the >>> add-to-root in random-users. Correct? If that's the case, how would I >>> then go about removing this user object from persistence? Would it be >>> by setting the indexed slot value to NIL? >> See above. >>> On another note, if I want to create a collection of users, I don't >>> have to store these users in a collection. Simply making them inherit >>> from persistent-metaclass and indexing them would do so automatically >>> (just like the example above). Right? How about this, then: >>> >>> (asdf:operate 'asdf:load-op :elephant-tests) >>> >>> (in-package :elephant-tests) >>> >>> (setf *default-spec* *testbdb-spec*) >>> >>> (open-store *default-spec*) >>> >>> (defpclass state () >>> ((abbr :type 'string :initform "" :initarg :abbr :accessor abbr-of >>> :index t) >>> (name :type 'string :initform "" :initarg :name :accessor name-of))) >>> >>> (defpclass zip-code () >>> ((zip :type 'string :initform "" :initarg :zip :accessor zip-of >>> :index t) >>> (city :type 'string :initform "" :initarg :city :accessor city-of) >>> (county :type 'string :initform "" :initarg :county :accessor >>> county-of) >>> (state :initform nil :initarg :state :accessor state-of))) >>> >>> (defmethod print-object ((obj state) stream) >>> (format stream "State (abbr, name) = (~A, ~A)" (abbr-of obj) >>> (name-of obj))) >>> >>> (defmethod print-object ((obj zip-code) stream) >>> (format stream "Zip (zip, city, county, state) = (~A, ~A, ~A, ~A)" >>> (zip-of obj) (city-of obj) (county-of obj) (state-of obj))) >>> >>> (let* ((s1 (make-instance 'state :abbr "FL" :name "Florida")) >>> (s2 (make-instance 'state :abbr "NY" :name "New York")) >>> (z1 (make-instance 'zip-code :zip "33015" :city "Miami" :county >>> "Dade" :state s1)) >>> (z2 (make-instance 'zip-code :zip "13605" :city "Adams" :county >>> "Jefferson" :state s2)) >>> (z2 (make-instance 'zip-code :zip "33160" :city "Sunny Isles >>> Beach" :county "Dade" :state s1))) >>> (print s1) >>> (print s2) >>> (print z1) >>> (print z2) >>> (print z3)) >>> >>> Here, I'm creating a couple of state objects and a couple of zip-code >>> objects. Since a zip-code can only belong to one state, they have a >>> reference back to the state in order to "quickly" determine the state >>> they belong to. >>> >>> A couple of questions/comments here: >>> >>> 1) If I wanted to ask a state to give me all the zip-code(s) within >>> it, would I create a slot in the state class to hold a collection of >>> zip-code references? Or would I simply create a state-class method like: >>> >>> (defmethod get-zip-codes ((obj state)) >>> (get-instances-by-value 'zip-code 'state obj)) >>> >>> This does not work because the state slot in zip-code is not indexed. >>> Also, from the code above, I don't know how to get a reference to the >>> btree in order to create a cursor so that I can linearly traverse the >>> zip-code(s) to return only those zip-code(s) which belong to that state. >> Yes, you have to decide what queries you want to do and make trade-offs >> between space and time. Of course it's easy to make the decision "lots >> of zip codes, small ranges in queries, large number of zip codes per >> state = index OR moderate # zip codes, large ranges per state, small >> number of states = add a slot and use a list or array" Then you add the >> :index marker to the zip code or add a new slot to state to keep track >> of the association. There is no general notion of association as you'll >> see below. However that might be a nice thing to add - you start to get >> relational notion into the simple object persistence model. >> >> I guess that's the confusion. Elephant is NOT an object database - it >> is a persistent object and collection facility. The DB functionality >> you (today) have to write yourself. It might be nice to add a layer or >> two that makes it more ODB like. But as Robert has done with DCM - you >> can craft an object/storage model that works for your application. >> Keeping references to objects in slots is an implicit DB but it does >> require that you think in a way that RDB's do not. >>> Of course, I would probably want to index the state slot of zip-code >>> because there are tens of thousands of zip codes and I wouldn't want >>> to linearly traverse them, but I just wanted to illustrate the problem >>> to get some additional feedback (the overhead of maintaining a >>> secondary index on state wouldn't matter too much to me because >>> changes to this class/objects are very rare but access is much more >>> frequent) >>> >>> 2) Maybe this is a totally different issue and mainly caused by my >>> lisp ignorance: >>> >>> If I have: >>> >>> (defparameter *s1* (get-instances-by-value 'state 'abbr "FL")) >>> (defparameter *s2* (get-instances-by-value 'state 'abbr "NY")) >>> (defparameter *z1* (get-instances-by-value 'zip-code 'zip "33015")) >>> (defparameter *z2* (get-instances-by-value 'zip-code 'zip "13605")) >>> (defparameter *z3* (get-instances-by-value 'zip-code 'zip "33160")) >>> >>> and I want to remove the association of zip code 33160 from the "FL" >>> state object, I would think all I have to do is: >>> >>> (setf (state-of *z3*) nil) >> First problem here is that get-instances (note the plural) returns a >> list. However that will work. You orphan the zip code by removing >> it's reference to a state. However there is a singular function that >> returns the first instance or nil. >> >> (defparameter *z3* (get-instance-by-value 'zip-code 'zip "13605))) >> >> (setf (state-of *z3*) nil) > From lists at infoway.net Wed Nov 15 17:16:10 2006 From: lists at infoway.net (Daniel Salama) Date: Wed, 15 Nov 2006 12:16:10 -0500 Subject: [elephant-devel] Querying Advice In-Reply-To: <455B00FA.6090206@csail.mit.edu> References: <948E9C76-5F70-4245-9697-A59E50ABF9D6@infoway.net> <1163364527.5145.244.camel@localhost.localdomain> <3D430634-748D-44BE-9B44-A8EB5AB5EBAB@infoway.net> <4558003C.2070104@csail.mit.edu> <88B23BCB-2A6C-4943-91E4-2F9CDC48C4C0@infoway.net> <4558670D.5090701@csail.mit.edu> <38E9823B-E4BD-4380-987B-8C33FFE252DE@infoway.net> <1163431348.5145.342.camel@localhost.localdomain> <20061114130803.GR12538@bateleur.arcanes.fr.eu.org> <4559CCE8.4050408@csail.mit.edu> <0A06D50B-317F-4682-8117-4482A46026B2@infoway.net> <455B00FA.6090206@csail.mit.edu> Message-ID: <8645F361-ABE9-4811-B550-22474804C31F@infoway.net> Ian, Thank you for the detailed explanation. I think, at least, I was under the impression that Elephant was an ODB. At least, that's how the first sentence of the web site presents it: "Elephant is an object database for Common Lisp" :) Maybe this could confuse other newcomers so you may want to revisit the web site. Now, under the understanding that it is a framework to "transparently" persist data, it makes it more apparent that any object model needs to be done in Lisp and in memory and Elephant will simply aid in persist the data in Lisp's memory (at least that which you want to persist). Going back to all this email thread, makes me wonder, which direction do you/we all want Elephant to go to. In your email you mention that it would be nice to incorporation some ODB-like features into Elephant. Well, I think the approach to query data may be affected as to the direction we want to take. If we just have a framework to query data, it will serve one purpose. If we incorporate a "layer or two" to make it more ODB-like, that could change the panorama of where Elephant goes, potentially involving changes to DCM and also whatever querying framework is "developed". After all, it could end up looking something like CL-SQL with the ability to _also_ use BDB as the backend :) As for the indexing and deletion, it makes more sense. I still need to understand better how class and slot indices work. From your explanation, it seems as if the Elephant machinery is automatically creating these BTree indices on the class/slots. I will look into the code for the get-instance* methods to learn more how it works behind the scenes (I suppose that's more like the Lispy mentality of learning by looking at the library's source code - something which I still need to get used to) Will probably bother more later as questions/concerns come up. However, as I mentioned before, I'd like to contribute to the project, specially in the querying layer/framework and possibly, if that's where we go, the ODB layer(s). I just need to know what will be the direction we want to go. Thanks again, Daniel On Nov 15, 2006, at 6:58 AM, Ian Eslick wrote: > Daniel, > > There are a couple of things about the Elephant model that should be > made clear. Elephant is not a very high level DB system. It's > focus is > persistence of data. In effect, it is a collection of indexing > mechanisms and a metaobject protocol that helps us to store/retrieve > data. There are three ways to persist data. > > 1) Put it in the root, or another BTree that you have manually created > and put in the rot > 2) Create a persistent object. A persistent object has two > manifestations: > a) A stub containing an OID and a reference to the controller the > object is stored in > b) A set of slot values stored in a master slot-value BTree > > When you do a slot access, the OID and slot name are used to go into > this (hidden) master slot-value BTree to find the actual slot value. > This is done as a tradeoff so if you are writing an integer slot in an > object with a large string, you don't have to load that string into > memory to get ACID properties on that integer slot. > > When you load a persistent object from the root or another index, all > that is loaded into memory is the stub. All slot accesses go > independently back to the on-disk store to get their values. This > means > that the on-disk value and locking together guarantee that any > number of > processes or threads within a process can use the same Elephant store > and guarantee ACID semantics (BDB & SQL handle locking for us). For > most lisp applications with a single image, bookkeeping like Robert's > system make more sense, but you have to worry about locking yourself. > DCM pre-loads all the slot values but updates the on-disk version on > writes for CID properties - atomicity has to be explicitly managed. > We're hoping to make slot caching a natural feature of elephant > eventually. > > 3) Use slot and class indices. Underneath, this is just #1 with > functions to make the BTrees simple to use. When you created an > indexed > object, the object is added to a BTree and a secondary index is > created > for slot values on top of the master slot-value index. It's the class > index that guarantees you can reach the object. > > Now if you are hacking on elephant, this is the mental model you need. > Eventually we're hoping that it becomes sophisticated enough you > rarely > need this level of detail (query language, flexible persistence > semantics, etc). However the persistence of lisp values and the > persistent vs. normal class objects means it will never be perfect. > > Manual deletions are actually quite hard to do in such a system, > because > if you delete an object and forget to delete the references then > you end > up with a corrupted data representation that will fail silently > (return > garbage or null slot values) if you reload a reference. Also deleting > an object means removing it from all referring data structures and > also > deleting (separately) each slot value. Thus the only safe method of > deletion is GC which is not yet implemented except by manually > migrating > your DB which copies everything reachable from root & class-root. > I'm a > little worried for large DBs that GC can't happen when the DB > itself is > some factor larger than the available working memory. I know how > to fix > that, but it's a little involved. > > Some more comments below. > > PS - All BDB log files start out at 10MB. They store bookkeeping > information so take up more room than you might expect from the stored > data itself. > > Daniel Salama wrote: >> Ian et al, >> >> Based on my comment to Robert and that of Pierre, could you, or >> anyone, please clarify this for me (and maybe others): >> >> If making a class persistent means that there is no need to add it to >> root or to any persistent collection, when I look at Robert's sample >> code, I see (excerpts): >> >> (defclass User () >> ((username :type 'string :initform "" :initarg :uname :accessor >> username-of) >> (password :type 'string :initform "" :initarg :pword :accessor >> password-of) >> (email :type 'string :initform "" :initarg :email :accessor >> email-of) >> (fullname :type 'string :initform "" :initarg :fullname :accessor >> fullname-of) >> (balance :type 'integer :initform 0 :initarg :balance :accessor >> balance-of))) >> >> (defun random-users (n) >> (dotimes (x n) >> (let ((u (make-instance >> 'User >> :uname (format nil "user~A" x) >> :pword (random-password) >> :email (format nil "user~A at .nowheresville.org" x) >> :fullname (format nil "~A~A ~A~A" (random-password) x >> (random-password) x) >> :balance (random 100)))) >> (add-to-root x u)))) >> >> There is an explicit add-to-root in random-users. I suppose the >> reason >> for this is because User class does not inherit from >> persistent-metaclass and in order to be able to "search for" or >> retrieve that object (could this also be the reason for the >> additional >> storage overhead, as you pointed out yesterday?). Right? So, if my >> understanding is correct, defining User with defpclass instead would >> mean that you don't have to add-to-root because it will be >> automatically persisted. However, after the function exits, there >> will >> be no reference to that persistent object and will therefore be >> eventually garbage collected (whether or not the persistent space >> will >> be reclaimed is a different story, as you mentioned in your >> email). Is >> that right? If so, how could I avoid for the User objects to be >> garbage collected in this case, since there really is no other >> reference to these objects after creating them? Or, if the objects >> are >> NOT garbage collected, how could I manually "delete" any of them? >> >> Now, if I (or Robert) had defined the User class as: >> >> (defpclass User () >> ((username :type 'string :initform "" :initarg :uname :accessor >> username-of :index t) >> (password :type 'string :initform "" :initarg :pword :accessor >> password-of) >> (email :type 'string :initform "" :initarg :email :accessor >> email-of) >> (fullname :type 'string :initform "" :initarg :fullname :accessor >> fullname-of) >> (balance :type 'integer :initform 0 :initarg :balance :accessor >> balance-of))) >> >> where (notice how it's defined with defpclass) the username slot is >> indexed, the system would automatically store a reference to the >> object in the slot index, and there would be no need to use the >> add-to-root in random-users. Correct? If that's the case, how would I >> then go about removing this user object from persistence? Would it be >> by setting the indexed slot value to NIL? > See above. >> On another note, if I want to create a collection of users, I don't >> have to store these users in a collection. Simply making them inherit >> from persistent-metaclass and indexing them would do so automatically >> (just like the example above). Right? How about this, then: >> >> (asdf:operate 'asdf:load-op :elephant-tests) >> >> (in-package :elephant-tests) >> >> (setf *default-spec* *testbdb-spec*) >> >> (open-store *default-spec*) >> >> (defpclass state () >> ((abbr :type 'string :initform "" :initarg :abbr :accessor abbr-of >> :index t) >> (name :type 'string :initform "" :initarg :name :accessor name- >> of))) >> >> (defpclass zip-code () >> ((zip :type 'string :initform "" :initarg :zip :accessor zip-of >> :index t) >> (city :type 'string :initform "" :initarg :city :accessor city-of) >> (county :type 'string :initform "" :initarg :county :accessor >> county-of) >> (state :initform nil :initarg :state :accessor state-of))) >> >> (defmethod print-object ((obj state) stream) >> (format stream "State (abbr, name) = (~A, ~A)" (abbr-of obj) >> (name-of obj))) >> >> (defmethod print-object ((obj zip-code) stream) >> (format stream "Zip (zip, city, county, state) = (~A, ~A, ~A, ~A)" >> (zip-of obj) (city-of obj) (county-of obj) (state-of obj))) >> >> (let* ((s1 (make-instance 'state :abbr "FL" :name "Florida")) >> (s2 (make-instance 'state :abbr "NY" :name "New York")) >> (z1 (make-instance 'zip-code :zip "33015" :city "Miami" :county >> "Dade" :state s1)) >> (z2 (make-instance 'zip-code :zip "13605" :city "Adams" :county >> "Jefferson" :state s2)) >> (z2 (make-instance 'zip-code :zip "33160" :city "Sunny Isles >> Beach" :county "Dade" :state s1))) >> (print s1) >> (print s2) >> (print z1) >> (print z2) >> (print z3)) >> >> Here, I'm creating a couple of state objects and a couple of zip-code >> objects. Since a zip-code can only belong to one state, they have a >> reference back to the state in order to "quickly" determine the state >> they belong to. >> >> A couple of questions/comments here: >> >> 1) If I wanted to ask a state to give me all the zip-code(s) within >> it, would I create a slot in the state class to hold a collection of >> zip-code references? Or would I simply create a state-class method >> like: >> >> (defmethod get-zip-codes ((obj state)) >> (get-instances-by-value 'zip-code 'state obj)) >> >> This does not work because the state slot in zip-code is not indexed. >> Also, from the code above, I don't know how to get a reference to the >> btree in order to create a cursor so that I can linearly traverse the >> zip-code(s) to return only those zip-code(s) which belong to that >> state. > Yes, you have to decide what queries you want to do and make trade- > offs > between space and time. Of course it's easy to make the decision > "lots > of zip codes, small ranges in queries, large number of zip codes per > state = index OR moderate # zip codes, large ranges per state, small > number of states = add a slot and use a list or array" Then you > add the > :index marker to the zip code or add a new slot to state to keep track > of the association. There is no general notion of association as > you'll > see below. However that might be a nice thing to add - you start > to get > relational notion into the simple object persistence model. > > I guess that's the confusion. Elephant is NOT an object database - it > is a persistent object and collection facility. The DB functionality > you (today) have to write yourself. It might be nice to add a > layer or > two that makes it more ODB like. But as Robert has done with DCM - > you > can craft an object/storage model that works for your application. > Keeping references to objects in slots is an implicit DB but it does > require that you think in a way that RDB's do not. >> Of course, I would probably want to index the state slot of zip-code >> because there are tens of thousands of zip codes and I wouldn't want >> to linearly traverse them, but I just wanted to illustrate the >> problem >> to get some additional feedback (the overhead of maintaining a >> secondary index on state wouldn't matter too much to me because >> changes to this class/objects are very rare but access is much more >> frequent) >> >> 2) Maybe this is a totally different issue and mainly caused by my >> lisp ignorance: >> >> If I have: >> >> (defparameter *s1* (get-instances-by-value 'state 'abbr "FL")) >> (defparameter *s2* (get-instances-by-value 'state 'abbr "NY")) >> (defparameter *z1* (get-instances-by-value 'zip-code 'zip "33015")) >> (defparameter *z2* (get-instances-by-value 'zip-code 'zip "13605")) >> (defparameter *z3* (get-instances-by-value 'zip-code 'zip "33160")) >> >> and I want to remove the association of zip code 33160 from the "FL" >> state object, I would think all I have to do is: >> >> (setf (state-of *z3*) nil) > First problem here is that get-instances (note the plural) returns a > list. However that will work. You orphan the zip code by removing > it's reference to a state. However there is a singular function that > returns the first instance or nil. > > (defparameter *z3* (get-instance-by-value 'zip-code 'zip "13605))) > > (setf (state-of *z3*) nil) -------------- next part -------------- An HTML attachment was scrubbed... URL: From nowhere.man at levallois.eu.org Wed Nov 15 22:52:39 2006 From: nowhere.man at levallois.eu.org (Pierre THIERRY) Date: Wed, 15 Nov 2006 23:52:39 +0100 Subject: [elephant-devel] Concurrency problem? In-Reply-To: <20061110020041.GE12538@bateleur.arcanes.fr.eu.org> References: <20061031134344.GU7585@bateleur.arcanes.fr.eu.org> <1162306483.14560.285.camel@localhost.localdomain> <454CAF92.9010406@csail.mit.edu> <20061105203646.GB7160@bateleur.arcanes.fr.eu.org> <45539DF0.5090304@csail.mit.edu> <20061109231929.GD12538@bateleur.arcanes.fr.eu.org> <4553D4E6.4000609@csail.mit.edu> <20061110020041.GE12538@bateleur.arcanes.fr.eu.org> Message-ID: <20061115225239.GB30093@bateleur.arcanes.fr.eu.org> Scribit Pierre THIERRY dies 10/11/2006 hora 03:00: > > Could be the new SBCL not playing nice with some assumption left in > > 0.6.0. > I'll install a sarge chroot and test this within it. Well, in a sarge chroot, I could run the test suite for the BDB backend with SBCL. I even figured how to do the migration tests, by setfing *test-spec-{primary,secondary}*. I got the following: [...] Fetching MIGRATE-IPCLASS PREPARES-SLEEPYCATdatabase db not valid, so not runnning test test-seq1 TEST-SEQ1 TEST-SEQ2 CLEANSUP-SLEEPYCAT 1 out of 115 total tests failed: INDEXING-INHERIT. So you may be right. Do you see where the problem could be? I couldn't compile elephant HEAD with SBCL from sarge, but I didn't take time to see why. Quickly, Nowhere man -- nowhere.man at levallois.eu.org OpenPGP 0xD9D50D8A -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From eslick at csail.mit.edu Thu Nov 16 05:03:21 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Thu, 16 Nov 2006 00:03:21 -0500 Subject: [elephant-devel] Concurrency problem? In-Reply-To: <20061115225239.GB30093@bateleur.arcanes.fr.eu.org> References: <20061031134344.GU7585@bateleur.arcanes.fr.eu.org> <1162306483.14560.285.camel@localhost.localdomain> <454CAF92.9010406@csail.mit.edu> <20061105203646.GB7160@bateleur.arcanes.fr.eu.org> <45539DF0.5090304@csail.mit.edu> <20061109231929.GD12538@bateleur.arcanes.fr.eu.org> <4553D4E6.4000609@csail.mit.edu> <20061110020041.GE12538@bateleur.arcanes.fr.eu.org> <20061115225239.GB30093@bateleur.arcanes.fr.eu.org> Message-ID: <455BF119.7000108@csail.mit.edu> Ok, I'll be hacking on this tomorrow and just got some VM's running including the latest Ubuntu release so can test SBCL under it. I'm focusing on HEAD primarily with a goal of moving towards an 0.6.1 release in the near future. Make sure you clean the DB's using the script (delscript.sh or something like that) prior to running tests. They are not designed to be idempotent. Ian Pierre THIERRY wrote: > Scribit Pierre THIERRY dies 10/11/2006 hora 03:00: > >>> Could be the new SBCL not playing nice with some assumption left in >>> 0.6.0. >>> >> I'll install a sarge chroot and test this within it. >> > > Well, in a sarge chroot, I could run the test suite for the BDB backend > with SBCL. I even figured how to do the migration tests, by setfing > *test-spec-{primary,secondary}*. I got the following: > > [...] > Fetching > MIGRATE-IPCLASS PREPARES-SLEEPYCATdatabase db not valid, so not runnning test test-seq1 > TEST-SEQ1 TEST-SEQ2 CLEANSUP-SLEEPYCAT > 1 out of 115 total tests failed: INDEXING-INHERIT. > > So you may be right. Do you see where the problem could be? > > I couldn't compile elephant HEAD with SBCL from sarge, but I didn't take > time to see why. > > Quickly, > Nowhere man > > ------------------------------------------------------------------------ > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From nowhere.man at levallois.eu.org Thu Nov 16 06:59:55 2006 From: nowhere.man at levallois.eu.org (Pierre THIERRY) Date: Thu, 16 Nov 2006 07:59:55 +0100 Subject: [elephant-devel] Development tools Message-ID: <20061116065955.GF30093@bateleur.arcanes.fr.eu.org> If I ain't mistaken, currently Elephant only has a CVS, and no bug tracker of any sort. Before or after 0.6.1, could you consider switching to subversion and trac, as they are provided by common-list.net? It could help possible contributors to see what remains to be done (comments and attached patches to issues, assignments, etc.). Curiously, Nowhere man -- nowhere.man at levallois.eu.org OpenPGP 0xD9D50D8A -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From read at robertlread.net Thu Nov 16 14:26:32 2006 From: read at robertlread.net (Robert L. Read) Date: Thu, 16 Nov 2006 08:26:32 -0600 Subject: [elephant-devel] Development tools In-Reply-To: <20061116065955.GF30093@bateleur.arcanes.fr.eu.org> References: <20061116065955.GF30093@bateleur.arcanes.fr.eu.org> Message-ID: <1163687192.9130.26.camel@localhost.localdomain> Yes. I think we should probably do this. I am not experienced with subversion, and know nothing about Trac, but using a mailing list for everything is a concept we need to outgrow. On Thu, 2006-11-16 at 07:59 +0100, Pierre THIERRY wrote: > If I ain't mistaken, currently Elephant only has a CVS, and no bug > tracker of any sort. > > Before or after 0.6.1, could you consider switching to subversion and > trac, as they are provided by common-list.net? It could help possible > contributors to see what remains to be done (comments and attached > patches to issues, assignments, etc.). > > Curiously, > Nowhere man > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From eslick at csail.mit.edu Thu Nov 16 14:34:08 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Thu, 16 Nov 2006 09:34:08 -0500 Subject: [elephant-devel] Development tools In-Reply-To: <1163687192.9130.26.camel@localhost.localdomain> References: <20061116065955.GF30093@bateleur.arcanes.fr.eu.org> <1163687192.9130.26.camel@localhost.localdomain> Message-ID: <455C76E0.8010503@csail.mit.edu> I've used SVN a fair bit although I find managing branches in SVN to be quite annoying. I prefer darcs, actually, but am happy to use Subversion - any other elephant users want to chime in on this? Here's a set of Trac issues for UCW for example. http://trac.common-lisp.net/ucw/report/1 There's also a WIKI to keep the TODO file and DESIGN file contents live instead of hiding in the source tree. I'm all for switching to Trac sooner rather than later. Google has also put together a nice environment for hosting, but I think common-lisp.net is where folks go looking for lisp code. 0.6.1 is probably a few weeks away (although with x86 mac & parallels I can test all the lisp+os variations locally now; that should speed up the debugging cycle as I can reproduce anything except 64-bit problems locally - I think Marco and I resolved that) Unicode: By the way, I'm just cleaning up the last of my unicode updates. I kept having problems with the efficiency hacks in the current support for Unicode -- there was no canonical representation of strings in the database; each lisp+machine coded it differently. Also, even though most strings have codes in the ASCII or Latin-1 character set, SBCL was still storing 32-bit characters. It now uses the smallest coding size (8,16 or 32) necessary to represent the string. Support for 8 or 16 is fairly efficient but if you use unicode code pages > 0 there will be a performance and storage hit. I put in a convention in where all 16/32 bit unicode strings are stored little-endian (x86 is a little-endian machine) so I can use native string reader functions to pull shorts and ints out of the byte vectors when possible. This should greatly compact string storage on most unicode supporting systems (2x on allegro, 4x on SBCL). More string efficiency hacks coming your way soon... Cheers, Ian Robert L. Read wrote: > Yes. > > I think we should probably do this. I am not experienced with > subversion, and know nothing > about Trac, but using a mailing list for everything is a concept we > need to outgrow. > > > On Thu, 2006-11-16 at 07:59 +0100, Pierre THIERRY wrote: >> If I ain't mistaken, currently Elephant only has a CVS, and no bug >> tracker of any sort. >> >> Before or after 0.6.1, could you consider switching to subversion and >> trac, as they are provided by common-list.net? It could help possible >> contributors to see what remains to be done (comments and attached >> patches to issues, assignments, etc.). >> >> Curiously, >> Nowhere man >> _______________________________________________ >> elephant-devel site list >> elephant-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/elephant-devel >> > ------------------------------------------------------------------------ > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From read at robertlread.net Thu Nov 16 14:51:00 2006 From: read at robertlread.net (Robert L. Read) Date: Thu, 16 Nov 2006 08:51:00 -0600 Subject: [elephant-devel] Development tools In-Reply-To: <455C76E0.8010503@csail.mit.edu> References: <20061116065955.GF30093@bateleur.arcanes.fr.eu.org> <1163687192.9130.26.camel@localhost.localdomain> <455C76E0.8010503@csail.mit.edu> Message-ID: <1163688660.9130.28.camel@localhost.localdomain> I prefer darcs, as well, actually. On Thu, 2006-11-16 at 09:34 -0500, Ian Eslick wrote: > I've used SVN a fair bit although I find managing branches in SVN to be > quite annoying. I prefer darcs, actually, but am happy to use > Subversion - any other elephant users want to chime in on this? > > Here's a set of Trac issues for UCW for example. > http://trac.common-lisp.net/ucw/report/1 > > There's also a WIKI to keep the TODO file and DESIGN file contents live > instead of hiding in the source tree. I'm all for switching to Trac > sooner rather than later. > > Google has also put together a nice environment for hosting, but I think > common-lisp.net is where folks go looking for lisp code. 0.6.1 is > probably a few weeks away (although with x86 mac & parallels I can test > all the lisp+os variations locally now; that should speed up the > debugging cycle as I can reproduce anything except 64-bit problems > locally - I think Marco and I resolved that) > > Unicode: > > By the way, I'm just cleaning up the last of my unicode updates. I kept > having problems with the efficiency hacks in the current support for > Unicode -- there was no canonical representation of strings in the > database; each lisp+machine coded it differently. Also, even though > most strings have codes in the ASCII or Latin-1 character set, SBCL was > still storing 32-bit characters. It now uses the smallest coding size > (8,16 or 32) necessary to represent the string. Support for 8 or 16 is > fairly efficient but if you use unicode code pages > 0 there will be a > performance and storage hit. I put in a convention in where all 16/32 > bit unicode strings are stored little-endian (x86 is a little-endian > machine) so I can use native string reader functions to pull shorts and > ints out of the byte vectors when possible. This should greatly compact > string storage on most unicode supporting systems (2x on allegro, 4x on > SBCL). > > More string efficiency hacks coming your way soon... > > Cheers, > Ian > > Robert L. Read wrote: > > Yes. > > > > I think we should probably do this. I am not experienced with > > subversion, and know nothing > > about Trac, but using a mailing list for everything is a concept we > > need to outgrow. > > > > > > On Thu, 2006-11-16 at 07:59 +0100, Pierre THIERRY wrote: > >> If I ain't mistaken, currently Elephant only has a CVS, and no bug > >> tracker of any sort. > >> > >> Before or after 0.6.1, could you consider switching to subversion and > >> trac, as they are provided by common-list.net? It could help possible > >> contributors to see what remains to be done (comments and attached > >> patches to issues, assignments, etc.). > >> > >> Curiously, > >> Nowhere man > >> _______________________________________________ > >> elephant-devel site list > >> elephant-devel at common-lisp.net > >> http://common-lisp.net/mailman/listinfo/elephant-devel > >> > > ------------------------------------------------------------------------ > > > > _______________________________________________ > > elephant-devel site list > > elephant-devel at common-lisp.net > > http://common-lisp.net/mailman/listinfo/elephant-devel > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alain.Picard at memetrics.com Fri Nov 17 10:18:38 2006 From: Alain.Picard at memetrics.com (Alain Picard) Date: Fri, 17 Nov 2006 21:18:38 +1100 Subject: [elephant-devel] get-instances-by-range should work with string types In-Reply-To: <20061116065955.GF30093@bateleur.arcanes.fr.eu.org> (Pierre THIERRY's message of "Thu, 16 Nov 2006 07:59:55 +0100") References: <20061116065955.GF30093@bateleur.arcanes.fr.eu.org> Message-ID: <8764dewf41.fsf@memetrics.com> Hello to all Elephant developers, and many words of thanks for what appears to be a useful and well thought out library! I've recently started a project which will (I hope) make some good use of Elephant. I'm still wrapping my head around things, so go easy on me. As a show of good will, here's my first attempt at adding something to elephant. Suppose you have a class like this: (defclass test-user () ((name :reader name :initarg :name :type string :index t) (timestamp :reader timestamp :initform (get-universal-time) :type integer :index t)) (:metaclass persistent-metaclass)) (defun make-some-users () (let ((n 0) (*auto-commit* t)) (dotimes (i 1000) (make-instance 'test-user :name (format nil "User name ~D" (incf n)))) (get-instances-by-range 'test-user 'name "User name 10" "User name 20"))) (make-some-users) If you try to do something like (length (get-instances-by-range 'test-user 'timestamp 0 999999999999)) ==> 1000 It succeeds. But if you try on the name, (get-instances-by-range 'test-user 'name "User name 10" "User name 11") ==> Argument X is not a REAL: "User name 10" [Condition of type SIMPLE-TYPE-ERROR] Because the get-instances-by-range function (needlessly) assumes that the objects being returned can be compared with numeric equality. This little snippet fixes this problem: (change from the 0.6 distribution) =============cut================================================ (defun find-slot-type (class idx-name) (flet ((candidate-slot-p (slot) (and (eq (type-of slot) 'persistent-effective-slot-definition) (slot-value slot 'indexed) (eq (slot-definition-name slot) idx-name)))) (find-if #'candidate-slot-p (class-slots class)))) (defun find-index-comparison-function (class index) (let ((type (sb-pcl:slot-definition-type (find-indexed-slot class index)))) (cond ((subtypep type 'number) #'<=) ((subtypep type 'string) #'string<=) (t ;; We'll fall back to numerical, though it's not clear to ;; me that this is sensible. Maybe should just signal an error, ;; and force users to declare :type on all indexed slots? #'<=)))) (defmethod get-instances-by-range ((class persistent-metaclass) idx-name start end) (let ((comparison (find-index-comparison-function class idx-name))) (with-inverted-cursor (cur class idx-name) (labels ((next-range (instances) (multiple-value-bind (exists? skey val pkey) (cursor-pnext-nodup cur) (declare (ignore pkey)) (if (and exists? (funcall comparison skey end)) (next-in-range skey (cons val instances)) (nreverse instances)))) (next-in-range (key instances) (multiple-value-bind (exists? skey val pkey) (cursor-pnext-dup cur) (declare (ignore pkey skey)) (if exists? (next-in-range key (cons val instances)) (progn (cursor-pset-range cur key) (next-range instances)))))) (multiple-value-bind (exists? skey val pkey) (cursor-pset-range cur start) (declare (ignore pkey)) (if (and exists? (funcall comparison skey end)) (next-in-range skey (cons val nil)) nil)))))) =============cut================================================ Note that unfortunately FIND-INDEX-COMPARISON-FUNCTION makes use of SB-PCL:SLOT-DEFINITION-TYPE; I have not conditionalized this for other lisps, nor do I really know what forms are appropriate in any other lisps. Obviously, it is possible to extend this scheme to let users register their own comparison functions for more complex types; I'll let you judge if this is worth the effort. Hoping this helps someone, somewhere. If this was already covered in some other way, then consider this message a bug report on the documentation instead. :-) Alain Picard -- Please read about why Top Posting is evil at: http://en.wikipedia.org/wiki/Top-posting and http://www.dickalba.demon.co.uk/usenet/guide/faq_topp.html Please read about why HTML in email is evil at: http://www.birdhouse.org/etc/evilmail.html From nowhere.man at levallois.eu.org Fri Nov 17 10:23:26 2006 From: nowhere.man at levallois.eu.org (Pierre THIERRY) Date: Fri, 17 Nov 2006 11:23:26 +0100 Subject: [elephant-devel] get-instances-by-range should work with string types In-Reply-To: <8764dewf41.fsf@memetrics.com> References: <20061116065955.GF30093@bateleur.arcanes.fr.eu.org> <8764dewf41.fsf@memetrics.com> Message-ID: <20061117102326.GK30093@bateleur.arcanes.fr.eu.org> Scribit Alain Picard dies 17/11/2006 hora 21:18: > This little snippet fixes this problem: > (change from the 0.6 distribution) Could you just provide the patch, that would be more easy both to read and apply? (for me, at least) Quickly, Pierre -- nowhere.man at levallois.eu.org OpenPGP 0xD9D50D8A -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From Alain.Picard at memetrics.com Fri Nov 17 11:58:37 2006 From: Alain.Picard at memetrics.com (Alain Picard) Date: Fri, 17 Nov 2006 22:58:37 +1100 Subject: [elephant-devel] get-instances-by-range should work with string types In-Reply-To: <20061117102326.GK30093@bateleur.arcanes.fr.eu.org> (Pierre THIERRY's message of "Fri, 17 Nov 2006 11:23:26 +0100") References: <20061116065955.GF30093@bateleur.arcanes.fr.eu.org> <8764dewf41.fsf@memetrics.com> <20061117102326.GK30093@bateleur.arcanes.fr.eu.org> Message-ID: <871wo2wahe.fsf@memetrics.com> Pierre THIERRY writes: > Scribit Alain Picard dies 17/11/2006 hora 21:18: >> This little snippet fixes this problem: >> (change from the 0.6 distribution) > > Could you just provide the patch, that would be more easy both to read > and apply? (for me, at least) I can do that. Patch included. -------------- next part -------------- A non-text attachment was scrubbed... Name: patch Type: application/octet-stream Size: 2229 bytes Desc: patch against classindex.lisp URL: -------------- next part -------------- -- Please read about why Top Posting is evil at: http://en.wikipedia.org/wiki/Top-posting and http://www.dickalba.demon.co.uk/usenet/guide/faq_topp.html Please read about why HTML in email is evil at: http://www.birdhouse.org/etc/evilmail.html From eslick at csail.mit.edu Fri Nov 17 14:04:58 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Fri, 17 Nov 2006 09:04:58 -0500 Subject: [elephant-devel] get-instances-by-range should work with string types In-Reply-To: <8764dewf41.fsf@memetrics.com> References: <20061116065955.GF30093@bateleur.arcanes.fr.eu.org> <8764dewf41.fsf@memetrics.com> Message-ID: <455DC18A.9050504@csail.mit.edu> You have to be a little careful because in both BDB and SQL backends range queries are dependant on the ordering that the database uses for secondary indices. For instance, there is a C function that handles the ordering of elements in an index defined on a slot. The idea of a range query is to gain efficiency by only looking at a subset of objects. Now you can approximate this by mapping your non-integer objects into an integer space according to your own ordering. This is provided for in the (add-class-derived-index ...) which takes a function that is called by the backend to determine the value. This can be an arbitrary lisp function. Unless I'm mis-remembering, you can use the name of this derived index as a 'slotname' when you call (get-instances-by-range ...). However it may work for strings today because ordering is based on the order of binary bytes in the serialized representation of lisp data. I'll see if I can document all this better in the next release; it's helpful getting all this input to see what is confusing to new users and what features they want! Ian Alain Picard wrote: > Hello to all Elephant developers, and many words of thanks > for what appears to be a useful and well thought out library! > > I've recently started a project which will (I hope) make some > good use of Elephant. I'm still wrapping my head around things, > so go easy on me. As a show of good will, here's my first attempt > at adding something to elephant. > > Suppose you have a class like this: > > (defclass test-user () > ((name :reader name :initarg :name > :type string :index t) > (timestamp :reader timestamp > :initform (get-universal-time) > :type integer > :index t)) > (:metaclass persistent-metaclass)) > > (defun make-some-users () > (let ((n 0) > (*auto-commit* t)) > (dotimes (i 1000) > (make-instance 'test-user :name (format nil "User name ~D" (incf n)))) > (get-instances-by-range 'test-user 'name "User name 10" "User name 20"))) > > (make-some-users) > > If you try to do something like > > (length (get-instances-by-range 'test-user 'timestamp 0 999999999999)) > ==> 1000 > > It succeeds. > But if you try on the name, > > (get-instances-by-range 'test-user 'name "User name 10" "User name 11") > ==> > Argument X is not a REAL: "User name 10" > [Condition of type SIMPLE-TYPE-ERROR] > > Because the get-instances-by-range function (needlessly) assumes > that the objects being returned can be compared with numeric > equality. > > This little snippet fixes this problem: > (change from the 0.6 distribution) > =============cut================================================ > (defun find-slot-type (class idx-name) > (flet ((candidate-slot-p (slot) > (and (eq (type-of slot) 'persistent-effective-slot-definition) > (slot-value slot 'indexed) > (eq (slot-definition-name slot) idx-name)))) > (find-if #'candidate-slot-p (class-slots class)))) > > (defun find-index-comparison-function (class index) > (let ((type (sb-pcl:slot-definition-type (find-indexed-slot class index)))) > (cond ((subtypep type 'number) > #'<=) > ((subtypep type 'string) > #'string<=) > (t > ;; We'll fall back to numerical, though it's not clear to > ;; me that this is sensible. Maybe should just signal an error, > ;; and force users to declare :type on all indexed slots? > #'<=)))) > > (defmethod get-instances-by-range ((class persistent-metaclass) idx-name start end) > (let ((comparison (find-index-comparison-function class idx-name))) > (with-inverted-cursor (cur class idx-name) > (labels ((next-range (instances) > (multiple-value-bind (exists? skey val pkey) (cursor-pnext-nodup cur) > (declare (ignore pkey)) > (if (and exists? (funcall comparison skey end)) > (next-in-range skey (cons val instances)) > (nreverse instances)))) > (next-in-range (key instances) > (multiple-value-bind (exists? skey val pkey) (cursor-pnext-dup cur) > (declare (ignore pkey skey)) > (if exists? > (next-in-range key (cons val instances)) > (progn > (cursor-pset-range cur key) > (next-range instances)))))) > (multiple-value-bind (exists? skey val pkey) (cursor-pset-range cur start) > (declare (ignore pkey)) > (if (and exists? (funcall comparison skey end)) > (next-in-range skey (cons val nil)) > nil)))))) > =============cut================================================ > > Note that unfortunately FIND-INDEX-COMPARISON-FUNCTION makes > use of SB-PCL:SLOT-DEFINITION-TYPE; I have not conditionalized > this for other lisps, nor do I really know what forms are appropriate > in any other lisps. > > Obviously, it is possible to extend this scheme to let users register > their own comparison functions for more complex types; I'll let you > judge if this is worth the effort. > > Hoping this helps someone, somewhere. > > If this was already covered in some other way, then consider this > message a bug report on the documentation instead. :-) > > Alain Picard > > > From garry.steedman at gmail.com Sat Nov 18 01:29:50 2006 From: garry.steedman at gmail.com (Garry Steedman) Date: Sat, 18 Nov 2006 02:29:50 +0100 Subject: [elephant-devel] install problems on debian etch Message-ID: <28082d170611171729l57fa0863r56a4b266f70e24fe@mail.gmail.com> hello all, i'm trying to get elephant cvs running under debian etch and was daring to hope someone might be able to help with the following problem: ------------------------------------------------------------ ; registering # as UFFI /var/cache/common-lisp-controller/0/sbcl/elephant/src/memutil/libmemutil.so ; compiling file "/usr/share/common-lisp/source/elephant/src/memutil/memutil.lisp" (written 11 NOV 2006 11:53:13 PM): ; compiling (DEFPACKAGE ELEPHANT-MEMUTIL ...) ; compiling (IN-PACKAGE "ELEPHANT-MEMUTIL") ; compiling (DEF-TYPE POINTER-INT ...) ; compiling (DEF-TYPE POINTER-VOID ...) ; compiling (DEF-FOREIGN-TYPE ARRAY-OR-POINTER-CHAR ...) ; compiling (DEF-TYPE ARRAY-OR-POINTER-CHAR ...) ; compiling (DEF-FUNCTION ("copy_buf" COPY-BUFS) ...) ; compiling (DECLAIM (INLINE READ-INT ...)) ; compiling (DEFVAR +NULL-VOID+ ...) ; compiling (DEFVAR +NULL-CHAR+ ...) ; compiling (DEFVAR *BUFFER-STREAMS* ...) ; compiling (DEFSTRUCT BUFFER-STREAM ...) ; file: /usr/share/common-lisp/source/elephant/src/memutil/memutil.lisp ; in: DEFSTRUCT BUFFER-STREAM ; (UFFI:ALLOCATE-FOREIGN-OBJECT :CHAR 10) ; --> MAKE-ALIEN ; ==> ; (SB-ALIEN-INTERNALS:%SAP-ALIEN (SB-ALIEN::%MAKE-ALIEN (* 8 10)) ; '#) ; ; note: unable to ; optimize ; because: ; could not optimize away %SAP-ALIEN: forced to do runtime ; allocation of alien-value structure ; compiling (DEFUN GRAB-BUFFER-STREAM ...) ; file: /usr/share/common-lisp/source/elephant/src/memutil/memutil.lisp ; in: DEFUN GRAB-BUFFER-STREAM ; (LENGTH ELEPHANT-MEMUTIL::*BUFFER-STREAMS*) ; ; note: unable to ; optimize ; due to type uncertainty: ; The first argument is a SEQUENCE, not a (SIMPLE-ARRAY * (*)). ; ; note: unable to ; optimize ; due to type uncertainty: ; The first argument is a SEQUENCE, not a VECTOR. ; compiling (DEFUN RETURN-BUFFER-STREAM ...) ; compiling (DEFMACRO WITH-BUFFER-STREAMS ...) debugger invoked on a COMMON-LISP:UNDEFINED-FUNCTION in thread #: The function ELEPHANT-MEMUTIL::NEW-STYLE-COPY-P is undefined. Type HELP for debugger help, or (SB-EXT:QUIT) to exit from SBCL. restarts (invokable by number or by possibly-abbreviated name): 0: [RETRY ] Retry performing # on #. 1: [ACCEPT] Continue, treating # on # as having been successful. 2: [ABORT ] Exit debugger, returning to top level. ("bogus stack frame") ------------------------------------------------------------ any suggestions appreciated, cheers, gs -------------- next part -------------- An HTML attachment was scrubbed... URL: From read at robertlread.net Sat Nov 18 01:48:05 2006 From: read at robertlread.net (Robert L. Read) Date: Fri, 17 Nov 2006 19:48:05 -0600 Subject: [elephant-devel] install problems on debian etch In-Reply-To: <28082d170611171729l57fa0863r56a4b266f70e24fe@mail.gmail.com> References: <28082d170611171729l57fa0863r56a4b266f70e24fe@mail.gmail.com> Message-ID: <1163814485.9130.85.camel@localhost.localdomain> Under SBCL, apparently? Execute "nm" on libmemutil.so to see if the symbol is in the library. (Just a stab : make you you have remade libmemutil.) Those are just guesses, I'm afraid. On Sat, 2006-11-18 at 02:29 +0100, Garry Steedman wrote: > hello all, > > i'm trying to get elephant cvs running under debian etch and was > daring to hope someone might be able to help with the following > problem: > > ------------------------------------------------------------ > ; registering # as UFFI > /var/cache/common-lisp- > controller/0/sbcl/elephant/src/memutil/libmemutil.so > ; compiling file "/usr/share/common- > lisp/source/elephant/src/memutil/memutil.lisp" (written 11 NOV 2006 > 11:53:13 PM): > ; compiling (DEFPACKAGE ELEPHANT-MEMUTIL ...) > ; compiling (IN-PACKAGE "ELEPHANT-MEMUTIL") > ; compiling (DEF-TYPE POINTER-INT ...) > ; compiling (DEF-TYPE POINTER-VOID ...) > ; compiling (DEF-FOREIGN-TYPE ARRAY-OR-POINTER-CHAR ...) > ; compiling (DEF-TYPE ARRAY-OR-POINTER-CHAR ...) > ; compiling (DEF-FUNCTION ("copy_buf" COPY-BUFS) ...) > ; compiling (DECLAIM (INLINE READ-INT ...)) > ; compiling (DEFVAR +NULL-VOID+ ...) > ; compiling (DEFVAR +NULL-CHAR+ ...) > ; compiling (DEFVAR *BUFFER-STREAMS* ...) > ; compiling (DEFSTRUCT BUFFER-STREAM ...) > ; file: /usr/share/common- > lisp/source/elephant/src/memutil/memutil.lisp > ; in: DEFSTRUCT BUFFER-STREAM > ; (UFFI:ALLOCATE-FOREIGN-OBJECT :CHAR 10) > ; --> MAKE-ALIEN > ; ==> > ; (SB-ALIEN-INTERNALS:%SAP-ALIEN (SB-ALIEN::%MAKE-ALIEN (* 8 10)) > ; '# POINTER-TYPE (* > ; > (SIGNED > ; > 8))>) > ; > ; note: unable to > ; optimize > ; because: > ; could not optimize away %SAP-ALIEN: forced to do runtime > ; allocation of alien-value structure > > ; compiling (DEFUN GRAB-BUFFER-STREAM ...) > ; file: /usr/share/common- > lisp/source/elephant/src/memutil/memutil.lisp > ; in: DEFUN GRAB-BUFFER-STREAM > ; (LENGTH ELEPHANT-MEMUTIL::*BUFFER-STREAMS*) > ; > ; note: unable to > ; optimize > ; due to type uncertainty: > ; The first argument is a SEQUENCE, not a (SIMPLE-ARRAY * (*)). > ; > ; note: unable to > ; optimize > ; due to type uncertainty: > ; The first argument is a SEQUENCE, not a VECTOR. > > ; compiling (DEFUN RETURN-BUFFER-STREAM ...) > ; compiling (DEFMACRO WITH-BUFFER-STREAMS ...) > debugger invoked on a COMMON-LISP:UNDEFINED-FUNCTION in thread > #: > The function ELEPHANT-MEMUTIL::NEW-STYLE-COPY-P is undefined. > > Type HELP for debugger help, or (SB-EXT:QUIT) to exit from SBCL. > > restarts (invokable by number or by possibly-abbreviated name): > 0: [RETRY ] Retry performing # {AB9B451}> on > #. > 1: [ACCEPT] Continue, treating # {AB9B451}> > on # as having > been > successful. > 2: [ABORT ] Exit debugger, returning to top level. > > ("bogus stack frame") > ------------------------------------------------------------ > > any suggestions appreciated, > > cheers, > > gs > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From garry.steedman at gmail.com Sat Nov 18 02:03:38 2006 From: garry.steedman at gmail.com (Garry Steedman) Date: Sat, 18 Nov 2006 03:03:38 +0100 Subject: [elephant-devel] install problems on debian etch In-Reply-To: <1163814485.9130.85.camel@localhost.localdomain> References: <28082d170611171729l57fa0863r56a4b266f70e24fe@mail.gmail.com> <1163814485.9130.85.camel@localhost.localdomain> Message-ID: <28082d170611171803r58a02000i2b3a20c6a8b75af5@mail.gmail.com> Hi Robert, here's what i get: ------------------------------------------------------------ # nm /var/cache/common-lisp-controller/0/sbcl/elephant/src/memutil/libmemutil.so 000018cc A __bss_start 00000540 t call_gmon_start 000018cc b completed.5621 00000620 T copy_buf 000017c8 d __CTOR_END__ 000017c4 d __CTOR_LIST__ w __cxa_finalize@@GLIBC_2.1.3 00000760 t __do_global_ctors_aux 00000570 t __do_global_dtors_aux 000018c4 d __dso_handle 000017d0 d __DTOR_END__ 000017cc d __DTOR_LIST__ 000017d8 a _DYNAMIC 000018cc A _edata 000018d0 A _end 000007a4 T _fini 000005d0 t frame_dummy 000007c0 r __FRAME_END__ 000018ac a _GLOBAL_OFFSET_TABLE_ w __gmon_start__ 00000605 t __i686.get_pc_thunk.bx 000004e4 T _init 000017d4 d __JCR_END__ 000017d4 d __JCR_LIST__ w _Jv_RegisterClasses U memcpy@@GLIBC_2.0 00000610 T offset_charp 000018c8 d p.5619 000006e0 T read_double 00000700 T read_float 00000740 T read_int 00000720 T read_uint 00000660 T write_double 00000680 T write_float 000006c0 T write_int 000006a0 T write_uint # ------------------------------------------------------------ no help to me, maybe it is to you? btw, is the INSTALL file up to date for cvs? in section 4 there's stuff about running 'make' in elephant root, but there's no makefile... cheers, garry On 11/18/06, Robert L. Read wrote: > > Under SBCL, apparently? > > Execute "nm" on libmemutil.so to see if the symbol is in the library. > > (Just a stab : make you you have remade libmemutil.) > > Those are just guesses, I'm afraid. > > > On Sat, 2006-11-18 at 02:29 +0100, Garry Steedman wrote: > > hello all, > > i'm trying to get elephant cvs running under debian etch and was daring to > hope someone might be able to help with the following problem: > > ------------------------------------------------------------ > ; registering # as UFFI > > /var/cache/common-lisp-controller/0/sbcl/elephant/src/memutil/libmemutil.so > ; compiling file > "/usr/share/common-lisp/source/elephant/src/memutil/memutil.lisp" (written > 11 NOV 2006 11:53:13 PM): > ; compiling (DEFPACKAGE ELEPHANT-MEMUTIL ...) > ; compiling (IN-PACKAGE "ELEPHANT-MEMUTIL") > ; compiling (DEF-TYPE POINTER-INT ...) > ; compiling (DEF-TYPE POINTER-VOID ...) > ; compiling (DEF-FOREIGN-TYPE ARRAY-OR-POINTER-CHAR ...) > ; compiling (DEF-TYPE ARRAY-OR-POINTER-CHAR ...) > ; compiling (DEF-FUNCTION ("copy_buf" COPY-BUFS) ...) > ; compiling (DECLAIM (INLINE READ-INT ...)) > ; compiling (DEFVAR +NULL-VOID+ ...) > ; compiling (DEFVAR +NULL-CHAR+ ...) > ; compiling (DEFVAR *BUFFER-STREAMS* ...) > ; compiling (DEFSTRUCT BUFFER-STREAM ...) > ; file: /usr/share/common-lisp/source/elephant/src/memutil/memutil.lisp > ; in: DEFSTRUCT BUFFER-STREAM > ; (UFFI:ALLOCATE-FOREIGN-OBJECT :CHAR 10) > ; --> MAKE-ALIEN > ; ==> > ; (SB-ALIEN-INTERNALS:%SAP-ALIEN (SB-ALIEN::%MAKE-ALIEN (* 8 10)) > ; > '# ; > (SIGNED > ; > 8))>) > ; > ; note: unable to > ; optimize > ; because: > ; could not optimize away %SAP-ALIEN: forced to do runtime > ; allocation of alien-value structure > > ; compiling (DEFUN GRAB-BUFFER-STREAM ...) > ; file: /usr/share/common-lisp/source/elephant/src/memutil/memutil.lisp > ; in: DEFUN GRAB-BUFFER-STREAM > ; (LENGTH ELEPHANT-MEMUTIL::*BUFFER-STREAMS*) > ; > ; note: unable to > ; optimize > ; due to type uncertainty: > ; The first argument is a SEQUENCE, not a (SIMPLE-ARRAY * (*)). > ; > ; note: unable to > ; optimize > ; due to type uncertainty: > ; The first argument is a SEQUENCE, not a VECTOR. > > ; compiling (DEFUN RETURN-BUFFER-STREAM ...) > ; compiling (DEFMACRO WITH-BUFFER-STREAMS ...) > debugger invoked on a COMMON-LISP:UNDEFINED-FUNCTION in thread # "initial thread" {A7BC549}>: > The function ELEPHANT-MEMUTIL::NEW-STYLE-COPY-P is undefined. > > Type HELP for debugger help, or (SB-EXT:QUIT) to exit from SBCL. > > restarts (invokable by number or by possibly-abbreviated name): > 0: [RETRY ] Retry performing # {AB9B451}> on > #. > 1: [ACCEPT] Continue, treating # {AB9B451}> > on # as having been > > successful. > 2: [ABORT ] Exit debugger, returning to top level. > > ("bogus stack frame") > ------------------------------------------------------------ > > any suggestions appreciated, > > cheers, > > gs > > _______________________________________________elephant-devel site listelephant-devel at common-lisp.nethttp://common-lisp.net/mailman/listinfo/elephant-devel > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eslick at csail.mit.edu Sat Nov 18 02:16:24 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Fri, 17 Nov 2006 21:16:24 -0500 Subject: [elephant-devel] install problems on debian etch In-Reply-To: <28082d170611171803r58a02000i2b3a20c6a8b75af5@mail.gmail.com> References: <28082d170611171729l57fa0863r56a4b266f70e24fe@mail.gmail.com> <1163814485.9130.85.camel@localhost.localdomain> <28082d170611171803r58a02000i2b3a20c6a8b75af5@mail.gmail.com> Message-ID: <455E6CF8.3090203@csail.mit.edu> Sorry, CVS HEAD is churning quite a bit right now as I'm trying to get features in for 0.6.1 and I'm not testing incrementally under SBCL so I only catch SBCL-specific bugs (which this is) every so often. Let me track that down and I'll write back in a bit with a fix. Ian Garry Steedman wrote: > Hi Robert, > > here's what i get: > > ------------------------------------------------------------ > # nm > /var/cache/common-lisp-controller/0/sbcl/elephant/src/memutil/libmemutil.so > > 000018cc A __bss_start > 00000540 t call_gmon_start > 000018cc b completed.5621 > 00000620 T copy_buf > 000017c8 d __CTOR_END__ > 000017c4 d __CTOR_LIST__ > w __cxa_finalize@@GLIBC_2.1.3 > 00000760 t __do_global_ctors_aux > 00000570 t __do_global_dtors_aux > 000018c4 d __dso_handle > 000017d0 d __DTOR_END__ > 000017cc d __DTOR_LIST__ > 000017d8 a _DYNAMIC > 000018cc A _edata > 000018d0 A _end > 000007a4 T _fini > 000005d0 t frame_dummy > 000007c0 r __FRAME_END__ > 000018ac a _GLOBAL_OFFSET_TABLE_ > w __gmon_start__ > 00000605 t __i686.get_pc_thunk.bx > 000004e4 T _init > 000017d4 d __JCR_END__ > 000017d4 d __JCR_LIST__ > w _Jv_RegisterClasses > U memcpy@@GLIBC_2.0 > 00000610 T offset_charp > 000018c8 d p.5619 > 000006e0 T read_double > 00000700 T read_float > 00000740 T read_int > 00000720 T read_uint > 00000660 T write_double > 00000680 T write_float > 000006c0 T write_int > 000006a0 T write_uint > # > ------------------------------------------------------------ > > no help to me, maybe it is to you? > > btw, is the INSTALL file up to date for cvs? in section 4 there's > stuff about running 'make' in elephant root, but there's no makefile... > > cheers, > > garry > > > On 11/18/06, *Robert L. Read* > wrote: > > Under SBCL, apparently? > > Execute "nm" on libmemutil.so to see if the symbol is in the library. > > (Just a stab : make you you have remade libmemutil.) > > Those are just guesses, I'm afraid. > > > > On Sat, 2006-11-18 at 02:29 +0100, Garry Steedman wrote: >> hello all, >> >> i'm trying to get elephant cvs running under debian etch and was >> daring to hope someone might be able to help with the following >> problem: >> >> ------------------------------------------------------------ >> ; registering # as UFFI >> /var/cache/common-lisp-controller/0/sbcl/elephant/src/memutil/libmemutil.so >> ; compiling file >> "/usr/share/common-lisp/source/elephant/src/memutil/memutil.lisp" >> (written 11 NOV 2006 11:53:13 PM): >> ; compiling (DEFPACKAGE ELEPHANT-MEMUTIL ...) >> ; compiling (IN-PACKAGE "ELEPHANT-MEMUTIL") >> ; compiling (DEF-TYPE POINTER-INT ...) >> ; compiling (DEF-TYPE POINTER-VOID ...) >> ; compiling (DEF-FOREIGN-TYPE ARRAY-OR-POINTER-CHAR ...) >> ; compiling (DEF-TYPE ARRAY-OR-POINTER-CHAR ...) >> ; compiling (DEF-FUNCTION ("copy_buf" COPY-BUFS) ...) >> ; compiling (DECLAIM (INLINE READ-INT ...)) >> ; compiling (DEFVAR +NULL-VOID+ ...) >> ; compiling (DEFVAR +NULL-CHAR+ ...) >> ; compiling (DEFVAR *BUFFER-STREAMS* ...) >> ; compiling (DEFSTRUCT BUFFER-STREAM ...) >> ; file: >> /usr/share/common-lisp/source/elephant/src/memutil/memutil.lisp >> ; in: DEFSTRUCT BUFFER-STREAM >> ; (UFFI:ALLOCATE-FOREIGN-OBJECT :CHAR 10) >> ; --> MAKE-ALIEN >> ; ==> >> ; (SB-ALIEN-INTERNALS:%SAP-ALIEN (SB-ALIEN::%MAKE-ALIEN (* 8 10)) >> ; >> '#> ; >> (SIGNED >> ; >> 8))>) >> ; >> ; note: unable to >> ; optimize >> ; because: >> ; could not optimize away %SAP-ALIEN: forced to do runtime >> ; allocation of alien-value structure >> >> ; compiling (DEFUN GRAB-BUFFER-STREAM ...) >> ; file: >> /usr/share/common-lisp/source/elephant/src/memutil/memutil.lisp >> ; in: DEFUN GRAB-BUFFER-STREAM >> ; (LENGTH ELEPHANT-MEMUTIL::*BUFFER-STREAMS*) >> ; >> ; note: unable to >> ; optimize >> ; due to type uncertainty: >> ; The first argument is a SEQUENCE, not a (SIMPLE-ARRAY * (*)). >> ; >> ; note: unable to >> ; optimize >> ; due to type uncertainty: >> ; The first argument is a SEQUENCE, not a VECTOR. >> >> ; compiling (DEFUN RETURN-BUFFER-STREAM ...) >> ; compiling (DEFMACRO WITH-BUFFER-STREAMS ...) >> debugger invoked on a COMMON-LISP:UNDEFINED-FUNCTION in thread >> #: >> The function ELEPHANT-MEMUTIL::NEW-STYLE-COPY-P is undefined. >> >> Type HELP for debugger help, or (SB-EXT:QUIT) to exit from SBCL. >> >> restarts (invokable by number or by possibly-abbreviated name): >> 0: [RETRY ] Retry performing #> {AB9B451}> on >> #. >> 1: [ACCEPT] Continue, treating #> COMMON-LISP:NIL {AB9B451}> >> on # as >> having been >> successful. >> 2: [ABORT ] Exit debugger, returning to top level. >> >> ("bogus stack frame") >> ------------------------------------------------------------ >> >> any suggestions appreciated, >> >> cheers, >> >> gs >> _______________________________________________ >> elephant-devel site list >> elephant-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/elephant-devel >> >> > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > > http://common-lisp.net/mailman/listinfo/elephant-devel > > > ------------------------------------------------------------------------ > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From garry.steedman at gmail.com Sat Nov 18 02:23:57 2006 From: garry.steedman at gmail.com (Garry Steedman) Date: Sat, 18 Nov 2006 03:23:57 +0100 Subject: [elephant-devel] install problems on debian etch In-Reply-To: <455E6CF8.3090203@csail.mit.edu> References: <28082d170611171729l57fa0863r56a4b266f70e24fe@mail.gmail.com> <1163814485.9130.85.camel@localhost.localdomain> <28082d170611171803r58a02000i2b3a20c6a8b75af5@mail.gmail.com> <455E6CF8.3090203@csail.mit.edu> Message-ID: <28082d170611171823m6900011cs2615299b3ce32771@mail.gmail.com> appreciated! cheers, gs On 11/18/06, Ian Eslick wrote: > > Sorry, CVS HEAD is churning quite a bit right now as I'm trying to get > features in for 0.6.1 and I'm not testing incrementally under SBCL so I > only catch SBCL-specific bugs (which this is) every so often. > > Let me track that down and I'll write back in a bit with a fix. > > Ian > > > Garry Steedman wrote: > > Hi Robert, > > > > here's what i get: > > > > ------------------------------------------------------------ > > # nm > > > /var/cache/common-lisp-controller/0/sbcl/elephant/src/memutil/libmemutil.so > > > > 000018cc A __bss_start > > 00000540 t call_gmon_start > > 000018cc b completed.5621 > > 00000620 T copy_buf > > 000017c8 d __CTOR_END__ > > 000017c4 d __CTOR_LIST__ > > w __cxa_finalize@@GLIBC_2.1.3 > > 00000760 t __do_global_ctors_aux > > 00000570 t __do_global_dtors_aux > > 000018c4 d __dso_handle > > 000017d0 d __DTOR_END__ > > 000017cc d __DTOR_LIST__ > > 000017d8 a _DYNAMIC > > 000018cc A _edata > > 000018d0 A _end > > 000007a4 T _fini > > 000005d0 t frame_dummy > > 000007c0 r __FRAME_END__ > > 000018ac a _GLOBAL_OFFSET_TABLE_ > > w __gmon_start__ > > 00000605 t __i686.get_pc_thunk.bx > > 000004e4 T _init > > 000017d4 d __JCR_END__ > > 000017d4 d __JCR_LIST__ > > w _Jv_RegisterClasses > > U memcpy@@GLIBC_2.0 > > 00000610 T offset_charp > > 000018c8 d p.5619 > > 000006e0 T read_double > > 00000700 T read_float > > 00000740 T read_int > > 00000720 T read_uint > > 00000660 T write_double > > 00000680 T write_float > > 000006c0 T write_int > > 000006a0 T write_uint > > # > > ------------------------------------------------------------ > > > > no help to me, maybe it is to you? > > > > btw, is the INSTALL file up to date for cvs? in section 4 there's > > stuff about running 'make' in elephant root, but there's no makefile... > > > > cheers, > > > > garry > > > > > > On 11/18/06, *Robert L. Read* > > wrote: > > > > Under SBCL, apparently? > > > > Execute "nm" on libmemutil.so to see if the symbol is in the > library. > > > > (Just a stab : make you you have remade libmemutil.) > > > > Those are just guesses, I'm afraid. > > > > > > > > On Sat, 2006-11-18 at 02:29 +0100, Garry Steedman wrote: > >> hello all, > >> > >> i'm trying to get elephant cvs running under debian etch and was > >> daring to hope someone might be able to help with the following > >> problem: > >> > >> ------------------------------------------------------------ > >> ; registering # as UFFI > >> > /var/cache/common-lisp-controller/0/sbcl/elephant/src/memutil/libmemutil.so > >> ; compiling file > >> "/usr/share/common-lisp/source/elephant/src/memutil/memutil.lisp" > >> (written 11 NOV 2006 11:53:13 PM): > >> ; compiling (DEFPACKAGE ELEPHANT-MEMUTIL ...) > >> ; compiling (IN-PACKAGE "ELEPHANT-MEMUTIL") > >> ; compiling (DEF-TYPE POINTER-INT ...) > >> ; compiling (DEF-TYPE POINTER-VOID ...) > >> ; compiling (DEF-FOREIGN-TYPE ARRAY-OR-POINTER-CHAR ...) > >> ; compiling (DEF-TYPE ARRAY-OR-POINTER-CHAR ...) > >> ; compiling (DEF-FUNCTION ("copy_buf" COPY-BUFS) ...) > >> ; compiling (DECLAIM (INLINE READ-INT ...)) > >> ; compiling (DEFVAR +NULL-VOID+ ...) > >> ; compiling (DEFVAR +NULL-CHAR+ ...) > >> ; compiling (DEFVAR *BUFFER-STREAMS* ...) > >> ; compiling (DEFSTRUCT BUFFER-STREAM ...) > >> ; file: > >> /usr/share/common-lisp/source/elephant/src/memutil/memutil.lisp > >> ; in: DEFSTRUCT BUFFER-STREAM > >> ; (UFFI:ALLOCATE-FOREIGN-OBJECT :CHAR 10) > >> ; --> MAKE-ALIEN > >> ; ==> > >> ; (SB-ALIEN-INTERNALS:%SAP-ALIEN (SB-ALIEN::%MAKE-ALIEN (* 8 10)) > >> ; > >> '# >> ; > >> (SIGNED > >> ; > >> 8))>) > >> ; > >> ; note: unable to > >> ; optimize > >> ; because: > >> ; could not optimize away %SAP-ALIEN: forced to do runtime > >> ; allocation of alien-value structure > >> > >> ; compiling (DEFUN GRAB-BUFFER-STREAM ...) > >> ; file: > >> /usr/share/common-lisp/source/elephant/src/memutil/memutil.lisp > >> ; in: DEFUN GRAB-BUFFER-STREAM > >> ; (LENGTH ELEPHANT-MEMUTIL::*BUFFER-STREAMS*) > >> ; > >> ; note: unable to > >> ; optimize > >> ; due to type uncertainty: > >> ; The first argument is a SEQUENCE, not a (SIMPLE-ARRAY * (*)). > >> ; > >> ; note: unable to > >> ; optimize > >> ; due to type uncertainty: > >> ; The first argument is a SEQUENCE, not a VECTOR. > >> > >> ; compiling (DEFUN RETURN-BUFFER-STREAM ...) > >> ; compiling (DEFMACRO WITH-BUFFER-STREAMS ...) > >> debugger invoked on a COMMON-LISP:UNDEFINED-FUNCTION in thread > >> #: > >> The function ELEPHANT-MEMUTIL::NEW-STYLE-COPY-P is undefined. > >> > >> Type HELP for debugger help, or (SB-EXT:QUIT) to exit from SBCL. > >> > >> restarts (invokable by number or by possibly-abbreviated name): > >> 0: [RETRY ] Retry performing # >> {AB9B451}> on > >> #. > >> 1: [ACCEPT] Continue, treating # >> COMMON-LISP:NIL {AB9B451}> > >> on # as > >> having been > >> successful. > >> 2: [ABORT ] Exit debugger, returning to top level. > >> > >> ("bogus stack frame") > >> ------------------------------------------------------------ > >> > >> any suggestions appreciated, > >> > >> cheers, > >> > >> gs > >> _______________________________________________ > >> elephant-devel site list > >> elephant-devel at common-lisp.net elephant-devel at common-lisp.net> > >> http://common-lisp.net/mailman/listinfo/elephant-devel > >> > >> > > > > _______________________________________________ > > elephant-devel site list > > elephant-devel at common-lisp.net > > > > http://common-lisp.net/mailman/listinfo/elephant-devel > > > > > > ------------------------------------------------------------------------ > > > > _______________________________________________ > > elephant-devel site list > > elephant-devel at common-lisp.net > > http://common-lisp.net/mailman/listinfo/elephant-devel > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alain.Picard at memetrics.com Sat Nov 18 06:00:40 2006 From: Alain.Picard at memetrics.com (Alain Picard) Date: Sat, 18 Nov 2006 17:00:40 +1100 Subject: [elephant-devel] get-instances-by-range should work with string types In-Reply-To: <455DC18A.9050504@csail.mit.edu> (Ian Eslick's message of "Fri, 17 Nov 2006 09:04:58 -0500") References: <20061116065955.GF30093@bateleur.arcanes.fr.eu.org> <8764dewf41.fsf@memetrics.com> <455DC18A.9050504@csail.mit.edu> Message-ID: <87u00xl2ev.fsf@memetrics.com> Ian Eslick writes: > Now you can approximate this by mapping your non-integer objects into an > integer space according to your own ordering. This is provided for in > the (add-class-derived-index ...) ... [SNIP] Ah, ok. I missed that. That's kinda what I had in mind by a protocol to tell elephant how to define ordering and comparison on the objects indexed in a slot. I'll experiment with this and see if I can get it to do what I want. This is a more general mechanism than the trivial patch I submitted, so I wouldn't rush to commit what I sent. > However it may work for strings today because ordering is based on the > order of binary bytes in the serialized representation of lisp data. Okay. > I'll see if I can document all this better in the next release; it's > helpful getting all this input to see what is confusing to new users and > what features they want! Well, I'm happy to keep helping by asking questions, hopefully they won't all be stupid. ;-) --ap From gk1 at four-four.com Mon Nov 20 06:53:23 2006 From: gk1 at four-four.com (George Khouri) Date: Sun, 19 Nov 2006 22:53:23 -0800 Subject: [elephant-devel] Development tools In-Reply-To: <455C76E0.8010503@csail.mit.edu> Message-ID: Ian, Does the storing of unicode strings little-endian require little-endian input strings, or will/do you convert big-endian unicode strings as (I believe) are represented on the PPC (OpenMCL)? Thanks, George >Unicode: > >By the way, I'm just cleaning up the last of my unicode updates. I kept >having problems with the efficiency hacks in the current support for >Unicode -- there was no canonical representation of strings in the >database; each lisp+machine coded it differently. Also, even though >most strings have codes in the ASCII or Latin-1 character set, SBCL was >still storing 32-bit characters. It now uses the smallest coding size >(8,16 or 32) necessary to represent the string. Support for 8 or 16 is >fairly efficient but if you use unicode code pages > 0 there will be a >performance and storage hit. I put in a convention in where all 16/32 >bit unicode strings are stored little-endian (x86 is a little-endian >machine) so I can use native string reader functions to pull shorts and >ints out of the byte vectors when possible. This should greatly compact >string storage on most unicode supporting systems (2x on allegro, 4x on >SBCL). > -------- George Khouri gk1 at four-four.com From nowhere.man at levallois.eu.org Mon Nov 20 13:46:50 2006 From: nowhere.man at levallois.eu.org (Pierre THIERRY) Date: Mon, 20 Nov 2006 14:46:50 +0100 Subject: [elephant-devel] Pure-Lisp memutil? Message-ID: <20061120134650.GC8075@bateleur.arcanes.fr.eu.org> I was wondering why was C memutil needed. Couldn't it be possible to use memcpy and al. with FFI in CL code? Curiously, Pierre -- nowhere.man at levallois.eu.org OpenPGP 0xD9D50D8A -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From eslick at csail.mit.edu Mon Nov 20 14:53:38 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Mon, 20 Nov 2006 09:53:38 -0500 Subject: [elephant-devel] Development tools In-Reply-To: References: Message-ID: <4561C172.6030703@csail.mit.edu> George, On writes it does the conversion to little-endian manually using char-code and ldb to extract bytes. Similarly, the default read is also implemented using lisp routines code-char/dpb, but can exploit native byte-copy routines in some cases. I'll implement that later as a performance (re)-enhancement. The rest of the serializer still uses the underlying C representation of byte order so integers will reflect the endianness of the host machine and the width the host's native integer (i.e., 8 bytes on 64-bit machines). I decided for this version that lisp-independence and the bug freeness of string decoding was more useful than being machine-type independent. With some more work (and performance hits) the whole thing could become standardized on a particular byte order, but I won't do that unless asked. Ian George Khouri wrote: > Ian, > Does the storing of unicode strings little-endian require little-endian input strings, or will/do you convert big-endian unicode strings as (I believe) are represented on the PPC (OpenMCL)? > Thanks, > George > > >> Unicode: >> >> By the way, I'm just cleaning up the last of my unicode updates. I kept >> having problems with the efficiency hacks in the current support for >> Unicode -- there was no canonical representation of strings in the >> database; each lisp+machine coded it differently. Also, even though >> most strings have codes in the ASCII or Latin-1 character set, SBCL was >> still storing 32-bit characters. It now uses the smallest coding size >> (8,16 or 32) necessary to represent the string. Support for 8 or 16 is >> fairly efficient but if you use unicode code pages > 0 there will be a >> performance and storage hit. I put in a convention in where all 16/32 >> bit unicode strings are stored little-endian (x86 is a little-endian >> machine) so I can use native string reader functions to pull shorts and >> ints out of the byte vectors when possible. This should greatly compact >> string storage on most unicode supporting systems (2x on allegro, 4x on >> SBCL). >> >> > -------- > George Khouri > gk1 at four-four.com > From eslick at csail.mit.edu Mon Nov 20 14:56:24 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Mon, 20 Nov 2006 09:56:24 -0500 Subject: [elephant-devel] Serialization Message-ID: <4561C218.7050606@csail.mit.edu> Elephant users and developers, I've made the serializer thread safe for 0.6.1, but face an design tradeoff. The serializer is a critical bottleneck for Elephant's throughput and by adding locking into the equation I significantly slow down the calls to serialize that operate on a small amount of data (all symbols, integers and short strings which are typical slot values). There are several possible solutions, in rough order of performance from fastest to slowest: 1) Special start-elephant-thread macro which creates dynamic bindings for the new process of a global special variable that maintains the circularity buffer; this will allow us to be very efficient but requires elephant-specific code on thread creation for safe threading 2) Bind a C-library which provides a CPU independent compare-and-swap instruction and/or a lock-free queue so we can share a pool of circularity buffers among processes in a lock-free way 3) It occurs to me that most lisps do not allow C code to be run concurrently from multiple threads - perhaps just implementing a mutex in C would be enough to provide a cheap way of updating head and tail pointers in the queue. Capturing and releasing would be atomic. Is this true for all lisps? I know it's true for Allegro. It would be a hack, but a fairly harmless one given our existing dependence on the memutil library. 4) Stick with the portable locking approach already implemented Thoughts? Ian From eslick at csail.mit.edu Mon Nov 20 15:02:01 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Mon, 20 Nov 2006 10:02:01 -0500 Subject: [elephant-devel] Pure-Lisp memutil? In-Reply-To: <20061120134650.GC8075@bateleur.arcanes.fr.eu.org> References: <20061120134650.GC8075@bateleur.arcanes.fr.eu.org> Message-ID: <4561C369.3030502@csail.mit.edu> I suspect the original developers found that easier than doing all the boxing and unboxing conversions for ints, floats, etc by hand. Wrapping memcpy in a function means UFFI writes that code for you. Plus they got to use sizeof trivially which can change for different machines. Finally, they already had a C library compilation happening for BDB. Ian Pierre THIERRY wrote: > I was wondering why was C memutil needed. Couldn't it be possible to use > memcpy and al. with FFI in CL code? > > Curiously, > Pierre > > ------------------------------------------------------------------------ > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From eslick at csail.mit.edu Mon Nov 20 15:04:00 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Mon, 20 Nov 2006 10:04:00 -0500 Subject: [elephant-devel] Pure-Lisp memutil? In-Reply-To: <20061120134650.GC8075@bateleur.arcanes.fr.eu.org> References: <20061120134650.GC8075@bateleur.arcanes.fr.eu.org> Message-ID: <4561C3E0.5060102@csail.mit.edu> You'll notice that the cmu/sbcl implementation doesn't use the memutil C library routines and allegro doesn't need to (in fact it's faster), but then it conses which kicks in the GC more often. Reading through memutil.lisp should give you a feel for some of the design tradeoffs... Pierre THIERRY wrote: > I was wondering why was C memutil needed. Couldn't it be possible to use > memcpy and al. with FFI in CL code? > > Curiously, > Pierre > > ------------------------------------------------------------------------ > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From read at robertlread.net Mon Nov 20 15:55:20 2006 From: read at robertlread.net (Robert L. Read) Date: Mon, 20 Nov 2006 09:55:20 -0600 Subject: [elephant-devel] Serialization In-Reply-To: <4561C218.7050606@csail.mit.edu> References: <4561C218.7050606@csail.mit.edu> Message-ID: <1164038120.9130.230.camel@localhost.localdomain> Dear Ian, Since your doing all the work I'm a little hesitant to suggest a solution, but it seems to me the best solution is to use an all-lisp buffer pool, and use portable locking to atomically associate a serializing thread with a buffer. (This is very similar to solution #1 you propose, I think.) I'm actually not sure how your doing portable locking now; I just stuck some SBCL mutexes in there, but that is makes my personal branch SBCL dependent.) In the previous version (approximately v. 1.6 of the file serializer.lisp) the circularity hash is implemented as a hash-table. It's not clear to me why one couldn't just allocate this hashtable on every invocation, (perhpas lazily, when the first non- atomic type which might lead to a circularity is serialized.) Perhaps this allocation cost would dominate the serialization cost; but I would personally not be sure of that until a profiler gave such information. (However, I'm starting to worry that I might simply be terribly lazy, and covering up for the fact by chanting the mantra: no premature optimization. I trust your instincts and knowledge more than my own on this matter.) I'd like to stay away from any C-coding as much as possible; just as a matter of principle I can't imagine that that makes us more robust. Thanks for your work Ian ---- a better serializer will be a big performance boost for the whole project. On Mon, 2006-11-20 at 09:56 -0500, Ian Eslick wrote: > Elephant users and developers, > > I've made the serializer thread safe for 0.6.1, but face an design > tradeoff. The serializer is a critical bottleneck for Elephant's > throughput and by adding locking into the equation I significantly slow > down the calls to serialize that operate on a small amount of data (all > symbols, integers and short strings which are typical slot values). > There are several possible solutions, in rough order of performance from > fastest to slowest: > > 1) Special start-elephant-thread macro which creates dynamic bindings > for the new process of a global special variable that maintains the > circularity buffer; this will allow us to be very efficient but requires > elephant-specific code on thread creation for safe threading > > 2) Bind a C-library which provides a CPU independent compare-and-swap > instruction and/or a lock-free queue so we can share a pool of > circularity buffers among processes in a lock-free way > > 3) It occurs to me that most lisps do not allow C code to be run > concurrently from multiple threads - perhaps just implementing a mutex > in C would be enough to provide a cheap way of updating head and tail > pointers in the queue. Capturing and releasing would be atomic. Is > this true for all lisps? I know it's true for Allegro. It would be a > hack, but a fairly harmless one given our existing dependence on the > memutil library. > > 4) Stick with the portable locking approach already implemented > > Thoughts? > Ian > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From nowhere.man at levallois.eu.org Mon Nov 20 22:53:18 2006 From: nowhere.man at levallois.eu.org (Pierre THIERRY) Date: Mon, 20 Nov 2006 23:53:18 +0100 Subject: [elephant-devel] Pure-Lisp memutil? In-Reply-To: <4561C3E0.5060102@csail.mit.edu> References: <20061120134650.GC8075@bateleur.arcanes.fr.eu.org> <4561C3E0.5060102@csail.mit.edu> Message-ID: <20061120225318.GA24579@bateleur.arcanes.fr.eu.org> Scribit Ian Eslick dies 20/11/2006 hora 10:04: > Reading through memutil.lisp should give you a feel for some of the > design tradeoffs... I'm surprised in memutil.lisp by the optimize declarations. In the slot protocol also, there are some declarations for maximum speed, and here for minimum safety also. I'm not sure if it wouldn't be best to let the user choose. In a same vein, I may have a working Debian package by the end of the night, and I replaced -O3 in the Makefile by $(CFLAGS). I'm not sure but it may be a strong recommandation in Debian. Curiously, Pierre -- nowhere.man at levallois.eu.org OpenPGP 0xD9D50D8A -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From eslick at csail.mit.edu Tue Nov 21 01:38:21 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Mon, 20 Nov 2006 20:38:21 -0500 Subject: [elephant-devel] Pure-Lisp memutil? In-Reply-To: <20061120225318.GA24579@bateleur.arcanes.fr.eu.org> References: <20061120134650.GC8075@bateleur.arcanes.fr.eu.org> <4561C3E0.5060102@csail.mit.edu> <20061120225318.GA24579@bateleur.arcanes.fr.eu.org> Message-ID: <60DC2DDC-0E0C-487A-AD4A-19D8A68A3BFA@csail.mit.edu> Can you expand on your thinking as to why the user may want to turn off optimizations and by what mechanism you would suggest doing that? For production use the optimizations make sense. If you are conservative I think many lisps allow you to override the inlined declarations and limit the maximum and minimum value of optimization settings. If not, you can always evaluate things instead of compiling them for debugging or making sure the problem isn't in the library. In theory recent releases should be well tested and debugged, especially the paths through serialization, memutil and the slot protocol; they're speed critical so it's reasonable to run them fast and you can always turn that off by the above methods. Just my thinking on the subject! Ian On Nov 20, 2006, at 5:53 PM, Pierre THIERRY wrote: > Scribit Ian Eslick dies 20/11/2006 hora 10:04: >> Reading through memutil.lisp should give you a feel for some of the >> design tradeoffs... > > I'm surprised in memutil.lisp by the optimize declarations. In the > slot > protocol also, there are some declarations for maximum speed, and here > for minimum safety also. I'm not sure if it wouldn't be best to let > the > user choose. > > In a same vein, I may have a working Debian package by the end of the > night, and I replaced -O3 in the Makefile by $(CFLAGS). I'm not > sure but > it may be a strong recommandation in Debian. > > Curiously, > Pierre > -- > nowhere.man at levallois.eu.org > OpenPGP 0xD9D50D8A > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From nowhere.man at levallois.eu.org Tue Nov 21 05:07:18 2006 From: nowhere.man at levallois.eu.org (Pierre THIERRY) Date: Tue, 21 Nov 2006 06:07:18 +0100 Subject: [elephant-devel] Pure-Lisp memutil? In-Reply-To: <60DC2DDC-0E0C-487A-AD4A-19D8A68A3BFA@csail.mit.edu> References: <20061120134650.GC8075@bateleur.arcanes.fr.eu.org> <4561C3E0.5060102@csail.mit.edu> <20061120225318.GA24579@bateleur.arcanes.fr.eu.org> <60DC2DDC-0E0C-487A-AD4A-19D8A68A3BFA@csail.mit.edu> Message-ID: <20061121050718.GC24579@bateleur.arcanes.fr.eu.org> Scribit Ian Eslick dies 20/11/2006 hora 20:38: > Can you expand on your thinking as to why the user may want to turn > off optimizations and by what mechanism you would suggest doing that? I'm only really worried by the safety declarations. It should be a choice from the user to reduce safety, even for well-tested code. > For production use the optimizations make sense. Why not let the user choose it when in production? > they're speed critical so it's reasonable to run them fast and you > can always turn that off by the above methods. I can understand the rationale for declaring them so optimized. Are there numbers about the speed difference with and without optimizations? Curiously, Pierre -- nowhere.man at levallois.eu.org OpenPGP 0xD9D50D8A -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From nowhere.man at levallois.eu.org Tue Nov 21 05:40:10 2006 From: nowhere.man at levallois.eu.org (Pierre THIERRY) Date: Tue, 21 Nov 2006 06:40:10 +0100 Subject: [elephant-devel] Serialization In-Reply-To: <4561C218.7050606@csail.mit.edu> References: <4561C218.7050606@csail.mit.edu> Message-ID: <20061121054010.GD24579@bateleur.arcanes.fr.eu.org> Scribit Ian Eslick dies 20/11/2006 hora 09:56: > 1) Special start-elephant-thread macro I'm not sure it is viable to put this kind of burden on the user. > 2) Bind a C-library which provides a CPU independent compare-and-swap > instruction and/or a lock-free queue so we can share a pool of > circularity buffers among processes in a lock-free way Why couldn't it be achieved in Lisp? Curiously, Pierre -- nowhere.man at levallois.eu.org OpenPGP 0xD9D50D8A -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From eslick at csail.mit.edu Tue Nov 21 06:23:00 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Tue, 21 Nov 2006 01:23:00 -0500 Subject: [elephant-devel] Pure-Lisp memutil? In-Reply-To: <20061121050718.GC24579@bateleur.arcanes.fr.eu.org> References: <20061120134650.GC8075@bateleur.arcanes.fr.eu.org> <4561C3E0.5060102@csail.mit.edu> <20061120225318.GA24579@bateleur.arcanes.fr.eu.org> <60DC2DDC-0E0C-487A-AD4A-19D8A68A3BFA@csail.mit.edu> <20061121050718.GC24579@bateleur.arcanes.fr.eu.org> Message-ID: <2AE6592C-F815-4FE4-92BF-638BDD848EF6@csail.mit.edu> That's a fair question. You are right that (safety 0) vs. (safety 1) has a small effect. Here's a simple example of some variations over settings. I modified memutil.lisp to use declaim instead of local declares, then I ran a simple test case that mostly uses buffer-write- byte and buffer-write-int setting that declaim to various values. Serialize and deserializing 10M long symbols (new serializer): (optimize (speed 3) (safety 1) (space 0) (debug 0)): 8.12 sec (optimize (speed 3) (safety 1) (space 0) (debug 0)): 8.13 sec (optimize (speed 3) (safety 2) (space 1) (debug 2)): 11.5 sec (optimize (speed 2) (safety 3) (space 1) (debug 3)): 28.7 sec (optimize (speed 0) (safety 3) (space 2) (debug 3)): 28.5 sec Same test on original serializer: (optimize (speed 3) (safety 1) (space 0) (debug 0)): 43 sec (optimize (speed 2) (safety 3) (space 1) (debug 3)): 426 sec (extrapolated from a 1M iteration run) This was generated by Allegro 8.0 on an Core 2 MacBook Pro under Mac OS X. The only real benefit to safety 0 for this code is removing the extra check to ensure that an integer op did not overflow the maximum fixnum. For this code there are enough other instructions my guess is that a modern CPU can execute these extra checks in parallel to other bookkeeping functions without any impact on throughput. It might make a difference if you were doing alot more math than base +offset style computations. Ian On Nov 21, 2006, at 12:07 AM, Pierre THIERRY wrote: > Scribit Ian Eslick dies 20/11/2006 hora 20:38: >> Can you expand on your thinking as to why the user may want to turn >> off optimizations and by what mechanism you would suggest doing >> that? > > I'm only really worried by the safety declarations. It should be a > choice from the user to reduce safety, even for well-tested code. > >> For production use the optimizations make sense. > > Why not let the user choose it when in production? > >> they're speed critical so it's reasonable to run them fast and you >> can always turn that off by the above methods. > > I can understand the rationale for declaring them so optimized. Are > there numbers about the speed difference with and without > optimizations? > > Curiously, > Pierre > -- > nowhere.man at levallois.eu.org > OpenPGP 0xD9D50D8A > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From eslick at csail.mit.edu Tue Nov 21 06:30:42 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Tue, 21 Nov 2006 01:30:42 -0500 Subject: [elephant-devel] Serialization In-Reply-To: <20061121054010.GD24579@bateleur.arcanes.fr.eu.org> References: <4561C218.7050606@csail.mit.edu> <20061121054010.GD24579@bateleur.arcanes.fr.eu.org> Message-ID: <748CBD5B-A824-41A8-B175-70F57629E962@csail.mit.edu> On Nov 21, 2006, at 12:40 AM, Pierre THIERRY wrote: > Scribit Ian Eslick dies 20/11/2006 hora 09:56: >> 1) Special start-elephant-thread macro > > I'm not sure it is viable to put this kind of burden on the user. > >> 2) Bind a C-library which provides a CPU independent compare-and-swap >> instruction and/or a lock-free queue so we can share a pool of >> circularity buffers among processes in a lock-free way > > Why couldn't it be achieved in Lisp? The Common Lisp spec doesn't expose this hardware-specific feature that most CPUs have. It turns out that most implementations have a non-standard call 'without-interrupts' that allow you, on a single CPU architecture, to suspend interrupts and create a low-cost critical section. That seemed the best tradeoff so that's what I've used in the serializer and memutil buffer pools. However this isn't nearly as robust, high performing or elegant as the compare-and-swap, hardware-guaranteed atomicity used in a lock- free algorithm. (Allegro has an undocumented function that I think does this, but it's not easily portable among implementations and it's tricky to get the compiler to generate it correctly and they recommended i not use it - too many opportunities to shoot oneself in the foot, I imagine) Once you are outside the specification, there isn't anything that's 'in lisp' so to speak. If I had a CPU-generic assembler with all the interesting hardware features and/or a compiler-macro-like facility that allowed me to specify anything supported by the hardware then I could do everything in lisp (only system to do this was Genera and Lisp Machines). However, without getting inside the compiler the only way I can get access to instructions/capabilities of the hardware not exposed by my implementation is to call out to a C or assembly routine that can/does. Fortunately I didn't have to do it in this case. Cheers, Ian From nowhere.man at levallois.eu.org Tue Nov 21 17:36:25 2006 From: nowhere.man at levallois.eu.org (Pierre THIERRY) Date: Tue, 21 Nov 2006 18:36:25 +0100 Subject: [elephant-devel] Pure-Lisp memutil? In-Reply-To: <2AE6592C-F815-4FE4-92BF-638BDD848EF6@csail.mit.edu> References: <20061120134650.GC8075@bateleur.arcanes.fr.eu.org> <4561C3E0.5060102@csail.mit.edu> <20061120225318.GA24579@bateleur.arcanes.fr.eu.org> <60DC2DDC-0E0C-487A-AD4A-19D8A68A3BFA@csail.mit.edu> <20061121050718.GC24579@bateleur.arcanes.fr.eu.org> <2AE6592C-F815-4FE4-92BF-638BDD848EF6@csail.mit.edu> Message-ID: <20061121173625.GE24579@bateleur.arcanes.fr.eu.org> Scribit Ian Eslick dies 21/11/2006 hora 01:23: > Here's a simple example of some variations over settings. Could you provide your test, I'd like to try some other combinations. In production use, I guess the most sensible setting will look like (speed 3) (safety 3) (space 0) (debug 0), except on a space limited system. Would you accept a patch with conditional evaluation around these declare forms? Something like #-elephant-without-optimize(declare (optimize ...)) So that it wouldn't change the default behaviour and still let the user choose. At least I would use it. ;-) Quickly, Pierre -- nowhere.man at levallois.eu.org OpenPGP 0xD9D50D8A -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From nowhere.man at levallois.eu.org Tue Nov 21 17:42:58 2006 From: nowhere.man at levallois.eu.org (Pierre THIERRY) Date: Tue, 21 Nov 2006 18:42:58 +0100 Subject: [elephant-devel] Bug#399739: ITP: cl-elephant -- object database for Common Lisp Message-ID: <20061121174258.GF24579@bateleur.arcanes.fr.eu.org> Package: wnpp Severity: wishlist Owner: Pierre THIERRY * Package name : cl-elephant Version : 0.6.0 Upstream Author : Ian Eslick * URL : http://common-lisp.net/project/elephant/ * License : LLGPL Programming Lang: Lisp/C Description : object database for Common Lisp Elephant is an object database for Common Lisp. It supports storing CLOS objects and most lisp primitives, and access to BTrees. It can employ either Berkeley DB or a wide variety of relational databases (through cl-sql). See -- System Information: Debian Release: 4.0 APT prefers testing APT policy: (990, 'testing'), (501, 'stable'), (500, 'unstable'), (500, 'stable') Architecture: i386 (i686) Shell: /bin/sh linked to /bin/bash Kernel: Linux 2.6.17-2-k7 Locale: LANG=fr_FR at euro, LC_CTYPE=fr_FR at euro (charmap=ISO-8859-15) -- nowhere.man at levallois.eu.org OpenPGP 0xD9D50D8A -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From nowhere.man at levallois.eu.org Tue Nov 21 23:46:05 2006 From: nowhere.man at levallois.eu.org (Pierre THIERRY) Date: Wed, 22 Nov 2006 00:46:05 +0100 Subject: [elephant-devel] Debian package available Message-ID: <20061121234604.GI24579@bateleur.arcanes.fr.eu.org> Hi, I finally built a Debian package for Elephant. It's not perfect yet, but it seems to be already a good shape enough to be usable. There could be one or two things to correct, that I discovered while packaging: 1 ) there are still some CVS dirs in the tarball, which I removed before doing the real packaging (so they won't appear in the debian source package). 2) I tried to just modify the Makefile and config.lisp, but changing *clsql-foreign-lib-path* in the latter had no effect. I only managed to successfully load the SQL backend if I modify ele-clsql.asd instead. I wonder if I didn't understood something or if it was intended that the change in config.lisp was all that's needed. Previously, I just asdf-loaded cl-sql before opening the store, so I wonder if the order in which things are done couldn't be changed to let the CL-SQL package deal with how and where it finds its libraries. With the package as it is, once installed, I ran the test suite with an SQLite 3 store and it all succeeded, in SBCL. I also tried with two stores, IIUC I have to set *default-spec* and *test-spec-primary* to the same store and *test-spec*secondary* to another. Then in INDEXING-TIMING, I get a bunch of repeated warnings about *test-spec-primary* and *test-spec-secondary* being undefined The packages, source and binary, can be fetched with the following sources: deb http://arcanes.fr.eu.org/~pierre/debian/ sid main deb-src http://arcanes.fr.eu.org/~pierre/debian/ sid main I have been proposed to join the cl-debian alioth project, and I hope one of its DD will be able to sponsor my package soon. Nightly, Pierre -- nowhere.man at levallois.eu.org OpenPGP 0xD9D50D8A -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From Alain.Picard at memetrics.com Tue Nov 21 23:36:49 2006 From: Alain.Picard at memetrics.com (Alain Picard) Date: Wed, 22 Nov 2006 10:36:49 +1100 Subject: [elephant-devel] Serialization In-Reply-To: <4561C218.7050606@csail.mit.edu> (Ian Eslick's message of "Mon, 20 Nov 2006 09:56:24 -0500") References: <4561C218.7050606@csail.mit.edu> Message-ID: <87hcwsjrse.fsf@memetrics.com> Ian Eslick writes: > Elephant users and developers, > > 1) Special start-elephant-thread macro which creates dynamic bindings > for the new process of a global special variable that maintains the > circularity buffer; this will allow us to be very efficient but requires > elephant-specific code on thread creation for safe threading As long as you document and export the special which needs to be bound, this is not a big deal; certainly, in my code, I already have specialized thread-starting functions which for the same purpose. But I'd much prefer having the special exported than the thread-making function. (Could have both, of course). This option seems like a win to me. > 2) Bind a C-library which provides a CPU independent compare-and-swap > instruction and/or a lock-free queue so we can share a pool of > circularity buffers among processes in a lock-free way This would have to be well hidden and transparent, but would be OK. > 3) It occurs to me that most lisps do not allow C code to be run > concurrently from multiple threads - perhaps just implementing a mutex > in C would be enough to provide a cheap way of updating head and tail > pointers in the queue. Capturing and releasing would be atomic. Is > this true for all lisps? I know it's true for Allegro. It would be a > hack, but a fairly harmless one given our existing dependence on the > memutil library. It's pretty clear that SBCL is aiming for a full multicore support, and the commercial vendors will have to follow suit; they'll be dead in the water when 4/8 core cpus are commonplace and their lisps impose stringent restrictions on the programmer. So I'd say it's best to not make assumptions based on what implementations "currently do". > 4) Stick with the portable locking approach already implemented Doesn't seem good enough -- the point of a multithread lisp and berkeley DB is to allow concurrent access to the data. Serializing access... well... the cure is worse than the disease. > Thoughts? Just my 2 cents; it's your show. :-) --ap From nowhere.man at levallois.eu.org Wed Nov 22 00:04:09 2006 From: nowhere.man at levallois.eu.org (Pierre THIERRY) Date: Wed, 22 Nov 2006 01:04:09 +0100 Subject: [elephant-devel] Serialization In-Reply-To: <87hcwsjrse.fsf@memetrics.com> References: <4561C218.7050606@csail.mit.edu> <87hcwsjrse.fsf@memetrics.com> Message-ID: <20061122000409.GJ24579@bateleur.arcanes.fr.eu.org> Scribit Alain Picard dies 22/11/2006 hora 10:36: > > 4) Stick with the portable locking approach already implemented > Doesn't seem good enough -- the point of a multithread lisp and > berkeley DB is to allow concurrent access to the data. I think it is always best to aim at correctness first, then measure times, and only decide to optimize when measurements clearly indicate a problem. It is even more important with MT, because bugs are harder to reproduce, so, if the current implementation proves to be correct, let's keep it for the moment, and profile it to see where it could be made faster if needed. Theoretically, Pierre -- nowhere.man at levallois.eu.org OpenPGP 0xD9D50D8A -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From eslick at csail.mit.edu Wed Nov 22 00:18:44 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Tue, 21 Nov 2006 19:18:44 -0500 Subject: [elephant-devel] Serialization In-Reply-To: <20061122000409.GJ24579@bateleur.arcanes.fr.eu.org> References: <4561C218.7050606@csail.mit.edu> <87hcwsjrse.fsf@memetrics.com> <20061122000409.GJ24579@bateleur.arcanes.fr.eu.org> Message-ID: <3B0AF974-F20E-4A81-8BCD-89E78F210C8E@csail.mit.edu> The without-interrupts already fixed my problem and improved performance 4x or so. It has identical semantics to locking but avoids the OS overheads of traditional lisp locks. There's a nice discussion of all those issues here: If you would like to implement your optional safety compilation solution; there shouldn't be much conflicts with the changes I'm in the process of making (hope to checkin a whole set of features and bug fixes over the weekend) other than in memutil.lisp so feel free to go ahead and clean things up. I have no objections to the proposal you sent out earlier (i.e., extra *feature* and reader conditionals on optimizations) The routine I used to do timings was pretty trivial (paraphrased): (time (let ((bs (elephant-memutil::grab-buffer-stream))) (loop for i from 0 upto 10000000 do (reset-buffer-stream bs) (serialize 'this-is-a-test-symbol bs nil) (assert (eq 'this-is-a-test-symbol (deserialize bs)))))) I was looking at both the new and old serializers (I've kept the old around for migration purposes). Regards, Ian On Nov 21, 2006, at 7:04 PM, Pierre THIERRY wrote: > Scribit Alain Picard dies 22/11/2006 hora 10:36: >>> 4) Stick with the portable locking approach already implemented >> Doesn't seem good enough -- the point of a multithread lisp and >> berkeley DB is to allow concurrent access to the data. > > I think it is always best to aim at correctness first, then measure > times, and only decide to optimize when measurements clearly > indicate a > problem. > > It is even more important with MT, because bugs are harder to > reproduce, > so, if the current implementation proves to be correct, let's keep it > for the moment, and profile it to see where it could be made faster if > needed. > > Theoretically, > Pierre > -- > nowhere.man at levallois.eu.org > OpenPGP 0xD9D50D8A > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From nowhere.man at levallois.eu.org Wed Nov 22 01:52:10 2006 From: nowhere.man at levallois.eu.org (Pierre THIERRY) Date: Wed, 22 Nov 2006 02:52:10 +0100 Subject: [elephant-devel] Serialization In-Reply-To: <748CBD5B-A824-41A8-B175-70F57629E962@csail.mit.edu> References: <4561C218.7050606@csail.mit.edu> <20061121054010.GD24579@bateleur.arcanes.fr.eu.org> <748CBD5B-A824-41A8-B175-70F57629E962@csail.mit.edu> Message-ID: <20061122015210.GM24579@bateleur.arcanes.fr.eu.org> Scribit Ian Eslick dies 21/11/2006 hora 01:30: > It turns out that most implementations have a non-standard call > 'without-interrupts' that allow you, on a single CPU architecture, to > suspend interrupts and create a low-cost critical section. Well, after some reading on it, I understand that running code with interrupts disabled has a big hit on performance, for multiple CPUs architectures, that CAS has not. How do you intend to differentiate architectures that provide CAS? Lock-freely, Pierre -- nowhere.man at levallois.eu.org OpenPGP 0xD9D50D8A -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From nowhere.man at levallois.eu.org Wed Nov 22 01:34:35 2006 From: nowhere.man at levallois.eu.org (Pierre THIERRY) Date: Wed, 22 Nov 2006 02:34:35 +0100 Subject: [elephant-devel] Serialization In-Reply-To: <3B0AF974-F20E-4A81-8BCD-89E78F210C8E@csail.mit.edu> References: <4561C218.7050606@csail.mit.edu> <87hcwsjrse.fsf@memetrics.com> <20061122000409.GJ24579@bateleur.arcanes.fr.eu.org> <3B0AF974-F20E-4A81-8BCD-89E78F210C8E@csail.mit.edu> Message-ID: <20061122013435.GL24579@bateleur.arcanes.fr.eu.org> Scribit Ian Eslick dies 21/11/2006 hora 19:18: > I have no objections to the proposal you sent out earlier (i.e., > extra *feature* and reader conditionals on optimizations) I made the optimize form conditional everywhere it appears, here is the patch for HEAD. Quickly, Pierre -- nowhere.man at levallois.eu.org OpenPGP 0xD9D50D8A -------------- next part -------------- Index: src/db-acache/acache-controller.lisp =================================================================== RCS file: /project/elephant/cvsroot/elephant/src/db-acache/acache-controller.lisp,v retrieving revision 1.2 diff -u -r1.2 acache-controller.lisp --- src/db-acache/acache-controller.lisp 21 Feb 2006 19:40:02 -0000 1.2 +++ src/db-acache/acache-controller.lisp 22 Nov 2006 01:36:04 -0000 @@ -74,23 +74,23 @@ `(rplacd (rplaca *index-cons* ,oid) ,name)) (defmethod persistent-slot-reader ((sc acache-store-controller) instance name) - (declare (optimize (speed 3) (safety 1))) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 1))) (multiple-value-bind (val valid?) (map-value (controller-slots sc) (fast-key (oid instance) name)) (if valid? val (error "Slot ~A unbound in ~A" name instance)))) (defmethod persistent-slot-writer ((sc acache-store-controller) value instance name) - (declare (optimize (speed 3) (safety 1))) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 1))) (setf (map-value (controller-slots sc) (fast-key (oid instance) name)) value)) (defmethod persistent-slot-boundp ((sc acache-store-controller) instance name) - (declare (optimize (speed 3) (safety 1))) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 1))) (when (map-value (controller-slots sc) (fast-key (oid instance) name)) t)) (defmethod persistent-slot-makunbound ((sc acache-store-controller) instance name) - (declare (optimize (speed 3) (safety 1))) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 1))) (remove-from-map (controller-slots sc) (fast-key (oid instance) name))) Index: src/db-bdb/bdb-collections.lisp =================================================================== RCS file: /project/elephant/cvsroot/elephant/src/db-bdb/bdb-collections.lisp,v retrieving revision 1.10 diff -u -r1.10 bdb-collections.lisp --- src/db-bdb/bdb-collections.lisp 11 Nov 2006 18:41:10 -0000 1.10 +++ src/db-bdb/bdb-collections.lisp 22 Nov 2006 01:36:04 -0000 @@ -32,7 +32,7 @@ (make-instance 'bdb-btree :sc sc)) (defmethod get-value (key (bt bdb-btree)) - (declare (optimize (speed 3) (space 0) (safety 0))) + (declare #-elephant-without-optimize(optimize (speed 3) (space 0) (safety 0))) (let ((sc (get-con bt))) (with-buffer-streams (key-buf value-buf) (buffer-write-int (oid bt) key-buf) @@ -43,7 +43,7 @@ (values nil nil)))))) (defmethod existsp (key (bt bdb-btree)) - (declare (optimize (speed 3) (safety 0) (space 0))) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0) (space 0))) (with-buffer-streams (key-buf value-buf) (buffer-write-int (oid bt) key-buf) (serialize key key-buf) @@ -55,7 +55,7 @@ (defmethod (setf get-value) (value key (bt bdb-btree)) - (declare (optimize (speed 3) (safety 0) (space 0))) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0) (space 0))) (assert (or *auto-commit* (not (eq *current-transaction* 0)))) ;; (with-transaction (:store-controller (get-con bt)) (with-buffer-streams (key-buf value-buf) @@ -68,7 +68,7 @@ value)) (defmethod remove-kv (key (bt bdb-btree)) - (declare (optimize (speed 3) (space 0) (safety 0))) + (declare #-elephant-without-optimize(optimize (speed 3) (space 0) (safety 0))) (assert (or *auto-commit* (not (eq *current-transaction* 0)))) ;; (with-transaction (:store-controller (get-con bt)) (with-buffer-streams (key-buf) @@ -198,7 +198,7 @@ (defmethod remove-kv (key (bt bdb-indexed-btree)) "Remove a key / value pair, and update secondary indices." - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) (let ((sc (get-con bt))) (with-buffer-streams (key-buf secondary-buf) (buffer-write-int (oid bt) key-buf) @@ -234,7 +234,7 @@ (defmethod get-value (key (bt bdb-btree-index)) "Get the value in the primary DB from a secondary key." - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) (with-buffer-streams (key-buf value-buf) (buffer-write-int (oid bt) key-buf) (serialize key key-buf) @@ -245,7 +245,7 @@ (values nil nil))))) (defmethod get-primary-key (key (bt btree-index)) - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) (with-buffer-streams (key-buf value-buf) (buffer-write-int (oid bt) key-buf) (serialize key key-buf) @@ -263,19 +263,19 @@ (defmethod make-cursor ((bt bdb-btree)) "Make a cursor from a btree." - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) (make-instance 'bdb-cursor :btree bt :handle (db-cursor (controller-btrees (get-con bt))) :oid (oid bt))) (defmethod cursor-close ((cursor bdb-cursor)) - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) (db-cursor-close (cursor-handle cursor)) (setf (cursor-initialized-p cursor) nil)) (defmethod cursor-duplicate ((cursor bdb-cursor)) - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) (make-instance (type-of cursor) :initialized-p (cursor-initialized-p cursor) :oid (cursor-oid cursor) @@ -284,7 +284,7 @@ :position (cursor-initialized-p cursor)))) (defmethod cursor-current ((cursor bdb-cursor)) - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) (when (cursor-initialized-p cursor) (with-buffer-streams (key-buf value-buf) (multiple-value-bind (key val) @@ -299,7 +299,7 @@ (setf (cursor-initialized-p cursor) nil)))))) (defmethod cursor-first ((cursor bdb-cursor)) - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) (with-buffer-streams (key-buf value-buf) (buffer-write-int (cursor-oid cursor) key-buf) (multiple-value-bind (key val) @@ -315,7 +315,7 @@ ;;A bit of a hack..... (defmethod cursor-last ((cursor bdb-cursor)) - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) (with-buffer-streams (key-buf value-buf) (buffer-write-int (+ (cursor-oid cursor) 1) key-buf) (if (db-cursor-set-buffered (cursor-handle cursor) @@ -346,7 +346,7 @@ (setf (cursor-initialized-p cursor) nil)))))) (defmethod cursor-next ((cursor bdb-cursor)) - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) (if (cursor-initialized-p cursor) (with-buffer-streams (key-buf value-buf) (multiple-value-bind (key val) @@ -359,7 +359,7 @@ (cursor-first cursor))) (defmethod cursor-prev ((cursor bdb-cursor)) - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) (if (cursor-initialized-p cursor) (with-buffer-streams (key-buf value-buf) (multiple-value-bind (key val) @@ -372,7 +372,7 @@ (cursor-last cursor))) (defmethod cursor-set ((cursor bdb-cursor) key) - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) (with-buffer-streams (key-buf value-buf) (buffer-write-int (cursor-oid cursor) key-buf) (serialize key key-buf) @@ -385,7 +385,7 @@ (setf (cursor-initialized-p cursor) nil))))) (defmethod cursor-set-range ((cursor bdb-cursor) key) - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) (with-buffer-streams (key-buf value-buf) (buffer-write-int (cursor-oid cursor) key-buf) (serialize key key-buf) @@ -399,7 +399,7 @@ (setf (cursor-initialized-p cursor) nil))))) (defmethod cursor-get-both ((cursor bdb-cursor) key value) - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) (with-buffer-streams (key-buf value-buf) (buffer-write-int (cursor-oid cursor) key-buf) (serialize key key-buf) @@ -414,7 +414,7 @@ (setf (cursor-initialized-p cursor) nil))))) (defmethod cursor-get-both-range ((cursor bdb-cursor) key value) - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) (with-buffer-streams (key-buf value-buf) (buffer-write-int (cursor-oid cursor) key-buf) (serialize key key-buf) @@ -428,7 +428,7 @@ (setf (cursor-initialized-p cursor) nil))))) (defmethod cursor-delete ((cursor bdb-cursor)) - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) (if (cursor-initialized-p cursor) (with-buffer-streams (key-buf value-buf) (multiple-value-bind (key val) @@ -447,7 +447,7 @@ "Put by cursor. Not particularly useful since primaries don't support duplicates. Currently doesn't properly move the cursor." - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) (if key-specified-p (setf (get-value key (cursor-btree cursor)) value) (if (cursor-initialized-p cursor) @@ -471,7 +471,7 @@ (defmethod make-cursor ((bt bdb-btree-index)) "Make a secondary-cursor from a secondary index." - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) (make-instance 'bdb-secondary-cursor :btree bt :handle (db-cursor @@ -480,7 +480,7 @@ (defmethod cursor-pcurrent ((cursor bdb-secondary-cursor)) - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) (when (cursor-initialized-p cursor) (with-buffer-streams (key-buf pkey-buf value-buf) (multiple-value-bind (key pkey val) @@ -500,7 +500,7 @@ (setf (cursor-initialized-p cursor) nil)))))) (defmethod cursor-pfirst ((cursor bdb-secondary-cursor)) - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) (with-buffer-streams (key-buf pkey-buf value-buf) (buffer-write-int (cursor-oid cursor) key-buf) (multiple-value-bind (key pkey val) @@ -516,7 +516,7 @@ ;;A bit of a hack..... (defmethod cursor-plast ((cursor bdb-secondary-cursor)) - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) (with-buffer-streams (key-buf pkey-buf value-buf) (buffer-write-int (+ (cursor-oid cursor) 1) key-buf) (if (db-cursor-set-buffered (cursor-handle cursor) @@ -547,7 +547,7 @@ (setf (cursor-initialized-p cursor) nil)))))) (defmethod cursor-pnext ((cursor bdb-secondary-cursor)) - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) (if (cursor-initialized-p cursor) (with-buffer-streams (key-buf pkey-buf value-buf) (multiple-value-bind (key pkey val) @@ -563,7 +563,7 @@ (cursor-pfirst cursor))) (defmethod cursor-pprev ((cursor bdb-secondary-cursor)) - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) (if (cursor-initialized-p cursor) (with-buffer-streams (key-buf pkey-buf value-buf) (multiple-value-bind (key pkey val) @@ -579,7 +579,7 @@ (cursor-plast cursor))) (defmethod cursor-pset ((cursor bdb-secondary-cursor) key) - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) (with-buffer-streams (key-buf pkey-buf value-buf) (buffer-write-int (cursor-oid cursor) key-buf) (serialize key key-buf) @@ -593,7 +593,7 @@ (setf (cursor-initialized-p cursor) nil))))) (defmethod cursor-pset-range ((cursor bdb-secondary-cursor) key) - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) (with-buffer-streams (key-buf pkey-buf value-buf) (buffer-write-int (cursor-oid cursor) key-buf) (serialize key key-buf) @@ -608,7 +608,7 @@ (setf (cursor-initialized-p cursor) nil))))) (defmethod cursor-pget-both ((cursor bdb-secondary-cursor) key pkey) - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) (with-buffer-streams (key-buf pkey-buf value-buf) (let ((primary-oid (oid (primary (cursor-btree cursor))))) (buffer-write-int (cursor-oid cursor) key-buf) @@ -625,7 +625,7 @@ (setf (cursor-initialized-p cursor) nil)))))) (defmethod cursor-pget-both-range ((cursor bdb-secondary-cursor) key pkey) - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) (with-buffer-streams (key-buf pkey-buf value-buf) (let ((primary-oid (oid (primary (cursor-btree cursor))))) (buffer-write-int (cursor-oid cursor) key-buf) @@ -643,7 +643,7 @@ (defmethod cursor-delete ((cursor bdb-secondary-cursor)) "Delete by cursor: deletes ALL secondary indices." - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) (if (cursor-initialized-p cursor) (with-buffer-streams (key-buf pkey-buf value-buf) (multiple-value-bind (key pkey val) @@ -681,7 +681,7 @@ (error "Puts are forbidden on secondary indices. Try adding to the primary.")) (defmethod cursor-next-dup ((cursor bdb-secondary-cursor)) - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) (when (cursor-initialized-p cursor) (with-buffer-streams (key-buf value-buf) (multiple-value-bind (key val) @@ -693,7 +693,7 @@ (setf (cursor-initialized-p cursor) nil)))))) (defmethod cursor-next-nodup ((cursor bdb-secondary-cursor)) - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) (if (cursor-initialized-p cursor) (with-buffer-streams (key-buf value-buf) (multiple-value-bind (key val) @@ -706,7 +706,7 @@ (cursor-first cursor))) (defmethod cursor-prev-nodup ((cursor bdb-secondary-cursor)) - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) (if (cursor-initialized-p cursor) (with-buffer-streams (key-buf value-buf) (multiple-value-bind (key val) @@ -719,7 +719,7 @@ (cursor-last cursor))) (defmethod cursor-pnext-dup ((cursor bdb-secondary-cursor)) - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) (when (cursor-initialized-p cursor) (with-buffer-streams (key-buf pkey-buf value-buf) (multiple-value-bind (key pkey val) @@ -732,7 +732,7 @@ (setf (cursor-initialized-p cursor) nil)))))) (defmethod cursor-pnext-nodup ((cursor bdb-secondary-cursor)) - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) (if (cursor-initialized-p cursor) (with-buffer-streams (key-buf pkey-buf value-buf) (multiple-value-bind (key pkey val) @@ -746,7 +746,7 @@ (cursor-pfirst cursor))) (defmethod cursor-pprev-nodup ((cursor bdb-secondary-cursor)) - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) (if (cursor-initialized-p cursor) (with-buffer-streams (key-buf pkey-buf value-buf) (multiple-value-bind (key pkey val) Index: src/db-bdb/bdb-controller.lisp =================================================================== RCS file: /project/elephant/cvsroot/elephant/src/db-bdb/bdb-controller.lisp,v retrieving revision 1.13 diff -u -r1.13 bdb-controller.lisp --- src/db-bdb/bdb-controller.lisp 11 Nov 2006 18:41:10 -0000 1.13 +++ src/db-bdb/bdb-controller.lisp 22 Nov 2006 01:36:04 -0000 @@ -232,7 +232,7 @@ ;; (defmethod persistent-slot-reader ((sc bdb-store-controller) instance name) -;; (declare (optimize (speed 3) (safety 1) (space 1))) +;; (declare #-elephant-without-optimize(optimize (speed 3) (safety 1) (space 1))) (with-buffer-streams (key-buf value-buf) (buffer-write-int (oid instance) key-buf) (serialize name key-buf) @@ -245,7 +245,7 @@ (error 'unbound-slot :instance instance :name name))))) (defmethod persistent-slot-writer ((sc bdb-store-controller) new-value instance name) -;; (declare (optimize (speed 3) (safety 1) (space 1))) +;; (declare #-elephant-without-optimize(optimize (speed 3) (safety 1) (space 1))) ;; (format t "psw -- sc: ~A ct: ~A ac: ~A~%" *store-controller* *current-transaction* *auto-commit*) (with-buffer-streams (key-buf value-buf) (buffer-write-int (oid instance) key-buf) @@ -258,7 +258,7 @@ new-value)) (defmethod persistent-slot-boundp ((sc bdb-store-controller) instance name) -;; (declare (optimize (speed 3) (safety 1) (space 1))) +;; (declare #-elephant-without-optimize(optimize (speed 3) (safety 1) (space 1))) (with-buffer-streams (key-buf value-buf) (buffer-write-int (oid instance) key-buf) (serialize name key-buf) @@ -267,7 +267,7 @@ (if buf t nil)))) (defmethod persistent-slot-makunbound ((sc bdb-store-controller) instance name) -;; (declare (optimize (speed 3) (safety 1) (space 1))) +;; (declare #-elephant-without-optimize(optimize (speed 3) (safety 1) (space 1))) (with-buffer-streams (key-buf) (buffer-write-int (oid instance) key-buf) (serialize name key-buf) Index: src/db-bdb/berkeley-db.lisp =================================================================== RCS file: /project/elephant/cvsroot/elephant/src/db-bdb/berkeley-db.lisp,v retrieving revision 1.1 diff -u -r1.1 berkeley-db.lisp --- src/db-bdb/berkeley-db.lisp 11 Nov 2006 18:43:31 -0000 1.1 +++ src/db-bdb/berkeley-db.lisp 22 Nov 2006 01:36:04 -0000 @@ -607,7 +607,7 @@ a buffer-stream. Space for the value is passed in as a buffer-stream. On success the buffer-stream is returned for decoding, or NIL if nothing was found." - (declare (optimize (speed 3) (safety 0)) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type pointer-void db transaction) (type buffer-stream key-buffer-stream value-buffer-stream) (type boolean auto-commit get-both degree-2 read-committed dirty-read read-uncommitted)) @@ -658,7 +658,7 @@ string. Space for the value is passed in as a buffer-stream. On success the buffer-stream is returned for decoding, or NIL if nothing was found." - (declare (optimize (speed 3) (safety 0)) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type pointer-void db transaction) (type string key) (type buffer-stream value-buffer-stream) @@ -698,7 +698,7 @@ "Get a key / value pair from a DB. The key is passed as a string, and the value is returned as a string. If nothing is found, NIL is returned." - (declare (optimize (speed 3) (safety 0)) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type pointer-void db transaction) (type string key) (type fixnum key-size) @@ -749,7 +749,7 @@ "Put a key / value pair into a DB. The pair are encoded in buffer-streams. T on success, or nil if the key already exists and EXISTS-ERROR-P is NIL." - (declare (optimize (speed 3) (safety 0)) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type pointer-void db transaction) (type buffer-stream key-buffer-stream value-buffer-stream) (type boolean auto-commit exists-error-p)) @@ -784,7 +784,7 @@ (value-size (length value)) (transaction *current-transaction*)) :cstrings (key value) - :declarations (declare (optimize (speed 3) (safety 0)) + :declarations (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type pointer-void db transaction) (type string key value) (type fixnum key-size value-size) @@ -806,7 +806,7 @@ "Delete a key / value pair from a DB. The key is encoded in a buffer-stream. T on success, NIL if the key wasn't found." - (declare (optimize (speed 3) (safety 0)) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type pointer-void db transaction) (type buffer-stream key-buffer-stream) (type boolean auto-commit)) @@ -836,7 +836,7 @@ (transaction *current-transaction*)) "Delete a key / value pair from a DB. The key is a string. T on success, NIL if the key wasn't found." - (declare (optimize (speed 3) (safety 0)) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type pointer-void db transaction) (type string key) (type fixnum key-size) (type boolean auto-commit)) (with-cstrings ((key key)) @@ -868,7 +868,7 @@ duplicates. The key and value are encoded as buffer-streams. T on success, NIL if the key / value pair wasn't found." - (declare (optimize (speed 3) (safety 0)) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type pointer-void db transaction) (type buffer-stream key-buffer-stream value-buffer-stream)) (let ((errno (%db-delete-kv db transaction @@ -903,7 +903,7 @@ (defun db-compact (db start stop end &key (transaction *current-transaction*) freelist-only free-space) - (declare (optimize (speed 3) (safety 2)) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 2)) (type pointer-void db transaction) (type buffer-stream start stop) (type boolean freelist-only free-space)) @@ -943,7 +943,7 @@ (defun db-cursor (db &key (transaction *current-transaction*) degree-2 read-committed dirty-read read-uncommitted) "Create a cursor." - (declare (optimize (speed 3) (safety 0)) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type pointer-void db) (type boolean degree-2 read-committed dirty-read read-uncommitted) (type pointer-int *errno-buffer*)) @@ -969,7 +969,7 @@ (defun db-cursor-delete (cursor) "Delete by cursor." - (declare (optimize (speed 3) (safety 0)) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type pointer-void cursor)) (let ((errno (%db-cursor-delete cursor 0))) (declare (type fixnum errno)) @@ -990,7 +990,7 @@ (defun db-cursor-duplicate (cursor &key (position t)) "Duplicate a cursor." - (declare (optimize (speed 3) (safety 0)) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type pointer-void cursor)) (let* ((newc (%db-cursor-dup cursor (flags :position position) *errno-buffer*)) @@ -1021,7 +1021,7 @@ "Move a cursor, returning the key / value pair found. Supports current, first, last, next, next-dup, next-nodup, prev, prev-nodup." - (declare (optimize (speed 3) (safety 0)) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type pointer-void cursor) (type buffer-stream key-buffer-stream value-buffer-stream) (type boolean current first last next next-dup next-nodup prev @@ -1067,7 +1067,7 @@ &key set set-range dirty-read read-uncommitted) "Move a cursor to a key, returning the key / value pair found. Supports set and set-range." - (declare (optimize (speed 3) (safety 0)) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type pointer-void cursor) (type buffer-stream key-buffer-stream value-buffer-stream) (type boolean set set-range dirty-read read-uncommitted)) @@ -1108,7 +1108,7 @@ &key get-both get-both-range dirty-read read-uncommitted) "Move a cursor to a key / value pair, returning the key / value pair found. Supports get-both and get-both-range." - (declare (optimize (speed 3) (safety 0)) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type pointer-void cursor) (type buffer-stream key-buffer-stream value-buffer-stream) (type boolean get-both get-both-range dirty-read read-uncommitted)) @@ -1170,7 +1170,7 @@ "Move a secondary cursor, returning the key / value / primary triple found. Supports current, first, last, next, next-dup, next-nodup, prev, prev-nodup." - (declare (optimize (speed 3) (safety 0)) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type pointer-void cursor) (type buffer-stream key-buffer-stream pkey-buffer-stream value-buffer-stream) @@ -1226,7 +1226,7 @@ &key set set-range dirty-read) "Move a secondary cursor tp a key, returning the key / value / primary triple found. Supports set, set-range." - (declare (optimize (speed 3) (safety 0)) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type pointer-void cursor) (type buffer-stream key-buffer-stream pkey-buffer-stream value-buffer-stream) @@ -1278,7 +1278,7 @@ "Move a secondary cursor tp a key / primary pair, returning the key / value / primary triple found. Supports get, get-range." - (declare (optimize (speed 3) (safety 0)) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type pointer-void cursor) (type buffer-stream key-buffer-stream pkey-buffer-stream value-buffer-stream) @@ -1336,7 +1336,7 @@ &key after before current keyfirst keylast no-dup-data exists-error-p) "Put by cursor. The key and value are encoded as buffer-streams." - (declare (optimize (speed 3) (safety 0)) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type pointer-void cursor) (type buffer-stream key-buffer-stream value-buffer-stream) (type boolean after before current keyfirst keylast no-dup-data @@ -1375,7 +1375,7 @@ degree-2 read-committed dirty-read read-uncommitted txn-nosync txn-nowait txn-sync) "Start a transaction. Transactions may be nested." - (declare (optimize (speed 3) (safety 0)) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type pointer-void env parent) (type boolean degree-2 read-committed dirty-read read-uncommitted txn-nosync txn-nowait txn-sync) @@ -1401,7 +1401,7 @@ (wrap-errno (db-transaction-abort %db-txn-abort) (transaction) :keys ((transaction *current-transaction*)) - :declarations (declare (optimize (speed 3) (safety 0)) + :declarations (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type pointer-void transaction)) :documentation "Abort a transaction.") @@ -1413,7 +1413,7 @@ (wrap-errno (db-transaction-commit %db-txn-commit) (transaction flags) :keys ((transaction *current-transaction*)) :flags (txn-nosync txn-sync) - :declarations (declare (optimize (speed 3) (safety 0)) + :declarations (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type pointer-void transaction) (type boolean txn-nosync txn-sync)) :documentation "Commit a transaction.") @@ -1513,7 +1513,7 @@ (defun db-transaction-id (&optional (transaction *current-transaction*)) "Returns the ID of the transaction (for locking purposes.)" - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) (%db-transaction-id transaction)) (def-function ("db_env_lock_id" %db-env-lock-id) @@ -1703,7 +1703,7 @@ (defun db-sequence-create (db) "Create a new sequence." - (declare (optimize (speed 3) (safety 0)) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type pointer-void db) (type pointer-int *errno-buffer*)) (let* ((seq @@ -1751,7 +1751,7 @@ (defun db-sequence-get (sequence delta &key auto-commit txn-nosync (transaction *current-transaction*)) "Get the next element." - (declare (optimize (speed 3) (safety 0)) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type pointer-void sequence transaction) (type fixnum delta) (type boolean auto-commit txn-nosync)) @@ -1780,7 +1780,7 @@ (defun db-sequence-get-fixnum (sequence delta &key auto-commit txn-nosync (transaction *current-transaction*)) "Get the next element as a fixnum." - (declare (optimize (speed 3) (safety 0)) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type pointer-void sequence transaction) (type fixnum delta) (type boolean auto-commit txn-nosync)) Index: src/db-clsql/sql-collections.lisp =================================================================== RCS file: /project/elephant/cvsroot/elephant/src/db-clsql/sql-collections.lisp,v retrieving revision 1.6 diff -u -r1.6 sql-collections.lisp --- src/db-clsql/sql-collections.lisp 11 Nov 2006 18:41:11 -0000 1.6 +++ src/db-clsql/sql-collections.lisp 22 Nov 2006 01:36:04 -0000 @@ -21,7 +21,7 @@ (defmethod get-value (key (bt sql-btree-index)) "Get the value in the primary DB from a secondary key." - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) ;; Below, the take the oid and add it to the key, then look ;; thing up--- where? @@ -35,7 +35,7 @@ ))) (defmethod get-primary-key (key (bt sql-btree-index)) - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) (let* ((sc (get-con bt)) (con (controller-db sc)) ) @@ -53,7 +53,7 @@ (defmethod make-cursor ((bt sql-btree)) "Make a cursor from a btree." - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) (make-instance 'sql-cursor :btree bt :oid (oid bt))) @@ -68,7 +68,7 @@ ;; I'm not sure what cursor-duplicate is meant to do, and if ;; the other state needs to be copied or now. Probably soo... (defmethod cursor-duplicate ((cursor sql-cursor)) - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) (make-instance (type-of cursor) :initialized-p (cursor-initialized-p cursor) :oid (cursor-oid cursor) @@ -80,7 +80,7 @@ ;; :position (cursor-initialized-p cursor)))) (defmethod cursor-current ((cursor sql-cursor)) - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) (when (cursor-initialized-p cursor) (has-key-value cursor))) @@ -168,7 +168,7 @@ (defmethod cursor-first ((cursor sql-cursor)) - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) ;; Read all of the keys... ;; We need to get the contoller db from the btree somehow... (cursor-init cursor) @@ -197,7 +197,7 @@ (cursor-first cursor))) (defmethod cursor-prev ((cursor sql-cursor)) - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) (if (cursor-initialized-p cursor) (progn (decf (:sql-crsr-ck cursor)) @@ -205,7 +205,7 @@ (cursor-last cursor))) (defmethod cursor-set ((cursor sql-cursor) key) - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) (if (cursor-initialized-p cursor) (let ((p (position key (:sql-crsr-ks cursor) :test #'equal))) (if p @@ -228,7 +228,7 @@ (defmethod cursor-set-range ((cursor sql-cursor) key) - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) ;; I'm a little fuzzy on when I should leave a cursor in ;; the initialized state... (unless (cursor-initialized-p cursor) @@ -254,7 +254,7 @@ (defmethod cursor-get-both ((cursor sql-cursor) key value) - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) (let* ((bt (cursor-btree cursor)) (v (get-value key bt))) (if (equal v value) @@ -265,7 +265,7 @@ ;; This needs to be rewritten! (defmethod cursor-get-both-range ((cursor sql-cursor) key value) - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) (let* ((bt (cursor-btree cursor)) (v (get-value key bt))) ;; Since we don't allow duplicates in primary cursors, I @@ -278,7 +278,7 @@ (defmethod cursor-delete ((cursor sql-cursor)) - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) (if (cursor-initialized-p cursor) (multiple-value-bind (has k v) @@ -294,7 +294,7 @@ "Put by cursor. Not particularly useful since primaries don't support duplicates. Currently doesn't properly move the cursor." - (declare (optimize (speed 3)) + (declare #-elephant-without-optimize(optimize (speed 3)) (ignore key value key-specified-p)) (error "Puts on sql-cursors are not yet implemented, because I can't get them to work on BDB cursors!")) @@ -308,7 +308,7 @@ (defmethod make-cursor ((bt sql-btree-index)) "Make a secondary-cursor from a secondary index." - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) (make-instance 'sql-secondary-cursor :btree bt :oid (oid bt))) @@ -359,7 +359,7 @@ (cursor-prev-x cursor :returnpk t)) (defmethod cursor-pset ((cursor sql-secondary-cursor) key) - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) (unless (cursor-initialized-p cursor) (cursor-init cursor)) (let ((idx (position key (:sql-crsr-ks cursor)))) @@ -381,7 +381,7 @@ ) (defmethod cursor-pset-range ((cursor sql-secondary-cursor) key) - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) (unless (cursor-initialized-p cursor) (cursor-init cursor)) (let ((idx (array-index-if #'(lambda (x) (my-generic-at-most key x)) (:sql-crsr-ks cursor)))) @@ -399,7 +399,7 @@ ;; with secondary key equal to the key argument, and primary key greater or equal to the pkey argument. ;; Returns has-tuple / secondary key / value / primary key. (defmethod cursor-pget-both ((cursor sql-secondary-cursor) key pkey) - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) ;; It's better to get the value by the primary key, ;; as that is unique.. (let* ((bt (primary (cursor-btree cursor))) @@ -424,7 +424,7 @@ (cursor-un-init cursor :returnpk t)))) (defmethod cursor-pget-both-range ((cursor sql-secondary-cursor) key pkey) - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) ;; It's better to get the value by the primary key, ;; as that is unique.. (do ((vs @@ -452,7 +452,7 @@ (defmethod cursor-delete ((cursor sql-secondary-cursor)) "Delete by cursor: deletes ALL secondary indices." - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) (if (cursor-initialized-p cursor) (multiple-value-bind (m k v p) @@ -498,7 +498,7 @@ ) (defmethod cursor-first-x ((cursor sql-secondary-cursor) &key (returnpk nil)) - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) (setf (:dp-nmbr cursor) 0) (cursor-init cursor) (has-key-value-scnd cursor :returnpk returnpk) @@ -523,7 +523,7 @@ (cursor-prev-x cursor) ) (defmethod cursor-prev-x ((cursor sql-secondary-cursor) &key (returnpk nil)) - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) (if (cursor-initialized-p cursor) (progn (let ((cur-pk (get-current-key cursor))) @@ -548,7 +548,7 @@ ) (defmethod cursor-next-dup-x ((cursor sql-secondary-cursor) &key (returnpk nil)) -;; (declare (optimize (speed 3))) +;; (declare #-elephant-without-optimize(optimize (speed 3))) (when (cursor-initialized-p cursor) (let* ((cur-pk (aref (:sql-crsr-ks cursor) (:sql-crsr-ck cursor))) @@ -614,7 +614,7 @@ (cursor-prev-nodup-x cursor) ) (defmethod cursor-prev-nodup-x ((cursor sql-secondary-cursor) &key (returnpk nil)) - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) (if (cursor-initialized-p cursor) (progn (setf (:sql-crsr-ck cursor) (- (:sql-crsr-ck cursor) (+ 1 (:dp-nmbr cursor)))) Index: src/db-clsql/sql-controller.lisp =================================================================== RCS file: /project/elephant/cvsroot/elephant/src/db-clsql/sql-controller.lisp,v retrieving revision 1.12 diff -u -r1.12 sql-controller.lisp --- src/db-clsql/sql-controller.lisp 11 Nov 2006 18:41:11 -0000 1.12 +++ src/db-clsql/sql-controller.lisp 22 Nov 2006 01:36:04 -0000 @@ -188,7 +188,7 @@ (defmethod remove-kv (key (bt sql-indexed-btree)) "Remove a key / value pair, and update secondary indices." - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) (let* ( (sc (get-con bt)) (con (controller-db sc))) Index: src/elephant/classes.lisp =================================================================== RCS file: /project/elephant/cvsroot/elephant/src/elephant/classes.lisp,v retrieving revision 1.9 diff -u -r1.9 classes.lisp --- src/elephant/classes.lisp 26 Apr 2006 17:53:44 -0000 1.9 +++ src/elephant/classes.lisp 22 Nov 2006 01:36:04 -0000 @@ -235,13 +235,13 @@ (defmethod slot-value-using-class :around ((class persistent-metaclass) (instance persistent-object) (slot-def persistent-slot-definition)) "Get the slot value from the database." - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) (let ((name (slot-definition-name slot-def))) (persistent-slot-reader (get-con instance) instance name))) (defmethod (setf slot-value-using-class) :around (new-value (class persistent-metaclass) (instance persistent-object) (slot-def persistent-slot-definition)) "Set the slot value in the database." - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) (if (indexed class) (indexed-slot-writer class instance slot-def new-value) (let ((name (slot-definition-name slot-def))) @@ -249,13 +249,13 @@ (defmethod slot-boundp-using-class :around ((class persistent-metaclass) (instance persistent-object) (slot-def persistent-slot-definition)) "Checks if the slot exists in the database." - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) (let ((name (slot-definition-name slot-def))) (persistent-slot-boundp (get-con instance) instance name))) (defmethod slot-boundp-using-class :around ((class persistent-metaclass) (instance persistent-object) (slot-name symbol)) "Checks if the slot exists in the database." - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) (loop for slot in (class-slots class) for matches-p = (eq (slot-definition-name slot) slot-name) until matches-p @@ -266,7 +266,7 @@ (defmethod slot-makunbound-using-class :around ((class persistent-metaclass) (instance persistent-object) (slot-def persistent-slot-definition)) "Deletes the slot from the database." - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) ;; NOTE: call remove-indexed-slot here instead? (when (indexed slot-def) (unregister-indexed-slot class (slot-definition-name slot-def))) @@ -322,21 +322,21 @@ #+(or cmu sbcl) (defun make-persistent-reader (name) (lambda (instance) - (declare (optimize (speed 3)) + (declare #-elephant-without-optimize(optimize (speed 3)) (type persistent-object instance)) (persistent-slot-reader (get-con instance) instance name))) #+(or cmu sbcl) (defun make-persistent-writer (name) (lambda (new-value instance) - (declare (optimize (speed 3)) + (declare #-elephant-without-optimize(optimize (speed 3)) (type persistent-object instance)) (persistent-slot-writer (get-con instance) new-value instance name))) #+(or cmu sbcl) (defun make-persistent-slot-boundp (name) (lambda (instance) - (declare (optimize (speed 3)) + (declare #-elephant-without-optimize(optimize (speed 3)) (type persistent-object instance)) (persistent-slot-boundp (get-con instance) instance name))) Index: src/elephant/classindex-utils.lisp =================================================================== RCS file: /project/elephant/cvsroot/elephant/src/elephant/classindex-utils.lisp,v retrieving revision 1.3 diff -u -r1.3 classindex-utils.lisp --- src/elephant/classindex-utils.lisp 26 Apr 2006 17:53:44 -0000 1.3 +++ src/elephant/classindex-utils.lisp 22 Nov 2006 01:36:04 -0000 @@ -226,7 +226,7 @@ (simple-match-set (synch-rule-lhs rule) features)) (defun simple-match-set (a b) - (declare (optimize (speed 3) (safety 1))) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 1))) (cond ((null a) t) ((and (not (null a)) (null b)) nil) ((member (first a) b :test #'equal) @@ -252,7 +252,7 @@ (warn (warn "Performing slot synchronization actions: ~A" (synch-rule-rhs rule)))))) (defun apply-synch-rules (class records rule-set) - (declare (optimize (speed 3) (safety 1))) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 1))) (labels ((slotname (rec) (car rec)) (feature-set (rec) (cdr rec))) (loop for record in records do Index: src/elephant/classindex.lisp =================================================================== RCS file: /project/elephant/cvsroot/elephant/src/elephant/classindex.lisp,v retrieving revision 1.14 diff -u -r1.14 classindex.lisp --- src/elephant/classindex.lisp 21 Jul 2006 16:32:45 -0000 1.14 +++ src/elephant/classindex.lisp 22 Nov 2006 01:36:04 -0000 @@ -378,7 +378,7 @@ (defmethod get-instances-by-value ((class persistent-metaclass) slot-name value) ;; (declare -;; (optimize (speed 3) (safety 1) (space 1)) +;; #-elephant-without-optimize(optimize (speed 3) (safety 1) (space 1)) ;; (type (or string symbol) slot-name)) (let ((instances nil)) (with-btree-cursor (cur (find-inverted-index class slot-name)) @@ -407,7 +407,7 @@ (get-instances-by-range (find-class class) slot-name start end)) (defmethod get-instances-by-range ((class persistent-metaclass) idx-name start end) -;; (declare (optimize speed (safety 1) (space 1)) +;; (declare #-elephant-without-optimize(optimize speed (safety 1) (space 1)) ;; (type fixnum start end) ;; (type string idx-name)) (with-inverted-cursor (cur class idx-name) Index: src/elephant/collections.lisp =================================================================== RCS file: /project/elephant/cvsroot/elephant/src/elephant/collections.lisp,v retrieving revision 1.5 diff -u -r1.5 collections.lisp --- src/elephant/collections.lisp 19 Jun 2006 01:03:30 -0000 1.5 +++ src/elephant/collections.lisp 22 Nov 2006 01:36:04 -0000 @@ -135,7 +135,7 @@ (defmethod remove-kv (key (bt btree-index)) "Remove a key / value from the PRIMARY by a secondary lookup, updating ALL other secondary indices." - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) (remove-kv (get-primary-key key bt) (primary bt))) Index: src/elephant/serializer.lisp =================================================================== RCS file: /project/elephant/cvsroot/elephant/src/elephant/serializer.lisp,v retrieving revision 1.14 diff -u -r1.14 serializer.lisp --- src/elephant/serializer.lisp 11 Nov 2006 22:53:13 -0000 1.14 +++ src/elephant/serializer.lisp 22 Nov 2006 01:36:05 -0000 @@ -132,13 +132,13 @@ (defun serialize (frob bs) "Serialize a lisp value into a buffer-stream." - (declare (optimize (speed 3) (safety 0)) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type buffer-stream bs)) (let ((*lisp-obj-id* 0) (*circularity-hash* (get-circularity-hash))) (labels ((%serialize (frob) - (declare (optimize (speed 3) (safety 0))) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0))) (etypecase frob ((integer #.(- 1 (expt 2 31)) #.(1- (expt 2 31))) ;; fixnum (buffer-write-byte +fixnum+ bs) @@ -328,7 +328,7 @@ bs))) (defun slots-and-values (o) - (declare (optimize (speed 3) (safety 0))) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0))) (loop for sd in (compute-slots (class-of o)) for slot-name = (slot-definition-name sd) with ret = () @@ -342,12 +342,12 @@ (defun deserialize (buf-str &key sc) "Deserialize a lisp value from a buffer-stream." - (declare (optimize (speed 3) (safety 0)) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type (or null buffer-stream) buf-str)) (let ((*circularity-hash* (get-circularity-hash))) (labels ((%deserialize (bs) - (declare (optimize (speed 3) (safety 0)) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type buffer-stream bs)) (let ((tag (buffer-read-byte bs))) (declare (type foreign-char tag)) @@ -496,7 +496,7 @@ result)))))) (defun deserialize-bignum (bs length positive) - (declare (optimize (speed 3) (safety 0)) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type buffer-stream bs) (type fixnum length) (type boolean positive)) @@ -553,7 +553,7 @@ (the (unsigned-byte 8) (gethash ty array-type-to-byte))) (defun int-byte-spec (position) - (declare (optimize (speed 3) (safety 0)) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type (unsigned-byte 24) position)) #+(or cmu sbcl allegro) (progn (setf (cdr *resourced-byte-spec*) (* 32 position)) Index: src/memutil/memutil.lisp =================================================================== RCS file: /project/elephant/cvsroot/elephant/src/memutil/memutil.lisp,v retrieving revision 1.12 diff -u -r1.12 memutil.lisp --- src/memutil/memutil.lisp 11 Nov 2006 22:53:13 -0000 1.12 +++ src/memutil/memutil.lisp 22 Nov 2006 01:36:05 -0000 @@ -125,14 +125,14 @@ (defun grab-buffer-stream () "Grab a buffer-stream from the *buffer-streams* resource pool." - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) (if (= (length *buffer-streams*) 0) (make-buffer-stream) (vector-pop *buffer-streams*))) (defun return-buffer-stream (bs) "Return a buffer-stream to the *buffer-streams* resource pool." - (declare (optimize (speed 3))) + (declare #-elephant-without-optimize(optimize (speed 3))) (reset-buffer-stream bs) (vector-push-extend bs *buffer-streams*)) @@ -159,7 +159,7 @@ #+(or cmu sbcl) (defun read-int (buf offset) "Read a 32-bit signed integer from a foreign char buffer." - (declare (optimize (speed 3) (safety 0)) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type (alien (* char)) buf) (type fixnum offset)) (the (signed-byte 32) @@ -169,7 +169,7 @@ #+(or cmu sbcl) (defun read-uint (buf offset) "Read a 32-bit unsigned integer from a foreign char buffer." - (declare (optimize (speed 3) (safety 0)) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type (alien (* char)) buf) (type fixnum offset)) (the (unsigned-byte 32) @@ -179,7 +179,7 @@ #+(or cmu sbcl) (defun read-float (buf offset) "Read a single-float from a foreign char buffer." - (declare (optimize (speed 3) (safety 0)) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type (alien (* char)) buf) (type fixnum offset)) (the single-float @@ -189,7 +189,7 @@ #+(or cmu sbcl) (defun read-double (buf offset) "Read a double-float from a foreign char buffer." - (declare (optimize (speed 3) (safety 0)) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type (alien (* char)) buf) (type fixnum offset)) (the double-float @@ -199,7 +199,7 @@ #+(or cmu sbcl) (defun write-int (buf num offset) "Write a 32-bit signed integer to a foreign char buffer." - (declare (optimize (speed 3) (safety 0)) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type (alien (* char)) buf) (type (signed-byte 32) num) (type fixnum offset)) @@ -209,7 +209,7 @@ #+(or cmu sbcl) (defun write-uint (buf num offset) "Write a 32-bit unsigned integer to a foreign char buffer." - (declare (optimize (speed 3) (safety 0)) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type (alien (* char)) buf) (type (unsigned-byte 32) num) (type fixnum offset)) @@ -219,7 +219,7 @@ #+(or cmu sbcl) (defun write-float (buf num offset) "Write a single-float to a foreign char buffer." - (declare (optimize (speed 3) (safety 0)) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type (alien (* char)) buf) (type single-float num) (type fixnum offset)) @@ -229,7 +229,7 @@ #+(or cmu sbcl) (defun write-double (buf num offset) "Write a double-float to a foreign char buffer." - (declare (optimize (speed 3) (safety 0)) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type (alien (* char)) buf) (type double-float num) (type fixnum offset)) @@ -239,7 +239,7 @@ #+(or cmu sbcl) (defun offset-char-pointer (p offset) "Pointer arithmetic." - (declare (optimize (speed 3) (safety 0)) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type (alien (* char)) p) (type fixnum offset)) (sap-alien (sap+ (alien-sap p) offset) (* char))) @@ -345,7 +345,7 @@ #+(or cmu sbcl scl) (defun copy-str-to-buf (d do s so l) - (declare (optimize (speed 3) (safety 0)) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type array-or-pointer-char d) (type fixnum do so l) (type string s)) @@ -360,7 +360,7 @@ #+openmcl (defun copy-str-to-buf (dest dest-offset src src-offset length) "Copy a string to a foreign buffer. From Gary Byers." - (declare (optimize (speed 3) (safety 0)) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type string src) (type array-or-pointer-char dest) (type fixnum length src-offset dest-offset) @@ -374,7 +374,7 @@ ;; (defun copy-str-to-buf (dest dest-offset src src-offset length) ;; "Use build-in unicode handling and copying facilities. ;; NOTE: We need to validate the speed of this vs. default." -;; (declare (optimize (speed 3) (safety 0)) +;; (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) ;; (type string src) ;; (type array-or-pointer-char dest) ;; (type fixnum length src-offset dest-offset) @@ -386,7 +386,7 @@ #+(not (or cmu sbcl scl openmcl lispworks)) (defun copy-str-to-buf (dest dest-offset src src-offset length) "Copy a string to a foreign buffer." - (declare (optimize (speed 3) (safety 0)) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type string src) (type array-or-pointer-char dest) (type fixnum length src-offset dest-offset) @@ -419,7 +419,7 @@ (defun resize-buffer-stream (bs length) "Resize the underlying buffer of a buffer-stream, copying the old data." - (declare (optimize (speed 3) (safety 0)) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type buffer-stream bs) (type fixnum length)) (with-struct-slots ((buf buffer-stream-buffer) @@ -441,7 +441,7 @@ (defun resize-buffer-stream-no-copy (bs length) "Resize the underlying buffer of a buffer-stream." - (declare (optimize (speed 3) (safety 0)) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type buffer-stream bs) (type fixnum length)) (with-struct-slots ((buf buffer-stream-buffer) @@ -461,14 +461,14 @@ (defun reset-buffer-stream (bs) "'Empty' the buffer-stream." - (declare (optimize (speed 3) (safety 0)) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type buffer-stream bs)) (setf (buffer-stream-size bs) 0) (setf (buffer-stream-position bs) 0)) (defun buffer-write-byte (b bs) "Write a byte." - (declare (optimize (speed 3) (safety 0)) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type buffer-stream bs) (type (unsigned-byte 8) b)) (with-struct-slots ((buf buffer-stream-buffer) @@ -483,7 +483,7 @@ (defun buffer-write-int (i bs) "Write a 32-bit signed integer." - (declare (optimize (speed 3) (safety 0)) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type buffer-stream bs) (type (signed-byte 32) i)) (with-struct-slots ((buf buffer-stream-buffer) @@ -499,7 +499,7 @@ (defun buffer-write-uint (u bs) "Write a 32-bit unsigned integer." - (declare (optimize (speed 3) (safety 0)) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type buffer-stream bs) (type (unsigned-byte 32) u)) (with-struct-slots ((buf buffer-stream-buffer) @@ -515,7 +515,7 @@ (defun buffer-write-float (d bs) "Write a single-float." - (declare (optimize (speed 3) (safety 0)) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type buffer-stream bs) (type single-float d)) (with-struct-slots ((buf buffer-stream-buffer) @@ -531,7 +531,7 @@ (defun buffer-write-double (d bs) "Write a double-float." - (declare (optimize (speed 3) (safety 0)) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type buffer-stream bs) (type double-float d)) (with-struct-slots ((buf buffer-stream-buffer) @@ -548,7 +548,7 @@ (defun buffer-write-string (s bs) "Write the underlying bytes of a string. On Unicode Lisps, this is a 16-bit operation." - (declare (optimize (speed 3) (safety 0)) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type buffer-stream bs) (type string s)) (with-struct-slots ((buf buffer-stream-buffer) @@ -577,7 +577,7 @@ (defun buffer-read-byte (bs) "Read a byte." - (declare (optimize (speed 3) (safety 0)) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type buffer-stream bs)) (let ((position (buffer-stream-position bs))) (incf (buffer-stream-position bs)) @@ -586,7 +586,7 @@ (defun buffer-read-byte-vector (bs) "Read the whole buffer into byte vector." - (declare (optimize (speed 3) (safety 0)) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type buffer-stream bs)) (let* ((position (buffer-stream-position bs)) (size (buffer-stream-size bs)) @@ -599,7 +599,7 @@ (defun buffer-write-byte-vector (bs bv) "Read the whole buffer into byte vector." - (declare (optimize (speed 3) (safety 0)) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type buffer-stream bs)) (let* ((position (buffer-stream-position bs)) (size (buffer-stream-size bs)) @@ -611,7 +611,7 @@ (defun buffer-read-fixnum (bs) "Read a 32-bit signed integer, which is assumed to be a fixnum." - (declare (optimize (speed 3) (safety 0)) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type buffer-stream bs)) (let ((position (buffer-stream-position bs))) (setf (buffer-stream-position bs) (+ position 4)) @@ -619,7 +619,7 @@ (defun buffer-read-int (bs) "Read a 32-bit signed integer." - (declare (optimize (speed 3) (safety 0)) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type buffer-stream bs)) (let ((position (buffer-stream-position bs))) (setf (buffer-stream-position bs) (+ position 4)) @@ -627,7 +627,7 @@ (defun buffer-read-uint (bs) "Read a 32-bit unsigned integer." - (declare (optimize (speed 3) (safety 0)) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type buffer-stream bs)) (let ((position (buffer-stream-position bs))) (setf (buffer-stream-position bs) (+ position 4)) @@ -635,7 +635,7 @@ (defun buffer-read-float (bs) "Read a single-float." - (declare (optimize (speed 3) (safety 0)) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type buffer-stream bs)) (let ((position (buffer-stream-position bs))) (setf (buffer-stream-position bs) (+ position 4)) @@ -643,7 +643,7 @@ (defun buffer-read-double (bs) "Read a double-float." - (declare (optimize (speed 3) (safety 0)) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type buffer-stream bs)) (let ((position (buffer-stream-position bs))) (setf (buffer-stream-position bs) (+ position 8)) @@ -662,7 +662,7 @@ (defun buffer-read-ucs1-string (bs byte-length) "Read a UCS1 string." - (declare (optimize (speed 3) (safety 0)) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type buffer-stream bs) (type fixnum byte-length)) (let ((position (buffer-stream-position bs))) @@ -692,7 +692,7 @@ #+(or lispworks (and allegro ics)) (defun buffer-read-ucs2-string (bs byte-length) "Read a UCS2 string." - (declare (optimize (speed 3) (safety 0)) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type buffer-stream bs) (type fixnum byte-length)) (let ((position (buffer-stream-position bs))) @@ -711,7 +711,7 @@ #+(and sbcl sb-unicode) (defun buffer-read-ucs4-string (bs byte-length) "Read a UCS4 string." - (declare (optimize (speed 3) (safety 0)) + (declare #-elephant-without-optimize(optimize (speed 3) (safety 0)) (type buffer-stream bs) (type fixnum byte-length)) (let ((position (buffer-stream-position bs))) Index: tests/testsorter.lisp =================================================================== RCS file: /project/elephant/cvsroot/elephant/tests/testsorter.lisp,v retrieving revision 1.3 diff -u -r1.3 testsorter.lisp --- tests/testsorter.lisp 11 Nov 2006 18:41:11 -0000 1.3 +++ tests/testsorter.lisp 22 Nov 2006 01:36:05 -0000 @@ -55,14 +55,14 @@ :returning :double) (defun read-num (num) - (declare (optimize (speed 3)) + (declare #-elephant-without-optimize(optimize (speed 3)) (type integer num)) (with-buffer-streams (nb) (serialize num nb) (%read-num (buffer-stream-buffer nb)))) (defun num-test (num) - (declare (optimize (speed 3)) (type integer num)) + (declare #-elephant-without-optimize(optimize (speed 3)) (type integer num)) (loop with i of-type double-float = 0.0d0 for j fixnum from 0 below (ceiling (/ (integer-length num) 32)) for bs = (byte 32 (* j 32)) @@ -73,7 +73,7 @@ (defun find-bad-num (bot top) - (declare (optimize (speed 3)) + (declare #-elephant-without-optimize(optimize (speed 3)) (type integer bot top)) (cond ((= bot top) bot) ((= bot (- top 1)) @@ -85,7 +85,7 @@ (find-bad-num bot middle)))))) (defun rfind-bad-num (bot top) - (declare (optimize (speed 3)) + (declare #-elephant-without-optimize(optimize (speed 3)) (type integer bot top)) (cond ((= bot top) bot) ((= bot (- top 1)) -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From eslick at csail.mit.edu Fri Nov 24 22:32:18 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Fri, 24 Nov 2006 17:32:18 -0500 Subject: [elephant-devel] Version 0.5.0 vs. 0.6.0 Message-ID: Do we have any remaining users of Elephant version 0.5.0 out there? I want to remove support for 0.5.0->0.6.0 upgrades. This makes the code cleaner as I found a problem with DB version tagging support I put in for 0.6.0 - I won't need to do this in the future. Cheers, Ian From joubert at joubster.com Sat Nov 25 12:24:57 2006 From: joubert at joubster.com (Joubert Nel) Date: Sat, 25 Nov 2006 07:24:57 -0500 Subject: [elephant-devel] Version 0.5.0 vs. 0.6.0 In-Reply-To: References: Message-ID: <1164457497.9012.10.camel@localhost.localdomain> Hi Ian - I use v0.6.0. Not using 0.5.0 at all. On Fri, 2006-11-24 at 17:32 -0500, Ian Eslick wrote: > Do we have any remaining users of Elephant version 0.5.0 out there? > I want to remove support for 0.5.0->0.6.0 upgrades. This makes the > code cleaner as I found a problem with DB version tagging support I > put in for 0.6.0 - I won't need to do this in the future. > > Cheers, > Ian > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From nowhere.man at levallois.eu.org Sat Nov 25 15:11:44 2006 From: nowhere.man at levallois.eu.org (Pierre THIERRY) Date: Sat, 25 Nov 2006 16:11:44 +0100 Subject: [elephant-devel] Version 0.5.0 vs. 0.6.0 In-Reply-To: References: Message-ID: <20061125151144.GB19309@bateleur.arcanes.fr.eu.org> Scribit Ian Eslick dies 24/11/2006 hora 17:32: > I want to remove support for 0.5.0->0.6.0 upgrades. This makes the > code cleaner as I found a problem with DB version tagging support I > put in for 0.6.0 - I won't need to do this in the future. Could you describe the problem you found? We may try to at least write an upgrade script... Curiously, Pierre -- nowhere.man at levallois.eu.org OpenPGP 0xD9D50D8A -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From eslick at csail.mit.edu Sat Nov 25 18:37:12 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Sat, 25 Nov 2006 13:37:12 -0500 Subject: [elephant-devel] Version 0.5.0 vs. 0.6.0 In-Reply-To: <20061125151144.GB19309@bateleur.arcanes.fr.eu.org> References: <20061125151144.GB19309@bateleur.arcanes.fr.eu.org> Message-ID: Install 0.6.0, upgrade 0.5.0 -> 0.6.0 then install 0.6.1 and upgrade 0.6.0->0.6.1 Ian On Nov 25, 2006, at 10:11 AM, Pierre THIERRY wrote: > Scribit Ian Eslick dies 24/11/2006 hora 17:32: >> I want to remove support for 0.5.0->0.6.0 upgrades. This makes the >> code cleaner as I found a problem with DB version tagging support I >> put in for 0.6.0 - I won't need to do this in the future. > > Could you describe the problem you found? We may try to at least write > an upgrade script... > > Curiously, > Pierre > -- > nowhere.man at levallois.eu.org > OpenPGP 0xD9D50D8A > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From nowhere.man at levallois.eu.org Sat Nov 25 18:56:45 2006 From: nowhere.man at levallois.eu.org (Pierre THIERRY) Date: Sat, 25 Nov 2006 19:56:45 +0100 Subject: [elephant-devel] Version 0.5.0 vs. 0.6.0 In-Reply-To: References: <20061125151144.GB19309@bateleur.arcanes.fr.eu.org> Message-ID: <20061125185645.GA29568@bateleur.arcanes.fr.eu.org> Scribit Ian Eslick dies 25/11/2006 hora 13:37: > Install 0.6.0, upgrade 0.5.0 -> 0.6.0 then install 0.6.1 and upgrade > 0.6.0->0.6.1 Thansk, I'll try and see if there's anything I can do about it. BTW, is there a way to help you migrate to svn/trac, or any other combination of VCS/BTS? Quickly, Pierre -- nowhere.man at levallois.eu.org OpenPGP 0xD9D50D8A -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From eslick at csail.mit.edu Sat Nov 25 19:44:14 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Sat, 25 Nov 2006 14:44:14 -0500 Subject: [elephant-devel] Version 0.5.0 vs. 0.6.0 In-Reply-To: <20061125185645.GA29568@bateleur.arcanes.fr.eu.org> References: <20061125151144.GB19309@bateleur.arcanes.fr.eu.org> <20061125185645.GA29568@bateleur.arcanes.fr.eu.org> Message-ID: <83BF6578-3502-438E-9A89-051022F6448B@csail.mit.edu> Trac and SVN ------------------- I'm in the middle of a huge set of changes - I'm debugging the last set of BDB issues and planned to switch to SVN once that was done rather than pollute HEAD with a non-working version. Checkins to HEAD at this point would probably create a complex set of conflicts. Normally I would do this scope of change incrementally, but the supporting changes touched quite a few files and I fixed a set of long-outstanding problems along the way. Until recently Robert and I were the only ones willing to do coding and development work so working in my local copy for awhile wasn't an issue (as Robert is busy with other things). Moving to SVN and Trac should help us include more people in the design, bug-fix and feature enhancement process now that the user and potential developer community seems to be growing. My next task is to put all the current TODO items with an expanded description into Trac so everyone can comment, etc. Upgrading and Migration --------------------------------- I wouldn't bother with 0.5.0 upgrading - I'm happy to walk the 1 or 2 users through the process of taking existing data up to 0.6.0 if they need it. What would be nice, and be a good tool in general, is a general way of walking the database and dumping it to a clean external format (xml with lisp-readable primitives?) and a way to reload that external format. That's a good way of backing up data in a format that is highly transparent and independent of any binary serialization strategy. This could easily be based on the migration code in the current 0.6.0 or HEAD repository and would be orthogonal to the changes I'm making. Robert and I would like to minimize external library dependencies, so perhaps an optional xml dump utility using a 3rd party library could be provided as an asdf-loadable option? Additionally, there are a few migration issues (like following persistent object references that are embedded in arrays) that need to be cleaned up to allow a complete upgrade path. The other nice thing about completing upgrade/migration support is that it's a great way to compact the database after a long period of read/write (I have a 9 gig 0.6.0 database that I'll test migration on) The one issue with migration that I'm concerned about but haven't looked into is whether the object reference hash limits the database size to the size of all saved objects in memory - or whether the table is only the list of source OIDs to target OIDs which should fit in most modern machine's memory size (100m's of objects). I just haven't looked into this since last spring. Regards, Ian On Nov 25, 2006, at 1:56 PM, Pierre THIERRY wrote: > Scribit Ian Eslick dies 25/11/2006 hora 13:37: >> Install 0.6.0, upgrade 0.5.0 -> 0.6.0 then install 0.6.1 and upgrade >> 0.6.0->0.6.1 > > Thansk, I'll try and see if there's anything I can do about it. > > BTW, is there a way to help you migrate to svn/trac, or any other > combination of VCS/BTS? > > Quickly, > Pierre > -- > nowhere.man at levallois.eu.org > OpenPGP 0xD9D50D8A > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel